@kairos-es/read 0.0.0

package/src/superviseOnProgress.ts

@@ -0,0 +1,897 @@
+/**
+ * `superviseOnProgress` — the projection runner's SUPERVISOR: retry an effect for
+ * as long as an EXTERNALLY OBSERVED progress signal keeps moving, and give up when
+ * it stops.
+ *
+ * ## The mechanism
+ *
+ * A daemon that fails has to be restarted, and something has to decide when
+ * restarting has become pointless. The usual answer counts failures, which
+ * cannot tell "this keeps failing and nothing is happening" from "this keeps
+ * failing and the work is getting done anyway" — and those two want opposite
+ * decisions. So this module keys the decision on a SIGNAL the supervised effect
+ * does not own: a `Position` read from somewhere else, monotone by construction,
+ * whose movement means work landed no matter who landed it. Two mechanisms, kept
+ * deliberately apart:
+ *
+ * - **The schedule decides HOW FAST to restart** — a capped, jittered,
+ *   sawtoothing exponential that never terminates on its own.
+ * - **The circuit-breaker decides WHETHER to keep restarting** — by re-reading
+ *   `progress` after every failure and counting only the CONSECUTIVE restarts
+ *   that moved it not at all.
+ *
+ * Splitting them is what makes a competing writer behave. Two processes
+ * accidentally maintaining one materialisation — a deploy overlapping its
+ * predecessor, one store passed twice — both fail on every batch, the loser's
+ * guarded checkpoint advance being superseded every time; but the WINNER keeps
+ * advancing the shared stored position, so every one of the loser's restarts
+ * observes movement, its counter resets, and it resynchronises from the winner's
+ * position for as long as the winner keeps working: bounded duplicate effort, at
+ * the schedule's rate, with no hot loop and no spurious give-up. Only a signal
+ * that is genuinely stuck — a poison event whose apply always fails, a
+ * persistently broken view store — trips the breaker and fails
+ * `ProjectionStalled`.
+ *
+ * "Accidentally" is the word and it is doing work, because ONE read model
+ * materialised N ways is a DELIBERATE shape (`runProjections`) and is not this case
+ * at all. Those N runners share a `ProjectionId` but each owns its own checkpoint in
+ * its OWN store, so they never contend, no guard is ever lost between them, and
+ * nothing here is being relied on to make them safe. The tolerated fault above is
+ * two writers over ONE cursor, which is what the construction gate's collision rung
+ * refuses when it is a wiring rather than an overlapping deploy — this mechanism
+ * degrades gracefully under it and is deliberately not the place it is caught.
+ *
+ * ## Two observability channels, and which one is primary
+ *
+ * Every supervision outcome is reported twice: to a HOOK — `onRestart` on every
+ * restart, `onStalled` once at the give-up — and to a LOG LINE beside it. The
+ * hooks are the primary channel and carry the fault ITSELF, untouched. The log
+ * lines are the thin lossy default: every annotation on them is a string or a
+ * number — JSON-SAFE, that is, rather than uniformly stringy, because
+ * `effect@3.22`'s `Logger.json` throws outright on a `bigint` nested in an
+ * annotation value and `CheckpointSuperseded` carries one (the measurement, for
+ * each shipped logger, is at the annotations in `shouldRestart`). The one numeric
+ * annotation, `noProgressRestarts`, stays numeric on purpose: a serialiser has no
+ * quarrel with it and an aggregator can filter and graph on it, both of which a
+ * `String` around it would give up for nothing. The fault goes out through
+ * `String`, which is to say through the FAULT: `Error.prototype.toString` is
+ * `name: message`, so a fault with something to say sets `message` — as all four of
+ * the runner's own faults do, `CheckpointSuperseded` included, its `bigint`
+ * `expected` interpolated into a `string` and so hidden from every serialiser — and
+ * one with nothing to say beyond its tag renders as its tag. The supervisor needs
+ * no knowledge of the supervised effect's `E` to label it — it is generic in `E` —
+ * and no caller has to supply anything for the purpose.
+ *
+ * ## Why it is a module of its own, and why it is PACKAGE-INTERNAL
+ *
+ * Not because the mechanism is on offer for reuse. It is deliberately absent from
+ * `@kairos-es/read`'s entry point, and its signature says what it supervises: a
+ * `CheckpointKey` names the supervised thing, a `Position` is the signal, a
+ * `ProjectionStoreError` is what reading that signal can fail with, and
+ * `ProjectionStalled` is a projection-shaped failure on
+ * `ProjectionRunner.fibre`'s error channel.
+ *
+ * It is separate because it is the hardest thing in this package to test THROUGH
+ * the pipeline, and the repo has the before and after. Pinning the five-minute
+ * default no-progress budget — a quantity about supervision and nothing else —
+ * cost ~122 lines while the mechanism lived inside `runProjection`: an in-memory
+ * event store, a view store decorated never to commit, a read model, a log-capture
+ * layer, and a hand-driven loop stepping the clock some 1200 times. Against a stub
+ * it costs 52, and 22 of those are code: one `Effect` yielding a `Position`, one
+ * effect that fails, one `TestClock.adjust`, and the rest of the case is the
+ * arithmetic it asserts written out. (The figure is re-measured rather than
+ * inherited — it read ~37 until the three restart `Duration`s became explicit
+ * arguments and the case gained a comment saying why. The ratio is what the
+ * argument turns on, and that has not moved.) Every other subtle behaviour is
+ * pinned the same way — the
+ * `ORIGIN` seeding, the breaker being consulted before the sleep, the interruption
+ * trap on the re-read, the counter resetting on progress — because this module
+ * knows nothing of `subscribe`, of decoding, of micro-batching, of committing, of
+ * slices or of the `Serializer`. Those cases are three suites over one stub
+ * (`test/superviseOnProgress.{pacing,breaker,observability}.test.ts`, sharing
+ * `test/superviseOnProgress.fixture.ts`), split along the two mechanisms this
+ * section keeps apart plus the channel their outcomes leave through. A SEPARATE
+ * module bought all of that; a GENERAL one was never needed for any of it.
+ *
+ * And there is no second consumer here, nor one in prospect. The mechanism needs a
+ * caller with a failing effect worth restarting AND a progress signal owned by
+ * somebody else; in this library only a projection has both. A command runner's
+ * progress IS its own success. The store's poll retry lives in
+ * `@kairos-es/store-postgres` and already has a bounded retry with no supervisor
+ * over it. A lease renewer for competing consumers is out of scope by ADR-0007.
+ *
+ * Should such a consumer appear, generalising then is a smaller change than
+ * maintaining the claim now — so the cost is recorded here rather than
+ * re-derived: `key` becomes a bare `string` label, losing the two discrete
+ * `projection`/`partition` annotations an operator greps on (or forcing a generic
+ * annotation shape onto every caller); `Position`'s `!==` becomes an
+ * `Equivalence` on the signal type; `ProjectionStoreError` becomes a fourth type
+ * parameter, and collapsing it to `unknown` instead is what would cost the
+ * interruption trap its argument, since "`catchAll` touches only the typed
+ * `ProjectionStoreError`" stops being a claim a reader can check against the
+ * signature; and `ProjectionStalled` has to be renamed or made generic.
+ *
+ * ## Platform-neutral
+ *
+ * No `node:*`, no `Date`, no host RNG: the backoff is `Duration`s scheduled
+ * through the `Clock` and the jitter is Effect's `Random` service, so
+ * `TestClock` drives all of it.
+ */
+import { ORIGIN, type Position } from '@kairos-es/core'
+import { Data, type Duration, Effect, Ref, Schedule, Schema } from 'effect'
+import type { CheckpointKey, ProjectionStoreError } from './ProjectionStore'
+
+/**
+ * Consecutive no-progress restarts before the supervisor gives up: a branded
+ * POSITIVE integer.
+ *
+ * Branded because at `0` the breaker's `restarts >= maxNoProgressRestarts` test
+ * is true on the FIRST failure, so the supervisor gives up before it has retried
+ * once and before `onRestart` has ever fired — a `ProjectionStalled` for a fault
+ * that a single 100-millisecond retry would have cleared, and the flat
+ * contradiction of the five-minute budget `DEFAULT_MAX_NO_PROGRESS_RESTARTS`
+ * documents. Negative and fractional counts fail the same test the same way.
+ *
+ * `>= 1` rather than `>= 0` for that reason: "give up immediately" is not a
+ * supervision policy this module offers, and a caller who wants one has
+ * `Fiber.interrupt`.
+ *
+ * A distinct brand from `BatchSize` despite the identical refinement, because
+ * the two are still writable side by side in one flat options literal, are both
+ * bare integers, and differ by an order of magnitude in meaning — nominality is
+ * what makes a transposition fail to typecheck rather than silently retune the
+ * runner. That the two now tune different mechanisms in different modules is a
+ * second reason for the distinction, not a replacement for the first.
+ *
+ * Construct with `MaxNoProgressRestarts.make(n)` — and retune it by the WALL
+ * CLOCK the figure buys, per `DEFAULT_MAX_NO_PROGRESS_RESTARTS`.
+ */
+export const MaxNoProgressRestarts = Schema.Int.pipe(
+  Schema.positive(),
+  Schema.brand('MaxNoProgressRestarts'),
+)
+export type MaxNoProgressRestarts = typeof MaxNoProgressRestarts.Type
+
+/**
+ * Fast enough that a transient blip is invisible, slow enough not to spin.
+ *
+ * Exported — like the two below and unlike `DEFAULT_MAX_NO_PROGRESS_RESTARTS` —
+ * because `resolveProjection` is what RESOLVES it, for both entry points: the
+ * construction gate judges the
+ * figure that will actually run, so the resolution has to happen before the
+ * supervisor is built and the default has to be readable from there. The
+ * documentation stays HERE, against the sawtooth arithmetic the three of them
+ * define together, because that is what the figure is chosen for.
+ */
+export const DEFAULT_RESTART_MIN_DELAY: Duration.DurationInput = '100 millis'
+
+/**
+ * The store's own poll retry exhausts after roughly 40 seconds of sustained
+ * transient faults before `subscribe` dies, so a 30-second ceiling keeps a
+ * restart in the same order of magnitude as the fault it is recovering from —
+ * long enough not to hammer a struggling database, short enough that recovery is
+ * not measured in minutes.
+ *
+ * Exported for the reason `DEFAULT_RESTART_MIN_DELAY` gives.
+ */
+export const DEFAULT_RESTART_MAX_DELAY: Duration.DurationInput = '30 seconds'
+
+/**
+ * A minute of ELAPSED retrying, after which the pacing starts over from
+ * `restartMinDelay`.
+ *
+ * `Schedule.resetAfter` keys on `Schedule.elapsed` — the time since the schedule
+ * (or its own last reset) began, NOT the time since the last decision, verified
+ * against `effect@3.22`'s `internal/schedule.ts` — and both readings of that are
+ * wanted. A run that stays healthy for a minute pushes `elapsed` past the
+ * threshold on its way to the next fault, so its backoff starts from
+ * `restartMinDelay` again instead of inheriting a ceiling from a fault it
+ * recovered from hours ago. And a SUSTAINED outage, whose attempts fail fast,
+ * sawtooths back down to `restartMinDelay` every cycle rather than pinning at
+ * `restartMaxDelay` for ever, so a store that comes back is noticed within
+ * `restartMinDelay` rather than up to `restartMaxDelay` later.
+ *
+ * Exported for the reason `DEFAULT_RESTART_MIN_DELAY` gives.
+ */
+export const DEFAULT_RESTART_RESET_AFTER: Duration.DurationInput = '60 seconds'
+
+/**
+ * FORTY zero-progress restarts, chosen for the WALL CLOCK it buys — roughly five
+ * minutes — rather than for the count, which on its own means nothing.
+ *
+ * What the figure has to satisfy is `ProjectionStalled`'s own claim: a view store
+ * "broken rather than merely unavailable". So the budget must be long enough that
+ * a merely-unavailable view store — a failover, a rolling restart, a brief
+ * partition — is back inside it, and comfortably longer than the ~40 seconds the
+ * event store itself spends retrying before `subscribe` dies. The arithmetic is
+ * written out rather than asserted, because the pacing above is not obvious and
+ * the next reader should be able to CHECK the claim:
+ *
+ * Unjittered, the restart schedule sleeps `0.1, 0.2, 0.4, 0.8, 1.6, 3.2, 6.4,
+ * 12.8, 25.6, 30` seconds — the exponential, then the `restartMaxDelay` cap — and
+ * then starts over: by the eleventh decision 81.1s have elapsed, past
+ * `restartResetAfter`, so `resetAfter` resets the schedule. The pacing is
+ * therefore a SAWTOOTH of ten sleeps summing 81.1s. A give-up at the Nth
+ * consecutive no-progress failure has slept N-1 times (the breaker is consulted
+ * BEFORE each sleep), so 40 restarts is three whole cycles (243.3s) plus the
+ * first nine of a fourth (51.1s): 294.4s ≈ 4.9 minutes, which the ±20% jitter
+ * spreads to roughly 4.5–5.3. `superviseOnProgress.pacing.test.ts` pins that budget
+ * under a `TestClock` — and, beside it, the N-1 that this arithmetic turns on — so a
+ * retuning cannot quietly shrink either.
+ *
+ * Five restarts — the figure this shipped with first — bought 1.5s, which made a
+ * ten-second blip a PERMANENT stall and flatly contradicted the documentation
+ * above. That is the mistake this figure exists to avoid: if you lower it,
+ * recompute the wall clock.
+ *
+ * Two things make the effective budget longer still. When it is the SUBSCRIPTION
+ * that keeps dying rather than the view store refusing to answer, each attempt
+ * also burns the store's own ~40-second poll-retry budget before dying. And
+ * restarts that DO move the progress signal never count at all, so a busy
+ * projection that restarts often can never trip this however long it runs.
+ *
+ * PRIVATE, where the three restart `Duration`s beside it are exported: the count is
+ * BRANDED, so `MaxNoProgressRestarts.make(n)` refused a bad value at the caller's
+ * own line and there is nothing left for the construction gate to judge. Nobody
+ * outside needs to resolve it, so it is resolved here.
+ */
+const DEFAULT_MAX_NO_PROGRESS_RESTARTS: MaxNoProgressRestarts =
+  MaxNoProgressRestarts.make(40)
+
+/**
+ * How the supervisor is tuned: how fast it restarts, how long it keeps trying,
+ * and where each of its two OUTCOMES is reported — a restart to `onRestart`, a
+ * give-up to `onStalled`.
+ *
+ * Separated from the pipeline's own tuning (`ProjectionPipelineOptions`:
+ * `batchSize`, `batchWindow`) because they are dials on different machines —
+ * one sizes a transaction and a latency ceiling, the other sizes a RECOVERY
+ * POLICY measured in minutes of outage — and reading them as one undifferentiated
+ * bag is how `maxNoProgressRestarts` gets retuned as though it were a throughput
+ * knob. `ProjectionRunnerOptions` still composes the two into one flat object, so
+ * no call site pays for the distinction; the types are what carry it.
+ *
+ * Every figure is a build-time DEFAULT, not a contract: the mechanisms are fixed
+ * by ADR-0007, the numbers are judgement calls sized against
+ * `@kairos-es/store-postgres`'s own defaults and are meant to be overridden by a
+ * deployment that knows its own latency budget (and by tests, which want tiny
+ * delays).
+ *
+ * A judgement call is still not a free choice, though: zero, negative,
+ * fractional and infinite figures are not "aggressive tuning", they are wirings
+ * whose failure mode is a hot loop or a permanently stalled daemon with nothing
+ * on any error channel. So this surface fails at its boundary, by TWO mechanisms
+ * because the field types differ — the count is BRANDED, so a bad value is
+ * refused at the caller's own `.make(...)`, and the three durations are judged by
+ * the read side's construction gate (`projectionWiringFault`, which says why they
+ * cannot be branded), called by `runProjection` and `runProjections` alike before a
+ * supervisor, an attempt or a fibre exists, where the fault lands as a defect exactly as
+ * `assertServableQuery`'s does.
+ *
+ * Neither mechanism is this module's own, and that is deliberate: by the time a
+ * figure reaches `superviseOnProgress` it has been resolved and judged, which is
+ * why the three appear again as REQUIRED fields on `SuperviseOnProgressOptions`.
+ */
+export interface ProjectionSupervisionOptions {
+  /**
+   * First restart delay. Default `'100 millis'`. Positive and finite, checked at
+   * construction — it is the base `Schedule.exponential` multiplies, so a zero
+   * here zeroes the whole backoff.
+   */
+  readonly restartMinDelay?: Duration.DurationInput
+
+  /**
+   * Restart delay ceiling. Default `'30 seconds'` — the same order of magnitude
+   * as the store's own ~40-second poll-retry budget, so the runner does not back
+   * off far past the lifetime of the fault it is recovering from.
+   *
+   * Positive, finite, and not below `restartMinDelay`; all three are checked at
+   * construction.
+   */
+  readonly restartMaxDelay?: Duration.DurationInput
+
+  /**
+   * How long a run must stay up before the backoff resets to `restartMinDelay`.
+   * Default `'60 seconds'`. Without this, a projection that hits one fault an hour
+   * would still be waiting `restartMaxDelay` a day later.
+   *
+   * Positive and finite, checked at construction: both degenerate ends dissolve
+   * the sawtooth that `DEFAULT_MAX_NO_PROGRESS_RESTARTS`' budget is computed
+   * against.
+   */
+  readonly restartResetAfter?: Duration.DurationInput
+
+  /**
+   * Consecutive restarts that moved the progress signal NOT AT ALL before the
+   * supervisor gives up with `ProjectionStalled`. Default `40`, which on the
+   * default pacing is about five minutes of retrying — long enough that a view
+   * store that is merely unavailable is back inside it.
+   *
+   * Retune it by the WALL CLOCK it buys rather than by the count: with the
+   * sawtoothing backoff the two are not proportional, and the arithmetic is set
+   * out on `DEFAULT_MAX_NO_PROGRESS_RESTARTS`.
+   *
+   * A branded positive integer: write `MaxNoProgressRestarts.make(80)`, not
+   * `80`.
+   */
+  readonly maxNoProgressRestarts?: MaxNoProgressRestarts
+
+  /**
+   * Supervision observability seam for the NON-TERMINAL outcome: called on every
+   * restart, BEFORE the backoff sleep, so a metric or a trace records the restart
+   * at the moment it is decided rather than after the delay.
+   *
+   * THE PRIMARY CHANNEL for a restart, not a supplement to the log line beside
+   * it. `reason` is the RAW fault; the `reason` on the log line is that fault
+   * `String`ed, because a serialiser cannot take the `bigint` a
+   * `CheckpointSuperseded` carries (the measurement is at the annotations, in
+   * `shouldRestart`). So anything that needs to match on a tag, read a payload or
+   * count by fault type takes this hook, and the log line is the lossy convenience
+   * default for everyone else.
+   *
+   * Infallible and requirement-free by signature, so a hook cannot FAIL the
+   * daemon or widen the supervised effect's requirements. It carries no
+   * `position`, unlike `onStalled`: a restart is about to re-read the signal
+   * anyway, so where it stood at the restart is a number in flight rather than a
+   * fact. `onStalled` states the full rule the two share, including what happens
+   * when a hook dies.
+   */
+  readonly onRestart?: (info: {
+    readonly key: CheckpointKey
+    readonly reason: unknown
+    readonly noProgressRestarts: number
+  }) => Effect.Effect<void>
+
+  /**
+   * Supervision observability seam for the TERMINAL outcome: called ONCE, at the
+   * give-up, immediately after the error is logged and immediately before
+   * `ProjectionStalled` is raised. An application can fail a health check, exit
+   * non-zero, or page somebody.
+   *
+   * ## WHY, given `onRestart` already exists — the restart is the metric, the
+   * stall is the page
+   *
+   * The two outcomes are not equally important, and the asymmetry runs the
+   * OPPOSITE way to the one the older surface implied. A restart RESOLVES ITSELF:
+   * it is a rate to graph, and to alert on only if it climbs. A give-up does not —
+   * `ProjectionStalled` means a poison event or a view store broken rather than
+   * merely unavailable, and its own doc says both need a human. Yet under the
+   * wiring this library recommends (`projectionLayer`, which is
+   * `Layer.scopedDiscard`, so the daemon fibre is discarded ON PURPOSE — a
+   * projection is a background process and nothing should depend on it as a
+   * service) the terminal event's only route out of the process was the
+   * `Effect.logError` beside this call. A seam for the event that fixes itself and
+   * none for the event that does not is backwards; this is the missing half.
+   *
+   * ## The payload is the `ProjectionStalled` about to be raised
+   *
+   * The same four fields, carrying the same values, so a hook and a caller
+   * awaiting the daemon see one story rather than two. `position` is the STUCK
+   * signal — the position everything is failing just after, so the offending event
+   * is one query away — `noProgressRestarts` the budget that was spent, and
+   * `reason` the last underlying failure as the RAW fault rather than the reduced
+   * log label, because a programmatic callback can inspect a `bigint` where a log
+   * line cannot.
+   *
+   * With `onRestart` that makes the hooks the PRIMARY observability channel for
+   * both supervision outcomes, and the two log lines the thin lossy default: every
+   * value on those is a string or a number — JSON-SAFE, by measurement rather than
+   * by superstition (see the annotations in `shouldRestart`), which is the real
+   * bound and not "stringify everything" — whereas a hook gets the fault untouched.
+   *
+   * ## Infallible by signature, NOT defect-trapped — the same rule as `onRestart`
+   *
+   * `Effect<void>`, so a hook can neither fail the daemon nor widen the supervised
+   * effect's requirements. A DEFECT from a hook is a different matter and is
+   * deliberately left alone: it takes the fibre down exactly as one from
+   * `onRestart` does, because absorbing it would hide a bug in the application's
+   * own pager on the one path whose entire job is to make failures visible. What
+   * makes that safe is the ORDER — the `Effect.logError` has already landed by the
+   * time this is called, so the guaranteed route to an operator is never hostage
+   * to the hook, and a broken hook surfaces as a further defect rather than as
+   * silence. The one cost is that such a fibre dies with the hook's defect instead
+   * of failing `ProjectionStalled`, which is the honest report: the stall was
+   * announced, and then the announcer broke.
+   */
+  readonly onStalled?: (info: {
+    readonly key: CheckpointKey
+    readonly noProgressRestarts: number
+    readonly position: Position
+    readonly reason: unknown
+  }) => Effect.Effect<void>
+}
+
+/**
+ * The three restart figures that are RESOLVED before a supervisor exists:
+ * `resolveProjection` applies each default and the construction gate judges the
+ * result, so what reaches `superviseOnProgress` has been both settled and checked.
+ *
+ * A type of its own rather than three fields restated wherever they are needed,
+ * because it is the BOUNDARY between the two halves of the surface above and both
+ * halves are computed from it: `SuperviseOnProgressOptions` requires exactly these,
+ * and `UnresolvedSupervisionOptions` below is everything else. `ResolvedProjection`
+ * in `ProjectionRunner.ts` EXTENDS it, so "which figures the resolve step owns" is
+ * ONE declaration serving both ends rather than two that happen to agree today — a
+ * fourth resolved figure added here is a compile error in the resolve step until it
+ * produces one, and another at the forwarding call site until it names one.
+ *
+ * PACKAGE-INTERNAL, like `SuperviseOnProgressOptions` itself. The half a caller
+ * writes is `ProjectionSupervisionOptions`, where all three stay optional.
+ */
+export interface ResolvedSupervisionTuning {
+  /** First restart delay, already resolved and judged. */
+  readonly restartMinDelay: Duration.DurationInput
+
+  /** Restart delay ceiling, already resolved and judged. */
+  readonly restartMaxDelay: Duration.DurationInput
+
+  /** Backoff reset threshold, already resolved and judged. */
+  readonly restartResetAfter: Duration.DurationInput
+}
+
+/**
+ * The half of `ProjectionSupervisionOptions` that NOTHING resolves — the two hooks
+ * and the branded count — forwarded to the supervisor exactly as the caller wrote
+ * it.
+ *
+ * Defined by SUBTRACTION rather than by listing its members, and that is the whole
+ * value of it. `buildAndFork` spreads the forwardable remainder of the caller's own
+ * options — this type plus the two PIPELINE figures, which are inert here because
+ * nothing below reads them by name — and names the resolved three beside it, so the
+ * split between the two halves is carried by the types: a new OPTIONAL supervision
+ * field lands in `ProjectionSupervisionOptions`, arrives here
+ * automatically and travels with that spread untouched, while a new RESOLVED figure
+ * lands in `ResolvedSupervisionTuning`, drops OUT of this type, and leaves the spread
+ * unable to supply it — a missing required property at that one call site, rather
+ * than an unjudged raw value winning silently on spread order.
+ *
+ * `Omit` rather than a second `extends` clause, and not by preference: an interface
+ * cannot extend both `ProjectionSupervisionOptions` and `ResolvedSupervisionTuning`,
+ * because the same property optional in one parent and required in the other is a
+ * conflict TypeScript refuses outright (TS2320, "not identical").
+ */
+export type UnresolvedSupervisionOptions = Omit<
+  ProjectionSupervisionOptions,
+  keyof ResolvedSupervisionTuning
+>
+
+/**
+ * The supervisor gave up: `noProgressRestarts` consecutive restarts moved the
+ * progress signal not at all.
+ *
+ * This is the only way a supervised daemon's fibre ever fails, and for a
+ * projection it means one of two things: a POISON EVENT (an `apply` or a decode
+ * that fails deterministically on the event at the checkpoint, so every restart
+ * re-reads it and dies again), or a view store still not answering after the
+ * WHOLE no-progress budget — broken, that is, rather than merely unavailable.
+ * That distinction is only as true as the budget makes it, which is why the
+ * default is sized at roughly five minutes of retrying rather than at a tidy
+ * restart count (`DEFAULT_MAX_NO_PROGRESS_RESTARTS`). Both need a human; neither
+ * is fixed by retrying, which is why the supervisor stops instead of retrying for
+ * ever and calling that resilience.
+ *
+ * Getting it in FRONT of that human is `onStalled`'s job, and the reason that
+ * hook exists: this failure lands on the daemon's error channel, and the wiring
+ * this library recommends discards the daemon. Beside the hook there is one
+ * `Effect.logError`, and nothing else.
+ *
+ * `position` is the stuck signal — the position everything is failing just
+ * after — so the offending event is one query away. `reason` is the last
+ * underlying failure, kept as `unknown` because it spans the supervised effect's
+ * whole `E`, which this module is generic in.
+ */
+export class ProjectionStalled extends Data.TaggedError('ProjectionStalled')<{
+  readonly key: CheckpointKey
+  readonly noProgressRestarts: number
+  readonly position: Position
+  readonly reason: unknown
+}> {
+  /**
+   * The two facts that make a stall diagnosable — how much budget was spent, and
+   * where it was spent — as the `Error` field that says why, so the fault renders
+   * ITSELF.
+   *
+   * This class needs the field more than any other fault the read side raises,
+   * because of WHERE it lands. `ProjectionStalled` is what `ProjectionRunner.fibre`
+   * fails with, which makes it the only one of them a consumer meets without going
+   * anywhere near the `ProjectionStore` port: `CheckpointSuperseded` and
+   * `ProjectionStoreError` sit on `commit`'s public signature too, but reaching them
+   * means calling or implementing that port, whereas the `Fiber.await` route the
+   * runner's own documentation recommends ends at whatever this getter says — from a
+   * daemon the application only forked.
+   *
+   * `Data.TaggedError` sets `name` to the tag and leaves `message` empty, and
+   * `Cause.pretty` substitutes its own `'An error has occurred'` for an empty one —
+   * so unfilled, a caller who did exactly what they were told to do reads
+   * `'ProjectionStalled: An error has occurred'` about the single failure this whole
+   * module exists to report. Filled, the same call yields a sentence naming the
+   * budget that was spent and the position everything is failing just after, which
+   * is the first thing an operator queries the log at.
+   *
+   * ## Why THESE two fields, and why interpolating them is safe
+   *
+   * `noProgressRestarts` is a `number`. `position` is a branded `bigint`, and
+   * TEMPLATE INTERPOLATION of a `bigint` yields its decimal digits — so what leaves
+   * this getter is typed `string` and carries no `bigint` for any serialiser to
+   * meet. That distinction is worth being exact about, because this package
+   * measures a real `bigint` hazard and it is easy to over-apply: `Logger.json`
+   * throws on a `bigint` NESTED IN AN ANNOTATION VALUE and loses the whole line, so
+   * the reduction in `shouldRestart` hands the logger `String(stored)`. The hazard
+   * is about the values a logger is handed, never about a `string` rendered from
+   * one — and `String(fault)` is precisely how a fault carrying a `bigint` field
+   * crosses that boundary intact.
+   *
+   * `reason` is deliberately NOT in the label. It spans the supervised effect's
+   * whole `E` and arrives as `unknown`, so anything this class said about it would
+   * be a guess at somebody else's value — and there is no need to guess: the raw
+   * fault reaches `onStalled` untouched, and the give-up log line beside it already
+   * carries `String(reason)`, which is the same self-rendering rule applied one
+   * level down.
+   *
+   * A GETTER over two existing fields rather than a `message` field, so neither can
+   * disagree with the label. Why a prototype accessor is safe here, how far filling
+   * the field reaches and the structured shape it leaves a machine to parse are set
+   * out once for all four of these getters on `ProjectionStoreError` in
+   * `ProjectionStore.ts`.
+   */
+  override get message(): string {
+    return `no checkpoint progress in ${this.noProgressRestarts} consecutive restarts (stuck at ${this.position})`
+  }
+}
+
+/**
+ * Everything one supervision needs beyond the effect being supervised: WHAT is
+ * supervised (`key`), WHERE its progress is observed (`progress`), and the tuning
+ * and hooks `ProjectionSupervisionOptions` documents.
+ *
+ * One FLAT object, extending the two tuning interfaces rather than nesting either,
+ * for the reason `ProjectionRunnerOptions` gives at length: the
+ * `{ ...OPTIONS, oneField }` override this repo writes everywhere is a shallow
+ * spread, so a nested half would lose its siblings silently.
+ *
+ * Those two are exactly the halves of `ProjectionSupervisionOptions`:
+ * `UnresolvedSupervisionOptions`, which `buildAndFork` forwards by SPREADING the
+ * caller's own object, and `ResolvedSupervisionTuning`, which it names one by one
+ * beside that spread. So flatness costs this module nothing and hides nothing. A
+ * caller's whole unresolved half travels through in one expression — an optional
+ * field added there needs no edit anywhere to arrive — and a figure this module
+ * requires cannot arrive from that half at all: the spread's TYPE no longer carries
+ * one, and the OBJECT does not either, `buildAndFork` discarding the three by
+ * rest-destructure before it spreads what is left. The split is the types' to keep
+ * rather than an enumeration's, and that call site is where the runtime half of it
+ * is argued.
+ *
+ * FIVE fields are REQUIRED here where every field of the half a caller writes is
+ * optional, for two different reasons that happen to arrive at the same modifier.
+ *
+ * `key` and `progress` have no defensible default: a supervisor with no key could
+ * not name the thing it is reporting on, and one with no signal would be counting
+ * failures — the very thing this module exists not to do.
+ *
+ * The three restart `Duration`s do have defaults, and are required anyway, because
+ * by the time a figure arrives here `resolveProjection` has RESOLVED it against the
+ * defaults above and the construction gate has JUDGED the resolved value.
+ * Re-admitting an absent field would mean a second `?? DEFAULT` in the body below,
+ * and then the figure that RUNS could differ from the figure that was checked —
+ * which is the one failure mode a construction-time check cannot survive, since it
+ * would report a wiring sound and then run a different one. Required, this module
+ * cannot be handed a duration nothing validated. `maxNoProgressRestarts` is the
+ * exception that shows the rule: it stays optional and is defaulted below, because
+ * its brand did the judging at the caller's own `.make(...)` and there is no
+ * resolution for the gate to be inconsistent with.
+ */
+export interface SuperviseOnProgressOptions
+  extends UnresolvedSupervisionOptions,
+    ResolvedSupervisionTuning {
+  /**
+   * Names the supervised thing. It is what both log lines, both hook payloads and
+   * the `ProjectionStalled` failure report, because an operator's first question
+   * about a stalled projection is which one.
+   */
+  readonly key: CheckpointKey
+
+  /**
+   * The externally observed signal, re-read after EVERY failure — which is what
+   * makes the give-up decision about the world rather than about this process's
+   * own attempt count.
+   *
+   * A VALUE, not a thunk and not a service: nothing is read until the first
+   * failure, so building a supervision touches the view store not at all. That is
+   * what keeps the read side's construction-time defects honest — a runner the
+   * wiring gate rejected has read no checkpoint and opened no subscription, which
+   * every construction case in this package asserts.
+   */
+  readonly progress: Effect.Effect<Position, ProjectionStoreError>
+}
+
+/**
+ * Supervise `run`: retry it while `options.progress` moves, give up with
+ * `ProjectionStalled` when it does not.
+ *
+ * Generic in the supervised effect's `A`, `E` and `R`. The supervisor reads the
+ * failure only through `String` and hands it to `onRestart`/`onStalled` as
+ * `unknown`, so it has no reason to constrain `E`; its own error channel is exactly
+ * `ProjectionStalled`, because the breaker collapses everything else into another
+ * restart; and `R` passes through untouched, nothing here adding a requirement,
+ * which is what keeps either entry point's requirement list the pipeline's own.
+ *
+ * ONE CALL rather than a curried supervisor later applied to an attempt. The rank-2
+ * intermediate bought exactly one thing: the tuning could be validated before the
+ * attempt existed, which fixed the ORDER a mis-tuned supervisor was reported in
+ * relative to the runner's other construction-time checks. That argument is now
+ * gone outright rather than merely outweighed — the tuning is judged by
+ * `projectionWiringFault` before a supervisor, an attempt or a fibre exists, in an
+ * order that module declares — so what a curried form would leave behind is a type
+ * whose only application sat a line from its construction.
+ *
+ * It validates NOTHING, which is deliberate rather than an omission. Every
+ * construction-time rejection on the read side has ONE home — the wiring gate, whose
+ * own single caller is the shared `prepareProjection` phase both entry points run
+ * before anything is built — and this module is
+ * package-internal with exactly one caller, so a check here would be a SECOND home
+ * for one rule: two sentences for one mistake, free to drift, each reachable only by
+ * whichever of the two modules its author happened to open. What arrives here has
+ * therefore already been resolved and judged, which is why the three restart
+ * `Duration`s are REQUIRED on `SuperviseOnProgressOptions` and why nothing below
+ * reaches for a default. The mistake still lands as a DEFECT at construction,
+ * exactly as `assertServableQuery`'s does; the gate is merely where.
+ *
+ * Nothing is taken for RENDERING the supervised effect's `E` on the two log lines:
+ * the `reason` annotation is `String(fault)`, so the fault labels itself. That is a
+ * real reduction rather than a floor, because `Error.prototype.toString` is
+ * `name: message` and `Data.TaggedError` sets `name` to the tag — so a fault that
+ * fills `message` renders as `'SomeTag: why'`, and one that does not renders as
+ * `'SomeTag'`. A caller wanting a better label sets `message` on its own error,
+ * where the knowledge belongs and where `Cause.pretty` and every other reader
+ * benefit from it too, rather than handing a table to a supervisor that would be
+ * the only thing to use it.
+ *
+ * The returned effect is re-runnable: the two supervision `Ref`s are created INSIDE
+ * it rather than here, so every run starts a fresh supervision session instead of
+ * inheriting the breaker state a previous run left behind.
+ */
+export const superviseOnProgress = <A, E, R>(
+  run: Effect.Effect<A, E, R>,
+  options: SuperviseOnProgressOptions,
+): Effect.Effect<A, ProjectionStalled, R> => {
+  // The three restart durations are TAKEN, not defaulted, while the COUNT keeps its
+  // default: `SuperviseOnProgressOptions` above argues that asymmetry in full.
+  const { key, progress, restartMinDelay, restartMaxDelay, restartResetAfter } =
+    options
+  const maxNoProgressRestarts =
+    options.maxNoProgressRestarts ?? DEFAULT_MAX_NO_PROGRESS_RESTARTS
+  const onRestart = options.onRestart ?? (() => Effect.void)
+  const onStalled = options.onStalled ?? (() => Effect.void)
+
+  /**
+   * The restart pacing: a capped exponential that NEVER TERMINATES on its own.
+   *
+   * `Schedule.union` continues while EITHER arm continues and uses the SHORTER
+   * of the two delays, so unioning an unbounded exponential with a fixed
+   * `spaced(restartMaxDelay)` yields exponential growth up to the cap and then
+   * a flat cap — with no `recurs` arm, because the give-up decision belongs to
+   * the circuit-breaker (which knows whether anything is progressing) and not
+   * to a schedule (which only knows how many times it has fired).
+   *
+   * `jittered` because restarts are correlated: a database blip restarts every
+   * projection in the deployment at the same instant, and an unjittered backoff
+   * would march them all back in lockstep.
+   *
+   * `resetAfter` keys on ELAPSED retrying time rather than on the length of the
+   * last run (`Schedule.elapsed`, verified against the installed source), which
+   * buys two things at once: a run that stays up for `restartResetAfter` starts
+   * its next backoff from `restartMinDelay` instead of inheriting a ceiling from
+   * a fault it recovered from hours ago, and a sustained outage sawtooths from
+   * `restartMinDelay` up to the cap and back rather than pinning at the cap for
+   * ever. The sawtooth is also what makes the no-progress budget computable —
+   * the arithmetic is on `DEFAULT_MAX_NO_PROGRESS_RESTARTS`, and a test pins it.
+   *
+   * Effect v3 caveat: this is the THREE-type-parameter `Schedule<Out, In, R>`
+   * that `effect@3` exports. The currently-published `Schedule` docs show a
+   * newer four-type-parameter shape — do not copy them; this is written against
+   * the installed type definitions.
+   */
+  const restartSchedule = Schedule.union(
+    Schedule.exponential(restartMinDelay, 2),
+    Schedule.spaced(restartMaxDelay),
+  ).pipe(Schedule.jittered, Schedule.resetAfter(restartResetAfter))
+
+  return Effect.gen(function* () {
+    // Supervision state, owned OUTSIDE the retried effect so it survives a
+    // restart — which is the whole point: the breaker's question is about the
+    // sequence of restarts, not about any one of them.
+    const noProgressRestarts = yield* Ref.make(0)
+    // The signal observed at the previous restart. Seeded with `ORIGIN` rather
+    // than with a read of `progress` (which is not read until the first
+    // failure): a read model resuming from a non-zero durable checkpoint
+    // therefore scores its FIRST failure as progress and gets one extra restart
+    // before the counter engages. That errs towards retrying, which is the safe
+    // direction for a breaker.
+    const lastRestartPosition = yield* Ref.make(ORIGIN)
+
+    /**
+     * The circuit-breaker, evaluated on every failure and BEFORE the backoff
+     * sleep: does the progress signal show that anything is moving?
+     *
+     * Keying on the signal rather than on a failure count or on the error's tag
+     * is what makes competing runners behave — the module doc sets out that
+     * story in full. Conversely a poison event or a broken view store leaves the
+     * position pinned however many times the run restarts, so the breaker trips
+     * on exactly the cases retrying cannot fix.
+     */
+    const shouldRestart = (reason: E): Effect.Effect<boolean> =>
+      Effect.gen(function* () {
+        const previous = yield* Ref.get(lastRestartPosition)
+
+        // A view store too broken to answer `readCheckpoint` cannot report
+        // progress either, so a failed re-read is treated as "unchanged" and
+        // counts towards the breaker. That is the conservative reading: the
+        // alternative (retrying for ever because we cannot tell) is exactly the
+        // silent stall the breaker exists to end.
+        //
+        // `catchAll`, not `orElseSucceed`: the latter falls back on any cause
+        // carrying no defect, which includes INTERRUPTION, so a scope closing
+        // mid-re-read would be absorbed here — the daemon would go on to count a
+        // restart, log it and call `onRestart` while it was being torn down, and
+        // a teardown arriving on the last of the budget would report a
+        // `ProjectionStalled` that never happened — and now PAGE somebody about
+        // it, since `onStalled` fires on that branch. It is the same trap
+        // `catchAllDefect` avoids at the runner's stream boundary. `catchAll`
+        // touches only the typed `ProjectionStoreError`.
+        const stored = yield* Effect.catchAll(progress, () =>
+          Effect.succeed(previous),
+        )
+        const progressed = stored !== previous
+
+        const restarts = progressed
+          ? 0
+          : (yield* Ref.get(noProgressRestarts)) + 1
+        yield* Ref.set(noProgressRestarts, restarts)
+        yield* Ref.set(lastRestartPosition, stored)
+
+        // ## The log line is the LOSSY channel; the hooks are the faithful one
+        //
+        // Both supervision outcomes now reach a programmatic consumer carrying
+        // the fault ITSELF — `onRestart` at the foot of this function,
+        // `onStalled` just below — so this block is no longer the only record of
+        // what happened. It is the convenience channel: flat, greppable, and
+        // reduced to JSON-safe scalars on purpose. Anything that needs a fault's
+        // FIELDS takes a hook; anything that needs a line in a log takes this.
+        //
+        // ## Why every value is JSON-SAFE, MEASURED rather than assumed
+        //
+        // The bound is serialisability, not stringiness: `noProgressRestarts`
+        // below goes out as a NUMBER, which no serialiser objects to and which a
+        // log aggregator can filter, threshold and graph on — properties a
+        // `String` around it would throw away in exchange for nothing. What has
+        // to be reduced is the value class a serialiser genuinely chokes on.
+        //
+        // A `bigint` is not JSON-serialisable and these faults carry them
+        // (`CheckpointSuperseded.expected` is a `Position`). What each logger
+        // `effect@3.22` ships actually does with one was established by running
+        // it, not read off the documentation:
+        //
+        // - `Logger.json` special-cases a TOP-LEVEL `bigint` annotation value
+        //   (`internal/logger.ts`'s `structuredMessage`), but a `bigint` NESTED
+        //   inside an annotation's object survives `Inspectable.toJSON` into the
+        //   final `Inspectable.stringifyCircular`, which throws `TypeError: Do
+        //   not know how to serialize a BigInt`. Nothing catches it, so the throw
+        //   takes the WHOLE line — every other annotation with it — and lands as
+        //   a defect inside the supervisor, on the one path whose entire job is
+        //   to make failures visible.
+        // - The DEFAULT logger (`stringLogger`) does not throw: it renders each
+        //   annotation through `Inspectable.toStringUnknown`, whose `try/catch`
+        //   falls back to `String(value)`. The same loss, quieter — the offending
+        //   annotation degrades to a bare tag, or to `[object Object]`.
+        // - `Logger.pretty` hands the value to `console.log` and prints `42n`.
+        //
+        // So the reduction happens HERE, before the logger sees anything.
+        // `position` is `String`ed rather than handed over as a `Position`
+        // despite all three shipped loggers coping with a bare `bigint`: they
+        // render it identically to `String`, so there is nothing to win, and a
+        // third-party logger that JSON-encodes its annotation map naively is a
+        // real thing.
+        //
+        // `reason` is `String`ed too, and that is the WHOLE reduction — no table
+        // of a caller's fault union, because the faults can label themselves.
+        // `Error.prototype.toString` is `name: message` and `Data.TaggedError`
+        // sets `name` to the tag, so a fault that fills `message` arrives as
+        // `'ProjectionStoreError: connection reset'` and one that does not arrives
+        // as its bare tag. This module is generic in `E`, so that is also all it
+        // could honestly do: the choice of what to say is the fault's, made once
+        // on the class, and `Cause.pretty` and `String(fault)` everywhere else
+        // read the same field.
+        const annotations = {
+          projection: key.projection,
+          partition: key.partition,
+          position: String(stored),
+          noProgressRestarts: restarts,
+          reason: String(reason),
+        }
+
+        if (restarts >= maxNoProgressRestarts) {
+          // Logged HERE, not by the caller, because a stalled projection must
+          // never be silent: `runProjection` returns an infallible effect, so
+          // unless somebody awaits the fibre the `ProjectionStalled` failure has
+          // no other route to an operator.
+          //
+          // `Effect.annotateLogs`, not a second argument to `logError`. Effect's
+          // log constructors are VARIADIC IN THE MESSAGE (`logError: (...message:
+          // ReadonlyArray<any>) => Effect<void>`), so an object passed second is
+          // a second message and not an annotation at all — it used to arrive as
+          // a JSON blob wedged inside the line rather than as the structured
+          // fields the reduction above was built for. Annotating puts each field
+          // where a logger can render it in its own idiom: `reason=… position=…`
+          // under the default logger, discrete keys under `annotations` in
+          // `Logger.json`. That is the logger owning the presentation, which is
+          // its job; all this module owes it is values it can serialise.
+          yield* Effect.annotateLogs(
+            Effect.logError(
+              'kairos-es projection stalled: giving up after consecutive restarts with no checkpoint progress',
+            ),
+            annotations,
+          )
+
+          // ...and then the SEAM, because a log line is not an alert. The
+          // give-up is the outcome nothing recovers from, so it gets the hook
+          // the restart already had — an application can fail a health check,
+          // exit non-zero or page from here. `onStalled`'s doc carries the
+          // asymmetry argument in full.
+          //
+          // AFTER the log, deliberately: the log is the guaranteed route out and
+          // a defect from an application's hook must not be able to silence it.
+          // The payload is the `ProjectionStalled` this give-up is about to
+          // produce, field for field and value for value — `stored` is what was
+          // just written to `lastRestartPosition`, `restarts` what was just
+          // written to `noProgressRestarts`, and `reason` is the failure the
+          // `catchAll` below will carry — so a hook and a caller awaiting the
+          // daemon never see two versions of one stall. The RAW fault rather
+          // than the log label, for the same reason the annotations take the
+          // label: a callback can read a `bigint`, a serialiser cannot.
+          yield* onStalled({
+            key,
+            noProgressRestarts: restarts,
+            position: stored,
+            reason,
+          })
+          return false
+        }
+
+        yield* Effect.annotateLogs(
+          Effect.logWarning(
+            'kairos-es projection restarting from its stored checkpoint',
+          ),
+          annotations,
+        )
+        yield* onRestart({ key, reason, noProgressRestarts: restarts })
+        return true
+      })
+
+    /**
+     * `Effect.retry`'s `while` IS the breaker, which is why the retry can exit
+     * in exactly one way: the schedule never terminates, so the only path out
+     * with a failure is `shouldRestart` returning `false`. The residual error is
+     * therefore always the give-up case, and mapping it to `ProjectionStalled`
+     * is total rather than a fallback for an unreachable branch.
+     *
+     * A run that *succeeds* ends the daemon quietly. For a projection that is
+     * the honest translation of "there is nothing left to read", which the
+     * shipped stores' live tails never say.
+     */
+    return yield* Effect.retry(run, {
+      schedule: restartSchedule,
+      while: shouldRestart,
+    }).pipe(
+      Effect.catchAll((reason) =>
+        Effect.gen(function* () {
+          return yield* new ProjectionStalled({
+            key,
+            noProgressRestarts: yield* Ref.get(noProgressRestarts),
+            position: yield* Ref.get(lastRestartPosition),
+            reason,
+          })
+        }),
+      ),
+    )
+  })
+}