@kairos-es/read 0.0.0

package/dist/esm/superviseOnProgress.js

@@ -0,0 +1,571 @@
+/**
+ * `superviseOnProgress` — the projection runner's SUPERVISOR: retry an effect for
+ * as long as an EXTERNALLY OBSERVED progress signal keeps moving, and give up when
+ * it stops.
+ *
+ * ## The mechanism
+ *
+ * A daemon that fails has to be restarted, and something has to decide when
+ * restarting has become pointless. The usual answer counts failures, which
+ * cannot tell "this keeps failing and nothing is happening" from "this keeps
+ * failing and the work is getting done anyway" — and those two want opposite
+ * decisions. So this module keys the decision on a SIGNAL the supervised effect
+ * does not own: a `Position` read from somewhere else, monotone by construction,
+ * whose movement means work landed no matter who landed it. Two mechanisms, kept
+ * deliberately apart:
+ *
+ * - **The schedule decides HOW FAST to restart** — a capped, jittered,
+ *   sawtoothing exponential that never terminates on its own.
+ * - **The circuit-breaker decides WHETHER to keep restarting** — by re-reading
+ *   `progress` after every failure and counting only the CONSECUTIVE restarts
+ *   that moved it not at all.
+ *
+ * Splitting them is what makes a competing writer behave. Two processes
+ * accidentally maintaining one materialisation — a deploy overlapping its
+ * predecessor, one store passed twice — both fail on every batch, the loser's
+ * guarded checkpoint advance being superseded every time; but the WINNER keeps
+ * advancing the shared stored position, so every one of the loser's restarts
+ * observes movement, its counter resets, and it resynchronises from the winner's
+ * position for as long as the winner keeps working: bounded duplicate effort, at
+ * the schedule's rate, with no hot loop and no spurious give-up. Only a signal
+ * that is genuinely stuck — a poison event whose apply always fails, a
+ * persistently broken view store — trips the breaker and fails
+ * `ProjectionStalled`.
+ *
+ * "Accidentally" is the word and it is doing work, because ONE read model
+ * materialised N ways is a DELIBERATE shape (`runProjections`) and is not this case
+ * at all. Those N runners share a `ProjectionId` but each owns its own checkpoint in
+ * its OWN store, so they never contend, no guard is ever lost between them, and
+ * nothing here is being relied on to make them safe. The tolerated fault above is
+ * two writers over ONE cursor, which is what the construction gate's collision rung
+ * refuses when it is a wiring rather than an overlapping deploy — this mechanism
+ * degrades gracefully under it and is deliberately not the place it is caught.
+ *
+ * ## Two observability channels, and which one is primary
+ *
+ * Every supervision outcome is reported twice: to a HOOK — `onRestart` on every
+ * restart, `onStalled` once at the give-up — and to a LOG LINE beside it. The
+ * hooks are the primary channel and carry the fault ITSELF, untouched. The log
+ * lines are the thin lossy default: every annotation on them is a string or a
+ * number — JSON-SAFE, that is, rather than uniformly stringy, because
+ * `effect@3.22`'s `Logger.json` throws outright on a `bigint` nested in an
+ * annotation value and `CheckpointSuperseded` carries one (the measurement, for
+ * each shipped logger, is at the annotations in `shouldRestart`). The one numeric
+ * annotation, `noProgressRestarts`, stays numeric on purpose: a serialiser has no
+ * quarrel with it and an aggregator can filter and graph on it, both of which a
+ * `String` around it would give up for nothing. The fault goes out through
+ * `String`, which is to say through the FAULT: `Error.prototype.toString` is
+ * `name: message`, so a fault with something to say sets `message` — as all four of
+ * the runner's own faults do, `CheckpointSuperseded` included, its `bigint`
+ * `expected` interpolated into a `string` and so hidden from every serialiser — and
+ * one with nothing to say beyond its tag renders as its tag. The supervisor needs
+ * no knowledge of the supervised effect's `E` to label it — it is generic in `E` —
+ * and no caller has to supply anything for the purpose.
+ *
+ * ## Why it is a module of its own, and why it is PACKAGE-INTERNAL
+ *
+ * Not because the mechanism is on offer for reuse. It is deliberately absent from
+ * `@kairos-es/read`'s entry point, and its signature says what it supervises: a
+ * `CheckpointKey` names the supervised thing, a `Position` is the signal, a
+ * `ProjectionStoreError` is what reading that signal can fail with, and
+ * `ProjectionStalled` is a projection-shaped failure on
+ * `ProjectionRunner.fibre`'s error channel.
+ *
+ * It is separate because it is the hardest thing in this package to test THROUGH
+ * the pipeline, and the repo has the before and after. Pinning the five-minute
+ * default no-progress budget — a quantity about supervision and nothing else —
+ * cost ~122 lines while the mechanism lived inside `runProjection`: an in-memory
+ * event store, a view store decorated never to commit, a read model, a log-capture
+ * layer, and a hand-driven loop stepping the clock some 1200 times. Against a stub
+ * it costs 52, and 22 of those are code: one `Effect` yielding a `Position`, one
+ * effect that fails, one `TestClock.adjust`, and the rest of the case is the
+ * arithmetic it asserts written out. (The figure is re-measured rather than
+ * inherited — it read ~37 until the three restart `Duration`s became explicit
+ * arguments and the case gained a comment saying why. The ratio is what the
+ * argument turns on, and that has not moved.) Every other subtle behaviour is
+ * pinned the same way — the
+ * `ORIGIN` seeding, the breaker being consulted before the sleep, the interruption
+ * trap on the re-read, the counter resetting on progress — because this module
+ * knows nothing of `subscribe`, of decoding, of micro-batching, of committing, of
+ * slices or of the `Serializer`. Those cases are three suites over one stub
+ * (`test/superviseOnProgress.{pacing,breaker,observability}.test.ts`, sharing
+ * `test/superviseOnProgress.fixture.ts`), split along the two mechanisms this
+ * section keeps apart plus the channel their outcomes leave through. A SEPARATE
+ * module bought all of that; a GENERAL one was never needed for any of it.
+ *
+ * And there is no second consumer here, nor one in prospect. The mechanism needs a
+ * caller with a failing effect worth restarting AND a progress signal owned by
+ * somebody else; in this library only a projection has both. A command runner's
+ * progress IS its own success. The store's poll retry lives in
+ * `@kairos-es/store-postgres` and already has a bounded retry with no supervisor
+ * over it. A lease renewer for competing consumers is out of scope by ADR-0007.
+ *
+ * Should such a consumer appear, generalising then is a smaller change than
+ * maintaining the claim now — so the cost is recorded here rather than
+ * re-derived: `key` becomes a bare `string` label, losing the two discrete
+ * `projection`/`partition` annotations an operator greps on (or forcing a generic
+ * annotation shape onto every caller); `Position`'s `!==` becomes an
+ * `Equivalence` on the signal type; `ProjectionStoreError` becomes a fourth type
+ * parameter, and collapsing it to `unknown` instead is what would cost the
+ * interruption trap its argument, since "`catchAll` touches only the typed
+ * `ProjectionStoreError`" stops being a claim a reader can check against the
+ * signature; and `ProjectionStalled` has to be renamed or made generic.
+ *
+ * ## Platform-neutral
+ *
+ * No `node:*`, no `Date`, no host RNG: the backoff is `Duration`s scheduled
+ * through the `Clock` and the jitter is Effect's `Random` service, so
+ * `TestClock` drives all of it.
+ */
+import { ORIGIN } from '@kairos-es/core';
+import { Data, Effect, Ref, Schedule, Schema } from 'effect';
+/**
+ * Consecutive no-progress restarts before the supervisor gives up: a branded
+ * POSITIVE integer.
+ *
+ * Branded because at `0` the breaker's `restarts >= maxNoProgressRestarts` test
+ * is true on the FIRST failure, so the supervisor gives up before it has retried
+ * once and before `onRestart` has ever fired — a `ProjectionStalled` for a fault
+ * that a single 100-millisecond retry would have cleared, and the flat
+ * contradiction of the five-minute budget `DEFAULT_MAX_NO_PROGRESS_RESTARTS`
+ * documents. Negative and fractional counts fail the same test the same way.
+ *
+ * `>= 1` rather than `>= 0` for that reason: "give up immediately" is not a
+ * supervision policy this module offers, and a caller who wants one has
+ * `Fiber.interrupt`.
+ *
+ * A distinct brand from `BatchSize` despite the identical refinement, because
+ * the two are still writable side by side in one flat options literal, are both
+ * bare integers, and differ by an order of magnitude in meaning — nominality is
+ * what makes a transposition fail to typecheck rather than silently retune the
+ * runner. That the two now tune different mechanisms in different modules is a
+ * second reason for the distinction, not a replacement for the first.
+ *
+ * Construct with `MaxNoProgressRestarts.make(n)` — and retune it by the WALL
+ * CLOCK the figure buys, per `DEFAULT_MAX_NO_PROGRESS_RESTARTS`.
+ */
+export const MaxNoProgressRestarts = /*#__PURE__*/Schema.Int.pipe(/*#__PURE__*/Schema.positive(), /*#__PURE__*/Schema.brand('MaxNoProgressRestarts'));
+/**
+ * Fast enough that a transient blip is invisible, slow enough not to spin.
+ *
+ * Exported — like the two below and unlike `DEFAULT_MAX_NO_PROGRESS_RESTARTS` —
+ * because `resolveProjection` is what RESOLVES it, for both entry points: the
+ * construction gate judges the
+ * figure that will actually run, so the resolution has to happen before the
+ * supervisor is built and the default has to be readable from there. The
+ * documentation stays HERE, against the sawtooth arithmetic the three of them
+ * define together, because that is what the figure is chosen for.
+ */
+export const DEFAULT_RESTART_MIN_DELAY = '100 millis';
+/**
+ * The store's own poll retry exhausts after roughly 40 seconds of sustained
+ * transient faults before `subscribe` dies, so a 30-second ceiling keeps a
+ * restart in the same order of magnitude as the fault it is recovering from —
+ * long enough not to hammer a struggling database, short enough that recovery is
+ * not measured in minutes.
+ *
+ * Exported for the reason `DEFAULT_RESTART_MIN_DELAY` gives.
+ */
+export const DEFAULT_RESTART_MAX_DELAY = '30 seconds';
+/**
+ * A minute of ELAPSED retrying, after which the pacing starts over from
+ * `restartMinDelay`.
+ *
+ * `Schedule.resetAfter` keys on `Schedule.elapsed` — the time since the schedule
+ * (or its own last reset) began, NOT the time since the last decision, verified
+ * against `effect@3.22`'s `internal/schedule.ts` — and both readings of that are
+ * wanted. A run that stays healthy for a minute pushes `elapsed` past the
+ * threshold on its way to the next fault, so its backoff starts from
+ * `restartMinDelay` again instead of inheriting a ceiling from a fault it
+ * recovered from hours ago. And a SUSTAINED outage, whose attempts fail fast,
+ * sawtooths back down to `restartMinDelay` every cycle rather than pinning at
+ * `restartMaxDelay` for ever, so a store that comes back is noticed within
+ * `restartMinDelay` rather than up to `restartMaxDelay` later.
+ *
+ * Exported for the reason `DEFAULT_RESTART_MIN_DELAY` gives.
+ */
+export const DEFAULT_RESTART_RESET_AFTER = '60 seconds';
+/**
+ * FORTY zero-progress restarts, chosen for the WALL CLOCK it buys — roughly five
+ * minutes — rather than for the count, which on its own means nothing.
+ *
+ * What the figure has to satisfy is `ProjectionStalled`'s own claim: a view store
+ * "broken rather than merely unavailable". So the budget must be long enough that
+ * a merely-unavailable view store — a failover, a rolling restart, a brief
+ * partition — is back inside it, and comfortably longer than the ~40 seconds the
+ * event store itself spends retrying before `subscribe` dies. The arithmetic is
+ * written out rather than asserted, because the pacing above is not obvious and
+ * the next reader should be able to CHECK the claim:
+ *
+ * Unjittered, the restart schedule sleeps `0.1, 0.2, 0.4, 0.8, 1.6, 3.2, 6.4,
+ * 12.8, 25.6, 30` seconds — the exponential, then the `restartMaxDelay` cap — and
+ * then starts over: by the eleventh decision 81.1s have elapsed, past
+ * `restartResetAfter`, so `resetAfter` resets the schedule. The pacing is
+ * therefore a SAWTOOTH of ten sleeps summing 81.1s. A give-up at the Nth
+ * consecutive no-progress failure has slept N-1 times (the breaker is consulted
+ * BEFORE each sleep), so 40 restarts is three whole cycles (243.3s) plus the
+ * first nine of a fourth (51.1s): 294.4s ≈ 4.9 minutes, which the ±20% jitter
+ * spreads to roughly 4.5–5.3. `superviseOnProgress.pacing.test.ts` pins that budget
+ * under a `TestClock` — and, beside it, the N-1 that this arithmetic turns on — so a
+ * retuning cannot quietly shrink either.
+ *
+ * Five restarts — the figure this shipped with first — bought 1.5s, which made a
+ * ten-second blip a PERMANENT stall and flatly contradicted the documentation
+ * above. That is the mistake this figure exists to avoid: if you lower it,
+ * recompute the wall clock.
+ *
+ * Two things make the effective budget longer still. When it is the SUBSCRIPTION
+ * that keeps dying rather than the view store refusing to answer, each attempt
+ * also burns the store's own ~40-second poll-retry budget before dying. And
+ * restarts that DO move the progress signal never count at all, so a busy
+ * projection that restarts often can never trip this however long it runs.
+ *
+ * PRIVATE, where the three restart `Duration`s beside it are exported: the count is
+ * BRANDED, so `MaxNoProgressRestarts.make(n)` refused a bad value at the caller's
+ * own line and there is nothing left for the construction gate to judge. Nobody
+ * outside needs to resolve it, so it is resolved here.
+ */
+const DEFAULT_MAX_NO_PROGRESS_RESTARTS = /*#__PURE__*/MaxNoProgressRestarts.make(40);
+/**
+ * The supervisor gave up: `noProgressRestarts` consecutive restarts moved the
+ * progress signal not at all.
+ *
+ * This is the only way a supervised daemon's fibre ever fails, and for a
+ * projection it means one of two things: a POISON EVENT (an `apply` or a decode
+ * that fails deterministically on the event at the checkpoint, so every restart
+ * re-reads it and dies again), or a view store still not answering after the
+ * WHOLE no-progress budget — broken, that is, rather than merely unavailable.
+ * That distinction is only as true as the budget makes it, which is why the
+ * default is sized at roughly five minutes of retrying rather than at a tidy
+ * restart count (`DEFAULT_MAX_NO_PROGRESS_RESTARTS`). Both need a human; neither
+ * is fixed by retrying, which is why the supervisor stops instead of retrying for
+ * ever and calling that resilience.
+ *
+ * Getting it in FRONT of that human is `onStalled`'s job, and the reason that
+ * hook exists: this failure lands on the daemon's error channel, and the wiring
+ * this library recommends discards the daemon. Beside the hook there is one
+ * `Effect.logError`, and nothing else.
+ *
+ * `position` is the stuck signal — the position everything is failing just
+ * after — so the offending event is one query away. `reason` is the last
+ * underlying failure, kept as `unknown` because it spans the supervised effect's
+ * whole `E`, which this module is generic in.
+ */
+export class ProjectionStalled extends /*#__PURE__*/Data.TaggedError('ProjectionStalled') {
+  /**
+   * The two facts that make a stall diagnosable — how much budget was spent, and
+   * where it was spent — as the `Error` field that says why, so the fault renders
+   * ITSELF.
+   *
+   * This class needs the field more than any other fault the read side raises,
+   * because of WHERE it lands. `ProjectionStalled` is what `ProjectionRunner.fibre`
+   * fails with, which makes it the only one of them a consumer meets without going
+   * anywhere near the `ProjectionStore` port: `CheckpointSuperseded` and
+   * `ProjectionStoreError` sit on `commit`'s public signature too, but reaching them
+   * means calling or implementing that port, whereas the `Fiber.await` route the
+   * runner's own documentation recommends ends at whatever this getter says — from a
+   * daemon the application only forked.
+   *
+   * `Data.TaggedError` sets `name` to the tag and leaves `message` empty, and
+   * `Cause.pretty` substitutes its own `'An error has occurred'` for an empty one —
+   * so unfilled, a caller who did exactly what they were told to do reads
+   * `'ProjectionStalled: An error has occurred'` about the single failure this whole
+   * module exists to report. Filled, the same call yields a sentence naming the
+   * budget that was spent and the position everything is failing just after, which
+   * is the first thing an operator queries the log at.
+   *
+   * ## Why THESE two fields, and why interpolating them is safe
+   *
+   * `noProgressRestarts` is a `number`. `position` is a branded `bigint`, and
+   * TEMPLATE INTERPOLATION of a `bigint` yields its decimal digits — so what leaves
+   * this getter is typed `string` and carries no `bigint` for any serialiser to
+   * meet. That distinction is worth being exact about, because this package
+   * measures a real `bigint` hazard and it is easy to over-apply: `Logger.json`
+   * throws on a `bigint` NESTED IN AN ANNOTATION VALUE and loses the whole line, so
+   * the reduction in `shouldRestart` hands the logger `String(stored)`. The hazard
+   * is about the values a logger is handed, never about a `string` rendered from
+   * one — and `String(fault)` is precisely how a fault carrying a `bigint` field
+   * crosses that boundary intact.
+   *
+   * `reason` is deliberately NOT in the label. It spans the supervised effect's
+   * whole `E` and arrives as `unknown`, so anything this class said about it would
+   * be a guess at somebody else's value — and there is no need to guess: the raw
+   * fault reaches `onStalled` untouched, and the give-up log line beside it already
+   * carries `String(reason)`, which is the same self-rendering rule applied one
+   * level down.
+   *
+   * A GETTER over two existing fields rather than a `message` field, so neither can
+   * disagree with the label. Why a prototype accessor is safe here, how far filling
+   * the field reaches and the structured shape it leaves a machine to parse are set
+   * out once for all four of these getters on `ProjectionStoreError` in
+   * `ProjectionStore.ts`.
+   */
+  get message() {
+    return `no checkpoint progress in ${this.noProgressRestarts} consecutive restarts (stuck at ${this.position})`;
+  }
+}
+/**
+ * Supervise `run`: retry it while `options.progress` moves, give up with
+ * `ProjectionStalled` when it does not.
+ *
+ * Generic in the supervised effect's `A`, `E` and `R`. The supervisor reads the
+ * failure only through `String` and hands it to `onRestart`/`onStalled` as
+ * `unknown`, so it has no reason to constrain `E`; its own error channel is exactly
+ * `ProjectionStalled`, because the breaker collapses everything else into another
+ * restart; and `R` passes through untouched, nothing here adding a requirement,
+ * which is what keeps either entry point's requirement list the pipeline's own.
+ *
+ * ONE CALL rather than a curried supervisor later applied to an attempt. The rank-2
+ * intermediate bought exactly one thing: the tuning could be validated before the
+ * attempt existed, which fixed the ORDER a mis-tuned supervisor was reported in
+ * relative to the runner's other construction-time checks. That argument is now
+ * gone outright rather than merely outweighed — the tuning is judged by
+ * `projectionWiringFault` before a supervisor, an attempt or a fibre exists, in an
+ * order that module declares — so what a curried form would leave behind is a type
+ * whose only application sat a line from its construction.
+ *
+ * It validates NOTHING, which is deliberate rather than an omission. Every
+ * construction-time rejection on the read side has ONE home — the wiring gate, whose
+ * own single caller is the shared `prepareProjection` phase both entry points run
+ * before anything is built — and this module is
+ * package-internal with exactly one caller, so a check here would be a SECOND home
+ * for one rule: two sentences for one mistake, free to drift, each reachable only by
+ * whichever of the two modules its author happened to open. What arrives here has
+ * therefore already been resolved and judged, which is why the three restart
+ * `Duration`s are REQUIRED on `SuperviseOnProgressOptions` and why nothing below
+ * reaches for a default. The mistake still lands as a DEFECT at construction,
+ * exactly as `assertServableQuery`'s does; the gate is merely where.
+ *
+ * Nothing is taken for RENDERING the supervised effect's `E` on the two log lines:
+ * the `reason` annotation is `String(fault)`, so the fault labels itself. That is a
+ * real reduction rather than a floor, because `Error.prototype.toString` is
+ * `name: message` and `Data.TaggedError` sets `name` to the tag — so a fault that
+ * fills `message` renders as `'SomeTag: why'`, and one that does not renders as
+ * `'SomeTag'`. A caller wanting a better label sets `message` on its own error,
+ * where the knowledge belongs and where `Cause.pretty` and every other reader
+ * benefit from it too, rather than handing a table to a supervisor that would be
+ * the only thing to use it.
+ *
+ * The returned effect is re-runnable: the two supervision `Ref`s are created INSIDE
+ * it rather than here, so every run starts a fresh supervision session instead of
+ * inheriting the breaker state a previous run left behind.
+ */
+export const superviseOnProgress = (run, options) => {
+  // The three restart durations are TAKEN, not defaulted, while the COUNT keeps its
+  // default: `SuperviseOnProgressOptions` above argues that asymmetry in full.
+  const {
+    key,
+    progress,
+    restartMinDelay,
+    restartMaxDelay,
+    restartResetAfter
+  } = options;
+  const maxNoProgressRestarts = options.maxNoProgressRestarts ?? DEFAULT_MAX_NO_PROGRESS_RESTARTS;
+  const onRestart = options.onRestart ?? (() => Effect.void);
+  const onStalled = options.onStalled ?? (() => Effect.void);
+  /**
+   * The restart pacing: a capped exponential that NEVER TERMINATES on its own.
+   *
+   * `Schedule.union` continues while EITHER arm continues and uses the SHORTER
+   * of the two delays, so unioning an unbounded exponential with a fixed
+   * `spaced(restartMaxDelay)` yields exponential growth up to the cap and then
+   * a flat cap — with no `recurs` arm, because the give-up decision belongs to
+   * the circuit-breaker (which knows whether anything is progressing) and not
+   * to a schedule (which only knows how many times it has fired).
+   *
+   * `jittered` because restarts are correlated: a database blip restarts every
+   * projection in the deployment at the same instant, and an unjittered backoff
+   * would march them all back in lockstep.
+   *
+   * `resetAfter` keys on ELAPSED retrying time rather than on the length of the
+   * last run (`Schedule.elapsed`, verified against the installed source), which
+   * buys two things at once: a run that stays up for `restartResetAfter` starts
+   * its next backoff from `restartMinDelay` instead of inheriting a ceiling from
+   * a fault it recovered from hours ago, and a sustained outage sawtooths from
+   * `restartMinDelay` up to the cap and back rather than pinning at the cap for
+   * ever. The sawtooth is also what makes the no-progress budget computable —
+   * the arithmetic is on `DEFAULT_MAX_NO_PROGRESS_RESTARTS`, and a test pins it.
+   *
+   * Effect v3 caveat: this is the THREE-type-parameter `Schedule<Out, In, R>`
+   * that `effect@3` exports. The currently-published `Schedule` docs show a
+   * newer four-type-parameter shape — do not copy them; this is written against
+   * the installed type definitions.
+   */
+  const restartSchedule = Schedule.union(Schedule.exponential(restartMinDelay, 2), Schedule.spaced(restartMaxDelay)).pipe(Schedule.jittered, Schedule.resetAfter(restartResetAfter));
+  return Effect.gen(function* () {
+    // Supervision state, owned OUTSIDE the retried effect so it survives a
+    // restart — which is the whole point: the breaker's question is about the
+    // sequence of restarts, not about any one of them.
+    const noProgressRestarts = yield* Ref.make(0);
+    // The signal observed at the previous restart. Seeded with `ORIGIN` rather
+    // than with a read of `progress` (which is not read until the first
+    // failure): a read model resuming from a non-zero durable checkpoint
+    // therefore scores its FIRST failure as progress and gets one extra restart
+    // before the counter engages. That errs towards retrying, which is the safe
+    // direction for a breaker.
+    const lastRestartPosition = yield* Ref.make(ORIGIN);
+    /**
+     * The circuit-breaker, evaluated on every failure and BEFORE the backoff
+     * sleep: does the progress signal show that anything is moving?
+     *
+     * Keying on the signal rather than on a failure count or on the error's tag
+     * is what makes competing runners behave — the module doc sets out that
+     * story in full. Conversely a poison event or a broken view store leaves the
+     * position pinned however many times the run restarts, so the breaker trips
+     * on exactly the cases retrying cannot fix.
+     */
+    const shouldRestart = reason => Effect.gen(function* () {
+      const previous = yield* Ref.get(lastRestartPosition);
+      // A view store too broken to answer `readCheckpoint` cannot report
+      // progress either, so a failed re-read is treated as "unchanged" and
+      // counts towards the breaker. That is the conservative reading: the
+      // alternative (retrying for ever because we cannot tell) is exactly the
+      // silent stall the breaker exists to end.
+      //
+      // `catchAll`, not `orElseSucceed`: the latter falls back on any cause
+      // carrying no defect, which includes INTERRUPTION, so a scope closing
+      // mid-re-read would be absorbed here — the daemon would go on to count a
+      // restart, log it and call `onRestart` while it was being torn down, and
+      // a teardown arriving on the last of the budget would report a
+      // `ProjectionStalled` that never happened — and now PAGE somebody about
+      // it, since `onStalled` fires on that branch. It is the same trap
+      // `catchAllDefect` avoids at the runner's stream boundary. `catchAll`
+      // touches only the typed `ProjectionStoreError`.
+      const stored = yield* Effect.catchAll(progress, () => Effect.succeed(previous));
+      const progressed = stored !== previous;
+      const restarts = progressed ? 0 : (yield* Ref.get(noProgressRestarts)) + 1;
+      yield* Ref.set(noProgressRestarts, restarts);
+      yield* Ref.set(lastRestartPosition, stored);
+      // ## The log line is the LOSSY channel; the hooks are the faithful one
+      //
+      // Both supervision outcomes now reach a programmatic consumer carrying
+      // the fault ITSELF — `onRestart` at the foot of this function,
+      // `onStalled` just below — so this block is no longer the only record of
+      // what happened. It is the convenience channel: flat, greppable, and
+      // reduced to JSON-safe scalars on purpose. Anything that needs a fault's
+      // FIELDS takes a hook; anything that needs a line in a log takes this.
+      //
+      // ## Why every value is JSON-SAFE, MEASURED rather than assumed
+      //
+      // The bound is serialisability, not stringiness: `noProgressRestarts`
+      // below goes out as a NUMBER, which no serialiser objects to and which a
+      // log aggregator can filter, threshold and graph on — properties a
+      // `String` around it would throw away in exchange for nothing. What has
+      // to be reduced is the value class a serialiser genuinely chokes on.
+      //
+      // A `bigint` is not JSON-serialisable and these faults carry them
+      // (`CheckpointSuperseded.expected` is a `Position`). What each logger
+      // `effect@3.22` ships actually does with one was established by running
+      // it, not read off the documentation:
+      //
+      // - `Logger.json` special-cases a TOP-LEVEL `bigint` annotation value
+      //   (`internal/logger.ts`'s `structuredMessage`), but a `bigint` NESTED
+      //   inside an annotation's object survives `Inspectable.toJSON` into the
+      //   final `Inspectable.stringifyCircular`, which throws `TypeError: Do
+      //   not know how to serialize a BigInt`. Nothing catches it, so the throw
+      //   takes the WHOLE line — every other annotation with it — and lands as
+      //   a defect inside the supervisor, on the one path whose entire job is
+      //   to make failures visible.
+      // - The DEFAULT logger (`stringLogger`) does not throw: it renders each
+      //   annotation through `Inspectable.toStringUnknown`, whose `try/catch`
+      //   falls back to `String(value)`. The same loss, quieter — the offending
+      //   annotation degrades to a bare tag, or to `[object Object]`.
+      // - `Logger.pretty` hands the value to `console.log` and prints `42n`.
+      //
+      // So the reduction happens HERE, before the logger sees anything.
+      // `position` is `String`ed rather than handed over as a `Position`
+      // despite all three shipped loggers coping with a bare `bigint`: they
+      // render it identically to `String`, so there is nothing to win, and a
+      // third-party logger that JSON-encodes its annotation map naively is a
+      // real thing.
+      //
+      // `reason` is `String`ed too, and that is the WHOLE reduction — no table
+      // of a caller's fault union, because the faults can label themselves.
+      // `Error.prototype.toString` is `name: message` and `Data.TaggedError`
+      // sets `name` to the tag, so a fault that fills `message` arrives as
+      // `'ProjectionStoreError: connection reset'` and one that does not arrives
+      // as its bare tag. This module is generic in `E`, so that is also all it
+      // could honestly do: the choice of what to say is the fault's, made once
+      // on the class, and `Cause.pretty` and `String(fault)` everywhere else
+      // read the same field.
+      const annotations = {
+        projection: key.projection,
+        partition: key.partition,
+        position: String(stored),
+        noProgressRestarts: restarts,
+        reason: String(reason)
+      };
+      if (restarts >= maxNoProgressRestarts) {
+        // Logged HERE, not by the caller, because a stalled projection must
+        // never be silent: `runProjection` returns an infallible effect, so
+        // unless somebody awaits the fibre the `ProjectionStalled` failure has
+        // no other route to an operator.
+        //
+        // `Effect.annotateLogs`, not a second argument to `logError`. Effect's
+        // log constructors are VARIADIC IN THE MESSAGE (`logError: (...message:
+        // ReadonlyArray<any>) => Effect<void>`), so an object passed second is
+        // a second message and not an annotation at all — it used to arrive as
+        // a JSON blob wedged inside the line rather than as the structured
+        // fields the reduction above was built for. Annotating puts each field
+        // where a logger can render it in its own idiom: `reason=… position=…`
+        // under the default logger, discrete keys under `annotations` in
+        // `Logger.json`. That is the logger owning the presentation, which is
+        // its job; all this module owes it is values it can serialise.
+        yield* Effect.annotateLogs(Effect.logError('kairos-es projection stalled: giving up after consecutive restarts with no checkpoint progress'), annotations);
+        // ...and then the SEAM, because a log line is not an alert. The
+        // give-up is the outcome nothing recovers from, so it gets the hook
+        // the restart already had — an application can fail a health check,
+        // exit non-zero or page from here. `onStalled`'s doc carries the
+        // asymmetry argument in full.
+        //
+        // AFTER the log, deliberately: the log is the guaranteed route out and
+        // a defect from an application's hook must not be able to silence it.
+        // The payload is the `ProjectionStalled` this give-up is about to
+        // produce, field for field and value for value — `stored` is what was
+        // just written to `lastRestartPosition`, `restarts` what was just
+        // written to `noProgressRestarts`, and `reason` is the failure the
+        // `catchAll` below will carry — so a hook and a caller awaiting the
+        // daemon never see two versions of one stall. The RAW fault rather
+        // than the log label, for the same reason the annotations take the
+        // label: a callback can read a `bigint`, a serialiser cannot.
+        yield* onStalled({
+          key,
+          noProgressRestarts: restarts,
+          position: stored,
+          reason
+        });
+        return false;
+      }
+      yield* Effect.annotateLogs(Effect.logWarning('kairos-es projection restarting from its stored checkpoint'), annotations);
+      yield* onRestart({
+        key,
+        reason,
+        noProgressRestarts: restarts
+      });
+      return true;
+    });
+    /**
+     * `Effect.retry`'s `while` IS the breaker, which is why the retry can exit
+     * in exactly one way: the schedule never terminates, so the only path out
+     * with a failure is `shouldRestart` returning `false`. The residual error is
+     * therefore always the give-up case, and mapping it to `ProjectionStalled`
+     * is total rather than a fallback for an unreachable branch.
+     *
+     * A run that *succeeds* ends the daemon quietly. For a projection that is
+     * the honest translation of "there is nothing left to read", which the
+     * shipped stores' live tails never say.
+     */
+    return yield* Effect.retry(run, {
+      schedule: restartSchedule,
+      while: shouldRestart
+    }).pipe(Effect.catchAll(reason => Effect.gen(function* () {
+      return yield* new ProjectionStalled({
+        key,
+        noProgressRestarts: yield* Ref.get(noProgressRestarts),
+        position: yield* Ref.get(lastRestartPosition),
+        reason
+      });
+    })));
+  });
+};
+//# sourceMappingURL=superviseOnProgress.js.map

package/dist/esm/superviseOnProgress.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"superviseOnProgress.js","names":["ORIGIN","Data","Effect","Ref","Schedule","Schema","MaxNoProgressRestarts","Int","pipe","positive","brand","DEFAULT_RESTART_MIN_DELAY","DEFAULT_RESTART_MAX_DELAY","DEFAULT_RESTART_RESET_AFTER","DEFAULT_MAX_NO_PROGRESS_RESTARTS","make","ProjectionStalled","TaggedError","message","noProgressRestarts","position","superviseOnProgress","run","options","key","progress","restartMinDelay","restartMaxDelay","restartResetAfter","maxNoProgressRestarts","onRestart","void","onStalled","restartSchedule","union","exponential","spaced","jittered","resetAfter","gen","lastRestartPosition","shouldRestart","reason","previous","get","stored","catchAll","succeed","progressed","restarts","set","annotations","projection","partition","String","annotateLogs","logError","logWarning","retry","schedule","while"],"sources":["../../src/superviseOnProgress.ts"],"sourcesContent":[null],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAuHA,SAASA,MAAM,QAAuB,iBAAiB;AACvD,SAASC,IAAI,EAAiBC,MAAM,EAAEC,GAAG,EAAEC,QAAQ,EAAEC,MAAM,QAAQ,QAAQ;AAG3E;;;;;;;;;;;;;;;;;;;;;;;;;AAyBA,OAAO,MAAMC,qBAAqB,gBAAGD,MAAM,CAACE,GAAG,CAACC,IAAI,cAClDH,MAAM,CAACI,QAAQ,EAAE,eACjBJ,MAAM,CAACK,KAAK,CAAC,uBAAuB,CAAC,CACtC;AAGD;;;;;;;;;;;AAWA,OAAO,MAAMC,yBAAyB,GAA2B,YAAY;AAE7E;;;;;;;;;AASA,OAAO,MAAMC,yBAAyB,GAA2B,YAAY;AAE7E;;;;;;;;;;;;;;;;;AAiBA,OAAO,MAAMC,2BAA2B,GAA2B,YAAY;AAE/E;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAwCA,MAAMC,gCAAgC,gBACpCR,qBAAqB,CAACS,IAAI,CAAC,EAAE,CAAC;AA2NhC;;;;;;;;;;;;;;;;;;;;;;;;;AAyBA,OAAM,MAAOC,iBAAkB,sBAAQf,IAAI,CAACgB,WAAW,CAAC,mBAAmB,CAKzE;EACA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;EAgDA,IAAaC,OAAOA,CAAA;IAClB,OAAO,6BAA6B,IAAI,CAACC,kBAAkB,mCAAmC,IAAI,CAACC,QAAQ,GAAG;EAChH;;AAoEF;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AA8CA,OAAO,MAAMC,mBAAmB,GAAGA,CACjCC,GAA2B,EAC3BC,OAAmC,KACO;EAC1C;EACA;EACA,MAAM;IAAEC,GAAG;IAAEC,QAAQ;IAAEC,eAAe;IAAEC,eAAe;IAAEC;EAAiB,CAAE,GAC1EL,OAAO;EACT,MAAMM,qBAAqB,GACzBN,OAAO,CAACM,qBAAqB,IAAIf,gCAAgC;EACnE,MAAMgB,SAAS,GAAGP,OAAO,CAACO,SAAS,KAAK,MAAM5B,MAAM,CAAC6B,IAAI,CAAC;EAC1D,MAAMC,SAAS,GAAGT,OAAO,CAACS,SAAS,KAAK,MAAM9B,MAAM,CAAC6B,IAAI,CAAC;EAE1D;;;;;;;;;;;;;;;;;;;;;;;;;;;;EA4BA,MAAME,eAAe,GAAG7B,QAAQ,CAAC8B,KAAK,CACpC9B,QAAQ,CAAC+B,WAAW,CAACT,eAAe,EAAE,CAAC,CAAC,EACxCtB,QAAQ,CAACgC,MAAM,CAACT,eAAe,CAAC,CACjC,CAACnB,IAAI,CAACJ,QAAQ,CAACiC,QAAQ,EAAEjC,QAAQ,CAACkC,UAAU,CAACV,iBAAiB,CAAC,CAAC;EAEjE,OAAO1B,MAAM,CAACqC,GAAG,CAAC,aAAS;IACzB;IACA;IACA;IACA,MAAMpB,kBAAkB,GAAG,OAAOhB,GAAG,CAACY,IAAI,CAAC,CAAC,CAAC;IAC7C;IACA;IACA;IACA;IACA;IACA;IACA,MAAMyB,mBAAmB,GAAG,OAAOrC,GAAG,CAACY,IAAI,CAACf,MAAM,CAAC;IAEnD;;;;;;;;;;IAUA,MAAMyC,aAAa,GAAIC,MAAS,IAC9BxC,MAAM,CAACqC,GAAG,CAAC,aAAS;MAClB,MAAMI,QAAQ,GAAG,OAAOxC,GAAG,CAACyC,GAAG,CAACJ,mBAAmB,CAAC;MAEpD;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA,MAAMK,MAAM,GAAG,OAAO3C,MAAM,CAAC4C,QAAQ,CAACrB,QAAQ,EAAE,MAC9CvB,MAAM,CAAC6C,OAAO,CAACJ,QAAQ,CAAC,CACzB;MACD,MAAMK,UAAU,GAAGH,MAAM,KAAKF,QAAQ;MAEtC,MAAMM,QAAQ,GAAGD,UAAU,GACvB,CAAC,GACD,CAAC,OAAO7C,GAAG,CAACyC,GAAG,CAACzB,kBAAkB,CAAC,IAAI,CAAC;MAC5C,OAAOhB,GAAG,CAAC+C,GAAG,CAAC/B,kBAAkB,EAAE8B,QAAQ,CAAC;MAC5C,OAAO9C,GAAG,CAAC+C,GAAG,CAACV,mBAAmB,EAAEK,MAAM,CAAC;MAE3C;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA;MACA,MAAMM,WAAW,GAAG;QAClBC,UAAU,EAAE5B,GAAG,CAAC4B,UAAU;QAC1BC,SAAS,EAAE7B,GAAG,CAAC6B,SAAS;QACxBjC,QAAQ,EAAEkC,MAAM,CAACT,MAAM,CAAC;QACxB1B,kBAAkB,EAAE8B,QAAQ;QAC5BP,MAAM,EAAEY,MAAM,CAACZ,MAAM;OACtB;MAED,IAAIO,QAAQ,IAAIpB,qBAAqB,EAAE;QACrC;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA,OAAO3B,MAAM,CAACqD,YAAY,CACxBrD,MAAM,CAACsD,QAAQ,CACb,gGAAgG,CACjG,EACDL,WAAW,CACZ;QAED;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA;QACA,OAAOnB,SAAS,CAAC;UACfR,GAAG;UACHL,kBAAkB,EAAE8B,QAAQ;UAC5B7B,QAAQ,EAAEyB,MAAM;UAChBH;SACD,CAAC;QACF,OAAO,KAAK;MACd;MAEA,OAAOxC,MAAM,CAACqD,YAAY,CACxBrD,MAAM,CAACuD,UAAU,CACf,4DAA4D,CAC7D,EACDN,WAAW,CACZ;MACD,OAAOrB,SAAS,CAAC;QAAEN,GAAG;QAAEkB,MAAM;QAAEvB,kBAAkB,EAAE8B;MAAQ,CAAE,CAAC;MAC/D,OAAO,IAAI;IACb,CAAC,CAAC;IAEJ;;;;;;;;;;;IAWA,OAAO,OAAO/C,MAAM,CAACwD,KAAK,CAACpC,GAAG,EAAE;MAC9BqC,QAAQ,EAAE1B,eAAe;MACzB2B,KAAK,EAAEnB;KACR,CAAC,CAACjC,IAAI,CACLN,MAAM,CAAC4C,QAAQ,CAAEJ,MAAM,IACrBxC,MAAM,CAACqC,GAAG,CAAC,aAAS;MAClB,OAAO,OAAO,IAAIvB,iBAAiB,CAAC;QAClCQ,GAAG;QACHL,kBAAkB,EAAE,OAAOhB,GAAG,CAACyC,GAAG,CAACzB,kBAAkB,CAAC;QACtDC,QAAQ,EAAE,OAAOjB,GAAG,CAACyC,GAAG,CAACJ,mBAAmB,CAAC;QAC7CE;OACD,CAAC;IACJ,CAAC,CAAC,CACH,CACF;EACH,CAAC,CAAC;AACJ,CAAC","ignoreList":[]}