oracular 0.10.0 → 0.12.0

package/CHANGELOG.md

@@ -1,5 +1,27 @@
 # Changelog
 
+## 0.12.0
+
+### Minor Changes
+
+- [#62](https://github.com/Yuribenjamin/oracular/pull/62) [`0502b78`](https://github.com/Yuribenjamin/oracular/commit/0502b789049cb8ef8643af587ec94b78ce0c39b1) Thanks [@Yuribenjamin](https://github.com/Yuribenjamin)! - **The clock rule: one glitch cannot move the engine.** 0.10.0 made every write trust the latest event's time, so that one event stamped in the future could no longer hold an intent until the clock caught up. A production re-test found the cost: a clock that steps *backwards* (a phone that wakes up in 1970 before it syncs, a seconds-for-milliseconds bug) pulled every anchor back with it, and the next event decayed the whole gap again and closed the session, recording a miss nobody earned. The two cases are one ambiguity: after one backward step the engine cannot tell whether the new stamp or the old anchor is the wrong clock. It now waits one event. The engine keeps a clock, the latest event time it trusts; an event stamped behind it counts now, at the clock's time, and its stamp is held. If the next event is also behind the clock, the clock was corrected: every anchor ahead of the held stamp (evidence, learned counts, reveals, outcomes, the open episode, the episodes waiting under `attribution`) moves back to it, and decay and cooldowns resume from there. If the next event is at or after the clock, the held stamp was a glitch, and nothing moved. Monotone clocks see no difference at all, and every number in the docs is unchanged. What changes: a clock that was ahead heals two events after it is corrected rather than one, and heals every intent at once rather than only the ones later events touch; a reveal stamped ahead of the clock is pinned by the next event on any intent; a single backward stamp wipes nothing and closes nothing.
+  
+  Two costs, stated. The first event after a correction is processed at the uncorrected time, so the intents it touches are decayed to that time and their episodes waiting under `attribution` can settle as misses before the second event moves everything back. And two bad stamps in a row read as a correction, which costs exactly what 0.10 paid on every backward step: the anchors move to the bad time and the next good event decays the gap again. One case no rule covers: a device that boots in 1970 and syncs forward later makes one monotone jump of decades, indistinguishable from a long absence, so hold `observe` until the device clock is trusted.
+  
+  **State version 4.** The clock is stored, so `STATE_VERSION` is 4 and version 3 states (0.6.0 to 0.11.x) migrate on load: the clock is set to the latest time the stored anchors were written at, nothing is held, and no estimate changes. Under rule 5 of the stability policy, 0.11 reads a version 4 state as its own; what it loses is the clock: a backward stamp awaiting confirmation is forgotten, and the next event is trusted the way 0.11 trusted every event.
+
+### Patch Changes
+
+- [#62](https://github.com/Yuribenjamin/oracular/pull/62) [`0502b78`](https://github.com/Yuribenjamin/oracular/commit/0502b789049cb8ef8643af587ec94b78ce0c39b1) Thanks [@Yuribenjamin](https://github.com/Yuribenjamin)! - **Size.** The whole `oracular` entry goes from 7.72 kB to 8.09 kB minified and gzipped with the clock rule and its state version, the quickstart import (`createState`, `observe`, `shouldReveal`, `markRevealed`) from 4.63 kB to 4.85 kB. That crosses the 7.9 kB ceiling the stability page set in 0.8, so the ceiling moves as it did then, to the shipped size plus about 10%: 8.9 kB, with the quickstart budget at 5 kB.
+
+- [#62](https://github.com/Yuribenjamin/oracular/pull/62) [`0502b78`](https://github.com/Yuribenjamin/oracular/commit/0502b789049cb8ef8643af587ec94b78ce0c39b1) Thanks [@Yuribenjamin](https://github.com/Yuribenjamin)! - **Presets are read-only in the types too.** `presets` and `presetSources` are `DeepReadonly<...>`, matching the frozen objects: assigning into a preset is a type error, spreading one is unchanged.
+
+## 0.11.0
+
+### Minor Changes
+
+- [#58](https://github.com/Yuribenjamin/oracular/pull/58) [`d6d1e87`](https://github.com/Yuribenjamin/oracular/commit/d6d1e87e176fc0112384e47fb4782e12d6fa1d1a) Thanks [@Yuribenjamin](https://github.com/Yuribenjamin)! - **Learning no longer drifts without outcomes.** The outcome rates a signal is judged by, with it and without it, were Laplace-smoothed, which centres both on 0.5 with one pseudo-hit each: two groups with no outcome at all then differ only by their sizes, so with zero outcomes a common signal drifted negative and a rare one positive, and a weight could change sign on no evidence (a weight of 2 seen in 90 of 100 outcome-free episodes ended at −0.59). Both rates are now shrunk toward the intent's own base rate by two pseudo-episodes, `(hits + 2·q₀) / (n + 2)` with `q₀ = outcomes / episodes`: a signal whose outcome rate equals the base rate learns exactly zero whatever the sample sizes, with no outcome every rate is 0 and no weight moves away from the developer's, and a weight never changes sign without outcomes on at least one side. The association-rule baseline becomes the base rate itself. `tune` fits with the same smoothing, so its weights, intervals and lifts move a little on small samples (a signal seen once with the outcome over a low base rate is worth a little less than before); on large samples the change is in the third decimal. Every learned number in the documentation's generated tables was regenerated. The state format is unchanged, and existing learned counts are kept: only the weight read from them moves.
+
 ## 0.10.0
 
 ### Minor Changes

package/README.md

@@ -193,8 +193,9 @@ A signal's weight is then a blend of your prior and the learned estimate:
 ```
 nWith    = hits + misses                                      episodes with the signal
 nWithout = episodes − nWith                                   episodes without it
-qWith    = (hits + 1) / (nWith + 2)                           outcome rate with the signal
-qWithout = (outcomes − hits + 1) / (nWithout + 2)             outcome rate without it
+q₀       = outcomes / episodes                                the intent's own base rate
+qWith    = (hits + 2·q₀) / (nWith + 2)                        outcome rate with the signal
+qWithout = (outcomes − hits + 2·q₀) / (nWithout + 2)          outcome rate without it
 lift     = (qWith − qWithout) / (1 − qWithout)  if qWith ≥ qWithout,  else  (qWith − qWithout) / qWithout   → in [−1, 1]
 learned  = gain × |w₀| × lift                                 0 until the signal has history
 nEff     = (nWith × nWithout) / (nWith + nWithout)            0 if either side is 0
@@ -269,7 +270,7 @@ serialize(state): string
 deserialize(json: string, opts?: { config?: EngineConfig; migrate?: Migrate }): EngineState
 validateConfig(config: unknown): EngineConfig   // the checks createState runs, without a state; returns the normalized copy
 ms.seconds(n), ms.minutes(n), ms.hours(n), ms.days(n), ms.weeks(n)   // durations in milliseconds
-STATE_VERSION // 3
+STATE_VERSION // 4
 RECORD_VERSION // 1
 ```
 
@@ -339,7 +340,7 @@ interface SignalExplanation {
 
 ### Time
 
-`at` is a timestamp. Every duration (`halfLife`, `cooldown`, `episodeTimeout`, `learning.halfLife`) is in the same unit as `at`. Milliseconds from `Date.now()` are the obvious choice, but the library never calls `Date.now()` itself: you pass time in, which keeps every function deterministic and makes tests trivial. A clock that jumps backwards is tolerated: the evidence is held (no decay, no error) and the anchor moves to the new time, so decay resumes from there. One event stamped ahead of the clock therefore cannot hold an intent until the clock catches up, and a reveal or outcome stamped ahead of it counts as having happened now: its cooldown runs from the current time until the next event pins it there. When `at` is in milliseconds, `ms.days(7)`, `ms.hours(3)` and `ms.minutes(30)` spell the durations; they are plain multiplications.
+`at` is a timestamp. Every duration (`halfLife`, `cooldown`, `episodeTimeout`, `learning.halfLife`) is in the same unit as `at`. Milliseconds from `Date.now()` are the obvious choice, but the library never calls `Date.now()` itself: you pass time in, which keeps every function deterministic and makes tests trivial. The engine keeps a clock, the latest event time it trusts, and resolves every stamp against it. An event stamped behind the clock is tolerated: it counts now, at the clock's time, with no decay and no error, and its stamp is held. If the next event is also behind the clock, the clock was corrected (a device whose time was ahead caught up): every anchor ahead of the held stamp moves back to it, and decay and cooldowns resume from the corrected time. If the next event is at or after the clock, the held stamp was one glitch (a phone that woke up in 1970 before it synced) and nothing moved. So one event stamped years ahead holds an intent for at most the two events that follow it, and one stamped years behind wipes nothing and closes no session. A reveal or outcome stamped ahead of the clock counts as having happened now, and the next event pins it to its own time. A device that boots in 1970 and syncs forward later makes one monotone jump that no rule can tell from a long absence, so hold `observe` until its clock is trusted. When `at` is in milliseconds, `ms.days(7)`, `ms.hours(3)` and `ms.minutes(30)` spell the durations; they are plain multiplications.
 
 ## Persistence
 
@@ -357,7 +358,7 @@ const state = deserialize(saved, { config });
 
 The new config is validated and swapped in. Intents and signals you no longer declare are dropped (α and β are recomputed from what remains), newly declared intents start at their prior, and everything learned about signals that still exist is kept.
 
-States saved by an older version of this library are upgraded on load: `deserialize` migrates the state shapes written by oracular 0.1.0 to 0.5.x (`STATE_VERSION` 1 and 2) to the current one on its own. A `migrate` function is consulted only for a version the library does not know, such as a state written by a newer oracular before a rollback; without one, that throws a `RangeError`:
+States saved by an older version of this library are upgraded on load: `deserialize` migrates the state shapes written by oracular 0.1.0 to 0.11.x (`STATE_VERSION` 1 to 3) to the current one on its own. A `migrate` function is consulted only for a version the library does not know, such as a state written by a newer oracular before a rollback; without one, that throws a `RangeError`:
 
 ```ts
 deserialize(saved, {

package/dist/cli.js

@@ -437,6 +437,9 @@ import {
 function laplace(successes, total) {
   return (successes + 1) / (total + 2);
 }
+function smoothRate(successes, total, rate) {
+  return (successes + 2 * rate) / (total + 2);
+}
 function lift(q, b, type = "normalized-difference") {
   if (type === "absolute-difference") return q - b;
   if (type !== "normalized-difference") {
@@ -528,9 +531,10 @@ function fitSignal(signal, current, tally, strength, priorEpisodes, liftType) {
   const nWith = hits + misses;
   const without = Math.max(0, tally.episodes - nWith);
   const outcomesWithout = Math.min(without, Math.max(0, tally.outcomes - hits));
-  const qWith = laplace(hits, nWith);
-  const qWithout = laplace(outcomesWithout, without);
-  const baseline = liftType === "association-rule" ? (hits + outcomesWithout + 2) / (nWith + without + 4) : qWithout;
+  const base = tally.episodes === 0 ? 0 : tally.outcomes / tally.episodes;
+  const qWith = smoothRate(hits, nWith, base);
+  const qWithout = smoothRate(outcomesWithout, without, base);
+  const baseline = liftType === "association-rule" ? smoothRate(hits + outcomesWithout, nWith + without, base) : qWithout;
   const noContrast = without === 0;
   const rawLift = noContrast ? null : nWith === 0 ? 0 : lift(qWith, baseline, liftType);
   const score = rawLift === null ? null : nWith === 0 ? 0 : learningScore(rawLift, liftType);

package/dist/index.cjs

@@ -79,8 +79,8 @@ function lowerBound(alpha, beta, z, bound) {
 function repeatFactor(n) {
   return Math.log1p(1 / n) / Math.LN2;
 }
-function laplace(successes, total) {
-  return (successes + 1) / (total + 2);
+function smoothRate(successes, total, rate) {
+  return (successes + 2 * rate) / (total + 2);
 }
 function lift(q, b, type = "normalized-difference") {
   if (type === "absolute-difference") return q - b;
@@ -138,8 +138,9 @@ function mapRecord(record2, fn) {
 var NONE = { hits: 0, misses: 0 };
 function decayLearned(state, learning, at) {
   if (state.learnedAt === null) return { ...state, learnedAt: at };
+  if (!(at > state.learnedAt)) return state;
   const d = learning.halfLife === null ? 1 : decayFactor(at - state.learnedAt, learning.halfLife);
-  if (d === 1) return at === state.learnedAt ? state : { ...state, learnedAt: at };
+  if (d === 1) return { ...state, learnedAt: at };
   return {
     ...state,
     learnedAt: at,
@@ -154,12 +155,13 @@ function signalWeights(state, intent, learning, signal) {
   }
   const record2 = getOwn(state.learned, signal) ?? NONE;
   const nWith = record2.hits + record2.misses;
-  const qWith = laplace(record2.hits, nWith);
   const totalEpisodes = state.episodes.hits + state.episodes.misses;
   const nWithout = Math.max(0, totalEpisodes - nWith);
   const outcomesWithout = Math.min(nWithout, Math.max(0, state.episodes.hits - record2.hits));
-  const qWithout = laplace(outcomesWithout, nWithout);
-  const baseline = learning.lift === "association-rule" ? (record2.hits + outcomesWithout + 2) / (nWith + nWithout + 4) : qWithout;
+  const base = totalEpisodes === 0 ? 0 : state.episodes.hits / totalEpisodes;
+  const qWith = smoothRate(record2.hits, nWith, base);
+  const qWithout = smoothRate(outcomesWithout, nWithout, base);
+  const baseline = learning.lift === "association-rule" ? smoothRate(record2.hits + outcomesWithout, nWith + nWithout, base) : qWithout;
   const score = learningScore(lift(qWith, baseline, learning.lift), learning.lift);
   const learned = nWith === 0 ? 0 : learning.gain * Math.abs(initial) * score;
   const nEff = nWith + nWithout === 0 ? 0 : nWith * nWithout / (nWith + nWithout);
@@ -250,8 +252,9 @@ function withContributions(state, intent, contributions) {
 }
 function decayIntent(state, intent, at) {
   if (state.updatedAt === null) return { ...state, updatedAt: at };
+  if (!(at > state.updatedAt)) return state;
   const d = decayFactor(at - state.updatedAt, intent.halfLife);
-  if (d === 1) return at === state.updatedAt ? state : { ...state, updatedAt: at };
+  if (d === 1) return { ...state, updatedAt: at };
   const contributions = mapRecord(state.contributions, (c) => ({
     alpha: c.alpha * d,
     beta: c.beta * d
@@ -569,7 +572,7 @@ function markResolved(episode, intent) {
 }
 
 // src/version.ts
-var STATE_VERSION = 3;
+var STATE_VERSION = 4;
 var RECORD_VERSION = 1;
 
 // src/state.ts
@@ -579,10 +582,16 @@ function createState(config) {
   for (const [name, intent] of Object.entries(normalized.intents)) {
     intents[name] = initIntent(intent);
   }
-  return { version: STATE_VERSION, config: normalized, intents, episode: emptyEpisode() };
+  return {
+    version: STATE_VERSION,
+    config: normalized,
+    intents,
+    episode: emptyEpisode(),
+    clock: { at: null, behindAt: null }
+  };
 }
 function requireState(state) {
-  if (!isRecord(state) || state.version !== STATE_VERSION || !isRecord(state.config) || !isRecord(state.intents) || !isRecord(state.episode)) {
+  if (!isRecord(state) || state.version !== STATE_VERSION || !isRecord(state.config) || !isRecord(state.intents) || !isRecord(state.episode) || !isRecord(state.clock)) {
     throw new TypeError("oracular: state must be an EngineState from createState or deserialize");
   }
   return state;
@@ -719,8 +728,10 @@ function explain(state, intent, at, opts) {
 
 // src/observe.ts
 function observe(state, event) {
-  let s = requireState(state);
-  const { name, at } = validateEvent(event);
+  const { name, at: stamp } = validateEvent(event);
+  const synced = syncClock(requireState(state), stamp);
+  let s = synced.state;
+  const at = synced.at;
   const closeAt = timedOutAt(s.episode, s.config.episodeTimeout, at);
   if (closeAt !== null) s = closeEpisode(s, closeAt);
   const learning = resolveLearning(s.config);
@@ -733,7 +744,6 @@ function observe(state, event) {
     let is = getIntentState(s, intentName, intent);
     is = decayIntent(is, intent, at);
     is = settlePending(is, intent, learning, at);
-    is = pinAnchors(is, at);
     if (isSignal) {
       const { effective } = signalWeights(is, intent, learning, name);
       const n = occurrences(episode, intentName, name) + 1;
@@ -754,15 +764,36 @@ function observe(state, event) {
   }
   return { ...s, intents, episode };
 }
-function pinAnchors(is, at) {
-  const reveal = is.lastRevealedAt !== null && is.lastRevealedAt > at;
-  const outcome = is.lastOutcomeAt !== null && is.lastOutcomeAt > at;
-  if (!reveal && !outcome) return is;
-  return {
-    ...is,
-    lastRevealedAt: reveal ? at : is.lastRevealedAt,
-    lastOutcomeAt: outcome ? at : is.lastOutcomeAt
-  };
+function syncClock(state, stamp) {
+  const { at, behindAt } = state.clock;
+  if (at === null || stamp >= at) return { state: pinTo(state, stamp), at: stamp };
+  if (behindAt !== null) return syncClock(pinTo(state, behindAt), stamp);
+  return { state: { ...state, clock: { at, behindAt: stamp } }, at };
+}
+var INTENT_ANCHORS = ["updatedAt", "learnedAt", "lastRevealedAt", "lastOutcomeAt"];
+var EPISODE_ANCHORS = ["startedAt", "lastEventAt"];
+function pinTo(state, t) {
+  let intents = state.intents;
+  for (const [name, is] of Object.entries(state.intents)) {
+    let pinned = pinKeys(is, INTENT_ANCHORS, t);
+    if (is.pending.some((p) => p.endedAt > t)) {
+      pinned = {
+        ...pinned,
+        pending: is.pending.map((p) => p.endedAt > t ? { ...p, endedAt: t } : p)
+      };
+    }
+    if (pinned !== is) intents = { ...intents, [name]: pinned };
+  }
+  const episode = pinKeys(state.episode, EPISODE_ANCHORS, t);
+  return { ...state, intents, episode, clock: { at: t, behindAt: null } };
+}
+function pinKeys(obj, keys, t) {
+  let out = obj;
+  for (const key of keys) {
+    const value = obj[key];
+    if (typeof value === "number" && value > t) out = { ...out, [key]: t };
+  }
+  return out;
 }
 function endEpisode(state, at) {
   const s = requireState(state);
@@ -888,7 +919,8 @@ function createEndRecord(at, opts) {
 // src/migrate.ts
 var MIGRATIONS = {
   1: fromV1,
-  2: fromV2
+  2: fromV2,
+  3: fromV3
 };
 function canMigrate(version) {
   return hasOwn(MIGRATIONS, String(version));
@@ -929,6 +961,9 @@ function fromV2(stored) {
   }
   return { ...stored, version: 3, intents };
 }
+function fromV3(stored) {
+  return { ...stored, version: 4, clock: { at: null, behindAt: null } };
+}
 
 // src/serialize.ts
 function serialize(state) {
@@ -1074,6 +1109,13 @@ function episodeState(value, path) {
     intents: mapOwn(r.intents, `${path}.intents`, episodeIntent)
   };
 }
+function clockState(value, path) {
+  const r = record(value, path);
+  return {
+    at: nullOrFinite(r.at, `${path}.at`),
+    behindAt: nullOrFinite(r.behindAt, `${path}.behindAt`)
+  };
+}
 function parseState(value) {
   const r = record(value, "state");
   if (r.version !== STATE_VERSION) {
@@ -1085,7 +1127,8 @@ function parseState(value) {
     // when it is only stored, and the config passed in replaces it anyway.
     config: normalizeConfig(r.config, false),
     intents: mapOwn(r.intents, "state.intents", intentState),
-    episode: episodeState(r.episode, "state.episode")
+    episode: episodeState(r.episode, "state.episode"),
+    clock: clockState(r.clock, "state.clock")
   };
 }
 function requireComputable(state) {
@@ -1159,9 +1202,24 @@ function reconcile(raw, config) {
       startedAt: raw.episode.startedAt,
       lastEventAt: raw.episode.startedAt === null ? null : raw.episode.lastEventAt,
       intents: episodeIntents
-    }
+    },
+    clock: trustedClock(raw.clock, intents, raw.episode)
   };
 }
+function trustedClock(clock, intents, episode) {
+  let at = clock.at;
+  const see = (value) => {
+    if (value !== null && (at === null || value > at)) at = value;
+  };
+  see(episode.startedAt);
+  see(episode.lastEventAt);
+  for (const is of Object.values(intents)) {
+    see(is.updatedAt);
+    see(is.lastOutcomeAt);
+  }
+  const behindAt = clock.behindAt !== null && at !== null && clock.behindAt < at ? clock.behindAt : null;
+  return at === clock.at && behindAt === clock.behindAt ? clock : { at, behindAt };
+}
 
 // src/units.ts
 var ms = {