oracular 0.5.0 → 0.5.2

package/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 0.5.2
+
+### Patch Changes
+
+- [#30](https://github.com/Yuribenjamin/oracular/pull/30) [`8be17e0`](https://github.com/Yuribenjamin/oracular/commit/8be17e032bb1b7886ce1d7278701555c39bb03cf) Thanks [@Yuribenjamin](https://github.com/Yuribenjamin)! - `oracular tune` learned three rules from a public e-commerce clickstream of 2.7 million events. A signal seen in more than 90% of counted episodes is left out of the fitted config, whatever its lift: it cannot tell episodes apart, and the few episodes without it are rarely typical sessions, so the contrast measured on them says little (on that log, `item.viewed` sat in 99% of sessions and the contrast called it strongly negative; dropping it scored better than any weight). The current half-life is kept when the best grid point improves on it by less than 1%, and the grid is printed to four decimals so a flat grid reads as flat. The report says when the fitted intent still does not beat its own base rate. The JSON report gains `share` and `ubiquitous` per signal and `beatsBaseRate` per intent.
+
+## 0.5.1
+
+### Patch Changes
+
+- [#28](https://github.com/Yuribenjamin/oracular/pull/28) [`250a32d`](https://github.com/Yuribenjamin/oracular/commit/250a32d9764aeabdff020f99074c0a512ea7194f) Thanks [@Yuribenjamin](https://github.com/Yuribenjamin)! - `oracular replay` and `oracular tune` read logs of any size. A log of more than about a hundred thousand records made both commands fail with "Maximum call stack size exceeded", because every file's entries were spread into one `push` call. Found by replaying the Retailrocket clickstream (2.7 million events).
+
 ## 0.5.0
 
 ### Minor Changes

package/README.md

@@ -382,7 +382,7 @@ npx oracular replay shadow.ndjson --config oracular.config.json
 npx oracular tune shadow.ndjson --config oracular.config.json
 ```
 
-`tune` seeds a config from the same log: grouped by install, the base rate becomes the prior, the spread of per-install base rates the prior strength, each signal's lift over the episodes without it its weight, and the half-life that predicts the log best wins a grid search scored by `replay`. The counts come from the engine itself, so tune cannot disagree with what a device would learn. It prints sample sizes and a 90% interval per weight, and the config to paste; the engine corrects it per user from there. The [Shadow mode guide](https://yuribenjamin.github.io/oracular/guide/shadow-mode) has the workflow, the record format and both commands.
+`tune` seeds a config from the same log: grouped by install, the base rate becomes the prior, the spread of per-install base rates the prior strength, each signal's lift over the episodes without it its weight, and the half-life that predicts the log best wins a grid search scored by `replay`. The counts come from the engine itself, so tune cannot disagree with what a device would learn. It prints sample sizes and a 90% interval per weight, and the config to paste; the engine corrects it per user from there. Three rules learned from a real clickstream keep it honest: a signal seen in nearly every episode is left out, a flat grid keeps your half-life, and the report says when an intent still does not beat its base rate. The [Shadow mode guide](https://yuribenjamin.github.io/oracular/guide/shadow-mode) has the workflow, the record format and both commands.
 
 ## Design rules
 

package/dist/cli.js

@@ -444,6 +444,8 @@ var DEFAULT_HALF_LIVES = [HOUR2, 6 * HOUR2, DAY2, 3 * DAY2, 7 * DAY2, 14 * DAY2,
 var DEFAULT_PRIOR_EPISODES = 10;
 var MIN_INSTALLS_FOR_SPREAD = 10;
 var STRENGTH_RANGE = [2, 50];
+var UBIQUITY_SHARE = 0.9;
+var FLAT_GRID_GAIN = 0.01;
 var Z90 = 1.6449;
 var NONE = { hits: 0, misses: 0 };
 function stepDelta(prev, next) {
@@ -498,6 +500,8 @@ function fitSignal(signal, current, tally, strength, priorEpisodes) {
   const contrast = nWith + without === 0 ? 0 : nWith * without / (nWith + without);
   const [qLow, qHigh] = wilson(hits, nWith, Z90);
   const rounded = Math.round(strength * l * 100) / 100;
+  const share = tally.episodes === 0 ? 0 : nWith / tally.episodes;
+  const ubiquitous = nWith > UBIQUITY_SHARE * tally.episodes;
   return {
     signal,
     current,
@@ -508,7 +512,9 @@ function fitSignal(signal, current, tally, strength, priorEpisodes) {
     qWith,
     qWithout,
     lift: l,
-    weight: nWith === 0 || rounded === 0 ? null : rounded,
+    weight: nWith === 0 || ubiquitous || rounded === 0 ? null : rounded,
+    share,
+    ubiquitous,
     low: strength * lift(qLow, qWithout),
     high: strength * lift(qHigh, qWithout),
     contrast,
@@ -707,6 +713,7 @@ function tune(entries, config, options = {}) {
       return { halfLife: halfLife2, brier: scored?.brier ?? null, points: scored?.points ?? 0 };
     });
     const scoredPoints = curve.filter((p) => p.brier !== null);
+    const atCurrent = curve.find((p) => p.halfLife === intent.halfLife);
     let halfLife;
     if (scoredPoints.length === 0) {
       halfLife = {
@@ -723,17 +730,33 @@ function tune(entries, config, options = {}) {
       const ties = scoredPoints.filter((p) => p.brier - best <= 1e-12);
       const nearest = (p) => Math.abs(Math.log(p.halfLife / intent.halfLife));
       const chosen = ties.reduce((a, b) => nearest(b) < nearest(a) ? b : a);
-      halfLife = {
-        value: chosen.halfLife,
-        current: intent.halfLife,
-        source: "grid",
-        note: `lowest Brier score over the grid, ${best.toFixed(3)}${brierBaseline === null ? "" : ` (always predicting the base rate: ${brierBaseline.toFixed(3)})`}`,
-        window,
-        curve,
-        brierBaseline
-      };
+      const baseline = brierBaseline === null ? "" : ` (always predicting the base rate: ${brierBaseline.toFixed(4)})`;
+      const gain = atCurrent.brier === null || atCurrent.brier <= 0 ? Number.POSITIVE_INFINITY : (atCurrent.brier - best) / atCurrent.brier;
+      if (chosen.halfLife !== intent.halfLife && gain < FLAT_GRID_GAIN) {
+        halfLife = {
+          value: intent.halfLife,
+          current: intent.halfLife,
+          source: "kept",
+          note: `kept: the grid is flat, the best point (${formatDuration(chosen.halfLife)}, ${best.toFixed(4)}) improves on the current value (${atCurrent.brier.toFixed(4)}) by ${(gain * 100).toFixed(2)}%, less than ${FLAT_GRID_GAIN * 100}%${baseline}`,
+          window,
+          curve,
+          brierBaseline
+        };
+      } else {
+        halfLife = {
+          value: chosen.halfLife,
+          current: intent.halfLife,
+          source: "grid",
+          note: `lowest Brier score over the grid, ${best.toFixed(4)}${baseline}`,
+          window,
+          curve,
+          brierBaseline
+        };
+      }
     }
     fitted = withIntent(fitted, name, { ...fittedIntent, halfLife: halfLife.value });
+    const scoredAt = curve.find((p) => p.halfLife === halfLife.value);
+    const beatsBaseRate = scoredAt.brier === null || brierBaseline === null ? null : scoredAt.brier < brierBaseline;
     intents.push({
       intent: name,
       outcome: intent.outcome,
@@ -745,6 +768,7 @@ function tune(entries, config, options = {}) {
       priorStrength: strength,
       halfLife,
       priorEpisodes,
+      beatsBaseRate,
       signals,
       candidates
     });
@@ -771,7 +795,13 @@ var plural = (n, word) => `${n} ${word}${n === 1 ? "" : "s"}`;
 function note(fit, priorEpisodes) {
   if (fit.hits + fit.misses === 0) return "never seen in a counted episode";
   const notes = [];
-  if (fit.weight === null) notes.push("no measurable lift, left out of the config");
+  if (fit.ubiquitous) {
+    notes.push(
+      `in ${(fit.share * 100).toFixed(1)}% of episodes, so it cannot tell them apart; left out of the config`
+    );
+  } else if (fit.weight === null) {
+    notes.push("no measurable lift, left out of the config");
+  }
   if (fit.thin) notes.push(`thin: ${fit.contrast.toFixed(1)} of ${priorEpisodes} contrast`);
   return notes.join("; ");
 }
@@ -841,9 +871,18 @@ function formatIntent2(fit) {
   out.push("");
   out.push(
     `Half-life grid, Brier score against "the outcome follows within ${formatDuration(fit.halfLife.window)}" (lower is better): ${fit.halfLife.curve.map(
-      (p) => `${formatDuration(p.halfLife)} ${num(p.brier, 3)}${p.halfLife === fit.halfLife.value && fit.halfLife.source === "grid" ? " *" : ""}`
+      (p) => `${formatDuration(p.halfLife)} ${num(p.brier, 4)}${p.halfLife === fit.halfLife.value && p.brier !== null ? " *" : ""}`
     ).join(", ")}`
   );
+  if (fit.beatsBaseRate === false) {
+    const scored = fit.halfLife.curve.find((p) => p.halfLife === fit.halfLife.value);
+    const window = formatDuration(fit.halfLife.window);
+    const why = fit.signals.some((f) => f.weight !== null) ? `Its signals do not separate the events that ${fit.outcome} follows within ${window} from the rest.` : `No signal is left in the config, so p never leaves the prior, and the prior (the share of episodes that end with ${fit.outcome}) is not the share of events that ${fit.outcome} follows within ${window}.`;
+    out.push("");
+    out.push(
+      `This intent does not beat its base rate: the fitted config scores ${num(scored?.brier ?? null, 4)} where always predicting the base rate scores ${num(fit.halfLife.brierBaseline, 4)} (lower is better). ${why} Drop the intent, or find signals that tell its episodes apart (--all-events measures every other event in the log).`
+    );
+  }
   if (fit.candidates.length > 0) {
     out.push("");
     out.push(
@@ -1151,7 +1190,7 @@ function readLogs(files, io) {
     const parsed = parseLog(io.readFile(resolve(io.cwd, file)), file);
     for (const warning of parsed.warnings) io.stderr(`oracular: warning: ${warning}
 `);
-    entries.push(...parsed.entries);
+    for (const entry of parsed.entries) entries.push(entry);
   }
   if (entries.length === 0) throw new Error("the log holds no records");
   return entries;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "oracular",
-  "version": "0.5.0",
+  "version": "0.5.2",
   "description": "On-device intent engine. Your app learns when the user is ready. Nothing leaves the device.",
   "keywords": [
     "oracular",