trajectories 0.0.0 → 0.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (205) hide show
  1. package/CHANGELOG.md +36 -0
  2. package/LICENSE +201 -0
  3. package/README.md +288 -2
  4. package/dist/cli.js +30210 -0
  5. package/dist/convert.d.ts +31 -0
  6. package/dist/convert.js +431 -0
  7. package/dist/deepagents_checkpoint.py +362 -0
  8. package/dist/index.d.ts +7 -0
  9. package/dist/index.js +8402 -0
  10. package/dist/log-baseline.d.ts +26 -0
  11. package/dist/log-baseline.js +104 -0
  12. package/dist/log-content.d.ts +16 -0
  13. package/dist/log-content.js +80 -0
  14. package/dist/log-io.d.ts +10 -0
  15. package/dist/log-io.js +121 -0
  16. package/dist/log-read.d.ts +24 -0
  17. package/dist/log-read.js +117 -0
  18. package/dist/log-reconcile.d.ts +5 -0
  19. package/dist/log-reconcile.js +38 -0
  20. package/dist/log-records.d.ts +35 -0
  21. package/dist/log-records.js +85 -0
  22. package/dist/log-recovery.d.ts +18 -0
  23. package/dist/log-recovery.js +123 -0
  24. package/dist/log-stage.d.ts +14 -0
  25. package/dist/log-stage.js +138 -0
  26. package/dist/log-state.d.ts +89 -0
  27. package/dist/log-state.js +92 -0
  28. package/dist/log-switch.d.ts +3 -0
  29. package/dist/log-switch.js +98 -0
  30. package/dist/log-types.d.ts +129 -0
  31. package/dist/log-types.js +8 -0
  32. package/dist/log.d.ts +8 -0
  33. package/dist/log.js +62 -0
  34. package/dist/migrate.d.ts +29 -0
  35. package/dist/migrate.js +173 -0
  36. package/dist/normalization-runtime.d.ts +1 -0
  37. package/dist/normalization-runtime.js +3150 -0
  38. package/dist/portable.d.ts +9 -0
  39. package/dist/portable.js +26 -0
  40. package/dist/readers/canonical.d.ts +13 -0
  41. package/dist/readers/canonical.js +54 -0
  42. package/dist/readers/cline.d.ts +10 -0
  43. package/dist/readers/cline.js +107 -0
  44. package/dist/readers/cursor.d.ts +20 -0
  45. package/dist/readers/cursor.js +374 -0
  46. package/dist/readers/deepagents.d.ts +41 -0
  47. package/dist/readers/deepagents.js +163 -0
  48. package/dist/readers/fx.d.ts +11 -0
  49. package/dist/readers/fx.js +573 -0
  50. package/dist/readers/github-copilot.d.ts +8 -0
  51. package/dist/readers/github-copilot.js +216 -0
  52. package/dist/readers/grok-build.d.ts +12 -0
  53. package/dist/readers/grok-build.js +391 -0
  54. package/dist/readers/guards.d.ts +4 -0
  55. package/dist/readers/guards.js +121 -0
  56. package/dist/readers/pi-compaction.d.ts +7 -0
  57. package/dist/readers/pi-compaction.js +112 -0
  58. package/dist/readers/pi-context.d.ts +13 -0
  59. package/dist/readers/pi-context.js +64 -0
  60. package/dist/store.d.ts +25 -0
  61. package/dist/store.js +218 -0
  62. package/dist/types.d.ts +91 -0
  63. package/dist/types.js +1 -0
  64. package/dist/writers/claude-code.d.ts +5 -0
  65. package/dist/writers/claude-code.js +51 -0
  66. package/dist/writers/cline.d.ts +3 -0
  67. package/dist/writers/cline.js +77 -0
  68. package/dist/writers/codex.d.ts +3 -0
  69. package/dist/writers/codex.js +32 -0
  70. package/dist/writers/cursor.d.ts +3 -0
  71. package/dist/writers/cursor.js +197 -0
  72. package/dist/writers/deepagents.d.ts +3 -0
  73. package/dist/writers/deepagents.js +116 -0
  74. package/dist/writers/entropy.d.ts +5 -0
  75. package/dist/writers/entropy.js +33 -0
  76. package/dist/writers/fx.d.ts +3 -0
  77. package/dist/writers/fx.js +135 -0
  78. package/dist/writers/github-copilot.d.ts +3 -0
  79. package/dist/writers/github-copilot.js +71 -0
  80. package/dist/writers/grok-build.d.ts +3 -0
  81. package/dist/writers/grok-build.js +43 -0
  82. package/dist/writers/helpers.d.ts +17 -0
  83. package/dist/writers/helpers.js +75 -0
  84. package/dist/writers/omp.d.ts +3 -0
  85. package/dist/writers/omp.js +7 -0
  86. package/dist/writers/opencode.d.ts +3 -0
  87. package/dist/writers/opencode.js +65 -0
  88. package/dist/writers/pi.d.ts +3 -0
  89. package/dist/writers/pi.js +45 -0
  90. package/docs/cline-native-format.md +43 -0
  91. package/docs/copilot-native-format.md +78 -0
  92. package/docs/cursor-native-format.md +103 -0
  93. package/docs/deepagents-native-format.md +98 -0
  94. package/docs/evidence-api.md +170 -0
  95. package/docs/fx-native-format.md +87 -0
  96. package/docs/harness-parity-bootstrap.md +78 -0
  97. package/docs/harness-support-parity.md +107 -0
  98. package/docs/inspirations.md +103 -0
  99. package/docs/ledger-migration.md +141 -0
  100. package/docs/pi-compaction-context.md +75 -0
  101. package/docs/pi-provider-context.md +76 -0
  102. package/docs/research/2026-10-06-huihua-and-oar.md +176 -0
  103. package/docs/research/ledger/README.md +96 -0
  104. package/docs/research/ledger/confer/2026-09-13-round-10/README.md +61 -0
  105. package/docs/research/ledger/confer/2026-09-13-round-2/cass.md +72 -0
  106. package/docs/research/ledger/confer/2026-09-13-round-3/README.md +70 -0
  107. package/docs/research/ledger/confer/2026-09-13-round-5/README.md +61 -0
  108. package/docs/research/ledger/confer/2026-09-13-round-6/README.md +68 -0
  109. package/docs/research/ledger/confer/2026-09-13-round-7/README.md +64 -0
  110. package/docs/research/ledger/confer/2026-09-13-round-9/README.md +59 -0
  111. package/docs/research/ledger/confer/2026-09-14-wake/wake.md +146 -0
  112. package/docs/research/ledger/docs/inspirations.md +320 -0
  113. package/docs/research/ledger/docs/pr2-pr6-integration.md +53 -0
  114. package/docs/research/ledger/docs/research/2026-09-07-confer-inspirations.md +163 -0
  115. package/docs/research/ledger/docs/research/2026-09-07-letta-continuous-service.md +104 -0
  116. package/docs/research/ledger/docs/research/2026-09-08-claude-history-retrieval.md +117 -0
  117. package/docs/research/ledger/docs/research/2026-09-09-agentsview-native-discovery.md +87 -0
  118. package/docs/research/ledger/docs/research/2026-09-09-daemon-observation.md +231 -0
  119. package/docs/research/ledger/docs/research/2026-09-09-discovery-pagination.md +114 -0
  120. package/docs/research/ledger/docs/research/2026-09-09-harness-transcript-contract.md +145 -0
  121. package/docs/research/ledger/docs/research/2026-09-09-native-history-discovery.md +180 -0
  122. package/docs/research/ledger/docs/research/2026-09-09-native-history-implementation.md +137 -0
  123. package/docs/research/ledger/docs/research/2026-09-09-other-inspirations.md +279 -0
  124. package/docs/research/ledger/docs/research/2026-09-09-trajectory-and-letta-code.md +128 -0
  125. package/docs/research/ledger/docs/research/2026-09-13-diagnostics-coverage.md +116 -0
  126. package/docs/research/ledger/docs/research/2026-09-13-fts-rebuild.md +83 -0
  127. package/docs/research/ledger/docs/research/2026-09-13-read-path-split.md +74 -0
  128. package/docs/research/ledger/docs/research/2026-09-13-runtime-rollout.md +102 -0
  129. package/docs/research/ledger/docs/research/2026-09-13-scan-outcomes.md +80 -0
  130. package/docs/research/ledger/docs/research/2026-09-13-search-quality.md +105 -0
  131. package/docs/research/ledger/docs/research/2026-09-13-source-coverage.md +64 -0
  132. package/docs/research/ledger/docs/research/2026-09-13-sync-wall-budget.md +82 -0
  133. package/docs/research/ledger/docs/research/2026-09-13-typed-read-gate.md +81 -0
  134. package/docs/research/ledger/docs/research/2026-09-13-writer-lease.md +84 -0
  135. package/docs/research/ledger/docs/research/2026-09-16-c1-copy-fallback.md +227 -0
  136. package/docs/research/ledger/docs/research/2026-09-16-c10-kimi.md +414 -0
  137. package/docs/research/ledger/docs/research/2026-09-16-c2-derive-counter.md +216 -0
  138. package/docs/research/ledger/docs/research/2026-09-16-c3-echo-filter.md +305 -0
  139. package/docs/research/ledger/docs/research/2026-09-16-c4-title-index.md +349 -0
  140. package/docs/research/ledger/docs/research/2026-09-16-c5-trigram.md +268 -0
  141. package/docs/research/ledger/docs/research/2026-09-16-c6-recency-prior.md +438 -0
  142. package/docs/research/ledger/docs/research/2026-09-16-c7-tombstones.md +467 -0
  143. package/docs/research/ledger/docs/research/2026-09-16-c8-remote-truth.md +390 -0
  144. package/docs/research/ledger/docs/research/2026-09-16-c9-grok.md +380 -0
  145. package/docs/research/ledger/docs/research/2026-09-16-wake-port-program.md +38 -0
  146. package/docs/research/ledger/docs/research/2026-09-17-cursor-blobstore.md +157 -0
  147. package/docs/research/ledger/manifest.json +540 -0
  148. package/legacy/NOTICE.md +50 -0
  149. package/legacy/bin/ledger-core.ts +7 -0
  150. package/legacy/bin/ledger.ts +7 -0
  151. package/legacy/bin/traj-bridge.ts +7 -0
  152. package/legacy/scripts/import-claude-history.ts +130 -0
  153. package/legacy/scripts/native-history-smoke.ts +52 -0
  154. package/legacy/scripts/observe-daemon.ts +570 -0
  155. package/legacy/scripts/retitle-sessions.md +19 -0
  156. package/legacy/scripts/verify-runtime-rollout.ts +151 -0
  157. package/legacy/src/canonical.ts +108 -0
  158. package/legacy/src/cli.ts +1293 -0
  159. package/legacy/src/db.ts +2254 -0
  160. package/legacy/src/diagnostics.ts +87 -0
  161. package/legacy/src/discovery.ts +56 -0
  162. package/legacy/src/embeddings.ts +382 -0
  163. package/legacy/src/filter.ts +602 -0
  164. package/legacy/src/gemini_embeddings.ts +103 -0
  165. package/legacy/src/identity.ts +9 -0
  166. package/legacy/src/index.ts +10 -0
  167. package/legacy/src/ingest.ts +791 -0
  168. package/legacy/src/managed.ts +178 -0
  169. package/legacy/src/native/common.ts +68 -0
  170. package/legacy/src/native/cursor.ts +647 -0
  171. package/legacy/src/native/grok.ts +540 -0
  172. package/legacy/src/native/kimi.ts +706 -0
  173. package/legacy/src/native/opencode.ts +470 -0
  174. package/legacy/src/native/sqlite-ro.ts +281 -0
  175. package/legacy/src/native.ts +97 -0
  176. package/legacy/src/normalizer.ts +457 -0
  177. package/legacy/src/parentage.ts +562 -0
  178. package/legacy/src/pressure.ts +162 -0
  179. package/legacy/src/private-storage.ts +74 -0
  180. package/legacy/src/reader.ts +1255 -0
  181. package/legacy/src/scanner.ts +591 -0
  182. package/legacy/src/schema-gate.ts +263 -0
  183. package/legacy/src/schema.ts +829 -0
  184. package/legacy/src/semantic-top-k.ts +71 -0
  185. package/legacy/src/session-hash.ts +49 -0
  186. package/legacy/src/store-status.ts +218 -0
  187. package/legacy/src/sync-preparation.ts +45 -0
  188. package/legacy/src/sync-truth.ts +843 -0
  189. package/legacy/src/sync.ts +1233 -0
  190. package/legacy/src/types.ts +436 -0
  191. package/legacy/src/vocab.txt +30522 -0
  192. package/legacy/src/watch.ts +673 -0
  193. package/legacy/src/writer-lease.ts +310 -0
  194. package/ops/README.md +150 -0
  195. package/ops/install-cli.sh +63 -0
  196. package/ops/install-native.sh +35 -0
  197. package/ops/install-release.sh +49 -0
  198. package/ops/linux/install-systemd-user.sh +103 -0
  199. package/ops/macos/install-launchd.sh +72 -0
  200. package/ops/sync-settings.sh +37 -0
  201. package/package.json +21 -5
  202. package/skills/trajectories/SKILL.md +138 -0
  203. package/skills/trajectories/scripts/trajectories_recent.sh +26 -0
  204. package/skills/trajectories/scripts/trajectories_search.sh +26 -0
  205. package/index.js +0 -1
@@ -0,0 +1,438 @@
1
+ # C6: the recency prior before RRF, 2026-09-16
2
+
3
+ ## Outcome
4
+
5
+ Both lexical lanes now order by **bm25 × time decay**, Wake's shape: a session
6
+ from today is ×2, one 30 days old ×1.5, one a year old ×1.08
7
+ (`crates/wake-core/src/db.rs#L1546-L1560`, tag v0.6.5, MIT). The factor is keyed
8
+ on the *session's* `started_at`, so hits inside one session keep their bm25
9
+ order; it is the lane's `rank`, so within a lane the emitted `score` is the
10
+ ordering key; the clock is one `Date.now()` per query, shared by both lanes. The
11
+ semantic lane stays pure cosine, and hybrid inherits the prior through the
12
+ lexical leg's *positions* — which is what RRF reads. No counter moves, no DDL
13
+ changes, no new dependency: `FTS_FORMAT` stays `"3"`.
14
+
15
+ On a byte-consistent copy of the production store (1,135,804,416 B, 2,613
16
+ sessions, 230,451 records, 0 embeddings), the case the cycle exists for:
17
+ (Point-in-time fixture — measured 2026-09-16 on that copy; the live store
18
+ grew during review and later re-measurements of the same queries can flip
19
+ differently while still following the same crossover rule, which predicted
20
+ the winner in every re-measured case.)
21
+
22
+ | query | what changed | raw bm25 gap | factor gap | verdict |
23
+ | --- | --- | --- | --- | --- |
24
+ | `recency prior` (record lane) | rank 1 was a 3.62-day-old session, now today's (age 0.11 d) | today's match was **0.7% weaker** | **1.055×** | flips — the intended case |
25
+ | `gemini` (title lane) | rank 1 was a 34.81-day-old title, now a 2.11-day-old one | the new rank 1 is **16% weaker** | **1.322×** | flips |
26
+ | `memory cards` (record lane) | rank 1 unchanged: the 64.2-day-old session | old is **2.50× stronger** than today's best | 1.478× | holds — a relevance lead past the factor's span survives |
27
+
28
+ So the crossover is quantified, not assumed: **a newer hit takes the position of
29
+ an older one only when its raw bm25 is within the factor ratio** — today vs 30
30
+ days it needs ≥ 75% of the older hit's magnitude (2.0/1.5), today vs a 400-day
31
+ one ≥ 54% (2.0/1.07), and the `memory cards` row above is the other side of it
32
+ (40%, loses). Intra-session order is provably untouched: 17 hits of one session
33
+ scale by exactly the same factor (measured ratio 1.9784, identical across all
34
+ 17) and none of them moves.
35
+
36
+ The cost is one expression per matched row. Measured directly on the worst case
37
+ (48,120 matched rows, `run`, 258 ms): **+4.9 ms** for the chosen `unixepoch()`
38
+ route, where a `strftime('%s', …)` route costs +9.6–12.7 ms and `julianday`
39
+ +5.0–7.2 ms — the fastest epoch route is the one that shipped. At the reader
40
+ level (interleaved A/B, 25 runs each, three passes) the delta is inside the
41
+ host's noise: −1.42 ms to +0.12 ms per query, with the one reproducible residue
42
+ being the fastest query in the set (0.66 → 0.78 ms, +0.12 ms, where a fixed cost
43
+ is a measurable share).
44
+
45
+ ## The mechanism
46
+
47
+ Pinned upstream (`/tmp/ledger-confer/wake`, `d9f702b6`, tag v0.6.5):
48
+
49
+ ```sql
50
+ -- db.rs#L1552 (the constant), #L1604 (the message lane), #L1685 (the title lane)
51
+ ORDER BY bm25(messages_fts) * (1.0 + 1.0 / (1.0 + max(0, ? - s.updated_at) / 2592000000.0)) LIMIT ?
52
+ ```
53
+
54
+ Wake multiplies its message lane and its title lane by the same expression, keys
55
+ it on the session's `updated_at` (not the message's `ts`, so intra-session order
56
+ is untouched), binds the clock as a parameter, and uses only arithmetic and
57
+ `max()` — no SQLite math extension. Its degraded `LIKE` path keeps time ordering
58
+ because it has no bm25 to weigh; Ledger has no such path (its lexical fallback
59
+ is still FTS5), so there is nothing to mirror there.
60
+
61
+ Ledger's port, one shared fragment used by both lanes:
62
+
63
+ ```sql
64
+ bm25(records_fts) * (1.0 + 1.0 / (1.0 + max(0, ? - COALESCE(unixepoch(s.started_at) * 1000, 0)) / 2592000000.0)) AS rank
65
+ ORDER BY rank ASC, s.started_at DESC
66
+ ```
67
+
68
+ Four Ledger-specific shape decisions inside one expression:
69
+
70
+ - **`unixepoch()` is the epoch conversion**, because this store has no numeric
71
+ time column — `sessions.started_at` is ISO-8601 TEXT. It is core SQLite
72
+ (3.38+), and the read path only ever runs on Bun's bundled SQLite 3.51.0 (the
73
+ same runtime C5 verified its feature set against; the native SQLite window of
74
+ C1 never runs FTS ranking). Measured against the alternatives below.
75
+ - **`COALESCE(…, 0)`** puts an absent or unparsable timestamp at the epoch —
76
+ the factor's neutral end (×1.0). Without it, the arithmetic yields NULL, and
77
+ NULL sorts *first* in an `ASC` ORDER BY: an unparsable timestamp would have
78
+ become the strongest possible ranking signal. Verified on the store: all
79
+ 2,613 sessions parse (0 NULLs), so the clamp is a guard, not a behaviour.
80
+ - **The `?` is `Date.now()` fixed once per query** and passed to both lanes, so
81
+ a title hit and a record hit of one session are weighted identically and a
82
+ single query cannot see two clocks (Wake's `now_ms()`, bound the same way).
83
+ This is also what made the constant cheap: `strftime('%s','now')` inside the
84
+ expression was evaluated per row.
85
+ - **H is spelled as a real** (`${RECENCY_DECAY_MS}.0`): SQLite divides integers
86
+ with integer division, and a truncated divisor would silently make the factor
87
+ ×2 for every row in the corpus — a failure that would look like a stronger
88
+ prior rather than a bug.
89
+
90
+ ## Decisions (the cycle's open questions, resolved)
91
+
92
+ **1. Where the prior applies: both lexical lanes; not the semantic lane.**
93
+ Wake weighs both of its lanes, and Ledger's two lanes are one query's answer, so
94
+ both take the same factor. The semantic lane was decided separately, on the
95
+ measured shape of cosine rather than on symmetry: with the shipped model
96
+ (`all-MiniLM-L6-v2`, 384 dims) over 300 real records from the production store,
97
+ the whole top-50 span of a query's cosine scores is **1.77–2.60×**
98
+ (r1/r50 = 2.19 / 1.77 / 2.60 for three queries; r1 ∈ [0.34, 0.40], r50 ∈ [0.15,
99
+ 0.20]). That is the *entire* range a semantic query has to work with, and the
100
+ prior's own multiplier spans ×1–×2 over the same corpus — so the same factor on
101
+ cosine would order that lane by age nearly regardless of text, and it would stop
102
+ the reported `rank` from being a cosine at all. bm25 is the opposite kind of
103
+ score: unbounded, negative, and sparse (the measured flips above turn on
104
+ comparisons at 0.7% and 16%, and the no-flip on 2.5×). The program's promise —
105
+ cheap, arithmetic-only, no rescore pass — is kept by the lexical lanes, where
106
+ the prior is one `ORDER BY` term; the semantic lane is not left without recency,
107
+ it is left to the fusion, which is the layer designed to combine rank orders.
108
+
109
+ **2. The decay constant: 30 days, Wake's H, unchanged.** Measured distribution
110
+ of the store as it stands (read-only queries against the live file; the cycle
111
+ only ever read it):
112
+
113
+ | age of session start | sessions |
114
+ | --- | --- |
115
+ | ≤ 1 day | 17 |
116
+ | ≤ 7 days | 78 |
117
+ | ≤ 30 days | 132 |
118
+ | ≤ 90 days | 652 |
119
+ | ≤ 1 year | 2,605 |
120
+ | > 1 year | 8 |
121
+
122
+ and by month: 2025-08 8, 2025-09 33, 2025-10 37, 2025-11 1, 2025-12 11, 2026-01
123
+ 755, 2026-02 2, 2026-03 12, 2026-05 1,068, 2026-06 69, 2026-07 209, 2026-08 300,
124
+ 2026-09 108. Two things follow. The corpus is **not** "two weeks of heavy
125
+ activity" — it is a year wide with two backfilled lumps (2026-05, 2026-01) and a
126
+ live edge of ~78 sessions in the last week — so a prior tuned only to the last
127
+ fortnight would be tuned to 3% of the store. And 30 days is a scale that
128
+ separates the live edge from the lumps without erasing everything older: a
129
+ 2026-05 session still carries ×1.20 and a 2026-01 one ×1.11, while the last
130
+ week's work spans ×1.81–×2.0. The constant stays Wake's; nothing measured argues
131
+ for a different one, and a deviation would need a corpus whose live edge sits at
132
+ a different scale (a reopen condition, below).
133
+
134
+ **3. The key is `started_at`, not `updated_at` or `ended_at`.** `updated_at` is
135
+ the ingest write clock (`ingest.ts` stamps `new Date()`), and the measurement
136
+ makes it unusable: every one of the store's 2,613 sessions is stamped between
137
+ 2026-09-12 and 2026-09-16 — a four-day window that would make a backfill of
138
+ 2025 material rank as "today". `ended_at` is the content analog of Wake's
139
+ `updated_at` and would be the second-best choice: it moves the recency bucket of
140
+ only 5 of 2,613 sessions (active in the last 7 days: 78 by `started_at`, 83 by
141
+ `ended_at`; last 24 h: 17 vs 19; >1 day of spread between the two: 48 sessions,
142
+ 1.8%; `ended_at < started_at`: 0). With the two keys nearly equivalent on this
143
+ corpus, the lane keeps one timestamp: the one its `ORDER BY … started_at DESC`
144
+ tiebreak already names, and the one that never needs a fallback.
145
+
146
+ **4. RRF interplay: the prior must move the legs, and it does.** RRF reads rank
147
+ *positions* (k = 60), so a reorder inside a leg is exactly how a fused result
148
+ changes; the fusion code was not touched. Two consequences, both checked: a hit
149
+ whose raw relevance lead exceeds the factor ratio keeps its leg position (the
150
+ `memory cards` row), so the fusion keeps its winner; and a hit whose relevance is
151
+ comparable moves up its leg, which is the intended change (`gemini`: the
152
+ 2.11-day title was 16% weaker in raw bm25 and still takes rank 1 — 1/(60+1) vs
153
+ 1/(60+2) is a 1.6% RRF difference, and the fused order follows the leg). The
154
+ test suite pins the direction end-to-end: a fresh session that is *only* a
155
+ lexical match outranks a 500-day-old session that is *only* a semantic match,
156
+ because the lexical leg's position carries the recency and the semantic leg's
157
+ does not.
158
+
159
+ **5. Envelope honesty: nothing is disclosed, and `rank`'s meaning is
160
+ documented.** The prior is internal ranking, like bm25 itself — Wake's
161
+ `SearchHit` carries no score at all (`crates/wake-core/src/models.rs#L433-L440`),
162
+ and Ledger's envelope already discloses *mechanisms that change what could be
163
+ answered* (widening, fallback, count precision) rather than ranking internals.
164
+ What did change is what `score` means, so the api-surface lane-relative note now
165
+ says it: title and record hits carry bm25 weighted by the same prior, so within a
166
+ lane the score is the ordering key, and the stream's emission order (titles
167
+ first, then body) remains the contract; consumers must not re-sort across lanes.
168
+ The type comment on `SearchHitBase.rank` says the same for library callers. No
169
+ `recency_weighted` field, because a boolean that is always true is not
170
+ information.
171
+
172
+ **6. The constant is pinned by a test, not a comment.** The one place the number
173
+ 30 is asserted: two sessions with *identical records* (identical raw bm25) aged
174
+ 0 and exactly 30 days must report ranks in the ratio 0.75 —
175
+ `(1+1/(1+30/30)) / (1+1/(1+0)) = 1.5/2`. Move the constant and that ratio moves
176
+ with it (`packages/core/tests/recency-prior.test.ts`).
177
+
178
+ ## FTS_FORMAT: verified against C2's counter contract, and it does not move
179
+
180
+ C2's contract (`docs/research/2026-09-16-c2-derive-counter.md`, ADR "Derive-Rule
181
+ Counters"): the counter names **the rules a derived index was produced under**,
182
+ and "a mismatch → one full re-derivation" — it exists for changes to the derived
183
+ tables' *shape* (rows, columns, tokenizer, projection), which is exactly what
184
+ C3 (echo filter), C4 (title index) and the refused C5 (trigram) changed. C6
185
+ changes none of that: the same DDL, the same projections, the same row sets, the
186
+ same tokenizer — the same `records_fts` and `titles_fts` bytes. What changed is
187
+ one expression inside two read-side `SELECT`s. The verification is structural,
188
+ not a measurement: `git diff --stat` for this cycle touches `reader.ts` (both
189
+ lane SQLs plus comments), tests, `types.ts` (a doc comment) and this note's docs
190
+ — no `schema.ts`, no `db.ts`, no `ingest.ts`, no migration, and
191
+ `DERIVE_COUNTERS[].outputs` (`records_fts`, `records_echo`, `titles_fts`) are
192
+ untouched, so a store at `fts_format = "3"` is exactly as derivable under this
193
+ build as under C4. The `derive-counters` suite passes unchanged. The honest
194
+ consequence for operators: this cycle ships with **zero re-derivation cost** —
195
+ the opposite of C3/C4/C5, which each cost a one-shot rebuild.
196
+
197
+ ## Measurements (OBSERVED)
198
+
199
+ Host: Apple M1, macOS 25.6, APFS; Bun 1.3.14 (SQLite 3.51.0). Fixture: a copy of
200
+ the production store taken with `cp` of `db` + `-wal` + `-shm` into `/tmp` (the
201
+ live store was only ever read); "before" is the reader at HEAD `efa1b3a`
202
+ extracted with `git archive`, "after" is the working tree.
203
+
204
+ **The store's own timestamps** (read-only, production file): 2,613 sessions,
205
+ 230,451 records, 0 embeddings; `started_at` spans 2025-08-25T16:31:18.545Z →
206
+ 2026-09-16T08:14:08.018Z, `updated_at` spans 2026-09-12T08:50:56.960Z →
207
+ 2026-09-16T08:52:11.295Z; no NULL `started_at`/`ended_at`/`updated_at`; 0
208
+ timestamps that `unixepoch()` or `strftime('%s', …)` cannot parse, and the two
209
+ agree on all 2,613 values (checked before choosing `unixepoch`).
210
+
211
+ **Before/after orderings** (limit 50, the reader's own SQL; "before" ranks are
212
+ raw bm25, "after" ranks are the weighted value). The factor column is computed
213
+ from the session age at the moment of the run.
214
+
215
+ `recency prior` — the flip:
216
+
217
+ | | rank | session age | raw | factor | weighted |
218
+ | --- | --- | --- | --- | --- | --- |
219
+ | before 1 | 1 | 3.62 d | −8.8617 | 1.892 | — |
220
+ | before (today's session, lower down) | — | 0.11 d | −8.8014 | 1.996 | — |
221
+ | after 1 | 1 | 0.11 d | −8.8014 | 1.996 | −17.5700 |
222
+ | after 2 | 2 | 3.62 d | −8.8617 | 1.892 | −16.7684 |
223
+
224
+ Raw gap 1.007×, factor gap 1.055×: today's match needed ≥ 94.8% of the older
225
+ one's magnitude and had 99.3%.
226
+
227
+ `gemini` (title lane) — the flip with a wider raw deficit:
228
+
229
+ | | rank | session age | raw | factor | weighted |
230
+ | --- | --- | --- | --- | --- | --- |
231
+ | before 1 | 1 | 34.81 d | −8.4272 | 1.463 | — |
232
+ | before (the new rank 1) | 3 | 2.11 d | −7.2660 | 1.934 | — |
233
+ | after 1 | 1 | 2.11 d | −7.2660 | 1.934 | −14.0547 |
234
+ | after 2 | 2 | 34.81 d | −8.4272 | 1.463 | −12.3282 |
235
+
236
+ Raw gap 1.160×, factor gap 1.322×: the newer title needed ≥ 75.6% and had 86.2%.
237
+ A record-lane hit from 1.65 d (raw −6.9040) also moved from below the old title's
238
+ record-lane tail to overall rank 6.
239
+
240
+ `memory cards` — the crossover holds:
241
+
242
+ | | rank | session age | raw | factor | weighted |
243
+ | --- | --- | --- | --- | --- | --- |
244
+ | before 1 | 1 | 64.22 d | −14.7885 | 1.318 | — |
245
+ | after 1 | 1 | 64.22 d | −14.7885 | 1.318 | −19.4970 |
246
+ | after 2 | 2 | 1.65 d | −5.9181 | 1.948 | −11.5284 |
247
+
248
+ Raw gap 2.499×, factor gap 1.478×: today's best match needed ≥ 67.7% of the old
249
+ one's magnitude and has 40.0% — the relevance lead outlives the factor, and the
250
+ old session keeps rank 1.
251
+
252
+ **Intra-session invariance** (`wal_autocheckpoint`, one session, 17 hits): every
253
+ hit scaled by the same 1.9784 (the session's factor at 0.66 d), so the sequence
254
+ of weighted ranks is the sequence of raw ranks multiplied by a constant — no
255
+ within-session reordering is possible.
256
+
257
+ **Latency** (limit 50, warm, median of 25 interleaved runs in one process; hits =
258
+ returned hits; matched rows from a separate `COUNT(*) … MATCH`, given where it
259
+ matters):
260
+
261
+ | query | hits | matched rows | before | after | delta |
262
+ | --- | --- | --- | --- | --- | --- |
263
+ | `recency prior` | 25 | — | 1.57 ms | 1.59 ms | +0.02 ms |
264
+ | `memory cards` | 7 | — | 0.75 ms | 0.76 ms | +0.01 ms |
265
+ | `wal_autocheckpoint` | 50 | 84 | 8.67 ms | 8.72 ms | +0.04 ms |
266
+ | `gemini` | 50 | — | 41.90 ms | 40.48 ms | −1.42 ms |
267
+ | `ledger` | 50 | 13,807 | 0.68 ms | 0.77 ms | +0.08 ms |
268
+ | `the` | 50 | 74,074 | 3.09 ms | 3.04 ms | −0.05 ms |
269
+ | `test` | 50 | — | 0.32 ms | 0.37 ms | +0.04 ms |
270
+ | `run` | 50 | 48,120 | 258.33 ms | 257.57 ms | −0.76 ms |
271
+
272
+ Three passes of the same harness agreed within this: the largest positive delta
273
+ seen on any query in any pass was on `ledger` (0.68 → 0.77 ms, reproducible at
274
+ +0.08…+0.12 ms), and the 258 ms query measured −0.76 and +5.76 ms on the shipped
275
+ shape and +4.10 ms on the earlier `strftime` variant — i.e. inside the host's
276
+ noise band (±3 ms at that size), which is why the added term was also measured
277
+ *directly*: the same rank expression with and without the factor, in the
278
+ minimal SQL around it (no `snippet()` columns), 15 iterations, same store, query
279
+ `run`, 48,120 matched rows:
280
+
281
+ | rank expression | median |
282
+ | --- | --- |
283
+ | `bm25(records_fts)` (baseline) | 202.27 ms |
284
+ | `bm25 × strftime('%s', …)` prior | 215.00 ms (+12.7 ms) |
285
+ | `bm25 × julianday(…)` prior | 207.17 ms (+4.9 ms) |
286
+ | **`bm25 × unixepoch(…)` prior (shipped)** | **207.13 ms (+4.9 ms)** |
287
+
288
+ So the prior's true worst-case cost on this corpus is ~5 ms on a 258 ms query
289
+ (≈1.8%), the chosen epoch route is the cheapest of the three measured, and
290
+ everything that answers in under 10 ms pays ≤ 0.12 ms.
291
+
292
+ **Cosine spread** (the semantic decision, measured with the shipped
293
+ `OnnxEmbeddingProvider`): 300 real records' embedding text from the store copy,
294
+ three queries, cosines of the top 50:
295
+
296
+ | query | r1 | r10 | r50 | median | r1/r50 |
297
+ | --- | --- | --- | --- | --- | --- |
298
+ | `recency prior` | 0.3363 | 0.2338 | 0.1539 | 0.1908 | 2.19× |
299
+ | `sqlite wal checkpoint` | 0.3463 | 0.2520 | 0.1953 | 0.2203 | 1.77× |
300
+ | `gemini embeddings backfill` | 0.4027 | 0.2394 | 0.1550 | 0.1846 | 2.60× |
301
+
302
+ ## What is inferred (INFERRED, marked)
303
+
304
+ - The corpus conclusion ("30 days separates the live edge from the backfilled
305
+ lumps") is an inference from the session-start histogram, not a measurement of
306
+ retrieval quality: no relevance judgement exists for this store, so the cycle's
307
+ evidence is ordering deltas and arithmetic, not graded accuracy. A qrels set
308
+ would be the way to argue for a *different* constant.
309
+ - The +4.9 ms worst case is a lower bound for a bigger corpus: the expression is
310
+ O(matched rows), so a store several times this size would pay proportionally
311
+ more on its slowest queries (the same shape C5 measured for bm25 itself).
312
+ - The claim that a semantic-lane prior would order by age is an extrapolation
313
+ from the measured 1.8–2.6× span plus the factor's own ×1–×2 range; it is not a
314
+ measured end-to-end semantic ranking (the production store holds no vectors:
315
+ `records_embeddings` is empty, so the semantic lane answers lexically there by
316
+ design).
317
+ - `ended_at` as the key would likely read slightly better for resumed sessions
318
+ (the 5 sessions whose bucket moves), but nothing measured shows a user-visible
319
+ difference at this corpus size.
320
+
321
+ ## Tests
322
+
323
+ `packages/core/tests/recency-prior.test.ts` (7 cases, fixtures dated relative to
324
+ `now` because a literal timestamp cannot pin "today"). Three of them prove their
325
+ own premise by *inverting* the prior — `rawBm25(hit) = rank / (1 + 1/(1 +
326
+ age/30d))` recovers the raw bm25 the lane would have ranked by, so the test can
327
+ assert that the fresh hit really was the weaker raw match the prior reversed,
328
+ and that the crossover hit truly is stronger by more than the factor's span:
329
+
330
+ 1. the record lane prefers the session from today when the two matches are
331
+ comparable, with the premise asserted (today's raw bm25 is weaker);
332
+ 2. a much stronger old match still wins — the crossover, with the fixture's
333
+ measured raw gap (2.50×) against the factor's span (1.87×) in the case
334
+ comment and in the premise assertion;
335
+ 3. the title lane orders by the same prior (today's longer, weaker title beats
336
+ an older, stronger one), and an exact 400-day-old title still wins over
337
+ today's diluted one, again with both premises asserted;
338
+ 4. intra-session order is the session's bm25 order — the factor is the session's,
339
+ not the record's (the appended-today record still ranks below the older,
340
+ stronger one);
341
+ 5. the decay constant is 30 days: identical records 0 and 30 days old report
342
+ ranks in the exact ratio 0.75;
343
+ 6. the semantic lane stays pure cosine (a 500-day-old session wins on similarity
344
+ alone, and its reported rank is the cosine) while hybrid lifts the fresh
345
+ lexical match above it through the lexical leg's position;
346
+ 7. the lane contract survives: a title hit is emitted ahead of a record hit whose
347
+ rank is *more* negative, and both ranks are negative.
348
+
349
+ Five existing tests carried assumptions the prior invalidates, each fixed at the
350
+ assumption rather than around it:
351
+
352
+ - `search-quality.test.ts`, the `zonk` twins: they no longer share a rank (the
353
+ prior separates them by session age), so the assertion pins the direction
354
+ (newest rank most negative) instead of equality — the order assertion above it
355
+ is unchanged and still passes.
356
+ - `integration-migrations.test.ts`, `querySnapshot`: the rebuild comparison
357
+ cannot compare `rank` exactly any more, because rank is now a function of the
358
+ wall clock as well as the index; it compares at six significant digits, which
359
+ absorbs any time between the snapshots and is far tighter than any change a
360
+ rebuild could make to bm25.
361
+ - `robot.test.ts`, the v1 byte-identity case, and `cli-canonical.test.ts`, the
362
+ `search`/`query` alias case: two CLI *processes* can no longer render identical
363
+ last digits for `score`, so the score value is blanked (robot) or zeroed
364
+ (cli-canonical, whose normalizer already does the same for `elapsed_ms`) on
365
+ both sides, and every other byte is still compared as written.
366
+ - `title-index.test.ts`: the tie comment ("equal rank, so the newer session comes
367
+ first") became "equal relevance, so the prior decides" — the assertion is
368
+ unchanged.
369
+
370
+ ## Roadmap context and the open-source case (program doc, C6 row)
371
+
372
+ > **C6** | Recency prior before RRF (bm25 × time-decay, arithmetic-only) | All
373
+ > lanes order by relevance with `started_at DESC` tiebreak only; "the session
374
+ > from today" loses to an old strong match | Cheap relevance win every user
375
+ > feels; no extension dependency, single ORDER BY term
376
+
377
+ Both halves hold as measured: the fix is real and visible (`recency prior`,
378
+ `gemini`), it is one `ORDER BY` term with no extension and no rescore pass, and
379
+ its cost is ~5 ms at the corpus's slowest query and unmeasurable elsewhere. The
380
+ program's dependency note — C6 after C5, "ranking experiments on the final
381
+ tokenizer" — resolves cleanly: C5 refused the trigram swap, so the prior was
382
+ designed and measured against the porter index the store actually ships.
383
+
384
+ ## What would reopen this
385
+
386
+ 1. **A relevance judgement set.** Every ordering claim here is an ordering
387
+ delta; with graded relevance the constant (30 d) and the weight (W = 1) could
388
+ be tuned instead of inherited, and the crossover table is exactly the shape
389
+ that tuning would consume.
390
+ 2. **A corpus whose live edge sits elsewhere** — e.g. a store used hourly where
391
+ a 7-day scale separates the working set, or one whose backfill is recent
392
+ enough that `updated_at`-shaped reasoning tempts a re-look. Re-measure the
393
+ histogram first; the constant is one line.
394
+ 3. **Candidate capping for the common-term classes** (the C5 note's own reopen
395
+ item): the prior is O(matched rows) on top of bm25's existing O(matched rows),
396
+ and a match-set-bound query large enough to need capping would want the
397
+ recency term inside the capped plan rather than after it.
398
+ 4. **A semantic-lane recency decision with real vectors in the store.** The
399
+ decision here is argued from the cosine spread and defended by the fusion;
400
+ with a populated `records_embeddings` the same measurement could be run on
401
+ real semantic rankings.
402
+
403
+ ## Method (for reproducibility)
404
+
405
+ Copy the live store to `/tmp` (`cp` of `db`, `-wal`, `-shm`; the original was
406
+ never opened for writing). Extract HEAD into `/tmp/c6/before` with
407
+ `git archive HEAD packages/core | tar -x` and symlink `node_modules`, so both
408
+ codebases can be loaded side by side. The harnesses lived in `/tmp/c6` and were
409
+ deleted: an A/B probe that prints ranked hits for a query from a given tree, an
410
+ interleaved latency harness (25 runs per tree per query, medians, in one
411
+ process), a direct-expression benchmark, a cosine probe against the shipped
412
+ ONNX provider, and a fixture calibration script. Gates on the tree:
413
+ `cd packages/core && bun test` (**586 pass / 0 fail** across 41 files, of which
414
+ this cycle's 7 are `tests/recency-prior.test.ts`), `bun run typecheck`,
415
+ `bun run build`.
416
+
417
+ ## Wake pins (MIT, `iAmCorey/Wake@d9f702b6` tag v0.6.5)
418
+
419
+ | mechanism | pin |
420
+ | --- | --- |
421
+ | the decay factor, W = 1 and H = 30 days, as one ORDER BY term | `crates/wake-core/src/db.rs#L1552` (`RECENCY_BOOST`), `#L1604` |
422
+ | the same factor on the title lane | `db.rs#L1685` |
423
+ | keyed on the session's timestamp, so intra-session order is untouched | `db.rs#L1546-L1556` (the comment above the constant) |
424
+ | the clock bound as a parameter, arithmetic and `max()` only | `db.rs#L1553` (`now_ms()`), `#L1604` |
425
+ | the degraded path keeps time ordering (no bm25 to weigh) | `db.rs#L1632-L1650` |
426
+ | ranking internals are not part of the hit shape | `crates/wake-core/src/models.rs#L433-L440` (no score field) |
427
+
428
+ ## Files
429
+
430
+ - `packages/core/src/reader.ts` — `RECENCY_DECAY_MS`/`RECENCY_PRIOR_SQL`, the
431
+ record lane's `rank`, the title lane's `rank`, the per-query `now`, and the
432
+ semantic lane's decision comment.
433
+ - `packages/core/src/types.ts` — the `SearchHitBase.rank` doc comment.
434
+ - `packages/core/tests/recency-prior.test.ts` — this cycle's 7 cases.
435
+ - `packages/core/tests/{search-quality,integration-migrations,robot,cli-canonical,title-index}.test.ts`
436
+ — the five assumptions the prior invalidates, each fixed where it was wrong.
437
+ - `docs/api-surface.md` — the lane-relative `score` sentence (§1.4.4), the
438
+ `titles_fts` search bullet (§3.2), and the C6 ADR entry (§4).