trajectories 0.0.0 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +36 -0
- package/LICENSE +201 -0
- package/README.md +288 -2
- package/dist/cli.js +30210 -0
- package/dist/convert.d.ts +31 -0
- package/dist/convert.js +431 -0
- package/dist/deepagents_checkpoint.py +362 -0
- package/dist/index.d.ts +7 -0
- package/dist/index.js +8402 -0
- package/dist/log-baseline.d.ts +26 -0
- package/dist/log-baseline.js +104 -0
- package/dist/log-content.d.ts +16 -0
- package/dist/log-content.js +80 -0
- package/dist/log-io.d.ts +10 -0
- package/dist/log-io.js +121 -0
- package/dist/log-read.d.ts +24 -0
- package/dist/log-read.js +117 -0
- package/dist/log-reconcile.d.ts +5 -0
- package/dist/log-reconcile.js +38 -0
- package/dist/log-records.d.ts +35 -0
- package/dist/log-records.js +85 -0
- package/dist/log-recovery.d.ts +18 -0
- package/dist/log-recovery.js +123 -0
- package/dist/log-stage.d.ts +14 -0
- package/dist/log-stage.js +138 -0
- package/dist/log-state.d.ts +89 -0
- package/dist/log-state.js +92 -0
- package/dist/log-switch.d.ts +3 -0
- package/dist/log-switch.js +98 -0
- package/dist/log-types.d.ts +129 -0
- package/dist/log-types.js +8 -0
- package/dist/log.d.ts +8 -0
- package/dist/log.js +62 -0
- package/dist/migrate.d.ts +29 -0
- package/dist/migrate.js +173 -0
- package/dist/normalization-runtime.d.ts +1 -0
- package/dist/normalization-runtime.js +3150 -0
- package/dist/portable.d.ts +9 -0
- package/dist/portable.js +26 -0
- package/dist/readers/canonical.d.ts +13 -0
- package/dist/readers/canonical.js +54 -0
- package/dist/readers/cline.d.ts +10 -0
- package/dist/readers/cline.js +107 -0
- package/dist/readers/cursor.d.ts +20 -0
- package/dist/readers/cursor.js +374 -0
- package/dist/readers/deepagents.d.ts +41 -0
- package/dist/readers/deepagents.js +163 -0
- package/dist/readers/fx.d.ts +11 -0
- package/dist/readers/fx.js +573 -0
- package/dist/readers/github-copilot.d.ts +8 -0
- package/dist/readers/github-copilot.js +216 -0
- package/dist/readers/grok-build.d.ts +12 -0
- package/dist/readers/grok-build.js +391 -0
- package/dist/readers/guards.d.ts +4 -0
- package/dist/readers/guards.js +121 -0
- package/dist/readers/pi-compaction.d.ts +7 -0
- package/dist/readers/pi-compaction.js +112 -0
- package/dist/readers/pi-context.d.ts +13 -0
- package/dist/readers/pi-context.js +64 -0
- package/dist/store.d.ts +25 -0
- package/dist/store.js +218 -0
- package/dist/types.d.ts +91 -0
- package/dist/types.js +1 -0
- package/dist/writers/claude-code.d.ts +5 -0
- package/dist/writers/claude-code.js +51 -0
- package/dist/writers/cline.d.ts +3 -0
- package/dist/writers/cline.js +77 -0
- package/dist/writers/codex.d.ts +3 -0
- package/dist/writers/codex.js +32 -0
- package/dist/writers/cursor.d.ts +3 -0
- package/dist/writers/cursor.js +197 -0
- package/dist/writers/deepagents.d.ts +3 -0
- package/dist/writers/deepagents.js +116 -0
- package/dist/writers/entropy.d.ts +5 -0
- package/dist/writers/entropy.js +33 -0
- package/dist/writers/fx.d.ts +3 -0
- package/dist/writers/fx.js +135 -0
- package/dist/writers/github-copilot.d.ts +3 -0
- package/dist/writers/github-copilot.js +71 -0
- package/dist/writers/grok-build.d.ts +3 -0
- package/dist/writers/grok-build.js +43 -0
- package/dist/writers/helpers.d.ts +17 -0
- package/dist/writers/helpers.js +75 -0
- package/dist/writers/omp.d.ts +3 -0
- package/dist/writers/omp.js +7 -0
- package/dist/writers/opencode.d.ts +3 -0
- package/dist/writers/opencode.js +65 -0
- package/dist/writers/pi.d.ts +3 -0
- package/dist/writers/pi.js +45 -0
- package/docs/cline-native-format.md +43 -0
- package/docs/copilot-native-format.md +78 -0
- package/docs/cursor-native-format.md +103 -0
- package/docs/deepagents-native-format.md +98 -0
- package/docs/evidence-api.md +170 -0
- package/docs/fx-native-format.md +87 -0
- package/docs/harness-parity-bootstrap.md +78 -0
- package/docs/harness-support-parity.md +107 -0
- package/docs/inspirations.md +103 -0
- package/docs/ledger-migration.md +141 -0
- package/docs/pi-compaction-context.md +75 -0
- package/docs/pi-provider-context.md +76 -0
- package/docs/research/2026-10-06-huihua-and-oar.md +176 -0
- package/docs/research/ledger/README.md +96 -0
- package/docs/research/ledger/confer/2026-09-13-round-10/README.md +61 -0
- package/docs/research/ledger/confer/2026-09-13-round-2/cass.md +72 -0
- package/docs/research/ledger/confer/2026-09-13-round-3/README.md +70 -0
- package/docs/research/ledger/confer/2026-09-13-round-5/README.md +61 -0
- package/docs/research/ledger/confer/2026-09-13-round-6/README.md +68 -0
- package/docs/research/ledger/confer/2026-09-13-round-7/README.md +64 -0
- package/docs/research/ledger/confer/2026-09-13-round-9/README.md +59 -0
- package/docs/research/ledger/confer/2026-09-14-wake/wake.md +146 -0
- package/docs/research/ledger/docs/inspirations.md +320 -0
- package/docs/research/ledger/docs/pr2-pr6-integration.md +53 -0
- package/docs/research/ledger/docs/research/2026-09-07-confer-inspirations.md +163 -0
- package/docs/research/ledger/docs/research/2026-09-07-letta-continuous-service.md +104 -0
- package/docs/research/ledger/docs/research/2026-09-08-claude-history-retrieval.md +117 -0
- package/docs/research/ledger/docs/research/2026-09-09-agentsview-native-discovery.md +87 -0
- package/docs/research/ledger/docs/research/2026-09-09-daemon-observation.md +231 -0
- package/docs/research/ledger/docs/research/2026-09-09-discovery-pagination.md +114 -0
- package/docs/research/ledger/docs/research/2026-09-09-harness-transcript-contract.md +145 -0
- package/docs/research/ledger/docs/research/2026-09-09-native-history-discovery.md +180 -0
- package/docs/research/ledger/docs/research/2026-09-09-native-history-implementation.md +137 -0
- package/docs/research/ledger/docs/research/2026-09-09-other-inspirations.md +279 -0
- package/docs/research/ledger/docs/research/2026-09-09-trajectory-and-letta-code.md +128 -0
- package/docs/research/ledger/docs/research/2026-09-13-diagnostics-coverage.md +116 -0
- package/docs/research/ledger/docs/research/2026-09-13-fts-rebuild.md +83 -0
- package/docs/research/ledger/docs/research/2026-09-13-read-path-split.md +74 -0
- package/docs/research/ledger/docs/research/2026-09-13-runtime-rollout.md +102 -0
- package/docs/research/ledger/docs/research/2026-09-13-scan-outcomes.md +80 -0
- package/docs/research/ledger/docs/research/2026-09-13-search-quality.md +105 -0
- package/docs/research/ledger/docs/research/2026-09-13-source-coverage.md +64 -0
- package/docs/research/ledger/docs/research/2026-09-13-sync-wall-budget.md +82 -0
- package/docs/research/ledger/docs/research/2026-09-13-typed-read-gate.md +81 -0
- package/docs/research/ledger/docs/research/2026-09-13-writer-lease.md +84 -0
- package/docs/research/ledger/docs/research/2026-09-16-c1-copy-fallback.md +227 -0
- package/docs/research/ledger/docs/research/2026-09-16-c10-kimi.md +414 -0
- package/docs/research/ledger/docs/research/2026-09-16-c2-derive-counter.md +216 -0
- package/docs/research/ledger/docs/research/2026-09-16-c3-echo-filter.md +305 -0
- package/docs/research/ledger/docs/research/2026-09-16-c4-title-index.md +349 -0
- package/docs/research/ledger/docs/research/2026-09-16-c5-trigram.md +268 -0
- package/docs/research/ledger/docs/research/2026-09-16-c6-recency-prior.md +438 -0
- package/docs/research/ledger/docs/research/2026-09-16-c7-tombstones.md +467 -0
- package/docs/research/ledger/docs/research/2026-09-16-c8-remote-truth.md +390 -0
- package/docs/research/ledger/docs/research/2026-09-16-c9-grok.md +380 -0
- package/docs/research/ledger/docs/research/2026-09-16-wake-port-program.md +38 -0
- package/docs/research/ledger/docs/research/2026-09-17-cursor-blobstore.md +157 -0
- package/docs/research/ledger/manifest.json +540 -0
- package/legacy/NOTICE.md +50 -0
- package/legacy/bin/ledger-core.ts +7 -0
- package/legacy/bin/ledger.ts +7 -0
- package/legacy/bin/traj-bridge.ts +7 -0
- package/legacy/scripts/import-claude-history.ts +130 -0
- package/legacy/scripts/native-history-smoke.ts +52 -0
- package/legacy/scripts/observe-daemon.ts +570 -0
- package/legacy/scripts/retitle-sessions.md +19 -0
- package/legacy/scripts/verify-runtime-rollout.ts +151 -0
- package/legacy/src/canonical.ts +108 -0
- package/legacy/src/cli.ts +1293 -0
- package/legacy/src/db.ts +2254 -0
- package/legacy/src/diagnostics.ts +87 -0
- package/legacy/src/discovery.ts +56 -0
- package/legacy/src/embeddings.ts +382 -0
- package/legacy/src/filter.ts +602 -0
- package/legacy/src/gemini_embeddings.ts +103 -0
- package/legacy/src/identity.ts +9 -0
- package/legacy/src/index.ts +10 -0
- package/legacy/src/ingest.ts +791 -0
- package/legacy/src/managed.ts +178 -0
- package/legacy/src/native/common.ts +68 -0
- package/legacy/src/native/cursor.ts +647 -0
- package/legacy/src/native/grok.ts +540 -0
- package/legacy/src/native/kimi.ts +706 -0
- package/legacy/src/native/opencode.ts +470 -0
- package/legacy/src/native/sqlite-ro.ts +281 -0
- package/legacy/src/native.ts +97 -0
- package/legacy/src/normalizer.ts +457 -0
- package/legacy/src/parentage.ts +562 -0
- package/legacy/src/pressure.ts +162 -0
- package/legacy/src/private-storage.ts +74 -0
- package/legacy/src/reader.ts +1255 -0
- package/legacy/src/scanner.ts +591 -0
- package/legacy/src/schema-gate.ts +263 -0
- package/legacy/src/schema.ts +829 -0
- package/legacy/src/semantic-top-k.ts +71 -0
- package/legacy/src/session-hash.ts +49 -0
- package/legacy/src/store-status.ts +218 -0
- package/legacy/src/sync-preparation.ts +45 -0
- package/legacy/src/sync-truth.ts +843 -0
- package/legacy/src/sync.ts +1233 -0
- package/legacy/src/types.ts +436 -0
- package/legacy/src/vocab.txt +30522 -0
- package/legacy/src/watch.ts +673 -0
- package/legacy/src/writer-lease.ts +310 -0
- package/ops/README.md +150 -0
- package/ops/install-cli.sh +63 -0
- package/ops/install-native.sh +35 -0
- package/ops/install-release.sh +49 -0
- package/ops/linux/install-systemd-user.sh +103 -0
- package/ops/macos/install-launchd.sh +72 -0
- package/ops/sync-settings.sh +37 -0
- package/package.json +21 -5
- package/skills/trajectories/SKILL.md +138 -0
- package/skills/trajectories/scripts/trajectories_recent.sh +26 -0
- package/skills/trajectories/scripts/trajectories_search.sh +26 -0
- package/index.js +0 -1
|
@@ -0,0 +1,438 @@
|
|
|
1
|
+
# C6: the recency prior before RRF, 2026-09-16
|
|
2
|
+
|
|
3
|
+
## Outcome
|
|
4
|
+
|
|
5
|
+
Both lexical lanes now order by **bm25 × time decay**, Wake's shape: a session
|
|
6
|
+
from today is ×2, one 30 days old ×1.5, one a year old ×1.08
|
|
7
|
+
(`crates/wake-core/src/db.rs#L1546-L1560`, tag v0.6.5, MIT). The factor is keyed
|
|
8
|
+
on the *session's* `started_at`, so hits inside one session keep their bm25
|
|
9
|
+
order; it is the lane's `rank`, so within a lane the emitted `score` is the
|
|
10
|
+
ordering key; the clock is one `Date.now()` per query, shared by both lanes. The
|
|
11
|
+
semantic lane stays pure cosine, and hybrid inherits the prior through the
|
|
12
|
+
lexical leg's *positions* — which is what RRF reads. No counter moves, no DDL
|
|
13
|
+
changes, no new dependency: `FTS_FORMAT` stays `"3"`.
|
|
14
|
+
|
|
15
|
+
On a byte-consistent copy of the production store (1,135,804,416 B, 2,613
|
|
16
|
+
sessions, 230,451 records, 0 embeddings), the case the cycle exists for:
|
|
17
|
+
(Point-in-time fixture — measured 2026-09-16 on that copy; the live store
|
|
18
|
+
grew during review and later re-measurements of the same queries can flip
|
|
19
|
+
differently while still following the same crossover rule, which predicted
|
|
20
|
+
the winner in every re-measured case.)
|
|
21
|
+
|
|
22
|
+
| query | what changed | raw bm25 gap | factor gap | verdict |
|
|
23
|
+
| --- | --- | --- | --- | --- |
|
|
24
|
+
| `recency prior` (record lane) | rank 1 was a 3.62-day-old session, now today's (age 0.11 d) | today's match was **0.7% weaker** | **1.055×** | flips — the intended case |
|
|
25
|
+
| `gemini` (title lane) | rank 1 was a 34.81-day-old title, now a 2.11-day-old one | the new rank 1 is **16% weaker** | **1.322×** | flips |
|
|
26
|
+
| `memory cards` (record lane) | rank 1 unchanged: the 64.2-day-old session | old is **2.50× stronger** than today's best | 1.478× | holds — a relevance lead past the factor's span survives |
|
|
27
|
+
|
|
28
|
+
So the crossover is quantified, not assumed: **a newer hit takes the position of
|
|
29
|
+
an older one only when its raw bm25 is within the factor ratio** — today vs 30
|
|
30
|
+
days it needs ≥ 75% of the older hit's magnitude (2.0/1.5), today vs a 400-day
|
|
31
|
+
one ≥ 54% (2.0/1.07), and the `memory cards` row above is the other side of it
|
|
32
|
+
(40%, loses). Intra-session order is provably untouched: 17 hits of one session
|
|
33
|
+
scale by exactly the same factor (measured ratio 1.9784, identical across all
|
|
34
|
+
17) and none of them moves.
|
|
35
|
+
|
|
36
|
+
The cost is one expression per matched row. Measured directly on the worst case
|
|
37
|
+
(48,120 matched rows, `run`, 258 ms): **+4.9 ms** for the chosen `unixepoch()`
|
|
38
|
+
route, where a `strftime('%s', …)` route costs +9.6–12.7 ms and `julianday`
|
|
39
|
+
+5.0–7.2 ms — the fastest epoch route is the one that shipped. At the reader
|
|
40
|
+
level (interleaved A/B, 25 runs each, three passes) the delta is inside the
|
|
41
|
+
host's noise: −1.42 ms to +0.12 ms per query, with the one reproducible residue
|
|
42
|
+
being the fastest query in the set (0.66 → 0.78 ms, +0.12 ms, where a fixed cost
|
|
43
|
+
is a measurable share).
|
|
44
|
+
|
|
45
|
+
## The mechanism
|
|
46
|
+
|
|
47
|
+
Pinned upstream (`/tmp/ledger-confer/wake`, `d9f702b6`, tag v0.6.5):
|
|
48
|
+
|
|
49
|
+
```sql
|
|
50
|
+
-- db.rs#L1552 (the constant), #L1604 (the message lane), #L1685 (the title lane)
|
|
51
|
+
ORDER BY bm25(messages_fts) * (1.0 + 1.0 / (1.0 + max(0, ? - s.updated_at) / 2592000000.0)) LIMIT ?
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Wake multiplies its message lane and its title lane by the same expression, keys
|
|
55
|
+
it on the session's `updated_at` (not the message's `ts`, so intra-session order
|
|
56
|
+
is untouched), binds the clock as a parameter, and uses only arithmetic and
|
|
57
|
+
`max()` — no SQLite math extension. Its degraded `LIKE` path keeps time ordering
|
|
58
|
+
because it has no bm25 to weigh; Ledger has no such path (its lexical fallback
|
|
59
|
+
is still FTS5), so there is nothing to mirror there.
|
|
60
|
+
|
|
61
|
+
Ledger's port, one shared fragment used by both lanes:
|
|
62
|
+
|
|
63
|
+
```sql
|
|
64
|
+
bm25(records_fts) * (1.0 + 1.0 / (1.0 + max(0, ? - COALESCE(unixepoch(s.started_at) * 1000, 0)) / 2592000000.0)) AS rank
|
|
65
|
+
ORDER BY rank ASC, s.started_at DESC
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Four Ledger-specific shape decisions inside one expression:
|
|
69
|
+
|
|
70
|
+
- **`unixepoch()` is the epoch conversion**, because this store has no numeric
|
|
71
|
+
time column — `sessions.started_at` is ISO-8601 TEXT. It is core SQLite
|
|
72
|
+
(3.38+), and the read path only ever runs on Bun's bundled SQLite 3.51.0 (the
|
|
73
|
+
same runtime C5 verified its feature set against; the native SQLite window of
|
|
74
|
+
C1 never runs FTS ranking). Measured against the alternatives below.
|
|
75
|
+
- **`COALESCE(…, 0)`** puts an absent or unparsable timestamp at the epoch —
|
|
76
|
+
the factor's neutral end (×1.0). Without it, the arithmetic yields NULL, and
|
|
77
|
+
NULL sorts *first* in an `ASC` ORDER BY: an unparsable timestamp would have
|
|
78
|
+
become the strongest possible ranking signal. Verified on the store: all
|
|
79
|
+
2,613 sessions parse (0 NULLs), so the clamp is a guard, not a behaviour.
|
|
80
|
+
- **The `?` is `Date.now()` fixed once per query** and passed to both lanes, so
|
|
81
|
+
a title hit and a record hit of one session are weighted identically and a
|
|
82
|
+
single query cannot see two clocks (Wake's `now_ms()`, bound the same way).
|
|
83
|
+
This is also what made the constant cheap: `strftime('%s','now')` inside the
|
|
84
|
+
expression was evaluated per row.
|
|
85
|
+
- **H is spelled as a real** (`${RECENCY_DECAY_MS}.0`): SQLite divides integers
|
|
86
|
+
with integer division, and a truncated divisor would silently make the factor
|
|
87
|
+
×2 for every row in the corpus — a failure that would look like a stronger
|
|
88
|
+
prior rather than a bug.
|
|
89
|
+
|
|
90
|
+
## Decisions (the cycle's open questions, resolved)
|
|
91
|
+
|
|
92
|
+
**1. Where the prior applies: both lexical lanes; not the semantic lane.**
|
|
93
|
+
Wake weighs both of its lanes, and Ledger's two lanes are one query's answer, so
|
|
94
|
+
both take the same factor. The semantic lane was decided separately, on the
|
|
95
|
+
measured shape of cosine rather than on symmetry: with the shipped model
|
|
96
|
+
(`all-MiniLM-L6-v2`, 384 dims) over 300 real records from the production store,
|
|
97
|
+
the whole top-50 span of a query's cosine scores is **1.77–2.60×**
|
|
98
|
+
(r1/r50 = 2.19 / 1.77 / 2.60 for three queries; r1 ∈ [0.34, 0.40], r50 ∈ [0.15,
|
|
99
|
+
0.20]). That is the *entire* range a semantic query has to work with, and the
|
|
100
|
+
prior's own multiplier spans ×1–×2 over the same corpus — so the same factor on
|
|
101
|
+
cosine would order that lane by age nearly regardless of text, and it would stop
|
|
102
|
+
the reported `rank` from being a cosine at all. bm25 is the opposite kind of
|
|
103
|
+
score: unbounded, negative, and sparse (the measured flips above turn on
|
|
104
|
+
comparisons at 0.7% and 16%, and the no-flip on 2.5×). The program's promise —
|
|
105
|
+
cheap, arithmetic-only, no rescore pass — is kept by the lexical lanes, where
|
|
106
|
+
the prior is one `ORDER BY` term; the semantic lane is not left without recency,
|
|
107
|
+
it is left to the fusion, which is the layer designed to combine rank orders.
|
|
108
|
+
|
|
109
|
+
**2. The decay constant: 30 days, Wake's H, unchanged.** Measured distribution
|
|
110
|
+
of the store as it stands (read-only queries against the live file; the cycle
|
|
111
|
+
only ever read it):
|
|
112
|
+
|
|
113
|
+
| age of session start | sessions |
|
|
114
|
+
| --- | --- |
|
|
115
|
+
| ≤ 1 day | 17 |
|
|
116
|
+
| ≤ 7 days | 78 |
|
|
117
|
+
| ≤ 30 days | 132 |
|
|
118
|
+
| ≤ 90 days | 652 |
|
|
119
|
+
| ≤ 1 year | 2,605 |
|
|
120
|
+
| > 1 year | 8 |
|
|
121
|
+
|
|
122
|
+
and by month: 2025-08 8, 2025-09 33, 2025-10 37, 2025-11 1, 2025-12 11, 2026-01
|
|
123
|
+
755, 2026-02 2, 2026-03 12, 2026-05 1,068, 2026-06 69, 2026-07 209, 2026-08 300,
|
|
124
|
+
2026-09 108. Two things follow. The corpus is **not** "two weeks of heavy
|
|
125
|
+
activity" — it is a year wide with two backfilled lumps (2026-05, 2026-01) and a
|
|
126
|
+
live edge of ~78 sessions in the last week — so a prior tuned only to the last
|
|
127
|
+
fortnight would be tuned to 3% of the store. And 30 days is a scale that
|
|
128
|
+
separates the live edge from the lumps without erasing everything older: a
|
|
129
|
+
2026-05 session still carries ×1.20 and a 2026-01 one ×1.11, while the last
|
|
130
|
+
week's work spans ×1.81–×2.0. The constant stays Wake's; nothing measured argues
|
|
131
|
+
for a different one, and a deviation would need a corpus whose live edge sits at
|
|
132
|
+
a different scale (a reopen condition, below).
|
|
133
|
+
|
|
134
|
+
**3. The key is `started_at`, not `updated_at` or `ended_at`.** `updated_at` is
|
|
135
|
+
the ingest write clock (`ingest.ts` stamps `new Date()`), and the measurement
|
|
136
|
+
makes it unusable: every one of the store's 2,613 sessions is stamped between
|
|
137
|
+
2026-09-12 and 2026-09-16 — a four-day window that would make a backfill of
|
|
138
|
+
2025 material rank as "today". `ended_at` is the content analog of Wake's
|
|
139
|
+
`updated_at` and would be the second-best choice: it moves the recency bucket of
|
|
140
|
+
only 5 of 2,613 sessions (active in the last 7 days: 78 by `started_at`, 83 by
|
|
141
|
+
`ended_at`; last 24 h: 17 vs 19; >1 day of spread between the two: 48 sessions,
|
|
142
|
+
1.8%; `ended_at < started_at`: 0). With the two keys nearly equivalent on this
|
|
143
|
+
corpus, the lane keeps one timestamp: the one its `ORDER BY … started_at DESC`
|
|
144
|
+
tiebreak already names, and the one that never needs a fallback.
|
|
145
|
+
|
|
146
|
+
**4. RRF interplay: the prior must move the legs, and it does.** RRF reads rank
|
|
147
|
+
*positions* (k = 60), so a reorder inside a leg is exactly how a fused result
|
|
148
|
+
changes; the fusion code was not touched. Two consequences, both checked: a hit
|
|
149
|
+
whose raw relevance lead exceeds the factor ratio keeps its leg position (the
|
|
150
|
+
`memory cards` row), so the fusion keeps its winner; and a hit whose relevance is
|
|
151
|
+
comparable moves up its leg, which is the intended change (`gemini`: the
|
|
152
|
+
2.11-day title was 16% weaker in raw bm25 and still takes rank 1 — 1/(60+1) vs
|
|
153
|
+
1/(60+2) is a 1.6% RRF difference, and the fused order follows the leg). The
|
|
154
|
+
test suite pins the direction end-to-end: a fresh session that is *only* a
|
|
155
|
+
lexical match outranks a 500-day-old session that is *only* a semantic match,
|
|
156
|
+
because the lexical leg's position carries the recency and the semantic leg's
|
|
157
|
+
does not.
|
|
158
|
+
|
|
159
|
+
**5. Envelope honesty: nothing is disclosed, and `rank`'s meaning is
|
|
160
|
+
documented.** The prior is internal ranking, like bm25 itself — Wake's
|
|
161
|
+
`SearchHit` carries no score at all (`crates/wake-core/src/models.rs#L433-L440`),
|
|
162
|
+
and Ledger's envelope already discloses *mechanisms that change what could be
|
|
163
|
+
answered* (widening, fallback, count precision) rather than ranking internals.
|
|
164
|
+
What did change is what `score` means, so the api-surface lane-relative note now
|
|
165
|
+
says it: title and record hits carry bm25 weighted by the same prior, so within a
|
|
166
|
+
lane the score is the ordering key, and the stream's emission order (titles
|
|
167
|
+
first, then body) remains the contract; consumers must not re-sort across lanes.
|
|
168
|
+
The type comment on `SearchHitBase.rank` says the same for library callers. No
|
|
169
|
+
`recency_weighted` field, because a boolean that is always true is not
|
|
170
|
+
information.
|
|
171
|
+
|
|
172
|
+
**6. The constant is pinned by a test, not a comment.** The one place the number
|
|
173
|
+
30 is asserted: two sessions with *identical records* (identical raw bm25) aged
|
|
174
|
+
0 and exactly 30 days must report ranks in the ratio 0.75 —
|
|
175
|
+
`(1+1/(1+30/30)) / (1+1/(1+0)) = 1.5/2`. Move the constant and that ratio moves
|
|
176
|
+
with it (`packages/core/tests/recency-prior.test.ts`).
|
|
177
|
+
|
|
178
|
+
## FTS_FORMAT: verified against C2's counter contract, and it does not move
|
|
179
|
+
|
|
180
|
+
C2's contract (`docs/research/2026-09-16-c2-derive-counter.md`, ADR "Derive-Rule
|
|
181
|
+
Counters"): the counter names **the rules a derived index was produced under**,
|
|
182
|
+
and "a mismatch → one full re-derivation" — it exists for changes to the derived
|
|
183
|
+
tables' *shape* (rows, columns, tokenizer, projection), which is exactly what
|
|
184
|
+
C3 (echo filter), C4 (title index) and the refused C5 (trigram) changed. C6
|
|
185
|
+
changes none of that: the same DDL, the same projections, the same row sets, the
|
|
186
|
+
same tokenizer — the same `records_fts` and `titles_fts` bytes. What changed is
|
|
187
|
+
one expression inside two read-side `SELECT`s. The verification is structural,
|
|
188
|
+
not a measurement: `git diff --stat` for this cycle touches `reader.ts` (both
|
|
189
|
+
lane SQLs plus comments), tests, `types.ts` (a doc comment) and this note's docs
|
|
190
|
+
— no `schema.ts`, no `db.ts`, no `ingest.ts`, no migration, and
|
|
191
|
+
`DERIVE_COUNTERS[].outputs` (`records_fts`, `records_echo`, `titles_fts`) are
|
|
192
|
+
untouched, so a store at `fts_format = "3"` is exactly as derivable under this
|
|
193
|
+
build as under C4. The `derive-counters` suite passes unchanged. The honest
|
|
194
|
+
consequence for operators: this cycle ships with **zero re-derivation cost** —
|
|
195
|
+
the opposite of C3/C4/C5, which each cost a one-shot rebuild.
|
|
196
|
+
|
|
197
|
+
## Measurements (OBSERVED)
|
|
198
|
+
|
|
199
|
+
Host: Apple M1, macOS 25.6, APFS; Bun 1.3.14 (SQLite 3.51.0). Fixture: a copy of
|
|
200
|
+
the production store taken with `cp` of `db` + `-wal` + `-shm` into `/tmp` (the
|
|
201
|
+
live store was only ever read); "before" is the reader at HEAD `efa1b3a`
|
|
202
|
+
extracted with `git archive`, "after" is the working tree.
|
|
203
|
+
|
|
204
|
+
**The store's own timestamps** (read-only, production file): 2,613 sessions,
|
|
205
|
+
230,451 records, 0 embeddings; `started_at` spans 2025-08-25T16:31:18.545Z →
|
|
206
|
+
2026-09-16T08:14:08.018Z, `updated_at` spans 2026-09-12T08:50:56.960Z →
|
|
207
|
+
2026-09-16T08:52:11.295Z; no NULL `started_at`/`ended_at`/`updated_at`; 0
|
|
208
|
+
timestamps that `unixepoch()` or `strftime('%s', …)` cannot parse, and the two
|
|
209
|
+
agree on all 2,613 values (checked before choosing `unixepoch`).
|
|
210
|
+
|
|
211
|
+
**Before/after orderings** (limit 50, the reader's own SQL; "before" ranks are
|
|
212
|
+
raw bm25, "after" ranks are the weighted value). The factor column is computed
|
|
213
|
+
from the session age at the moment of the run.
|
|
214
|
+
|
|
215
|
+
`recency prior` — the flip:
|
|
216
|
+
|
|
217
|
+
| | rank | session age | raw | factor | weighted |
|
|
218
|
+
| --- | --- | --- | --- | --- | --- |
|
|
219
|
+
| before 1 | 1 | 3.62 d | −8.8617 | 1.892 | — |
|
|
220
|
+
| before (today's session, lower down) | — | 0.11 d | −8.8014 | 1.996 | — |
|
|
221
|
+
| after 1 | 1 | 0.11 d | −8.8014 | 1.996 | −17.5700 |
|
|
222
|
+
| after 2 | 2 | 3.62 d | −8.8617 | 1.892 | −16.7684 |
|
|
223
|
+
|
|
224
|
+
Raw gap 1.007×, factor gap 1.055×: today's match needed ≥ 94.8% of the older
|
|
225
|
+
one's magnitude and had 99.3%.
|
|
226
|
+
|
|
227
|
+
`gemini` (title lane) — the flip with a wider raw deficit:
|
|
228
|
+
|
|
229
|
+
| | rank | session age | raw | factor | weighted |
|
|
230
|
+
| --- | --- | --- | --- | --- | --- |
|
|
231
|
+
| before 1 | 1 | 34.81 d | −8.4272 | 1.463 | — |
|
|
232
|
+
| before (the new rank 1) | 3 | 2.11 d | −7.2660 | 1.934 | — |
|
|
233
|
+
| after 1 | 1 | 2.11 d | −7.2660 | 1.934 | −14.0547 |
|
|
234
|
+
| after 2 | 2 | 34.81 d | −8.4272 | 1.463 | −12.3282 |
|
|
235
|
+
|
|
236
|
+
Raw gap 1.160×, factor gap 1.322×: the newer title needed ≥ 75.6% and had 86.2%.
|
|
237
|
+
A record-lane hit from 1.65 d (raw −6.9040) also moved from below the old title's
|
|
238
|
+
record-lane tail to overall rank 6.
|
|
239
|
+
|
|
240
|
+
`memory cards` — the crossover holds:
|
|
241
|
+
|
|
242
|
+
| | rank | session age | raw | factor | weighted |
|
|
243
|
+
| --- | --- | --- | --- | --- | --- |
|
|
244
|
+
| before 1 | 1 | 64.22 d | −14.7885 | 1.318 | — |
|
|
245
|
+
| after 1 | 1 | 64.22 d | −14.7885 | 1.318 | −19.4970 |
|
|
246
|
+
| after 2 | 2 | 1.65 d | −5.9181 | 1.948 | −11.5284 |
|
|
247
|
+
|
|
248
|
+
Raw gap 2.499×, factor gap 1.478×: today's best match needed ≥ 67.7% of the old
|
|
249
|
+
one's magnitude and has 40.0% — the relevance lead outlives the factor, and the
|
|
250
|
+
old session keeps rank 1.
|
|
251
|
+
|
|
252
|
+
**Intra-session invariance** (`wal_autocheckpoint`, one session, 17 hits): every
|
|
253
|
+
hit scaled by the same 1.9784 (the session's factor at 0.66 d), so the sequence
|
|
254
|
+
of weighted ranks is the sequence of raw ranks multiplied by a constant — no
|
|
255
|
+
within-session reordering is possible.
|
|
256
|
+
|
|
257
|
+
**Latency** (limit 50, warm, median of 25 interleaved runs in one process; hits =
|
|
258
|
+
returned hits; matched rows from a separate `COUNT(*) … MATCH`, given where it
|
|
259
|
+
matters):
|
|
260
|
+
|
|
261
|
+
| query | hits | matched rows | before | after | delta |
|
|
262
|
+
| --- | --- | --- | --- | --- | --- |
|
|
263
|
+
| `recency prior` | 25 | — | 1.57 ms | 1.59 ms | +0.02 ms |
|
|
264
|
+
| `memory cards` | 7 | — | 0.75 ms | 0.76 ms | +0.01 ms |
|
|
265
|
+
| `wal_autocheckpoint` | 50 | 84 | 8.67 ms | 8.72 ms | +0.04 ms |
|
|
266
|
+
| `gemini` | 50 | — | 41.90 ms | 40.48 ms | −1.42 ms |
|
|
267
|
+
| `ledger` | 50 | 13,807 | 0.68 ms | 0.77 ms | +0.08 ms |
|
|
268
|
+
| `the` | 50 | 74,074 | 3.09 ms | 3.04 ms | −0.05 ms |
|
|
269
|
+
| `test` | 50 | — | 0.32 ms | 0.37 ms | +0.04 ms |
|
|
270
|
+
| `run` | 50 | 48,120 | 258.33 ms | 257.57 ms | −0.76 ms |
|
|
271
|
+
|
|
272
|
+
Three passes of the same harness agreed within this: the largest positive delta
|
|
273
|
+
seen on any query in any pass was on `ledger` (0.68 → 0.77 ms, reproducible at
|
|
274
|
+
+0.08…+0.12 ms), and the 258 ms query measured −0.76 and +5.76 ms on the shipped
|
|
275
|
+
shape and +4.10 ms on the earlier `strftime` variant — i.e. inside the host's
|
|
276
|
+
noise band (±3 ms at that size), which is why the added term was also measured
|
|
277
|
+
*directly*: the same rank expression with and without the factor, in the
|
|
278
|
+
minimal SQL around it (no `snippet()` columns), 15 iterations, same store, query
|
|
279
|
+
`run`, 48,120 matched rows:
|
|
280
|
+
|
|
281
|
+
| rank expression | median |
|
|
282
|
+
| --- | --- |
|
|
283
|
+
| `bm25(records_fts)` (baseline) | 202.27 ms |
|
|
284
|
+
| `bm25 × strftime('%s', …)` prior | 215.00 ms (+12.7 ms) |
|
|
285
|
+
| `bm25 × julianday(…)` prior | 207.17 ms (+4.9 ms) |
|
|
286
|
+
| **`bm25 × unixepoch(…)` prior (shipped)** | **207.13 ms (+4.9 ms)** |
|
|
287
|
+
|
|
288
|
+
So the prior's true worst-case cost on this corpus is ~5 ms on a 258 ms query
|
|
289
|
+
(≈1.8%), the chosen epoch route is the cheapest of the three measured, and
|
|
290
|
+
everything that answers in under 10 ms pays ≤ 0.12 ms.
|
|
291
|
+
|
|
292
|
+
**Cosine spread** (the semantic decision, measured with the shipped
|
|
293
|
+
`OnnxEmbeddingProvider`): 300 real records' embedding text from the store copy,
|
|
294
|
+
three queries, cosines of the top 50:
|
|
295
|
+
|
|
296
|
+
| query | r1 | r10 | r50 | median | r1/r50 |
|
|
297
|
+
| --- | --- | --- | --- | --- | --- |
|
|
298
|
+
| `recency prior` | 0.3363 | 0.2338 | 0.1539 | 0.1908 | 2.19× |
|
|
299
|
+
| `sqlite wal checkpoint` | 0.3463 | 0.2520 | 0.1953 | 0.2203 | 1.77× |
|
|
300
|
+
| `gemini embeddings backfill` | 0.4027 | 0.2394 | 0.1550 | 0.1846 | 2.60× |
|
|
301
|
+
|
|
302
|
+
## What is inferred (INFERRED, marked)
|
|
303
|
+
|
|
304
|
+
- The corpus conclusion ("30 days separates the live edge from the backfilled
|
|
305
|
+
lumps") is an inference from the session-start histogram, not a measurement of
|
|
306
|
+
retrieval quality: no relevance judgement exists for this store, so the cycle's
|
|
307
|
+
evidence is ordering deltas and arithmetic, not graded accuracy. A qrels set
|
|
308
|
+
would be the way to argue for a *different* constant.
|
|
309
|
+
- The +4.9 ms worst case is a lower bound for a bigger corpus: the expression is
|
|
310
|
+
O(matched rows), so a store several times this size would pay proportionally
|
|
311
|
+
more on its slowest queries (the same shape C5 measured for bm25 itself).
|
|
312
|
+
- The claim that a semantic-lane prior would order by age is an extrapolation
|
|
313
|
+
from the measured 1.8–2.6× span plus the factor's own ×1–×2 range; it is not a
|
|
314
|
+
measured end-to-end semantic ranking (the production store holds no vectors:
|
|
315
|
+
`records_embeddings` is empty, so the semantic lane answers lexically there by
|
|
316
|
+
design).
|
|
317
|
+
- `ended_at` as the key would likely read slightly better for resumed sessions
|
|
318
|
+
(the 5 sessions whose bucket moves), but nothing measured shows a user-visible
|
|
319
|
+
difference at this corpus size.
|
|
320
|
+
|
|
321
|
+
## Tests
|
|
322
|
+
|
|
323
|
+
`packages/core/tests/recency-prior.test.ts` (7 cases, fixtures dated relative to
|
|
324
|
+
`now` because a literal timestamp cannot pin "today"). Three of them prove their
|
|
325
|
+
own premise by *inverting* the prior — `rawBm25(hit) = rank / (1 + 1/(1 +
|
|
326
|
+
age/30d))` recovers the raw bm25 the lane would have ranked by, so the test can
|
|
327
|
+
assert that the fresh hit really was the weaker raw match the prior reversed,
|
|
328
|
+
and that the crossover hit truly is stronger by more than the factor's span:
|
|
329
|
+
|
|
330
|
+
1. the record lane prefers the session from today when the two matches are
|
|
331
|
+
comparable, with the premise asserted (today's raw bm25 is weaker);
|
|
332
|
+
2. a much stronger old match still wins — the crossover, with the fixture's
|
|
333
|
+
measured raw gap (2.50×) against the factor's span (1.87×) in the case
|
|
334
|
+
comment and in the premise assertion;
|
|
335
|
+
3. the title lane orders by the same prior (today's longer, weaker title beats
|
|
336
|
+
an older, stronger one), and an exact 400-day-old title still wins over
|
|
337
|
+
today's diluted one, again with both premises asserted;
|
|
338
|
+
4. intra-session order is the session's bm25 order — the factor is the session's,
|
|
339
|
+
not the record's (the appended-today record still ranks below the older,
|
|
340
|
+
stronger one);
|
|
341
|
+
5. the decay constant is 30 days: identical records 0 and 30 days old report
|
|
342
|
+
ranks in the exact ratio 0.75;
|
|
343
|
+
6. the semantic lane stays pure cosine (a 500-day-old session wins on similarity
|
|
344
|
+
alone, and its reported rank is the cosine) while hybrid lifts the fresh
|
|
345
|
+
lexical match above it through the lexical leg's position;
|
|
346
|
+
7. the lane contract survives: a title hit is emitted ahead of a record hit whose
|
|
347
|
+
rank is *more* negative, and both ranks are negative.
|
|
348
|
+
|
|
349
|
+
Five existing tests carried assumptions the prior invalidates, each fixed at the
|
|
350
|
+
assumption rather than around it:
|
|
351
|
+
|
|
352
|
+
- `search-quality.test.ts`, the `zonk` twins: they no longer share a rank (the
|
|
353
|
+
prior separates them by session age), so the assertion pins the direction
|
|
354
|
+
(newest rank most negative) instead of equality — the order assertion above it
|
|
355
|
+
is unchanged and still passes.
|
|
356
|
+
- `integration-migrations.test.ts`, `querySnapshot`: the rebuild comparison
|
|
357
|
+
cannot compare `rank` exactly any more, because rank is now a function of the
|
|
358
|
+
wall clock as well as the index; it compares at six significant digits, which
|
|
359
|
+
absorbs any time between the snapshots and is far tighter than any change a
|
|
360
|
+
rebuild could make to bm25.
|
|
361
|
+
- `robot.test.ts`, the v1 byte-identity case, and `cli-canonical.test.ts`, the
|
|
362
|
+
`search`/`query` alias case: two CLI *processes* can no longer render identical
|
|
363
|
+
last digits for `score`, so the score value is blanked (robot) or zeroed
|
|
364
|
+
(cli-canonical, whose normalizer already does the same for `elapsed_ms`) on
|
|
365
|
+
both sides, and every other byte is still compared as written.
|
|
366
|
+
- `title-index.test.ts`: the tie comment ("equal rank, so the newer session comes
|
|
367
|
+
first") became "equal relevance, so the prior decides" — the assertion is
|
|
368
|
+
unchanged.
|
|
369
|
+
|
|
370
|
+
## Roadmap context and the open-source case (program doc, C6 row)
|
|
371
|
+
|
|
372
|
+
> **C6** | Recency prior before RRF (bm25 × time-decay, arithmetic-only) | All
|
|
373
|
+
> lanes order by relevance with `started_at DESC` tiebreak only; "the session
|
|
374
|
+
> from today" loses to an old strong match | Cheap relevance win every user
|
|
375
|
+
> feels; no extension dependency, single ORDER BY term
|
|
376
|
+
|
|
377
|
+
Both halves hold as measured: the fix is real and visible (`recency prior`,
|
|
378
|
+
`gemini`), it is one `ORDER BY` term with no extension and no rescore pass, and
|
|
379
|
+
its cost is ~5 ms at the corpus's slowest query and unmeasurable elsewhere. The
|
|
380
|
+
program's dependency note — C6 after C5, "ranking experiments on the final
|
|
381
|
+
tokenizer" — resolves cleanly: C5 refused the trigram swap, so the prior was
|
|
382
|
+
designed and measured against the porter index the store actually ships.
|
|
383
|
+
|
|
384
|
+
## What would reopen this
|
|
385
|
+
|
|
386
|
+
1. **A relevance judgement set.** Every ordering claim here is an ordering
|
|
387
|
+
delta; with graded relevance the constant (30 d) and the weight (W = 1) could
|
|
388
|
+
be tuned instead of inherited, and the crossover table is exactly the shape
|
|
389
|
+
that tuning would consume.
|
|
390
|
+
2. **A corpus whose live edge sits elsewhere** — e.g. a store used hourly where
|
|
391
|
+
a 7-day scale separates the working set, or one whose backfill is recent
|
|
392
|
+
enough that `updated_at`-shaped reasoning tempts a re-look. Re-measure the
|
|
393
|
+
histogram first; the constant is one line.
|
|
394
|
+
3. **Candidate capping for the common-term classes** (the C5 note's own reopen
|
|
395
|
+
item): the prior is O(matched rows) on top of bm25's existing O(matched rows),
|
|
396
|
+
and a match-set-bound query large enough to need capping would want the
|
|
397
|
+
recency term inside the capped plan rather than after it.
|
|
398
|
+
4. **A semantic-lane recency decision with real vectors in the store.** The
|
|
399
|
+
decision here is argued from the cosine spread and defended by the fusion;
|
|
400
|
+
with a populated `records_embeddings` the same measurement could be run on
|
|
401
|
+
real semantic rankings.
|
|
402
|
+
|
|
403
|
+
## Method (for reproducibility)
|
|
404
|
+
|
|
405
|
+
Copy the live store to `/tmp` (`cp` of `db`, `-wal`, `-shm`; the original was
|
|
406
|
+
never opened for writing). Extract HEAD into `/tmp/c6/before` with
|
|
407
|
+
`git archive HEAD packages/core | tar -x` and symlink `node_modules`, so both
|
|
408
|
+
codebases can be loaded side by side. The harnesses lived in `/tmp/c6` and were
|
|
409
|
+
deleted: an A/B probe that prints ranked hits for a query from a given tree, an
|
|
410
|
+
interleaved latency harness (25 runs per tree per query, medians, in one
|
|
411
|
+
process), a direct-expression benchmark, a cosine probe against the shipped
|
|
412
|
+
ONNX provider, and a fixture calibration script. Gates on the tree:
|
|
413
|
+
`cd packages/core && bun test` (**586 pass / 0 fail** across 41 files, of which
|
|
414
|
+
this cycle's 7 are `tests/recency-prior.test.ts`), `bun run typecheck`,
|
|
415
|
+
`bun run build`.
|
|
416
|
+
|
|
417
|
+
## Wake pins (MIT, `iAmCorey/Wake@d9f702b6` tag v0.6.5)
|
|
418
|
+
|
|
419
|
+
| mechanism | pin |
|
|
420
|
+
| --- | --- |
|
|
421
|
+
| the decay factor, W = 1 and H = 30 days, as one ORDER BY term | `crates/wake-core/src/db.rs#L1552` (`RECENCY_BOOST`), `#L1604` |
|
|
422
|
+
| the same factor on the title lane | `db.rs#L1685` |
|
|
423
|
+
| keyed on the session's timestamp, so intra-session order is untouched | `db.rs#L1546-L1556` (the comment above the constant) |
|
|
424
|
+
| the clock bound as a parameter, arithmetic and `max()` only | `db.rs#L1553` (`now_ms()`), `#L1604` |
|
|
425
|
+
| the degraded path keeps time ordering (no bm25 to weigh) | `db.rs#L1632-L1650` |
|
|
426
|
+
| ranking internals are not part of the hit shape | `crates/wake-core/src/models.rs#L433-L440` (no score field) |
|
|
427
|
+
|
|
428
|
+
## Files
|
|
429
|
+
|
|
430
|
+
- `packages/core/src/reader.ts` — `RECENCY_DECAY_MS`/`RECENCY_PRIOR_SQL`, the
|
|
431
|
+
record lane's `rank`, the title lane's `rank`, the per-query `now`, and the
|
|
432
|
+
semantic lane's decision comment.
|
|
433
|
+
- `packages/core/src/types.ts` — the `SearchHitBase.rank` doc comment.
|
|
434
|
+
- `packages/core/tests/recency-prior.test.ts` — this cycle's 7 cases.
|
|
435
|
+
- `packages/core/tests/{search-quality,integration-migrations,robot,cli-canonical,title-index}.test.ts`
|
|
436
|
+
— the five assumptions the prior invalidates, each fixed where it was wrong.
|
|
437
|
+
- `docs/api-surface.md` — the lane-relative `score` sentence (§1.4.4), the
|
|
438
|
+
`titles_fts` search bullet (§3.2), and the C6 ADR entry (§4).
|