pi-memory-evolution 0.3.1 → 0.3.2

package/CHANGELOG.md

@@ -2,6 +2,13 @@
 
 All notable changes to pi-memory-evolution are documented here.
 
+## [0.3.2](https://github.com/btnalit/pi-memory-evolution/compare/v0.3.1...v0.3.2) (2026-09-10)
+
+
+### Bug Fixes
+
+* apply the candidate cap after the containment filter, not before ([#26](https://github.com/btnalit/pi-memory-evolution/issues/26)) ([58cccac](https://github.com/btnalit/pi-memory-evolution/commit/58cccac3ee9367a3c9c386ac9d0686198493a926))
+
 ## [0.3.1](https://github.com/btnalit/pi-memory-evolution/compare/v0.3.0...v0.3.1) (2026-09-10)
 
 

package/docs/core-quality.md

@@ -67,8 +67,9 @@ not inherit old utility/accuracy feedback. A literal correction clears old alias
 feedback; undo restores the actual prior metadata.
 
 **Limit:** conflict detection still depends on the model identifying a `replaces` target
-in its candidate set, which the host now selects by what the source mentions rather than by
-what is recent. Records the source never mentions are out of reach for that source. Arbitrary contradictory additions, paraphrases
+in its candidate set, which the host selects by what the source mentions, then caps by recency.
+Records the source never mentions are out of reach for that source, and so are mentioned ones
+older than the 32 most recent that qualify. Arbitrary contradictory additions, paraphrases
 and cross-origin identities are not automatically resolved. Multiple source events are
 not treated as independent corroboration; repeated summaries may share the same root
 observation. There is no reinforcement count or model-generated confidence score.

package/docs/design.md

@@ -178,6 +178,10 @@ Jaccard, because a source is orders of magnitude longer than a claim. A progress
 uses exactly the records nominated in `targets`.
 
 This is a **filter, never a ranking**, and the qualifying records keep the original recency order.
+The cap applies **after** the filter, not before: capping by recency first meant a scope holding more
+than 32 records could never show an older one again, however squarely the source was about it, so it
+could never be superseded — only accumulated alongside. Reach is still bounded by the 32 most recent
+qualifying records, and ordering stays by update time, never by recency of confirmation.
 Containment is highest for a record the source merely restates and lower for the one it
 contradicts, because the changed value is exactly the term that is missing; ordering by it and
 cutting to a small cap would drop the record that most needed superseding, and both versions

package/docs/usage.md

@@ -111,7 +111,9 @@ rename the tool to hide the conflict; an old installation would still run its ho
 - Each processing attempt makes at most one background model call, using up to 32
   recently updated active memories from that source's capture origin **that the source
   actually mentions** — the host filters the rest out, so a source cannot replace a
-  record it never talks about. This is a
+  record it never talks about. The filter runs **before** the cap, so an older record the
+  source is squarely about is no longer hidden behind newer unrelated ones; reach is still
+  bounded by the 32 most recent records that qualify. This is a
   conservative automatic-replacement safeguard, **not a recall restriction**.
   It defaults to **the current Pi session model and Pi's own provider/auth resolution**.
   With no session override, this is Pi's configured default. Quota/rate limits or repeated

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "pi-memory-evolution",
-	"version": "0.3.1",
+	"version": "0.3.2",
 	"description": "Memory that maintains itself. Pi learns what matters, injects what this session needs, and recalls the rest — nothing to configure, no commands to learn.",
 	"license": "MIT",
 	"publishConfig": {

package/src/memory/limits.ts

@@ -35,9 +35,21 @@ export const MAX_SEARCH_TERM_CHARS = 64;
  * source merely restates and lower for the one it contradicts — the changed value is exactly
  * the term that is missing — so ordering by it drops the record that most needs superseding.
  * IDF weighting makes that worse, not better: the missing term is the rare one. Qualifying
- * records therefore keep the original recency order, and the cap only bounds the payload. */
+ * records therefore keep the original recency order, and the cap below takes the most recent. */
 export const RELATED_CONTAINMENT = 0.4;
-/** Unchanged from the recency-only selection this filter narrows, so nothing once shown is cut. */
+/** How many qualifying records may be sent. Applied AFTER the containment filter, never before:
+ * capping by recency first meant a scope with more than 32 records could never show an older one
+ * again, however squarely the source was about it, so it could never be superseded — only
+ * accumulated alongside. Measured on a live 89-record scope, 46 relevant records were unreachable
+ * that way, including the exact record a user correction was aimed at (rank 63, top containment).
+ *
+ * Nothing once shown is cut: a record inside the recency top-32 overall is necessarily among the
+ * 32 most recent qualifying records, so the old selection is a subset of this one.
+ *
+ * Residual, deliberately accepted: reach is still bounded by "the 32 most recent that qualify".
+ * Containment saturates on long summary sources (38 of 89 records scored 1.00), so in a busy scope
+ * the oldest still do not re-enter. Ordering stays by `updatedAt` and must not become recency of
+ * confirmation, or records would be shown because they were recently shown. */
 export const MAX_CANDIDATES = 32;
 
 /** What a reply may cost us, derived from the contract above rather than invented. These are

package/src/memory/memory-store.ts

@@ -314,6 +314,39 @@ export class MemoryStore {
 			.get(...(scope === undefined ? [] : [scope]), now);
 		return row ? String(row.id) : undefined;
 	}
+	/** The single definition of what a source may reason about locally and what it may be shown.
+	 * The reservation estimate and the run itself must call this same function: an estimate cheaper
+	 * than the payload it authorizes is how a call gets admitted that the provider then refuses.
+	 *
+	 * A progress source arrives with its targets already nominated, so those are its candidates.
+	 * For everything else the host drops records this source never mentions: it cannot supersede
+	 * a fact it does not talk about, and retrieval is the host's job — deterministic and free —
+	 * not something to pay a model to do by handing it every recent record to search through.
+	 *
+	 * `memories` is NOT read-only: `finishEvolution` writes through it, replacing `searchTerms`
+	 * wholesale on an exact-content match and refreshing a duplicate's evidence. Those writes were
+	 * always bounded by the recency window, and must stay bounded, or model output would rewrite
+	 * records the model was never shown — losing aliases it could not have preserved and resetting
+	 * the aging clock on records it never named. So `memories` is the recency window plus whatever
+	 * was actually shown, and nothing else: `candidates` stays a subset, and every record the host
+	 * may write through is one that was either recent or in front of the model. */
+	private selectCandidates(source: Source): { memories: DurableMemory[]; candidates: DurableMemory[] } {
+		const scoped = this.readMemories(source.scope).filter((m) => m.scope === source.scope && active(m)
+			&& (source.kind !== "progress" || (m.kind === "project_state" && source.targets!.includes(m.id))))
+			.sort((a,b) => Date.parse(b.updatedAt)-Date.parse(a.updatedAt));
+		// Order is left alone deliberately. Containment filters; it must never rank. See limits.ts.
+		// The cap is applied AFTER the filter. Capping first hid every matching record that had aged
+		// past the 32 most recent, so in any scope with more than 32 records an older one could never
+		// be shown again, and therefore never superseded — only accumulated alongside.
+		const vocabulary = source.kind === "progress" ? undefined : features(source.content);
+		const candidates = (vocabulary === undefined ? scoped
+			: scoped.filter((m) => mentions(vocabulary, m.content, m.searchTerms) >= RELATED_CONTAINMENT)).slice(0, MAX_CANDIDATES);
+		const recent = scoped.slice(0, MAX_CANDIDATES);
+		const known = new Set(recent.map((m) => m.id));
+		// Older shown records follow the recency window in age order, so this stays recency-ordered.
+		return { memories: [...recent, ...candidates.filter((m) => !known.has(m.id))], candidates };
+	}
+
 	beginEvolution(id: string, retry: RetryMode = false, timeoutMs = EVOLUTION_TIMEOUT_MS, now = Date.now(), model?: string, call?: CallOptions): EvolutionRun | undefined {
 		this.assertLearningReady();
 		return this.transaction(() => {
@@ -324,14 +357,18 @@ export class MemoryStore {
 			this.db.prepare('UPDATE sources SET last_checked=? WHERE id=?').run(now, id);
 			const priorModels = parseModels(row.call_models);
 			const correctOutput = Number(row.corrections) === 0 && Number(row.output_failures) === 1 && ['invalid_output','output_limit'].includes(String(row.last_error));
+			const source = parseSource(row.data);
+			// Selecting candidates scans the whole scope; this transaction holds the write lock, so it is
+			// computed at most once per attempt and only once a route is actually available. The estimate
+			// and the run must see the same set anyway — a cheaper estimate authorizes a larger payload.
+			let selected: { memories: DurableMemory[]; candidates: DurableMemory[] } | undefined;
+			const select = () => (selected ??= this.selectCandidates(source));
 			if (model !== undefined) {
 				model = modelLabel(model);
 				if (retry !== true && !priorModels.includes(model) && priorModels.length >= this.policy.sourceModels) return undefined;
 				const provider = modelLabel(call?.provider ?? model.split('/')[0]);
 				if (retry !== true && routeUntil(this.db, model, provider, now) > now) return undefined;
-				const source = parseSource(row.data);
-				const bytes = Buffer.byteLength(JSON.stringify(source)) + this.readMemories(source.scope)
-					.filter(active).sort((a,b) => Date.parse(b.updatedAt)-Date.parse(a.updatedAt)).slice(0,32)
+				const bytes = Buffer.byteLength(JSON.stringify(source)) + select().candidates
 					.reduce((sum,m) => sum + Buffer.byteLength(JSON.stringify(m)), 0);
 				const reserve = estimatedCost(bytes, call);
 				if (budgetUntil(this.db, model, now, this.policy, reserve) > now) return undefined;
@@ -340,19 +377,8 @@ export class MemoryStore {
 			}
 			timeoutMs = Math.min(timeoutMs, this.policy.timeoutMs, retry === true ? timeoutMs : Math.max(1, this.policy.sourceTimeMs - Number(row.call_ms)));
 			this.db.prepare(`UPDATE sources SET state='running', attempt=attempt+1, lease=? WHERE id=?`).run(now + timeoutMs + LEASE_GRACE_MS, id);
-			const source = parseSource(row.data);
 			if (source.id !== id) throw new Error("Invalid source identity");
-			const memories = this.readMemories(source.scope).filter((m) => m.scope === source.scope && active(m)
-				&& (source.kind !== "progress" || (m.kind === "project_state" && source.targets!.includes(m.id))))
-				.sort((a,b) => Date.parse(b.updatedAt)-Date.parse(a.updatedAt)).slice(0, MAX_CANDIDATES);
-			// A progress source arrives with its targets already nominated, so those are its candidates.
-			// For everything else the host drops records this source never mentions: it cannot supersede
-			// a fact it does not talk about, and retrieval is the host's job — deterministic and free —
-			// not something to pay a model to do by handing it every recent record to search through.
-			// Order is left alone deliberately. Containment filters; it must never rank. See limits.ts.
-			const vocabulary = source.kind === "progress" ? undefined : features(source.content);
-			const candidates = vocabulary === undefined ? memories
-				: memories.filter((m) => mentions(vocabulary, m.content, m.searchTerms) >= RELATED_CONTAINMENT);
+			const { memories, candidates } = select();
 			// The stored diagnostic explains the last completed outcome. Claiming an attempt must not erase it:
 			// a cancelled or interrupted run would otherwise leave a paused source with no recorded reason.
 			return { source, attempt: Number(row.attempt) + 1, generation: this.generation(source.scope), memories, candidates, timeoutMs, correctOutput,