pi-memory-evolution 0.3.0 → 0.3.2

package/CHANGELOG.md

@@ -2,6 +2,20 @@
 
 All notable changes to pi-memory-evolution are documented here.
 
+## [0.3.2](https://github.com/btnalit/pi-memory-evolution/compare/v0.3.1...v0.3.2) (2026-09-10)
+
+
+### Bug Fixes
+
+* apply the candidate cap after the containment filter, not before ([#26](https://github.com/btnalit/pi-memory-evolution/issues/26)) ([58cccac](https://github.com/btnalit/pi-memory-evolution/commit/58cccac3ee9367a3c9c386ac9d0686198493a926))
+
+## [0.3.1](https://github.com/btnalit/pi-memory-evolution/compare/v0.3.0...v0.3.1) (2026-09-10)
+
+
+### Bug Fixes
+
+* let a model correct a broken output contract instead of losing the source ([#24](https://github.com/btnalit/pi-memory-evolution/issues/24)) ([97cde15](https://github.com/btnalit/pi-memory-evolution/commit/97cde1593d466993ec37ca0de81c0fcba09ef2cc))
+
 ## [0.3.0](https://github.com/btnalit/pi-memory-evolution/compare/v0.2.7...v0.3.0) (2026-09-10)
 
 

package/docs/core-quality.md

@@ -67,8 +67,9 @@ not inherit old utility/accuracy feedback. A literal correction clears old alias
 feedback; undo restores the actual prior metadata.
 
 **Limit:** conflict detection still depends on the model identifying a `replaces` target
-in its candidate set, which the host now selects by what the source mentions rather than by
-what is recent. Records the source never mentions are out of reach for that source. Arbitrary contradictory additions, paraphrases
+in its candidate set, which the host selects by what the source mentions, then caps by recency.
+Records the source never mentions are out of reach for that source, and so are mentioned ones
+older than the 32 most recent that qualify. Arbitrary contradictory additions, paraphrases
 and cross-origin identities are not automatically resolved. Multiple source events are
 not treated as independent corroboration; repeated summaries may share the same root
 observation. There is no reinforcement count or model-generated confidence score.

package/docs/design.md

@@ -178,6 +178,10 @@ Jaccard, because a source is orders of magnitude longer than a claim. A progress
 uses exactly the records nominated in `targets`.
 
 This is a **filter, never a ranking**, and the qualifying records keep the original recency order.
+The cap applies **after** the filter, not before: capping by recency first meant a scope holding more
+than 32 records could never show an older one again, however squarely the source was about it, so it
+could never be superseded — only accumulated alongside. Reach is still bounded by the 32 most recent
+qualifying records, and ordering stays by update time, never by recency of confirmation.
 Containment is highest for a record the source merely restates and lower for the one it
 contradicts, because the changed value is exactly the term that is missing; ordering by it and
 cutting to a small cap would drop the record that most needed superseding, and both versions
@@ -272,8 +276,11 @@ cooldowns never contaminate a source's retry_at. Generic runtime backoff is 1 mi
 5 minutes, 15 minutes, then 1 hour with up to 20% jitter; quota/auth/rate-limit errors can
 immediately use an allowed other provider. Two recent transport failures pause a model.
 Defaults allow 4 reserved calls / 2 models / 300 seconds per source, one format correction,
-and 20 reservations/hour shared across providers and processes. Unsafe writes/refusals
-pause immediately. Three output failures or five generic failures also pause work.
+and 20 reservations/hour shared across providers and processes. A write refused on the store's own authority — pinned,
+cross-origin, or a record already newer than the source — pauses immediately, because the same
+evidence would be refused again. A model that breaks the output contract is not the same thing:
+it is reported as `invalid_output` carrying the rule it broke, and is corrected and retried.
+Three output failures or five generic failures also pause work.
 Shutdown adds no failures, but an already reserved request may still cost money.
 `/memory evolve` overrides source delay/caps for one attempt, never shared ceilings.
 Completed/retired jobs are never forced to run again. A source resumed in another directory

package/docs/recovery.md

@@ -38,12 +38,26 @@ Last-checked ordering moves temporarily unroutable sources behind other eligible
 | HTTP 429 without a known quota code | Cool down that **model** for at least 60 seconds and at least `Retry-After`, since tpm/rpm ceilings are usually per model; try another provider, or an allowlisted model of the same provider. |
 | HTTP 408, timeout, network/5xx failure | Delayed retry; two recent transport failures cool down that model for 15 minutes and permit fallback. |
 | Context overflow or HTTP 400/404/422 | Do not resend unchanged requests to that model immediately; cool it down for 1 hour and permit a compatible backup. |
-| Invalid JSON / output truncation | Initial attempt, at most one corrective prompt, then an alternate model. Three output failures pause the source; two failures do not authorize repeatedly probing the same model. |
+| Invalid JSON / output truncation / **broken output contract** | Initial attempt, at most one corrective prompt naming the rule that was broken, then an alternate model. Three output failures pause the source; two failures do not authorize repeatedly probing the same model. |
 | Unsafe/unauthorized write or recognized safety/refusal | Reject and pause that source. No fallback to circumvent safety or write guards. |
 | Stale result | Re-read on a bounded delayed retry; not counted as a provider-health failure. |
 | Cancellation / shutdown / reload | Release the lease without adding a failure; already-reserved requests may still have consumed quota. |
 | Unknown error | Safe generic category and bounded transport retry; no guessing that arbitrary error prose means insufficient credit. |
 
+A claim the store refuses is classified by *whose* mistake it is, because the two need opposite
+handling. A **broken output contract** — a progress source answering with a plain addition instead
+of replacing its nominated record, a replacement id that was never shown, the same target replaced
+twice, a replacement that changes evidence kind, a cyclic batch, or aliases that fail validation —
+is the model's error. It is reported as `invalid_output` with a
+`reason` naming the rule, so the correction prompt can cite it and a sibling model may try. A
+**refusal on the store's own authority** — a pinned record, another origin's record, a record
+already newer than the source, or model output that still redacts to a placeholder — is not
+correctable: the same evidence is refused however often it
+is offered, so it stays `write_rejected` and stops the source rather than burning the budget. The
+last of those is a deliberate refusal to retry rather than an inability: a correction would resend
+the same unredacted source to another call and, because `invalid_output` permits cross-provider
+fallback, to another vendor. One exposure and a stop is the cheaper outcome.
+
 Generic source backoff is 1 minute, 5 minutes, 15 minutes, then 1 hour, with up to 20%
 positive jitter on runtime failures. Route failures can try one alternate immediately;
 there are at most two immediate attempts per queue item, not a sleep/retry loop. The

package/docs/usage.md

@@ -111,7 +111,9 @@ rename the tool to hide the conflict; an old installation would still run its ho
 - Each processing attempt makes at most one background model call, using up to 32
   recently updated active memories from that source's capture origin **that the source
   actually mentions** — the host filters the rest out, so a source cannot replace a
-  record it never talks about. This is a
+  record it never talks about. The filter runs **before** the cap, so an older record the
+  source is squarely about is no longer hidden behind newer unrelated ones; reach is still
+  bounded by the 32 most recent records that qualify. This is a
   conservative automatic-replacement safeguard, **not a recall restriction**.
   It defaults to **the current Pi session model and Pi's own provider/auth resolution**.
   With no session override, this is Pi's configured default. Quota/rate limits or repeated
@@ -139,7 +141,9 @@ rename the tool to hide the conflict; an old installation would still run its ho
   backoff is 1 minute, 5 minutes, 15 minutes, then 1 hour, plus up to 20% jitter. Defaults
   cap each source at **4 reserved calls / 2 models / 300 seconds**, and all models and
   processes share **20 calls per rolling hour**. Three output failures or five generic
-  failures also pause work; unsafe writes/refusals pause immediately. `/memory status`
+  failures also pause work; a write refused on the store's own authority (pinned, cross-origin,
+  or a record already newer than the source) pauses immediately, while a model that breaks the
+  output contract is corrected and retried like any other invalid output. `/memory status`
   explains routes, budgets and actual outcomes. `/memory evolve` overrides source limits
   for one attempt, never shared ceilings or settled jobs. [Full policy/configuration](recovery.md).
 - A job lease lasts **150 seconds** (120-second deadline plus 30-second grace). The timer

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "pi-memory-evolution",
-	"version": "0.3.0",
+	"version": "0.3.2",
 	"description": "Memory that maintains itself. Pi learns what matters, injects what this session needs, and recalls the rest — nothing to configure, no commands to learn.",
 	"license": "MIT",
 	"publishConfig": {

package/src/memory/diagnostics.ts

@@ -4,6 +4,10 @@ export const OUTPUT_PROTOCOL_VERSION = 2;
 export const DIAGNOSTIC_REASONS = ['empty_text', 'missing_final', 'ambiguous_final', 'unexpected_tool',
  'json_syntax', 'ambiguous_json', 'output_too_large', 'result_shape', 'too_many_claims', 'claim_shape',
  'unknown_field', 'invalid_kind', 'content_type', 'content_length', 'invalid_replaces',
+ // Output-contract violations the store finds, not the parser. They name the rule the model broke
+ // so a correction can say which one, instead of repeating the whole schema back at it.
+ 'progress_contract', 'unknown_replaces', 'duplicate_replaces', 'replaces_kind', 'cyclic_replaces',
+ 'invalid_aliases',
  'http_error', 'abnormal_stop', 'request_failed', 'legacy_import_failed'] as const;
 export type DiagnosticReason = typeof DIAGNOSTIC_REASONS[number];
 /** Structural metadata only. Never add output snippets, arbitrary keys, headers or exception messages. */

package/src/memory/limits.ts

@@ -35,9 +35,21 @@ export const MAX_SEARCH_TERM_CHARS = 64;
  * source merely restates and lower for the one it contradicts — the changed value is exactly
  * the term that is missing — so ordering by it drops the record that most needs superseding.
  * IDF weighting makes that worse, not better: the missing term is the rare one. Qualifying
- * records therefore keep the original recency order, and the cap only bounds the payload. */
+ * records therefore keep the original recency order, and the cap below takes the most recent. */
 export const RELATED_CONTAINMENT = 0.4;
-/** Unchanged from the recency-only selection this filter narrows, so nothing once shown is cut. */
+/** How many qualifying records may be sent. Applied AFTER the containment filter, never before:
+ * capping by recency first meant a scope with more than 32 records could never show an older one
+ * again, however squarely the source was about it, so it could never be superseded — only
+ * accumulated alongside. Measured on a live 89-record scope, 46 relevant records were unreachable
+ * that way, including the exact record a user correction was aimed at (rank 63, top containment).
+ *
+ * Nothing once shown is cut: a record inside the recency top-32 overall is necessarily among the
+ * 32 most recent qualifying records, so the old selection is a subset of this one.
+ *
+ * Residual, deliberately accepted: reach is still bounded by "the 32 most recent that qualify".
+ * Containment saturates on long summary sources (38 of 89 records scored 1.00), so in a busy scope
+ * the oldest still do not re-enter. Ordering stays by `updatedAt` and must not become recency of
+ * confirmation, or records would be shown because they were recently shown. */
 export const MAX_CANDIDATES = 32;
 
 /** What a reply may cost us, derived from the contract above rather than invented. These are

package/src/memory/memory-store.ts

@@ -1,6 +1,6 @@
 import { openDatabase, type Database } from "./sqlite.ts";
 import { features, mentions } from "./search.ts";
-import { MAX_CANDIDATES, RELATED_CONTAINMENT } from "./limits.ts";
+import { MAX_CANDIDATES, MAX_CLAIMS, RELATED_CONTAINMENT } from "./limits.ts";
 import { chmodSync, closeSync, lstatSync, mkdirSync, openSync } from "node:fs";
 import { join, resolve } from "node:path";
 import { randomUUID } from "node:crypto";
@@ -11,7 +11,7 @@ import { clipBytes, fingerprint, redact } from "./privacy.ts";
 import { validSearchTerms } from "./search.ts";
 import { sourceEvidence, validEvidence, validFeedback, mayReplace, FEEDBACK_VERDICTS, type Evidence, type MemoryFeedback, type FeedbackVerdict } from "./quality.ts";
 import { EVOLUTION_TIMEOUT_MS, LEASE_GRACE_MS, MAX_FAILURES, MAX_OUTPUT_FAILURES, PAUSED_SQL, FAILURE_CODES, EvolutionError, retryAt, type FailureCode } from "./recovery.ts";
-import { modelLabel, OUTPUT_PROTOCOL_VERSION, parseDiagnostic, validDiagnostic, type Diagnostic } from './diagnostics.ts';
+import { modelLabel, OUTPUT_PROTOCOL_VERSION, parseDiagnostic, validDiagnostic, type Diagnostic, type DiagnosticReason } from './diagnostics.ts';
 import { SCHEMA_VERSION, SUPPORTED_SCHEMAS } from './limits.ts';
 import { budgetUntil, reserveCall, finishCall, takeNotice, routeUntil, estimatedCost, type CallOptions } from './processing-state.ts';
 import { loadRoutingPolicy, type RoutingPolicy } from './routing-policy.ts';
@@ -314,6 +314,39 @@ export class MemoryStore {
 			.get(...(scope === undefined ? [] : [scope]), now);
 		return row ? String(row.id) : undefined;
 	}
+	/** The single definition of what a source may reason about locally and what it may be shown.
+	 * The reservation estimate and the run itself must call this same function: an estimate cheaper
+	 * than the payload it authorizes is how a call gets admitted that the provider then refuses.
+	 *
+	 * A progress source arrives with its targets already nominated, so those are its candidates.
+	 * For everything else the host drops records this source never mentions: it cannot supersede
+	 * a fact it does not talk about, and retrieval is the host's job — deterministic and free —
+	 * not something to pay a model to do by handing it every recent record to search through.
+	 *
+	 * `memories` is NOT read-only: `finishEvolution` writes through it, replacing `searchTerms`
+	 * wholesale on an exact-content match and refreshing a duplicate's evidence. Those writes were
+	 * always bounded by the recency window, and must stay bounded, or model output would rewrite
+	 * records the model was never shown — losing aliases it could not have preserved and resetting
+	 * the aging clock on records it never named. So `memories` is the recency window plus whatever
+	 * was actually shown, and nothing else: `candidates` stays a subset, and every record the host
+	 * may write through is one that was either recent or in front of the model. */
+	private selectCandidates(source: Source): { memories: DurableMemory[]; candidates: DurableMemory[] } {
+		const scoped = this.readMemories(source.scope).filter((m) => m.scope === source.scope && active(m)
+			&& (source.kind !== "progress" || (m.kind === "project_state" && source.targets!.includes(m.id))))
+			.sort((a,b) => Date.parse(b.updatedAt)-Date.parse(a.updatedAt));
+		// Order is left alone deliberately. Containment filters; it must never rank. See limits.ts.
+		// The cap is applied AFTER the filter. Capping first hid every matching record that had aged
+		// past the 32 most recent, so in any scope with more than 32 records an older one could never
+		// be shown again, and therefore never superseded — only accumulated alongside.
+		const vocabulary = source.kind === "progress" ? undefined : features(source.content);
+		const candidates = (vocabulary === undefined ? scoped
+			: scoped.filter((m) => mentions(vocabulary, m.content, m.searchTerms) >= RELATED_CONTAINMENT)).slice(0, MAX_CANDIDATES);
+		const recent = scoped.slice(0, MAX_CANDIDATES);
+		const known = new Set(recent.map((m) => m.id));
+		// Older shown records follow the recency window in age order, so this stays recency-ordered.
+		return { memories: [...recent, ...candidates.filter((m) => !known.has(m.id))], candidates };
+	}
+
 	beginEvolution(id: string, retry: RetryMode = false, timeoutMs = EVOLUTION_TIMEOUT_MS, now = Date.now(), model?: string, call?: CallOptions): EvolutionRun | undefined {
 		this.assertLearningReady();
 		return this.transaction(() => {
@@ -324,14 +357,18 @@ export class MemoryStore {
 			this.db.prepare('UPDATE sources SET last_checked=? WHERE id=?').run(now, id);
 			const priorModels = parseModels(row.call_models);
 			const correctOutput = Number(row.corrections) === 0 && Number(row.output_failures) === 1 && ['invalid_output','output_limit'].includes(String(row.last_error));
+			const source = parseSource(row.data);
+			// Selecting candidates scans the whole scope; this transaction holds the write lock, so it is
+			// computed at most once per attempt and only once a route is actually available. The estimate
+			// and the run must see the same set anyway — a cheaper estimate authorizes a larger payload.
+			let selected: { memories: DurableMemory[]; candidates: DurableMemory[] } | undefined;
+			const select = () => (selected ??= this.selectCandidates(source));
 			if (model !== undefined) {
 				model = modelLabel(model);
 				if (retry !== true && !priorModels.includes(model) && priorModels.length >= this.policy.sourceModels) return undefined;
 				const provider = modelLabel(call?.provider ?? model.split('/')[0]);
 				if (retry !== true && routeUntil(this.db, model, provider, now) > now) return undefined;
-				const source = parseSource(row.data);
-				const bytes = Buffer.byteLength(JSON.stringify(source)) + this.readMemories(source.scope)
-					.filter(active).sort((a,b) => Date.parse(b.updatedAt)-Date.parse(a.updatedAt)).slice(0,32)
+				const bytes = Buffer.byteLength(JSON.stringify(source)) + select().candidates
 					.reduce((sum,m) => sum + Buffer.byteLength(JSON.stringify(m)), 0);
 				const reserve = estimatedCost(bytes, call);
 				if (budgetUntil(this.db, model, now, this.policy, reserve) > now) return undefined;
@@ -340,19 +377,8 @@ export class MemoryStore {
 			}
 			timeoutMs = Math.min(timeoutMs, this.policy.timeoutMs, retry === true ? timeoutMs : Math.max(1, this.policy.sourceTimeMs - Number(row.call_ms)));
 			this.db.prepare(`UPDATE sources SET state='running', attempt=attempt+1, lease=? WHERE id=?`).run(now + timeoutMs + LEASE_GRACE_MS, id);
-			const source = parseSource(row.data);
 			if (source.id !== id) throw new Error("Invalid source identity");
-			const memories = this.readMemories(source.scope).filter((m) => m.scope === source.scope && active(m)
-				&& (source.kind !== "progress" || (m.kind === "project_state" && source.targets!.includes(m.id))))
-				.sort((a,b) => Date.parse(b.updatedAt)-Date.parse(a.updatedAt)).slice(0, MAX_CANDIDATES);
-			// A progress source arrives with its targets already nominated, so those are its candidates.
-			// For everything else the host drops records this source never mentions: it cannot supersede
-			// a fact it does not talk about, and retrieval is the host's job — deterministic and free —
-			// not something to pay a model to do by handing it every recent record to search through.
-			// Order is left alone deliberately. Containment filters; it must never rank. See limits.ts.
-			const vocabulary = source.kind === "progress" ? undefined : features(source.content);
-			const candidates = vocabulary === undefined ? memories
-				: memories.filter((m) => mentions(vocabulary, m.content, m.searchTerms) >= RELATED_CONTAINMENT);
+			const { memories, candidates } = select();
 			// The stored diagnostic explains the last completed outcome. Claiming an attempt must not erase it:
 			// a cancelled or interrupted run would otherwise leave a paused source with no recorded reason.
 			return { source, attempt: Number(row.attempt) + 1, generation: this.generation(source.scope), memories, candidates, timeoutMs, correctOutput,
@@ -369,9 +395,25 @@ export class MemoryStore {
 			const targets = new Set<string>();
 			let weakerConflicts = 0;
 			const incoming = sourceEvidence(run.source, "model");
+			// Two different things used to throw the same bare Error and land on write_rejected, which
+			// pauses a source for good and never even tells the model what it got wrong. They are not the
+			// same: a broken output contract is the model's mistake, correctable and worth another model;
+			// a refusal grounded in the store's own authority is not, because the same evidence will be
+			// refused again. Only the first becomes invalid_output. The second keeps write_rejected below.
+			const broke: (reason: DiagnosticReason, index: number, field?: string) => never = (reason, index, field) => {
+				// The diagnostic field path only admits the contract's own indices; anything else stays 'result'.
+				const at = index <= MAX_CLAIMS - 1 ? `memories[${index}]${field ? `.${field}` : ''}` : 'result';
+				throw new EvolutionError('invalid_output', { ...diagnostic, reason, field: at });
+			};
 			const stage = (memory: DurableMemory) => {
 				if (![...after.values()].some((m) => active(m) && fingerprint(m.content) === fingerprint(memory.content))) after.set(memory.id, memory);
 			};
+			// `claim()` below throws when the model's own output still redacts to a placeholder, meaning
+			// it echoed something credential-shaped that the source-side redaction did not catch. Do NOT
+			// wrap that in a correctable error: retrying resends the same unredacted source to another
+			// call and, since invalid_output is sibling-eligible, to another provider. One exposure then
+			// a stop is the cheap outcome; re-sending a secret to a second vendor is not. It stays a bare
+			// Error, and therefore write_rejected, deliberately.
 			const annotate = (memory: DurableMemory | undefined, claim: Claim) => {
 				if (!memory || !claim.searchTerms || memory.layer === "pinned") return;
 				const current = after.get(memory.id) ?? memory;
@@ -379,17 +421,22 @@ export class MemoryStore {
 				if (active(current) && JSON.stringify(current.searchTerms) !== JSON.stringify(searchTerms))
 					after.set(memory.id, { ...current, searchTerms, revision: memory.revision + 1 });
 			};
-			for (const claim of claims) {
-				if (!validSearchTerms(claim.searchTerms)) throw new Error("Invalid search terms");
+			for (const [index, claim] of claims.entries()) {
+				if (!validSearchTerms(claim.searchTerms)) broke('invalid_aliases', index, 'searchTerms');
 				if (run.source.kind === "progress" && (claim.kind !== "project_state" || !claim.replaces || !run.source.targets!.includes(claim.replaces)))
-					throw new Error("Progress observations may only update nominated project-state records");
+					broke('progress_contract', index);
 				if (claim.replaces) {
 					const old = run.candidates.find((m) => m.id === claim.replaces);
-					if (!old || old.scope !== run.source.scope || targets.has(old.id) || old.layer === "pinned"
-						|| (run.source.kind === "progress" && old.kind !== "project_state")
+					// The model is shown exactly the records it may name, so naming another is its own error.
+					if (!old) broke('unknown_replaces', index, 'replaces');
+					if (targets.has(old.id)) broke('duplicate_replaces', index, 'replaces');
+					if (run.source.kind === "progress" && old.kind !== "project_state") broke('replaces_kind', index, 'replaces');
+					// Authority, not shape: a pinned record, another origin's record, or one already newer
+					// than this source will refuse the same evidence however many times it is offered.
+					if (old.scope !== run.source.scope || old.layer === "pinned"
 						|| Date.parse(old.updatedAt) > Date.parse(run.source.createdAt)) throw new Error("Invalid replacement target");
 					targets.add(old.id);
-					if (claim.kind !== old.kind) throw new Error("Replacement cannot change evidence kind");
+					if (claim.kind !== old.kind) broke('replaces_kind', index, 'kind');
 					if (fingerprint(old.content) === fingerprint(claim.content)) {
 						if (run.source.kind !== "summary" && mayReplace(old, incoming)) after.set(old.id, { ...old, sourceEntryId: run.source.id,
 							updatedAt: run.source.createdAt, evidence: incoming, status: "provisional", revision: old.revision + 1 });
@@ -399,7 +446,7 @@ export class MemoryStore {
 					// Local extraction may already have added the replacement from this source.
 					const existing = run.memories.find((m) => m.id !== old.id && m.kind === claim.kind && fingerprint(m.content) === fingerprint(claim.content));
 					if ((!next && !existing) || (existing && !active(after.get(existing.id) ?? existing))) continue;
-					if (existing && claims.some((c) => c.replaces === existing.id)) throw new Error("Cyclic memory replacement");
+					if (existing && claims.some((c) => c.replaces === existing.id)) broke('cyclic_replaces', index, 'replaces');
 					if (!mayReplace(old, incoming)) {
 						// Quarantine only this source's weaker variant; preserve stronger evidence.
 						const weaker = next ?? (existing?.sourceEntryId === run.source.id ? existing : undefined);