pi-observational-memory 2.1.3 → 2.4.0

package/README.md

@@ -22,31 +22,34 @@ What the agent sees after compaction looks like this:
 
 ```
 ## Reflections
-User works at Acme Corp building Acme Dashboard on Next.js 15 with Supabase auth.
-Hard constraint: ship by January 22nd 2026.
-Public API uses GraphQL (switched from REST to reduce mobile over-fetching).
+[a1b2c3d4e5f6] User works at Acme Corp building Acme Dashboard on Next.js 15 with Supabase auth.
+[b2c3d4e5f6a1] Hard constraint: ship by January 22nd 2026.
+[c3d4e5f6a1b2] Public API uses GraphQL (switched from REST to reduce mobile over-fetching).
 
 ## Observations
-2026-01-15 14:30 [high] User decided to switch from REST to GraphQL for the public API; motivation was reducing over-fetching on mobile clients.
-2026-01-15 14:35 [medium] Agent scaffolded GraphQL schema in src/schema.ts.
-2026-01-15 14:50 [medium] GraphQL migration completed; user confirmed queries working.
-2026-01-15 15:10 [critical] User wants rate limiting on all public endpoints; prefers token bucket algorithm at 100 req/min per API key.
+[d4e5f6a1b2c3] 2026-01-15 14:30 [high] User decided to switch from REST to GraphQL for the public API; motivation was reducing over-fetching on mobile clients.
+[e5f6a1b2c3d4] 2026-01-15 14:35 [medium] Agent scaffolded GraphQL schema in src/schema.ts.
+[f6a1b2c3d4e5] 2026-01-15 14:50 [medium] GraphQL migration completed; user confirmed queries working.
+[a6b1c2d3e4f5] 2026-01-15 15:10 [critical] User wants rate limiting on all public endpoints; prefers token bucket algorithm at 100 req/min per API key.
 ```
 
 Two layers of memory, two different jobs:
 
-- **Reflections** are durable patterns — who you are, what you've decided, hard constraints. Plain prose, no timestamps. They crystallize once and persist across every future compaction.
-- **Observations** are timestamped events with a per-entry relevance tier (`low` / `medium` / `high` / `critical`). They're written near-real-time, then pruned over time — but never paraphrased.
+- **Reflections** are durable patterns — who you are, what you've decided, hard constraints. They render as plain prose with an id handle when recallable, and persist across future compactions.
+- **Observations** are timestamped events with an id and a per-entry relevance tier (`low` / `medium` / `high` / `critical`). They're written near-real-time, then pruned over time — but never paraphrased.
+
+Those ids are not decoration. When the agent needs exact evidence behind a compacted memory item, it can call the agent-facing `recall` tool with a reflection or observation id. The TUI shows a compact evidence summary, while the agent receives the full raw source context that produced the memory.
 
 Hour six should feel like hour one. The agent knows who you are, what you've built together, and what's left to do.
 
 ## What you actually get from it
 
-- **Continuity that survives many compactions.** The summary is built by mechanical concatenation, not an LLM rewrite. What survives one compaction survives all of them, byte-identical — there's no compounding drift across cycles.
+- **Continuity across many compactions.** The summary is built by mechanical concatenation, not an LLM rewrite. Kept observations and reflections are carried forward without paraphrase; observations may be pruned later, but they are never rewritten into summary-of-summary drift.
 - **Temporal reasoning.** Every observation carries a per-minute timestamp. The agent can reason about *when* something happened, not just *that* it happened.
+- **Source-backed recall.** Observation and reflection ids let the agent recover the exact prior conversation/tool evidence behind compacted memory when precision matters.
 - **Relevance-aware pruning.** Four relevance tiers drive what gets dropped first when the observation pool grows. Trivia goes; user assertions, decisions, and verbatim errors stay.
 - **Reflections that crystallize.** Identity, constraints, and durable preferences settle into a separate layer that doesn't get re-paraphrased on each compaction.
-- **Predictable token cost.** Properly configured for your use case, this can save real money. The reflector + pruner only run above a configurable gate, so most compactions cost **zero LLM calls** — just bookkeeping. The observer can be pointed at a cheap fast model independently of your main coding model.
+- **Predictable token cost.** Properly configured for your use case, this can save real money. The reflector + pruner only run above a configurable gate, so below-gate compactions skip those LLM calls; they only call a model if sync catch-up observation is needed. The observer can be pointed at a cheap fast model independently of your main coding model.
 - **Cache-friendly by design.** Memory updates are batched at compaction boundaries instead of injected into every turn, so prompt prefix caching keeps working between compactions.
 - **Fewer mid-work surprises.** The extension proactively triggers compaction when the agent is idle, and this will not affect your current work as after compaction you still keep the tail of your session intact.
 
@@ -73,19 +76,19 @@ flowchart TD
     Conv([Conversation accumulates])
     Obs[Observer<br/>async, fire-and-forget<br/>compresses each chunk into timestamped,<br/>relevance-tagged observations<br/>stored as silent tree entries]
     Comp[Compaction<br/>extension-owned; merges accumulated<br/>observations with prior compaction state]
-    RP[Reflector + Pruner<br/>Reflector appends new reflections<br/>crystallized from the pool<br/>Pruner drops observations by id<br/>across up to 5 passes]
-    Sum[Summary mechanically assembled<br/>## Reflections<br/>&nbsp;&nbsp;plain prose lines<br/>## Observations<br/>&nbsp;&nbsp;YYYY-MM-DD HH:MM relevance ...<br/>Becomes the compactionSummary<br/>the agent sees on the next turn]
+    RP[Reflector + Pruner<br/>Reflector runs focused passes<br/>to crystallize durable patterns<br/>Pruner drops observations by id<br/>across up to 5 passes]
+    Sum[Summary mechanically assembled<br/>## Reflections<br/>&nbsp;&nbsp;[id] durable insight<br/>## Observations<br/>&nbsp;&nbsp;[id] YYYY-MM-DD HH:MM relevance ...<br/>Becomes the compactionSummary<br/>the agent sees on the next turn]
 
     Conv -->|every ~1k raw tokens since last bound| Obs
-    Obs -->|every ~50k raw tokens since last compaction| Comp
+    Obs -->|live tail reaches ~50k raw tokens| Comp
     Comp -->|observation pool ≥ 30k tokens| RP
     RP --> Sum
-    Comp -.->|pool below gate — skip LLM calls| Sum
+    Comp -.->|pool below gate — skip reflector/pruner| Sum
 ```
 
 - **Observer** runs in the background as turns complete. The user never waits on it.
 - **Compaction** is owned by the extension. The summary is *mechanically concatenated* from current reflections + current observations — never an LLM rewrite. This is what eliminates the summary-of-a-summary problem.
-- **Reflector + Pruner** run as an inseparable pair, and only when there's enough material to crystallize. Below the gate, compaction does **zero LLM calls**.
+- **Reflector + Pruner** run as an inseparable pair, and only when there's enough material to crystallize. Below the gate, those roles are skipped; compaction only calls a model if sync catch-up observation is needed for uncovered raw history.
 
 The agent only ever sees the most recent compaction summary, packaged as a normal `compactionSummary` message. Observations and reflections are never injected into the live message stream — that would invalidate prefix caching with every observation. By batching memory updates at compaction boundaries, the prefix stays stable between compactions and prefix caching keeps working.
 
@@ -104,7 +107,8 @@ Settings live in Pi's `settings.json` — globally at `~/.pi/agent/settings.json
   "observational-memory": {
     "observationThresholdTokens": 1000,
     "compactionThresholdTokens": 50000,
-    "reflectionThresholdTokens": 30000
+    "reflectionThresholdTokens": 30000,
+    "passive": false
   },
   "compaction": {
     "keepRecentTokens": 20000
@@ -112,26 +116,40 @@ Settings live in Pi's `settings.json` — globally at `~/.pi/agent/settings.json
 }
 ```
 
-The five settings most worth knowing:
+To run the background memory work (observer, reflector, pruner) on a cheaper / faster model than your main coding agent — often the single biggest cost lever the extension exposes — add `compactionModel`:
+
+```json
+{
+  "observational-memory": {
+    "compactionModel": { "provider": "openrouter", "id": "google/gemma-4-31b-it" }
+  }
+}
+```
+
+The six settings most worth knowing:
 
 | Setting | Default | What it controls |
 |---|---|---|
 | `observationThresholdTokens` | `1,000` | How often the observer fires in the background |
 | `compactionThresholdTokens` | `50,000` | How often the extension proactively triggers compaction |
 | `reflectionThresholdTokens` | `30,000` | The observation pool size at which reflector + pruner engage |
+| `passive` | `false` | Disables proactive observation and extension-triggered compaction while keeping manual/Pi compaction and commands active |
 | `compactionModel` | session model | Which model runs the observer / reflector / pruner — point at a cheaper one to save cost |
 | `compaction.keepRecentTokens` | `20,000` | How much recent conversation Pi keeps verbatim post-compaction (Pi setting; structural to the extension) |
 
+For shell/session-level control, `PI_OBSERVATIONAL_MEMORY_PASSIVE` overrides global and project settings. Use `1`, `true`, `yes`, or `on` to enable passive mode; use `0`, `false`, `no`, or `off` to force it off.
+
 For the full list and tuning recipes, see **[docs/configuration.md](docs/configuration.md)**.
 
 > **Upgrading from `pi-observational-memory@1.x`?** The config keys changed: v1's `observationThreshold` is now `compactionThresholdTokens`, v1's `reflectionThreshold` is now `reflectionThresholdTokens`, and `observationThresholdTokens` is new. Old v1 keys are silently ignored — update your `settings.json`.
 
-## Commands
+## Commands and agent tool
 
-| Command | What it does |
+| Surface | What it does |
 |---|---|
-| `/om-status` | Memory totals, percent-to-threshold for each gate, and in-flight flags for observer and compaction |
-| `/om-view` | Full dump of memory state: every reflection, every committed observation, every pending observation. Each observation line is `[id] YYYY-MM-DD HH:MM [relevance] content` |
+| `/om-status` | Memory totals, percent-to-threshold for each gate, passive-mode status, and in-flight flags for observer and compaction |
+| `/om-view` | Full dump of memory state: every reflection, every committed observation, every pending observation. Reflection and observation ids shown here can be used with `recall` |
+| `recall` agent tool | Recovers exact source evidence for a specific reflection or observation id on the current branch. It is for agent self-recovery and provenance, not a user command, semantic search, or transcript browser |
 
 ## Credits
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-observational-memory",
-  "version": "2.1.3",
+  "version": "2.4.0",
   "description": "Observational memory extension for pi — cache-friendly tiered compaction with observations and reflections.",
   "type": "module",
   "license": "MIT",
@@ -30,18 +30,22 @@
     "README.md"
   ],
   "scripts": {
-    "typecheck": "tsc --noEmit"
+    "typecheck": "tsc --noEmit",
+    "test": "vitest run"
   },
   "peerDependencies": {
-    "@mariozechner/pi-coding-agent": "*",
+    "@mariozechner/pi-agent-core": "*",
     "@mariozechner/pi-ai": "*",
-    "@mariozechner/pi-agent-core": "*"
+    "@mariozechner/pi-coding-agent": "*",
+    "@mariozechner/pi-tui": "*"
   },
   "devDependencies": {
-    "@mariozechner/pi-ai": "^0.66.1",
     "@mariozechner/pi-agent-core": "^0.66.1",
+    "@mariozechner/pi-ai": "^0.66.1",
     "@mariozechner/pi-coding-agent": "^0.66.1",
+    "@mariozechner/pi-tui": "^0.66.1",
     "@types/node": "^22.0.0",
-    "typescript": "^5.6.0"
+    "typescript": "^5.6.0",
+    "vitest": "^4.1.5"
   }
 }

package/src/branch.ts

@@ -1,15 +1,17 @@
 import {
 	OBSERVATION_CUSTOM_TYPE,
-	isMemoryDetails,
 	isObservationEntryData,
-	type MemoryDetails,
+	isReflectionRecord,
+	isSupportedMemoryDetails,
+	type MemoryReflection,
 	type ObservationEntryData,
 	type ObservationRecord,
-	type Reflection,
+	type ReflectionRecord,
+	type SupportedMemoryDetails,
 } from "./types.js";
 import { estimateEntryTokens } from "./tokens.js";
 
-type Entry = {
+export type Entry = {
 	type: string;
 	id: string;
 	timestamp?: string;
@@ -25,10 +27,130 @@ type Entry = {
 
 const RAW_TYPES = new Set(["message", "custom_message", "branch_summary"]);
 
+export function isSourceEntry(entry: Entry): boolean {
+	return RAW_TYPES.has(entry.type);
+}
+
 function isObservationEntry(entry: Entry): boolean {
 	return entry.type === "custom" && entry.customType === OBSERVATION_CUSTOM_TYPE;
 }
 
+export type RecallObservationMatch =
+	| {
+			status: "ok";
+			observation: ObservationRecord;
+			observationEntryId: string;
+			sourceEntryIds: string[];
+			sourceEntries: Entry[];
+		}
+	| {
+			status: "no_source";
+			observation: ObservationRecord;
+			observationEntryId: string;
+		}
+	| {
+			status: "source_unavailable";
+			observation: ObservationRecord;
+			observationEntryId: string;
+			sourceEntryIds: string[];
+			missingSourceEntryIds: string[];
+			nonSourceEntryIds: string[];
+		};
+
+export type RecallObservationSourcesResult =
+	| {
+			status: "not_found";
+			observationId: string;
+			matches: [];
+			collision: false;
+		}
+	| {
+			status: "found";
+			observationId: string;
+			matches: RecallObservationMatch[];
+			collision: boolean;
+		};
+
+export type RecallMemoryObservation =
+	| {
+			status: "ok";
+			observation: ObservationRecord;
+			observationEntryId: string;
+			observationRecordIndex: number;
+			sourceEntryIds: string[];
+			sourceEntries: Entry[];
+		}
+	| {
+			status: "no_source";
+			observation: ObservationRecord;
+			observationEntryId: string;
+			observationRecordIndex: number;
+		}
+	| {
+			status: "source_unavailable";
+			observation: ObservationRecord;
+			observationEntryId: string;
+			observationRecordIndex: number;
+			sourceEntryIds: string[];
+			sourceEntries: Entry[];
+			missingSourceEntryIds: string[];
+			nonSourceEntryIds: string[];
+		};
+
+export type RecallMemoryReflectionMatch = {
+	reflection: ReflectionRecord;
+	reflectionIndex: number;
+};
+
+export type RecallUnavailableSupportingObservation = {
+	reflection: ReflectionRecord;
+	reflectionIndex: number;
+	observationId: string;
+};
+
+export type RecallUnavailableReflectionProvenance = {
+	reflection: ReflectionRecord;
+	reflectionIndex: number;
+	reason: "legacy";
+};
+
+export type RecallMemorySourcesResult =
+	| {
+			status: "not_found";
+			memoryId: string;
+			reflectionMatches: [];
+			directObservationMatches: [];
+			observations: [];
+			sourceEntries: [];
+			unavailableSupportingObservations: [];
+			unavailableReflectionProvenance: [];
+			missingSourceEntryIds: [];
+			nonSourceEntryIds: [];
+			collision: false;
+			partial: false;
+		}
+	| {
+			status: "found";
+			memoryId: string;
+			reflectionMatches: RecallMemoryReflectionMatch[];
+			directObservationMatches: RecallMemoryObservation[];
+			observations: RecallMemoryObservation[];
+			sourceEntries: Entry[];
+			unavailableSupportingObservations: RecallUnavailableSupportingObservation[];
+			unavailableReflectionProvenance: RecallUnavailableReflectionProvenance[];
+			missingSourceEntryIds: string[];
+			nonSourceEntryIds: string[];
+			collision: boolean;
+			partial: boolean;
+		};
+
+type IndexedObservation = {
+	observation: ObservationRecord;
+	observationEntryId: string;
+	observationRecordIndex: number;
+	branchIndex: number;
+};
+
 export function findLastCompactionIndex(entries: Entry[]): number {
 	for (let i = entries.length - 1; i >= 0; i--) {
 		if (entries[i].type === "compaction") return i;
@@ -103,16 +225,281 @@ export function rawTailEntriesBetween(entries: Entry[], fromId: string, untilId:
 
 	const result: Entry[] = [];
 	for (let i = fromIdx; i <= untilIdx; i++) {
-		if (RAW_TYPES.has(entries[i].type)) result.push(entries[i]);
+		if (isSourceEntry(entries[i])) result.push(entries[i]);
 	}
 	return result;
 }
 
-function getPriorMemoryDetails(entries: Entry[]): MemoryDetails | undefined {
+function uniqueIds(ids: string[]): string[] {
+	return Array.from(new Set(ids));
+}
+
+function collectIndexedObservations(entries: Entry[]): IndexedObservation[] {
+	const observations: IndexedObservation[] = [];
+	for (let branchIndex = 0; branchIndex < entries.length; branchIndex++) {
+		const entry = entries[branchIndex];
+		if (!isObservationEntry(entry)) continue;
+		if (!isObservationEntryData(entry.data)) continue;
+		entry.data.records.forEach((observation, observationRecordIndex) => {
+			observations.push({ observation, observationEntryId: entry.id, observationRecordIndex, branchIndex });
+		});
+	}
+	return observations;
+}
+
+function observationKey(observation: Pick<IndexedObservation, "observationEntryId" | "observationRecordIndex">): string {
+	return `${observation.observationEntryId}:${observation.observationRecordIndex}`;
+}
+
+function resolveSourceEntries(entries: Entry[], sourceEntryIds: string[]): {
+	status: "ok" | "source_unavailable";
+	sourceEntryIds: string[];
+	sourceEntries: Entry[];
+	missingSourceEntryIds: string[];
+	nonSourceEntryIds: string[];
+} {
+	const requested = uniqueIds(sourceEntryIds);
+	const requestedSet = new Set(requested);
+	const entriesById = new Map(entries.map((entry) => [entry.id, entry]));
+	const missingSourceEntryIds = requested.filter((id) => !entriesById.has(id));
+	const nonSourceEntryIds = requested.filter((id) => {
+		const entry = entriesById.get(id);
+		return entry !== undefined && !isSourceEntry(entry);
+	});
+	if (missingSourceEntryIds.length > 0 || nonSourceEntryIds.length > 0) {
+		return {
+			status: "source_unavailable",
+			sourceEntryIds: requested,
+			sourceEntries: [],
+			missingSourceEntryIds,
+			nonSourceEntryIds,
+		};
+	}
+
+	const sourceEntries = entries.filter((entry) => requestedSet.has(entry.id));
+	return {
+		status: "ok",
+		sourceEntryIds: sourceEntries.map((entry) => entry.id),
+		sourceEntries,
+		missingSourceEntryIds: [],
+		nonSourceEntryIds: [],
+	};
+}
+
+function resolveSourceEntriesPartial(entries: Entry[], sourceEntryIds: string[]): {
+	status: "ok" | "source_unavailable";
+	sourceEntryIds: string[];
+	sourceEntries: Entry[];
+	missingSourceEntryIds: string[];
+	nonSourceEntryIds: string[];
+} {
+	const requested = uniqueIds(sourceEntryIds);
+	const requestedSet = new Set(requested);
+	const entriesById = new Map(entries.map((entry) => [entry.id, entry]));
+	const missingSourceEntryIds = requested.filter((id) => !entriesById.has(id));
+	const nonSourceEntryIds = requested.filter((id) => {
+		const entry = entriesById.get(id);
+		return entry !== undefined && !isSourceEntry(entry);
+	});
+	const sourceEntries = entries.filter((entry) => requestedSet.has(entry.id) && isSourceEntry(entry));
+	return {
+		status: missingSourceEntryIds.length > 0 || nonSourceEntryIds.length > 0 ? "source_unavailable" : "ok",
+		sourceEntryIds: requested,
+		sourceEntries,
+		missingSourceEntryIds,
+		nonSourceEntryIds,
+	};
+}
+
+function memoryObservationFromIndexed(entries: Entry[], indexed: IndexedObservation): RecallMemoryObservation {
+	const { observation, observationEntryId, observationRecordIndex } = indexed;
+	if (!observation.sourceEntryIds || observation.sourceEntryIds.length === 0) {
+		return { status: "no_source", observation, observationEntryId, observationRecordIndex };
+	}
+	const resolved = resolveSourceEntriesPartial(entries, observation.sourceEntryIds);
+	if (resolved.status === "source_unavailable") {
+		return {
+			status: "source_unavailable",
+			observation,
+			observationEntryId,
+			observationRecordIndex,
+			sourceEntryIds: resolved.sourceEntryIds,
+			sourceEntries: resolved.sourceEntries,
+			missingSourceEntryIds: resolved.missingSourceEntryIds,
+			nonSourceEntryIds: resolved.nonSourceEntryIds,
+		};
+	}
+	return {
+		status: "ok",
+		observation,
+		observationEntryId,
+		observationRecordIndex,
+		sourceEntryIds: resolved.sourceEntryIds,
+		sourceEntries: resolved.sourceEntries,
+	};
+}
+
+function uniqueSourceEntriesInBranchOrder(entries: Entry[], observations: RecallMemoryObservation[]): Entry[] {
+	const requested = new Set<string>();
+	for (const observation of observations) {
+		if (observation.status === "ok" || observation.status === "source_unavailable") {
+			for (const entry of observation.sourceEntries) requested.add(entry.id);
+		}
+	}
+	return entries.filter((entry) => requested.has(entry.id) && isSourceEntry(entry));
+}
+
+function uniqueUnavailableSourceIds(
+	observations: RecallMemoryObservation[],
+	field: "missingSourceEntryIds" | "nonSourceEntryIds",
+): string[] {
+	const ids: string[] = [];
+	for (const observation of observations) {
+		if (observation.status !== "source_unavailable") continue;
+		ids.push(...observation[field]);
+	}
+	return uniqueIds(ids);
+}
+
+export function recallObservationSources(entries: Entry[], observationId: string): RecallObservationSourcesResult {
+	const matches: RecallObservationMatch[] = [];
+	for (const entry of entries) {
+		if (!isObservationEntry(entry)) continue;
+		if (!isObservationEntryData(entry.data)) continue;
+		for (const observation of entry.data.records) {
+			if (observation.id !== observationId) continue;
+			if (!observation.sourceEntryIds || observation.sourceEntryIds.length === 0) {
+				matches.push({ status: "no_source", observation, observationEntryId: entry.id });
+				continue;
+			}
+
+			const resolved = resolveSourceEntries(entries, observation.sourceEntryIds);
+			if (resolved.status === "source_unavailable") {
+				matches.push({
+					status: "source_unavailable",
+					observation,
+					observationEntryId: entry.id,
+					sourceEntryIds: resolved.sourceEntryIds,
+					missingSourceEntryIds: resolved.missingSourceEntryIds,
+					nonSourceEntryIds: resolved.nonSourceEntryIds,
+				});
+				continue;
+			}
+
+			matches.push({
+				status: "ok",
+				observation,
+				observationEntryId: entry.id,
+				sourceEntryIds: resolved.sourceEntryIds,
+				sourceEntries: resolved.sourceEntries,
+			});
+		}
+	}
+
+	if (matches.length === 0) return { status: "not_found", observationId, matches: [], collision: false };
+	return { status: "found", observationId, matches, collision: matches.length > 1 };
+}
+
+function getPriorMemoryDetails(entries: Entry[]): SupportedMemoryDetails | undefined {
 	const idx = findLastCompactionIndex(entries);
 	if (idx === -1) return undefined;
 	const details = entries[idx].details;
-	return isMemoryDetails(details) ? details : undefined;
+	return isSupportedMemoryDetails(details) ? details : undefined;
+}
+
+export function recallMemorySources(entries: Entry[], memoryId: string): RecallMemorySourcesResult {
+	const indexedObservations = collectIndexedObservations(entries);
+	const observationsById = new Map<string, IndexedObservation[]>();
+	for (const observation of indexedObservations) {
+		const matches = observationsById.get(observation.observation.id) ?? [];
+		matches.push(observation);
+		observationsById.set(observation.observation.id, matches);
+	}
+
+	const priorDetails = getPriorMemoryDetails(entries);
+	const reflectionMatches: RecallMemoryReflectionMatch[] = [];
+	if (priorDetails) {
+		priorDetails.reflections.forEach((reflection, reflectionIndex) => {
+			if (!isReflectionRecord(reflection) || reflection.id !== memoryId) return;
+			reflectionMatches.push({ reflection, reflectionIndex });
+		});
+	}
+
+	const directIndexedObservations = observationsById.get(memoryId) ?? [];
+	const observationsByKey = new Map<string, IndexedObservation>();
+	for (const observation of directIndexedObservations) {
+		observationsByKey.set(observationKey(observation), observation);
+	}
+
+	const unavailableSupportingObservations: RecallUnavailableSupportingObservation[] = [];
+	const unavailableReflectionProvenance: RecallUnavailableReflectionProvenance[] = [];
+	for (const reflectionMatch of reflectionMatches) {
+		if (reflectionMatch.reflection.legacy === true) {
+			unavailableReflectionProvenance.push({ ...reflectionMatch, reason: "legacy" });
+			continue;
+		}
+		for (const observationId of reflectionMatch.reflection.supportingObservationIds) {
+			const supportingObservations = observationsById.get(observationId);
+			if (!supportingObservations || supportingObservations.length === 0) {
+				unavailableSupportingObservations.push({ ...reflectionMatch, observationId });
+				continue;
+			}
+			for (const observation of supportingObservations) {
+				observationsByKey.set(observationKey(observation), observation);
+			}
+		}
+	}
+
+	const indexedObservationBag = Array.from(observationsByKey.values()).sort((a, b) => {
+		if (a.branchIndex !== b.branchIndex) return a.branchIndex - b.branchIndex;
+		return a.observationRecordIndex - b.observationRecordIndex;
+	});
+	const observations = indexedObservationBag.map((observation) => memoryObservationFromIndexed(entries, observation));
+	const directObservationKeys = new Set(directIndexedObservations.map(observationKey));
+	const directObservationMatches = observations.filter((observation) => directObservationKeys.has(observationKey(observation)));
+	const sourceEntries = uniqueSourceEntriesInBranchOrder(entries, observations);
+	const missingSourceEntryIds = uniqueUnavailableSourceIds(observations, "missingSourceEntryIds");
+	const nonSourceEntryIds = uniqueUnavailableSourceIds(observations, "nonSourceEntryIds");
+
+	if (reflectionMatches.length === 0 && directObservationMatches.length === 0) {
+		return {
+			status: "not_found",
+			memoryId,
+			reflectionMatches: [],
+			directObservationMatches: [],
+			observations: [],
+			sourceEntries: [],
+			unavailableSupportingObservations: [],
+			unavailableReflectionProvenance: [],
+			missingSourceEntryIds: [],
+			nonSourceEntryIds: [],
+			collision: false,
+			partial: false,
+		};
+	}
+
+	const hasNoSourceObservationInMemoryRecall =
+		reflectionMatches.length > 0 && observations.some((observation) => observation.status === "no_source");
+
+	return {
+		status: "found",
+		memoryId,
+		reflectionMatches,
+		directObservationMatches,
+		observations,
+		sourceEntries,
+		unavailableSupportingObservations,
+		unavailableReflectionProvenance,
+		missingSourceEntryIds,
+		nonSourceEntryIds,
+		collision: reflectionMatches.length + directObservationMatches.length > 1,
+		partial:
+			unavailableReflectionProvenance.length > 0 ||
+			hasNoSourceObservationInMemoryRecall ||
+			unavailableSupportingObservations.length > 0 ||
+			missingSourceEntryIds.length > 0 ||
+			nonSourceEntryIds.length > 0,
+	};
 }
 
 export function collectObservationsByCoverage(
@@ -174,7 +561,7 @@ function collectObservationsPendingNextCompaction(entries: Entry[]): Observation
 }
 
 export interface MemoryState {
-	reflections: Reflection[];
+	reflections: MemoryReflection[];
 	committedObs: ObservationRecord[];
 	pendingObs: ObservationRecord[];
 }