pi-observational-memory 3.0.0 → 3.0.1

package/README.md

@@ -87,7 +87,7 @@ Examples:
 * the project prefers minimal abstractions over framework-heavy patterns
 * the branch is about improving long-session agent memory
 
-Reflections help the agent stay oriented over time.
+Reflections help the agent stay oriented over time. The reflector treats coverage as stewardship: every active observation it reviews includes a `none`, `partial`, or `strong` coverage tier, but those tiers are review context rather than quotas. When the reflector emits a durable reflection, its support ids should cover all and only the observations whose durable meaning is actually preserved, because those ids later become dropper coverage evidence.
 
 Together, observations and reflections let Pi carry the important parts of the session forward without depending on fragile summary chains.
 
@@ -229,14 +229,14 @@ Most users can start with the defaults and tune only if they have a specific rea
 | Setting                     | Default       | Meaning                                                                                           |
 | --------------------------- | ------------- | ------------------------------------------------------------------------------------------------- |
 | `observeAfterTokens`        | `10000`       | Raw/source token threshold for observation runs.                                                  |
-| `reflectAfterTokens`        | `20000`       | Raw/source token threshold for reflection and memory maintenance.                                 |
+| `reflectAfterTokens`        | `20000`       | Raw/source token threshold for reflection runs; successful reflection creates dropper opportunities. |
 | `compactAfterTokens`        | `81000`       | Raw/source token threshold for proactive auto-compaction.                                         |
 | `observationsPoolMaxTokens` | `20000`       | Observation-token budget used for compaction full-fold pressure.                                  |
-| `observationsPoolTargetTokens` | half of max | Active-ledger observation target used by post-reflection dropper maintenance.                      |
+| `observationsPoolTargetTokens` | half of max | Active observation target used by post-reflection dropper maintenance.                            |
 | `agentMaxTurns`             | `16`          | Shared turn cap for background memory-agent loops.                                                |
 | `model`                     | session model | Optional memory-worker model override: `{ provider, id, thinking }`.                              |
 | `passive`                   | `false`       | Disables proactive background observation, reflection, maintenance, and auto-compaction triggers. |
-| `debugLog`                  | `false`       | Writes extension debug events to Pi's agent directory.                                            |
+| `debugLog`                  | `false`       | Writes opt-in per-session extension debug events to Pi's agent directory.                         |
 
 Valid `model.thinking` values are:
 
@@ -249,6 +249,12 @@ Valid `model.thinking` values are:
 
 If no `model` is configured, memory workers use the session model.
 
+`observationsPoolMaxTokens` and `observationsPoolTargetTokens` intentionally describe different pools. Max tokens control when compaction performs a full fold over visible memory. Target tokens control the folded active observation pool that the dropper maintains after successful reflection. If the target is omitted, it defaults to half of max.
+
+Dropper pruning balances age, relevance, and reflection coverage. Relevance is importance/resistance, not a permanent active-memory pin: `critical` observations require the strongest evidence but can be dropped when they are older and safely represented by reflections, superseded by newer memory, redundant, or obsolete. Dropper input annotates each active observation with deterministic coverage evidence: `none`, `partial`, or `strong`; coverage guides model judgment and is not an automatic drop rule. Dropping removes observations from active memory, not ledger history.
+
+When `debugLog` is enabled, debug events are written as local NDJSON files under Pi's agent directory. Normal sessions write to `observational-memory/debug/<session-id>.ndjson`; contexts without a session id fall back to `observational-memory/debug.ndjson`. Debug rows include `sessionId` and per-consolidation `runId`, so a session file can still be filtered to one observer/reflector/dropper run.
+
 For details and tuning guidance, see [`docs/configuration.md`](docs/configuration.md).
 
 ---
@@ -257,7 +263,7 @@ For details and tuning guidance, see [`docs/configuration.md`](docs/configuratio
 
 | Surface             | What it does                                                                                                                                    |
 | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
-| `/om-status`        | Shows memory counts, plain `+N` / `-N` visible/full drift suffixes, progress clocks, visible and active-ledger memory pressure, passive/in-flight state, and last worker errors. |
+| `/om-status`        | Shows memory counts, plain `+N` / `-N` visible/full drift suffixes, progress clocks, visible and active observation pool pressure, passive/in-flight state, and last worker errors. |
 | `/om-view`          | Shows current visible memory and attempts to copy the rendered memory text to the clipboard.                                                   |
 | `/om-view full`     | Shows the full current memory state for the branch and attempts to copy the rendered memory text to the clipboard.                             |
 | `recall` agent tool | Recovers source evidence for a 12-character observation/reflection id on the current branch. It is not semantic search or a transcript browser. |
@@ -304,7 +310,7 @@ Current behavior:
 * **Fast compaction.** `session_before_compact` does not call a model or wait for background workers. It renders the current prepared memory state.
 * **Background memory work.** Observation and reflection work run from `turn_end` when their token clocks are due; dropper work runs only after successful reflection and prunes the folded active observation ledger toward `observationsPoolTargetTokens`.
 * **Source-backed recall.** Observations and reflections can be traced back through the `recall` tool.
-* **Visible/full views.** `/om-view` shows visible memory and `/om-view full` shows the full current memory state. Use `/om-status` for visible-vs-full drift and for the separate visible observation pool vs active ledger pool.
+* **Visible/full views.** `/om-view` shows visible memory and `/om-view full` shows the full current memory state. Use `/om-status` for visible-vs-full drift and for the separate visible observation pool vs active observation pool.
 * **No V2 compatibility layer.** Old V2 settings and memory entries are ignored rather than migrated.
 
 ---
@@ -325,7 +331,7 @@ What this means in practice:
 | ---------------------------- | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
 | `observationThresholdTokens` | `observeAfterTokens`                                    | Rename. Same rough role: observation cadence based on raw/source tokens.                                                                       |
 | `compactionThresholdTokens`  | `compactAfterTokens`                                    | Rename. Same rough role: proactive compaction cadence.                                                                                         |
-| `reflectionThresholdTokens`  | `reflectAfterTokens`, `observationsPoolMaxTokens`, and/or `observationsPoolTargetTokens` | Split. Use `reflectAfterTokens` for reflection scheduling, `observationsPoolMaxTokens` for compaction full-fold pressure, and `observationsPoolTargetTokens` for dropper active-ledger maintenance. |
+| `reflectionThresholdTokens`  | `reflectAfterTokens`, `observationsPoolMaxTokens`, and/or `observationsPoolTargetTokens` | Split. Use `reflectAfterTokens` for reflection scheduling, `observationsPoolMaxTokens` for compaction full-fold pressure, and `observationsPoolTargetTokens` for dropper active observation maintenance. |
 | `compactionModel`            | `model`                                                 | Move `{ provider, id }` to `model`.                                                                                                            |
 | `thinkingLevel`              | `model.thinking`                                        | Move under `model`.                                                                                                                            |
 | `observerMaxTurnsPerRun`     | `agentMaxTurns`                                         | Replace with the shared memory-agent turn cap.                                                                                                 |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-observational-memory",
-  "version": "3.0.0",
+  "version": "3.0.1",
   "description": "Observational memory extension for pi — cache-friendly tiered compaction with observations and reflections.",
   "type": "module",
   "license": "MIT",

package/src/agents/dropper/agent.ts

@@ -2,9 +2,18 @@ import { agentLoop, type AgentContext, type AgentLoopConfig, type AgentTool } fr
 import type { Message, Model, ModelThinkingLevel } from "@earendil-works/pi-ai";
 import { Type } from "@earendil-works/pi-ai";
 import type { Static } from "typebox";
+import { debugLog } from "../../debug-log.js";
 import { AGENT_LOOP_MAX_TOKENS, boundedMaxTokens } from "../../model-budget.js";
-import { observationToSummaryLine, reflectionToSummaryLine, type Observation, type Reflection } from "../../session-ledger/index.js";
+import { reflectionToSummaryLine, type Observation, type Reflection } from "../../session-ledger/index.js";
 import { DROPPER_SYSTEM } from "./prompts.js";
+import {
+	REFLECTION_COVERAGE_DROP_RANK,
+	coverageTierForObservation,
+	reflectionCoverageMap,
+	summarizeCoverageByRelevance,
+	summarizeCoverageByRelevanceForIds,
+	observationToDropperLine,
+} from "./coverage.js";
 import { observationPoolMetrics } from "./pool.js";
 export {
 	maxDropCountForPool,
@@ -12,6 +21,19 @@ export {
 	observationPoolMetrics,
 } from "./pool.js";
 export type { ObservationPoolMetrics } from "./pool.js";
+export {
+	REFLECTION_COVERAGE_TIERS,
+	coverageTierForObservation,
+	emptyCoverageSummaryByRelevance,
+	observationToDropperLine,
+	reflectionCoverageMap,
+	reflectionCoverageTierForCount,
+	reflectionSupportCounts,
+	summarizeCoverageByRelevance,
+	summarizeCoverageByRelevanceForIds,
+	summarizeCoverageTransitionsByRelevance,
+} from "./coverage.js";
+export type { CoverageSummaryByRelevance, CoverageTransitionSummaryByRelevance, ReflectionCoverageTier } from "./coverage.js";
 
 interface RunDropperArgs {
 	model: Model<any>;
@@ -44,6 +66,13 @@ function joinOrEmpty(items: string[]): string {
 	return items.length ? items.join("\n") : "(none yet)";
 }
 
+function relevanceCounts(observations: readonly Observation[]): Record<Observation["relevance"], number> {
+	return observations.reduce<Record<Observation["relevance"], number>>((counts, observation) => {
+		counts[observation.relevance]++;
+		return counts;
+	}, { low: 0, medium: 0, high: 0, critical: 0 });
+}
+
 export function normalizeDropObservationIds(
 	ids: readonly string[] | undefined,
 	observations: readonly Observation[],
@@ -55,7 +84,6 @@ export function normalizeDropObservationIds(
 	for (const id of ids) {
 		const observation = allowed.get(id);
 		if (!observation) continue;
-		if (observation.relevance === "critical") continue;
 		if (seen.has(id)) continue;
 		seen.add(id);
 		result.push(id);
@@ -63,14 +91,21 @@ export function normalizeDropObservationIds(
 	return result.length > 0 ? result : undefined;
 }
 
+function timestampRank(timestamp: string): number {
+	const parsed = Date.parse(timestamp);
+	return Number.isFinite(parsed) ? parsed : Number.POSITIVE_INFINITY;
+}
+
 export function selectDropCandidates(
 	ids: readonly string[],
 	observations: readonly Observation[],
 	maxDrops: number,
+	reflections: readonly Reflection[] = [],
 ): string[] {
 	if (maxDrops <= 0 || ids.length === 0) return [];
 
 	const byId = new Map(observations.map((observation) => [observation.id, observation]));
+	const coverageById = reflectionCoverageMap(observations, reflections);
 	const firstProposalIndex = new Map<string, number>();
 	for (let i = 0; i < ids.length; i++) {
 		const id = ids[i];
@@ -80,11 +115,14 @@ export function selectDropCandidates(
 	return Array.from(firstProposalIndex.entries())
 		.map(([id, index]) => ({ id, index, observation: byId.get(id) }))
 		.filter((candidate): candidate is { id: string; index: number; observation: Observation } =>
-			candidate.observation !== undefined && candidate.observation.relevance !== "critical"
+			candidate.observation !== undefined
 		)
 		.sort((a, b) => {
+			const coverageDelta = REFLECTION_COVERAGE_DROP_RANK[coverageTierForObservation(a.observation, coverageById)]
+				- REFLECTION_COVERAGE_DROP_RANK[coverageTierForObservation(b.observation, coverageById)];
 			const relevanceDelta = RELEVANCE_DROP_RANK[a.observation.relevance] - RELEVANCE_DROP_RANK[b.observation.relevance];
-			return relevanceDelta || a.index - b.index;
+			const ageDelta = timestampRank(a.observation.timestamp) - timestampRank(b.observation.timestamp);
+			return coverageDelta || relevanceDelta || ageDelta || a.index - b.index;
 		})
 		.slice(0, maxDrops)
 		.map((candidate) => candidate.id);
@@ -96,10 +134,42 @@ export async function runDropper(args: RunDropperArgs): Promise<string[] | undef
 
 	const metrics = observationPoolMetrics(observations, targetTokens);
 	const { observationTokens, fullness, tokensOverTarget, maxDropsAllowed } = metrics;
-	if (maxDropsAllowed <= 0) return undefined;
+	const coverageById = reflectionCoverageMap(observations, reflections);
+	const coverageSummaryByRelevance = summarizeCoverageByRelevance(observations, coverageById);
+	debugLog("dropper.agent_start", {
+		activeObservationCount: observations.length,
+		reflectionCount: reflections.length,
+		observationTokens,
+		targetTokens,
+		tokensOverTarget,
+		fullness,
+		maxDropsAllowed,
+		relevanceCounts: relevanceCounts(observations),
+		coverageSummaryByRelevance,
+	});
+	if (maxDropsAllowed <= 0) {
+		debugLog("dropper.result", {
+			reason: "not_over_target",
+			toolCallCount: 0,
+			rawRequestedIdsCount: 0,
+			acceptedCandidateCount: 0,
+			selectedDropsCount: 0,
+			selectedDropTokens: 0,
+			selectedCoverageSummaryByRelevance: summarizeCoverageByRelevanceForIds([], observations, coverageById),
+			maxDropsAllowed,
+		});
+		return undefined;
+	}
 
 	const proposedDropIds: string[] = [];
 	const proposed = new Set<string>();
+	const allowed = new Map(observations.map((observation) => [observation.id, observation]));
+	let toolCallCount = 0;
+	let rawRequestedIdsCount = 0;
+	let missingIdsCount = 0;
+	let criticalCandidateIdsCount = 0;
+	let duplicateInRequestCount = 0;
+	let duplicateInRunCount = 0;
 
 	const dropObservations: AgentTool<typeof DropObservationsSchema> = {
 		name: "drop_observations",
@@ -107,14 +177,51 @@ export async function runDropper(args: RunDropperArgs): Promise<string[] | undef
 		description: "Propose active observation ids that are safe to remove from compacted memory.",
 		parameters: DropObservationsSchema,
 		execute: async (_id, params: DropObservationsArgs) => {
-			const normalized = normalizeDropObservationIds(params.ids, observations) ?? [];
+			toolCallCount++;
+			rawRequestedIdsCount += params.ids.length;
+			const seenInRequest = new Set<string>();
 			let added = 0;
-			for (const id of normalized) {
-				if (proposed.has(id)) continue;
+			let requestMissingIds = 0;
+			let requestCriticalCandidateIds = 0;
+			let requestDuplicateIds = 0;
+			let requestDuplicateInRunIds = 0;
+			for (const id of params.ids) {
+				const observation = allowed.get(id);
+				if (!observation) {
+					missingIdsCount++;
+					requestMissingIds++;
+					continue;
+				}
+				if (seenInRequest.has(id)) {
+					duplicateInRequestCount++;
+					requestDuplicateIds++;
+					continue;
+				}
+				seenInRequest.add(id);
+				if (proposed.has(id)) {
+					duplicateInRunCount++;
+					requestDuplicateInRunIds++;
+					continue;
+				}
 				proposed.add(id);
 				proposedDropIds.push(id);
+				if (observation.relevance === "critical") {
+					criticalCandidateIdsCount++;
+					requestCriticalCandidateIds++;
+				}
 				added++;
 			}
+			debugLog("dropper.tool_call", {
+				toolCallCount,
+				rawRequestedIdsCount: params.ids.length,
+				acceptedIdsCount: added,
+				missingIdsCount: requestMissingIds,
+				criticalCandidateIdsCount: requestCriticalCandidateIds,
+				duplicateInRequestCount: requestDuplicateIds,
+				duplicateInRunCount: requestDuplicateInRunIds,
+				totalCandidates: proposedDropIds.length,
+				maxDropsAllowed,
+			});
 			return {
 				content: [{ type: "text", text: `Queued ${added} drop candidate${added === 1 ? "" : "s"}. Candidates this run: ${proposedDropIds.length}. Maximum drops allowed: ${maxDropsAllowed}.` }],
 				details: { added, totalCandidates: proposedDropIds.length, maxDropsAllowed },
@@ -123,7 +230,7 @@ export async function runDropper(args: RunDropperArgs): Promise<string[] | undef
 	};
 
 	const fullnessPercent = Math.round(fullness * 100);
-	const userText = `CURRENT REFLECTIONS:\n${joinOrEmpty(reflections.map(reflectionToSummaryLine))}\n\nCURRENT OBSERVATIONS:\n${joinOrEmpty(observations.map(observationToSummaryLine))}\n\nActive observation pool: ~${observationTokens.toLocaleString()} tokens; target: ~${targetTokens.toLocaleString()} tokens; fullness against target: ~${fullnessPercent.toLocaleString()}%; over target by ~${tokensOverTarget.toLocaleString()} tokens.\nMaximum drops allowed this run: ${maxDropsAllowed.toLocaleString()} observation${maxDropsAllowed === 1 ? "" : "s"}. This maximum is sized to move the active pool toward the target if every proposed drop is clearly safe.\nThis maximum is a hard upper bound, not a target. Drop fewer or none if fewer observations are clearly safe.`;
+	const userText = `CURRENT REFLECTIONS:\n${joinOrEmpty(reflections.map(reflectionToSummaryLine))}\n\nCURRENT OBSERVATIONS:\n${joinOrEmpty(observations.map((observation) => observationToDropperLine(observation, coverageTierForObservation(observation, coverageById))))}\n\nActive observation pool: ~${observationTokens.toLocaleString()} tokens; target: ~${targetTokens.toLocaleString()} tokens; fullness against target: ~${fullnessPercent.toLocaleString()}%; over target by ~${tokensOverTarget.toLocaleString()} tokens.\nMaximum drops allowed this run: ${maxDropsAllowed.toLocaleString()} observation${maxDropsAllowed === 1 ? "" : "s"}. This maximum is sized to move the active pool toward the target if every proposed drop is clearly safe.\nThis maximum is a hard upper bound, not a target. Drop fewer or none if fewer observations are clearly safe.`;
 	const prompts: Message[] = [{ role: "user", content: [{ type: "text", text: userText }], timestamp: Date.now() }];
 	const context: AgentContext = { systemPrompt: DROPPER_SYSTEM, messages: [], tools: [dropObservations as AgentTool<any>] };
 	const reasoning = (model as { reasoning?: unknown }).reasoning;
@@ -147,6 +254,28 @@ export async function runDropper(args: RunDropperArgs): Promise<string[] | undef
 		// Tool execution collects candidate ids.
 	}
 	await stream.result();
-	const droppedIds = selectDropCandidates(proposedDropIds, observations, maxDropsAllowed);
+	const droppedIds = selectDropCandidates(proposedDropIds, observations, maxDropsAllowed, reflections);
+	const reason = droppedIds.length > 0
+		? "selected_nonempty"
+		: toolCallCount === 0
+			? "no_tool_call"
+			: proposedDropIds.length === 0
+				? "all_filtered"
+				: "selected_empty";
+	const selectedDropTokens = droppedIds.reduce((sum, id) => sum + (allowed.get(id)?.tokenCount ?? 0), 0);
+	debugLog("dropper.result", {
+		reason,
+		toolCallCount,
+		rawRequestedIdsCount,
+		missingIdsCount,
+		criticalCandidateIdsCount,
+		duplicateInRequestCount,
+		duplicateInRunCount,
+		acceptedCandidateCount: proposedDropIds.length,
+		selectedDropsCount: droppedIds.length,
+		selectedDropTokens,
+		selectedCoverageSummaryByRelevance: summarizeCoverageByRelevanceForIds(droppedIds, observations, coverageById),
+		maxDropsAllowed,
+	});
 	return droppedIds.length > 0 ? droppedIds : undefined;
 }

package/src/agents/dropper/coverage.ts

@@ -0,0 +1,128 @@
+import type { Observation, Reflection } from "../../session-ledger/index.js";
+
+export const REFLECTION_COVERAGE_TIERS = ["none", "partial", "strong"] as const;
+export type ReflectionCoverageTier = typeof REFLECTION_COVERAGE_TIERS[number];
+
+type Relevance = Observation["relevance"];
+
+type CoverageBucket = Record<ReflectionCoverageTier, { count: number; tokens: number }>;
+export type CoverageSummaryByRelevance = Record<Relevance, CoverageBucket>;
+export type CoverageTransitionSummaryByRelevance = Record<Relevance, Record<string, { count: number; tokens: number }>>;
+
+export const REFLECTION_COVERAGE_DROP_RANK: Record<ReflectionCoverageTier, number> = {
+	strong: 0,
+	partial: 1,
+	none: 2,
+};
+
+export function reflectionSupportCounts(reflections: readonly Reflection[]): Map<string, number> {
+	const counts = new Map<string, number>();
+	for (const reflection of reflections) {
+		const uniqueIds = new Set(reflection.supportingObservationIds);
+		for (const id of uniqueIds) counts.set(id, (counts.get(id) ?? 0) + 1);
+	}
+	return counts;
+}
+
+export function reflectionCoverageTierForCount(count: number): ReflectionCoverageTier {
+	if (count <= 0) return "none";
+	if (count === 1) return "partial";
+	return "strong";
+}
+
+export function reflectionCoverageMap(
+	observations: readonly Observation[],
+	reflections: readonly Reflection[],
+): Map<string, ReflectionCoverageTier> {
+	const counts = reflectionSupportCounts(reflections);
+	return new Map(observations.map((observation) => [
+		observation.id,
+		reflectionCoverageTierForCount(counts.get(observation.id) ?? 0),
+	]));
+}
+
+function emptyCoverageBucket(): CoverageBucket {
+	return {
+		none: { count: 0, tokens: 0 },
+		partial: { count: 0, tokens: 0 },
+		strong: { count: 0, tokens: 0 },
+	};
+}
+
+export function emptyCoverageSummaryByRelevance(): CoverageSummaryByRelevance {
+	return {
+		low: emptyCoverageBucket(),
+		medium: emptyCoverageBucket(),
+		high: emptyCoverageBucket(),
+		critical: emptyCoverageBucket(),
+	};
+}
+
+export function summarizeCoverageByRelevance(
+	observations: readonly Observation[],
+	coverageById: ReadonlyMap<string, ReflectionCoverageTier>,
+): CoverageSummaryByRelevance {
+	const summary = emptyCoverageSummaryByRelevance();
+	for (const observation of observations) {
+		const tier = coverageById.get(observation.id) ?? "none";
+		const bucket = summary[observation.relevance][tier];
+		bucket.count++;
+		bucket.tokens += observation.tokenCount;
+	}
+	return summary;
+}
+
+export function summarizeCoverageByRelevanceForIds(
+	ids: readonly string[],
+	observations: readonly Observation[],
+	coverageById: ReadonlyMap<string, ReflectionCoverageTier>,
+): CoverageSummaryByRelevance {
+	const byId = new Map(observations.map((observation) => [observation.id, observation]));
+	const selected = ids.flatMap((id) => {
+		const observation = byId.get(id);
+		return observation ? [observation] : [];
+	});
+	return summarizeCoverageByRelevance(selected, coverageById);
+}
+
+export function emptyCoverageTransitionSummaryByRelevance(): CoverageTransitionSummaryByRelevance {
+	return {
+		low: {},
+		medium: {},
+		high: {},
+		critical: {},
+	};
+}
+
+export function summarizeCoverageTransitionsByRelevance(
+	observations: readonly Observation[],
+	beforeCoverageById: ReadonlyMap<string, ReflectionCoverageTier>,
+	afterCoverageById: ReadonlyMap<string, ReflectionCoverageTier>,
+): CoverageTransitionSummaryByRelevance {
+	const summary = emptyCoverageTransitionSummaryByRelevance();
+	for (const observation of observations) {
+		const before = beforeCoverageById.get(observation.id) ?? "none";
+		const after = afterCoverageById.get(observation.id) ?? "none";
+		if (before === after) continue;
+		const key = `${before}->${after}`;
+		const bucket = summary[observation.relevance][key] ?? { count: 0, tokens: 0 };
+		bucket.count++;
+		bucket.tokens += observation.tokenCount;
+		summary[observation.relevance][key] = bucket;
+	}
+	return summary;
+}
+
+export function observationToDropperLine(
+	observation: Observation,
+	coverage: ReflectionCoverageTier,
+): string {
+	return `[${observation.id}] ${observation.timestamp} [${observation.relevance}] [coverage: ${coverage}] ${observation.content}`;
+}
+
+export function coverageTierForObservation(
+	observation: Observation,
+	coverageById: ReadonlyMap<string, ReflectionCoverageTier>,
+): ReflectionCoverageTier {
+	return coverageById.get(observation.id) ?? "none";
+}

package/src/agents/dropper/pool.ts

@@ -23,7 +23,7 @@ export function observationPoolFullness(observationTokens: number, targetTokens:
 }
 
 export function droppableObservationCount(observations: readonly Observation[]): number {
-	return observations.filter((observation) => observation.relevance !== "critical").length;
+	return observations.length;
 }
 
 export function maxDropCountForPool(observations: readonly Observation[], observationTokens: number, targetTokens: number): number {

package/src/agents/dropper/prompts.ts

@@ -6,25 +6,30 @@ Your job is to identify only the safest active observations to remove from compa
 
 Active-memory framing. Dropping an observation removes it from active compacted memory; it does not erase the ledger history or source evidence. Still, future compressed context will no longer show the observation, so only drop it when its durable meaning is safely captured elsewhere or it is genuinely low-signal and carries no unique future value.
 
-The user message includes the active observation pool target and "Maximum drops allowed this run". The maximum is a hard upper bound sized to move the pool toward the target if every proposed drop is clearly safe. It is not a target. Never try to hit it. Drop fewer or none when fewer observations are safely removable.
+The user message includes the active observation pool target and "Maximum drops allowed this run". The maximum is a hard upper bound sized to move the pool toward the target if every proposed drop is clearly safe. It is not a target. Do not try to fill it. Drop fewer or none when fewer observations are safely removable. When the active pool is far over target, make a thorough pass over safe candidates rather than stopping after a few obvious examples.
 
 What to drop, in priority order:
 - Redundant observations whose durable meaning is already captured by current reflections with equivalent fidelity.
 - Superseded observations where a later observation clearly replaces the older state.
 - Repeated routine tool acknowledgements or low-signal progress updates that do not carry decisions, constraints, exact errors, or user-specific facts.
-- Older medium observations that no longer carry working context and are covered by a reflection or a newer observation.
+- Older observations that no longer carry working context and are covered by a reflection or a newer observation.
 
-Age-gradient rule. Recent observations carry working context the assistant may still need; older observations have usually been summarized elsewhere or are no longer load-bearing. Prefer older safe drops before newer working context.
+Age-gradient rule. Recent observations carry working context the assistant may still need; older observations have usually been summarized elsewhere or are no longer load-bearing. Prefer older safe drops before newer working context, but age alone is not enough to drop important or uniquely load-bearing observations.
 
-Relevance guidance:
+Reflection coverage guidance. Each observation line includes [coverage: none|partial|strong]. Coverage is evidence, not an automatic decision:
+- none: no current reflection cites this observation id. Be cautious, especially for high or critical observations.
+- partial: one current reflection cites this observation id. Compare the observation to the reflection before dropping.
+- strong: two or more current reflections cite this observation id. This is stronger evidence that the durable meaning is preserved, but you must still keep uniquely load-bearing or uncertain observations.
+
+Relevance guidance. Relevance is importance/resistance, not an absolute keep/drop lock:
 - low: consider first, but drop only when it carries no unique detail, decision, state, error, identifier, or user-specific fact.
 - medium: drop when redundant with reflections or other observations, or when the work state is clearly obsolete.
 - high: drop only when clearly superseded or already captured by a reflection with equivalent fidelity.
-- critical: NEVER drop. Code also rejects critical ids, but you must avoid proposing them.
+- critical: highest importance and strongest resistance. Do not drop fresh or uniquely load-bearing critical observations. Critical observations may be dropped only with strong semantic evidence such as age plus partial/strong reflection coverage, supersession by newer memory, redundancy, or clear obsolescence.
 
-User assertions and concrete completions are never droppable, even at non-critical relevance, unless a current reflection preserves the exact assertion/completion and its important details with equivalent fidelity.
+User assertions and concrete completions must be preserved unless a current reflection or newer observation preserves the exact assertion/completion and its important details with equivalent fidelity.
 
-Preservation floor. Regardless of relevance label, urgency, budget pressure, or age, do not drop observations that uniquely carry any of the following:
+Preservation floor. Regardless of relevance label, budget pressure, coverage, or age, do not drop observations that uniquely carry any of the following:
 - User preferences, constraints, corrections, or identity/role facts.
 - Concrete completions that future runs must not redo.
 - Named identifiers, file paths, function names, package names, tickets, commit SHAs, handles, or exact commands.

package/src/agents/observer/prompts.ts

@@ -101,7 +101,7 @@ If a detail is non-obvious from the code or git history, it belongs in the obser
 
 Relevance levels (pick one per observation; this field drives future dropping):
 
-- critical: user assertions about identity, role, or persistent preferences; explicit corrections ("no, don't do X"); concrete completions that future runs MUST NOT redo. These are load-bearing and will NEVER be dropped. Why this matters: if a "critical" item is lost, the assistant may redo finished work, contradict a correction, or misrepresent who the user is.
+- critical: user assertions about identity, role, or persistent preferences; explicit corrections ("no, don't do X"); concrete completions that future runs MUST NOT redo. These are highest-resistance, load-bearing observations and require the strongest evidence before leaving active memory. Why this matters: if a "critical" item is lost, the assistant may redo finished work, contradict a correction, or misrepresent who the user is.
 - high: non-trivial technical decisions, architectural direction, unresolved blockers, key constraints. Worth keeping across many compactions.
 - medium: task-level context that helps within the current work but isn't durable. The default when you are unsure between medium and high.
 - low: routine tool-call acks, repetitive status updates, content trivially re-derivable from recent messages. The dropper will drop these first.

package/src/agents/reflector/agent.ts

@@ -2,12 +2,20 @@ import { agentLoop, type AgentContext, type AgentLoopConfig, type AgentTool } fr
 import type { Message, Model, ModelThinkingLevel } from "@earendil-works/pi-ai";
 import { Type } from "@earendil-works/pi-ai";
 import type { Static } from "typebox";
+import { debugLog } from "../../debug-log.js";
 import { hashId } from "../../ids.js";
 import { AGENT_LOOP_MAX_TOKENS, boundedMaxTokens } from "../../model-budget.js";
 import { truncateRecordContent } from "../../serialize.js";
 import { REFLECTOR_SYSTEM } from "./prompts.js";
 import { estimateStringTokens } from "../../tokens.js";
-import { observationToSummaryLine, reflectionToSummaryLine, type Observation, type Reflection } from "../../session-ledger/index.js";
+import { reflectionToSummaryLine, type Observation, type Reflection } from "../../session-ledger/index.js";
+import {
+	coverageTierForObservation,
+	reflectionCoverageMap,
+	summarizeCoverageByRelevance,
+	summarizeCoverageTransitionsByRelevance,
+	type ReflectionCoverageTier,
+} from "../dropper/coverage.js";
 
 interface RunReflectorArgs {
 	model: Model<any>;
@@ -37,6 +45,38 @@ function joinOrEmpty(items: string[]): string {
 	return items.length ? items.join("\n") : "(none yet)";
 }
 
+export function observationToReflectorLine(
+	observation: Observation,
+	coverage: ReflectionCoverageTier,
+): string {
+	return `[${observation.id}] ${observation.timestamp} [${observation.relevance}] [coverage: ${coverage}] ${observation.content}`;
+}
+
+export function summarizeSupportIdCounts(reflections: readonly Reflection[]): {
+	reflectionCount: number;
+	totalSupportIds: number;
+	minSupportIds: number;
+	maxSupportIds: number;
+	averageSupportIds: number;
+	histogram: Record<string, number>;
+} {
+	if (reflections.length === 0) {
+		return { reflectionCount: 0, totalSupportIds: 0, minSupportIds: 0, maxSupportIds: 0, averageSupportIds: 0, histogram: {} };
+	}
+	const counts = reflections.map((reflection) => reflection.supportingObservationIds.length);
+	const totalSupportIds = counts.reduce((sum, count) => sum + count, 0);
+	const histogram: Record<string, number> = {};
+	for (const count of counts) histogram[String(count)] = (histogram[String(count)] ?? 0) + 1;
+	return {
+		reflectionCount: reflections.length,
+		totalSupportIds,
+		minSupportIds: Math.min(...counts),
+		maxSupportIds: Math.max(...counts),
+		averageSupportIds: totalSupportIds / reflections.length,
+		histogram,
+	};
+}
+
 export function normalizeSupportingObservationIds(
 	supportingObservationIds: readonly string[] | undefined,
 	allowedObservationIds: readonly string[],
@@ -66,9 +106,21 @@ export async function runReflector(args: RunReflectorArgs): Promise<Reflection[]
 	const { model, apiKey, headers, reflections, observations, signal } = args;
 	if (observations.length === 0) return undefined;
 
+	const coverageById = reflectionCoverageMap(observations, reflections);
+	debugLog("reflector.agent_start", {
+		activeObservationCount: observations.length,
+		reflectionCount: reflections.length,
+		coverageSummaryByRelevance: summarizeCoverageByRelevance(observations, coverageById),
+	});
+
 	const allowedObservationIds = observations.map((observation) => observation.id);
 	const existingReflectionIds = new Set(reflections.map((reflection) => reflection.id));
 	const accumulated = new Map<string, Reflection>();
+	let toolCallCount = 0;
+	let rawProposedReflectionCount = 0;
+	let acceptedReflectionCount = 0;
+	let duplicateReflectionCount = 0;
+	let rejectedReflectionCount = 0;
 
 	const recordReflections: AgentTool<typeof RecordReflectionsSchema> = {
 		name: "record_reflections",
@@ -76,6 +128,8 @@ export async function runReflector(args: RunReflectorArgs): Promise<Reflection[]
 		description: "Record new durable reflections with supporting observation ids.",
 		parameters: RecordReflectionsSchema,
 		execute: async (_id, params: RecordReflectionsArgs) => {
+			toolCallCount++;
+			rawProposedReflectionCount += params.reflections.length;
 			let added = 0;
 			let duplicates = 0;
 			let rejected = 0;
@@ -99,6 +153,9 @@ export async function runReflector(args: RunReflectorArgs): Promise<Reflection[]
 				});
 				added++;
 			}
+			acceptedReflectionCount += added;
+			duplicateReflectionCount += duplicates;
+			rejectedReflectionCount += rejected;
 			return {
 				content: [{ type: "text", text: `Recorded ${added} reflection${added === 1 ? "" : "s"}; ${duplicates} duplicate${duplicates === 1 ? "" : "s"}; ${rejected} rejected. Total this run: ${accumulated.size}.` }],
 				details: { added, duplicates, rejected, total: accumulated.size },
@@ -106,7 +163,7 @@ export async function runReflector(args: RunReflectorArgs): Promise<Reflection[]
 		},
 	};
 
-	const userText = `CURRENT REFLECTIONS:\n${joinOrEmpty(reflections.map(reflectionToSummaryLine))}\n\nCURRENT OBSERVATIONS:\n${joinOrEmpty(observations.map(observationToSummaryLine))}\n\nCrystallize any missing durable facts or patterns into new reflections. If nothing is stable enough, do not call the tool.`;
+	const userText = `CURRENT REFLECTIONS:\n${joinOrEmpty(reflections.map(reflectionToSummaryLine))}\n\nCURRENT OBSERVATIONS:\n${joinOrEmpty(observations.map((observation) => observationToReflectorLine(observation, coverageTierForObservation(observation, coverageById))))}\n\nCrystallize any missing durable facts or patterns into new reflections. If nothing is stable enough, do not call the tool.`;
 	const prompts: Message[] = [{ role: "user", content: [{ type: "text", text: userText }], timestamp: Date.now() }];
 	const context: AgentContext = { systemPrompt: REFLECTOR_SYSTEM, messages: [], tools: [recordReflections as AgentTool<any>] };
 	const reasoning = (model as { reasoning?: unknown }).reasoning;
@@ -130,5 +187,17 @@ export async function runReflector(args: RunReflectorArgs): Promise<Reflection[]
 		// Tool execution collects records.
 	}
 	await stream.result();
-	return accumulated.size > 0 ? Array.from(accumulated.values()) : undefined;
+	const acceptedReflections = Array.from(accumulated.values());
+	const afterCoverageById = reflectionCoverageMap(observations, [...reflections, ...acceptedReflections]);
+	debugLog("reflector.result", {
+		reason: acceptedReflections.length > 0 ? "accepted_nonempty" : toolCallCount === 0 ? "no_tool_call" : "all_filtered",
+		toolCallCount,
+		rawProposedReflectionCount,
+		acceptedReflectionCount,
+		duplicateReflectionCount,
+		rejectedReflectionCount,
+		acceptedSupportIdCounts: summarizeSupportIdCounts(acceptedReflections),
+		coverageTransitionsByRelevance: summarizeCoverageTransitionsByRelevance(observations, coverageById, afterCoverageById),
+	});
+	return acceptedReflections.length > 0 ? acceptedReflections : undefined;
 }

package/src/agents/reflector/prompts.ts

@@ -6,7 +6,8 @@ Your task is different from the observer's: you are not recording events, you ar
 
 You receive:
 - Current reflections: durable facts already crystallized.
-- Current observations: active timestamped evidence lines, each shown as "[id] YYYY-MM-DD HH:MM [relevance] content".
+- Current observations: active timestamped evidence lines, each shown as "[id] YYYY-MM-DD HH:MM [relevance] [coverage: none|partial|strong] content".
+- Coverage tiers are review context: none means no current reflection supports the observation id, partial means exactly one current reflection supports it, and strong means two or more current reflections support it. Coverage is not a quota, target, priority score, or instruction to emit reflections.
 
 What to emit:
 - Emit only new durable reflections not already present in current reflections.
@@ -39,13 +40,16 @@ Focus on:
 - Completed outcomes future runs must not redo.
 - Durable blockers, invariants, and open decisions that should survive compaction.
 
-Support ids:
+Support ids and coverage stewardship:
 - Every reflection must include supportingObservationIds from the current observations list.
-- supportingObservationIds are a coverage/provenance set: include current observation ids whose durable meaning is preserved by the reflection and can later be treated as redundant active-memory detail.
-- supportingObservationIds are not a checklist to cover every observation. They are evidence for a reflection that already passed the durable-value bar.
+- First decide whether the reflection content passes the durable-value bar. Then audit support ids for that already-worthy reflection.
+- supportingObservationIds are a coverage/provenance set and downstream dropper coverage evidence: include all current observation ids whose durable meaning is preserved by the reflection with equivalent fidelity and can later be treated as redundant active-memory detail.
+- supportingObservationIds are not a checklist to cover every observation. Do not add ids merely to improve coverage counts, maximize support ids, maximize strong coverage, or unlock the dropper.
+- False or inflated support ids can cause unsafe downstream dropper pruning, including removal of high-resistance active observations whose meaning was not actually preserved.
 - Include additional observation ids only when the reflection preserves their durable meaning with equivalent fidelity.
 - Leave observations unsupported when their details are still active working state, too specific to compress safely, or not yet durable enough.
 - Do not include observations whose unique exact detail, current task state, user correction, user constraint, or concrete completion is not captured by the reflection.
+- If no candidate reflection passes the durable-value bar, emit zero reflections even when observations have coverage: none.
 - Never invent observation ids. Proposals with missing, empty, or invalid supportingObservationIds are rejected.
 
 User assertions are authoritative. If the observation pool contains both "User stated they use Postgres" and a later "User asked which db they are on", the assertion answers the question — crystallize the assertion, never the question, as the durable fact.

package/src/debug-log.ts

@@ -5,10 +5,13 @@ import { getAgentDir } from "@earendil-works/pi-coding-agent";
 
 export const DEBUG_LOG_MAX_BYTES = 10 * 1024 * 1024;
 export const DEBUG_LOG_RELATIVE_PATH = join("observational-memory", "debug.ndjson");
+export const DEBUG_LOG_SESSION_DIR_RELATIVE_PATH = join("observational-memory", "debug");
 
-interface DebugLogContext {
+export interface DebugLogContext {
 	enabled: boolean;
 	cwd?: string;
+	sessionId?: string;
+	sessionFile?: string;
 	runId?: string;
 }
 
@@ -19,18 +22,38 @@ export function withDebugLogContext<T>(context: DebugLogContext, fn: () => T): T
 	return storage.run({ ...parent, ...context }, fn);
 }
 
+export function safeDebugLogSessionId(sessionId: string | undefined): string | undefined {
+	const trimmed = sessionId?.trim();
+	if (!trimmed) return undefined;
+	const sanitized = trimmed
+		.replace(/[^A-Za-z0-9._-]+/g, "_")
+		.replace(/^_+|_+$/g, "")
+		.slice(0, 128);
+	if (!/[A-Za-z0-9]/.test(sanitized)) return undefined;
+	return sanitized;
+}
+
+export function debugLogRelativePath(context: Pick<DebugLogContext, "sessionId">): string {
+	const safeSessionId = safeDebugLogSessionId(context.sessionId);
+	return safeSessionId
+		? join(DEBUG_LOG_SESSION_DIR_RELATIVE_PATH, `${safeSessionId}.ndjson`)
+		: DEBUG_LOG_RELATIVE_PATH;
+}
+
 export function debugLog(event: string, data: Record<string, unknown> = {}): void {
 	const context = storage.getStore();
 	if (context?.enabled !== true) return;
 
 	try {
-		const path = join(getAgentDir(), DEBUG_LOG_RELATIVE_PATH);
+		const path = join(getAgentDir(), debugLogRelativePath(context));
 		mkdirSync(dirname(path), { recursive: true });
 		rotateIfNeeded(path);
 		const payload = {
 			ts: new Date().toISOString(),
 			event,
 			cwd: context.cwd,
+			sessionId: context.sessionId,
+			sessionFile: context.sessionFile,
 			runId: context.runId,
 			data,
 		};

package/src/hooks/consolidation-trigger.ts

@@ -35,7 +35,11 @@ type ConsolidationCtx = {
 	ui?: { notify: (message: string, type?: "warning" | "info" | "error") => void };
 	model: unknown;
 	modelRegistry: any;
-	sessionManager: { getBranch: () => unknown };
+	sessionManager: {
+		getBranch: () => unknown;
+		getSessionId?: () => string;
+		getSessionFile?: () => string | undefined;
+	};
 };
 
 type StageOutcome = "continue" | "abort";
@@ -100,6 +104,17 @@ export function registerConsolidationTrigger(pi: ExtensionAPI, runtime: Runtime)
 	pi.on("turn_end", launch);
 }
 
+function debugSessionMetadata(ctx: ConsolidationCtx): { sessionId?: string; sessionFile?: string } {
+	try {
+		return {
+			sessionId: ctx.sessionManager.getSessionId?.(),
+			sessionFile: ctx.sessionManager.getSessionFile?.(),
+		};
+	} catch {
+		return {};
+	}
+}
+
 function maybeLaunchConsolidation(pi: ExtensionAPI, runtime: Runtime, ctx: ConsolidationCtx): void {
 	runtime.ensureConfig(ctx.cwd);
 	if (runtime.config.passive === true) return;
@@ -118,7 +133,13 @@ function maybeLaunchConsolidation(pi: ExtensionAPI, runtime: Runtime, ctx: Conso
 		sessionManager: ctx.sessionManager,
 	};
 
-	void runtime.launchConsolidationTask(ctx, async () => withDebugLogContext({ enabled: runtime.config.debugLog === true, cwd: ctx.cwd, runId }, async () => {
+	const sessionMetadata = debugSessionMetadata(ctx);
+	void runtime.launchConsolidationTask(ctx, async () => withDebugLogContext({
+		enabled: runtime.config.debugLog === true,
+		cwd: ctx.cwd,
+		...sessionMetadata,
+		runId,
+	}, async () => {
 		await runConsolidationPipeline(pi, runtime, consolidationCtx);
 	}));
 }
@@ -221,7 +242,6 @@ async function runObserverStage(
 		count: observations.length,
 		observationTokens: observations.reduce((sum, observation) => sum + observation.tokenCount, 0),
 		coversUpToId,
-		observations,
 	});
 	appendEntry(pi, OM_OBSERVATIONS_RECORDED, data);
 	debugLog("observer.appended", { count: observations.length, coversUpToId });
@@ -305,6 +325,17 @@ async function runDropperStage(
 		});
 		return "continue";
 	}
+	debugLog("dropper.stage_start", {
+		observationCoverageId,
+		sameRunReflectionCoverageId,
+		sameRunReflectionCount: sameRunReflections.length,
+		activeObservationCount: metrics.activeObservationCount,
+		observationTokens: metrics.observationTokens,
+		targetTokens: metrics.targetTokens,
+		tokensOverTarget: metrics.tokensOverTarget,
+		fullness: metrics.fullness,
+		maxDropsAllowed: metrics.maxDropsAllowed,
+	});
 
 	if (ctx.hasUI) ctx.ui?.notify(
 		`Observational memory: dropper running after reflection — active observation pool ~${metrics.observationTokens.toLocaleString()} / ${metrics.targetTokens.toLocaleString()} target tokens (${Math.round(metrics.fullness * 100).toLocaleString()}%)`,
@@ -326,6 +357,12 @@ async function runDropperStage(
 	});
 	const coversUpToId = earlierCoverageMarkerId(entries, observationCoverageId, sameRunReflectionCoverageId);
 	const data = coversUpToId && droppedIds ? buildObservationsDroppedData(droppedIds, coversUpToId) : undefined;
+	debugLog("dropper.append", {
+		droppedIdsCount: droppedIds?.length ?? 0,
+		coversUpToId,
+		dataBuilt: data !== undefined,
+		appended: data !== undefined,
+	});
 	if (data) appendEntry(pi, OM_OBSERVATIONS_DROPPED, data);
 	return "continue";
 }