@dzhechkov/keysarium 1.2.0 → 1.3.0

package/templates/lib/memory-protocol.md

@@ -2,6 +2,8 @@
 
 Core protocol for persistent memory in the Keysarium pipeline. Provides `memory_query()` before tasks and `memory_store()` after tasks.
 
+**Protocol version: 1.1** — adds 2-tier index, record lifecycle (HOT/WARM/COLD/PURGE), and brain container manifest.
+
 ## Memory Namespace
 
 All memory files live in `.keysarium/memory/`:
@@ -9,13 +11,18 @@ All memory files live in `.keysarium/memory/`:
 ```
 .keysarium/memory/
 ├── config.json                                    ← Global configuration
+├── index.json                                     ← 2-tier index (v1.1)
 ├── _patterns/
 │   └── domain-patterns.json                       ← Detected domain patterns
 ├── _stats/
-│   └── reward-summary.json                        ← Aggregate statistics
+│   ├── reward-summary.json                        ← Aggregate statistics
+│   └── imported-baselines.json                    ← Imported reward baselines
 ├── {domain}/                                      ← banking | retail | enterprise | healthcare
-│   └── {case-slug}/
-│       └── {phase}_{timestamp}.json               ← Individual reward records
+│   ├── {case-slug}/
+│   │   └── {phase}_{timestamp}.json               ← HOT/WARM reward records
+│   └── _archive/                                  ← COLD compressed records (v1.1)
+│       └── {case-slug}/
+│           └── {phase}_{timestamp}.json            ← Archived records
 ```
 
 ## Configuration
@@ -24,7 +31,7 @@ All memory files live in `.keysarium/memory/`:
 
 ```json
 {
-  "version": "1.0",
+  "version": "1.1",
   "retention_days": 90,
   "max_results_per_query": 10,
   "enabled": true,
@@ -34,10 +41,62 @@ All memory files live in `.keysarium/memory/`:
     "good": 0.7,
     "needs_work": 0.3,
     "failed": 0.0
+  },
+  "tier_hot_days": 30,
+  "tier_warm_days": 90,
+  "tier_cold_days": 180
+}
+```
+
+**Tier fields (v1.1):** If `tier_hot_days` is absent, fall back to v1.0 behavior (flat `retention_days` with no tiering). When present, `retention_days` is ignored in favor of the tier thresholds.
+
+## Index File (v1.1)
+
+The index provides a compact catalog of all records, enabling targeted reads instead of full directory scans.
+
+### index.json Schema
+
+```json
+{
+  "version": "1.0",
+  "last_updated": "2026-03-02T14:00:00Z",
+  "record_count": 47,
+  "by_domain": {
+    "banking": {
+      "count": 23,
+      "slugs": ["case-a", "case-b"],
+      "avg_reward": 0.82
+    }
+  },
+  "by_phase": {
+    "phase-0": { "count": 8, "avg_reward": 0.91 },
+    "phase-2": { "count": 12, "avg_reward": 0.65 }
+  },
+  "by_skill": {
+    "goap-research-ed25519": { "count": 10, "avg_reward": 0.78 }
+  },
+  "records": {
+    "bank_kc_automation_phase-2_2026-03-01T12-30-00": {
+      "domain": "banking",
+      "phase": "phase-2",
+      "slug": "bank_kc_automation",
+      "reward": 0.7,
+      "skill": "goap-research-ed25519",
+      "tier": "hot",
+      "file": "banking/bank_kc_automation/phase-2_2026-03-01T12-30-00.json",
+      "expires_at": "2026-08-28T12:30:00Z"
+    }
   }
 }
 ```
 
+### Index Behavior
+
+- **Created:** On first `memory_store()` call when tier config is present (or `index.json` already exists)
+- **Updated:** On every `memory_store()`, archive, and purge operation
+- **Fallback:** If missing, `memory_query()` uses v1.0 full-scan algorithm
+- **Rebuild:** If corrupted, delete `index.json` — it will be recreated on next `memory_store()`
+
 ## Protocol: memory_query(context)
 
 Call at the **start** of each pipeline phase to load relevant historical patterns.
@@ -56,12 +115,22 @@ Call at the **start** of each pipeline phase to load relevant historical pattern
 ### Algorithm
 
 1. **Check enabled:** Read `.keysarium/memory/config.json`. If `enabled` is false or directory does not exist, return empty list.
-2. **Resolve path:** Construct `.keysarium/memory/{domain}/` (scan all slugs, not just current).
-3. **Scan records:** Read all `{phase}_*.json` files matching the requested phase across all slugs in the domain.
+2. **Read index (v1.1):** If `.keysarium/memory/index.json` exists, read it and filter `records` entries by `domain` + `phase`. Go to step 2a. If index does not exist, fall back to step 3 (v1.0 full scan).
+   - 2a. **Index hit:** If filtered record count ≤ `max_results_per_query`, read only those files by `file` path (targeted read). Go to step 4.
+   - 2b. **Index overflow:** If filtered count > limit, rank by `reward` from index entries, select top-N `file` paths, read those files. Go to step 4.
+3. **Full scan (fallback):** Construct `.keysarium/memory/{domain}/` (scan all slugs). Read all `{phase}_*.json` files matching the requested phase.
 4. **Filter expired:** Exclude records where `expires_at < current_date`.
-5. **Sort:** By `reward` DESC, then `timestamp` DESC.
-6. **Limit:** Return top `max_results_per_query` records (default 10).
-7. **Enrich:** Also load `.keysarium/memory/_patterns/domain-patterns.json` for matching domain patterns.
+5. **Archive old records (v1.1):** If tier config is present, check record ages. For records older than `tier_warm_days`:
+   - Read full record
+   - Generate `summary` from `reward_reason` + `outcome.checkpoint_response` (first 200 chars)
+   - Write compressed record to `.keysarium/memory/{domain}/_archive/{slug}/{filename}`
+   - Delete original file from active directory
+   - Update `index.json`: set `tier` to `"cold"`, update `file` path
+   - Log: "Archived {count} records to cold storage in {domain}"
+6. **Purge ancient records (v1.1):** Delete records from `_archive/` older than `tier_cold_days`. In v1.0 mode, delete records where `expires_at < current_date`. Update `index.json`. Log: "Purged {count} expired records from {domain}"
+7. **Sort:** By `reward` DESC, then `timestamp` DESC. HOT and WARM records first; COLD records included only if result count < `max_results_per_query`.
+8. **Limit:** Return top `max_results_per_query` records (default 10).
+9. **Enrich:** Also load `.keysarium/memory/_patterns/domain-patterns.json` for matching domain patterns.
 
 ### Output
 
@@ -136,17 +205,25 @@ Call at each **checkpoint** after the user responds, to persist the phase outcom
 
 1. **Ensure directory exists:** Create `.keysarium/memory/{domain}/{slug}/` if it does not exist.
 2. **Build record:** Construct full RewardRecord JSON from input.
-3. **Compute expires_at:** `current_date + retention_days` (from config, default 90 days).
+3. **Compute expires_at:** If tier config is present, use `tier_cold_days` (default 180 days). Otherwise, use `retention_days` (default 90 days). Formula: `current_date + threshold`.
 4. **Generate filename:** `{phase}_{ISO-timestamp}.json` (e.g., `phase-2_2026-03-01T12-30-00.json`).
 5. **Write file:** Write JSON to `.keysarium/memory/{domain}/{slug}/{filename}`.
-6. **Log:** "Stored reward {reward} ({reward_label}) for {phase} of {slug}"
+6. **Update index (v1.1):** Read `.keysarium/memory/index.json` (or create if absent and tier config is present):
+   - Add record entry to `records` map: key = record id, value = `{domain, phase, slug, reward, skill, tier: "hot", file, expires_at}`
+   - Increment `record_count`
+   - Update `by_domain` aggregate: increment count, add slug if new, recalculate `avg_reward`
+   - Update `by_phase` aggregate: increment count, recalculate `avg_reward`
+   - Update `by_skill` aggregate: increment count, recalculate `avg_reward`
+   - Set `last_updated` to current timestamp
+   - Write updated `index.json`
+7. **Log:** "Stored reward {reward} ({reward_label}) for {phase} of {slug}"
 
 ### RewardRecord JSON Schema
 
 ```json
 {
   "id": "bank_kc_automation_phase-2_2026-03-01T12-30-00",
-  "version": "1.0",
+  "version": "1.1",
   "timestamp": "2026-03-01T12:30:00Z",
   "case_slug": "bank_kc_automation",
   "domain": "banking",
@@ -171,10 +248,45 @@ Call at each **checkpoint** after the user responds, to persist the phase outcom
     "promise_emitted": "RESEARCH_PARANOID_PASSED"
   },
   "promise_tag": "RESEARCH_PARANOID_PASSED",
-  "expires_at": "2026-05-30T12:30:00Z"
+  "expires_at": "2026-08-28T12:30:00Z"
 }
 ```
 
+## Record Lifecycle (v1.1)
+
+Records transition through tiers based on age:
+
+| Tier | Age Range | Location | Fields |
+|------|-----------|----------|--------|
+| **HOT** | 0 – 30 days | `{domain}/{slug}/` | All fields |
+| **WARM** | 30 – 90 days | `{domain}/{slug}/` | All fields (deprioritized in search) |
+| **COLD** | 90 – 180 days | `{domain}/_archive/{slug}/` | Compressed (see below) |
+| **PURGE** | > 180 days | Deleted | — |
+
+Tier thresholds are configurable via `config.json` fields: `tier_hot_days`, `tier_warm_days`, `tier_cold_days`.
+
+### COLD Record Schema (compressed)
+
+```json
+{
+  "id": "bank_kc_automation_phase-2_2026-01-01T12-00-00",
+  "phase": "phase-2",
+  "domain": "banking",
+  "slug": "bank_kc_automation",
+  "reward": 0.7,
+  "reward_label": "good",
+  "skill_used": "goap-research-ed25519",
+  "promise_tag": "RESEARCH_PARANOID_PASSED",
+  "summary": "User requested minor edits to competitor analysis section. ok, but expand the FinTech competitor section",
+  "archived_at": "2026-03-02T14:00:00Z",
+  "original_timestamp": "2026-01-01T12:00:00Z"
+}
+```
+
+### Backward Compatibility
+
+If `tier_hot_days` is NOT present in `config.json`, the lifecycle system is disabled. Records use the v1.0 flat `retention_days` for expiration. No archiving occurs.
+
 ## Reward Assignment Rules
 
 The reward score is determined by the user's response at the checkpoint:
@@ -193,15 +305,6 @@ The reward score is determined by the user's response at the checkpoint:
 - **Positive feedback with minor note** ("great, just fix the typo in section 3") = 1.0
 - **No explicit response** (session ends) = do not store (no record created)
 
-## Purge Protocol
-
-To prevent unbounded growth, expired records should be cleaned up:
-
-1. **Trigger:** At each `memory_query()` call, check for expired records
-2. **Scan:** Within the queried domain directory, find records where `expires_at < current_date`
-3. **Delete:** Remove expired JSON files
-4. **Log:** "Purged {count} expired records from {domain}"
-
 ## Error Handling
 
 | Error | Behavior |
@@ -211,6 +314,8 @@ To prevent unbounded growth, expired records should be cleaned up:
 | Malformed JSON file | Skip that record, log warning, continue |
 | Write fails (permissions, disk) | Log error, continue pipeline without storing |
 | Domain not in `known_domains` | Use "unknown" as domain directory |
+| `index.json` corrupted | Delete and rebuild on next `memory_store()` call |
+| Archive directory write fails | Log warning, keep record in active directory |
 
 ## Integration with Brain Export