@dzhechkov/keysarium-core 1.0.0 → 1.1.0

package/README.md

@@ -13,7 +13,7 @@ Think of it as the "operating system" for multi-agent workflows. You bring the d
 | Module | Purpose | Key Protocols |
 |--------|---------|---------------|
 | **governance/** | Structural rules and human checkpoints | Constitution, shard protocol, promise tags |
-| **memory/** | Persistent learning across executions | memory_query, memory_store, reward tracking, dream cycles |
+| **memory/** | Persistent learning across executions | memory_query, memory_store, reward tracking, dream cycles, **2-tier index, COW branching, record lifecycle (v1.1)** |
 | **orchestration/** | Agent coordination and model routing | Queen protocol, 6 topologies, background workers, 3-tier routing |
 | **verification/** | Cryptographic integrity and isolation proofs | SHA-256 witness chain, judge attestation, audit trail |
 | **trust-tiers/** | Skill/artifact quality classification | 4-tier system (Advisory to Verified), promotion protocol |
@@ -71,9 +71,13 @@ keysarium-core has **zero dependencies** on either consumer package. Both consum
 - `shard-protocol.md` — How to create per-stage governance rules that are reloaded at each stage to prevent context drift
 - `checkpoint-protocol.md` — Human synchronization points with promise tags for machine-readable completion signals
 
-### Memory
+### Memory (v1.1)
 
-- `memory-protocol.md` — Core `memory_query()` / `memory_store()` protocol for loading and persisting execution patterns with reward scores
+- `memory-protocol.md` — Core `memory_query()` / `memory_store()` protocol with reward scores. **v1.1 enhancements:**
+  - **2-Tier Index** — `index.json` catalog for targeted reads instead of full directory scans
+  - **Brain Container Schema** — Portable brain exports with SHA-256 manifest and checksum verification
+  - **COW Branching** — Delta brain exports via JSON Patch (RFC 6902) for incremental knowledge transfer
+  - **Record Lifecycle** — HOT/WARM/COLD/PURGE tiering with automatic archive compression
 - `reward-tracker.md` — Analytics engine: per-stage averages, per-domain breakdowns, bottleneck detection, trend analysis
 - `dream-engine.md` — Background consolidation: builds concept graphs from accumulated data and generates cross-domain insights
 
@@ -120,6 +124,7 @@ keysarium-core was extracted from the [Keysarium](https://github.com/dzhechko/pr
 - **Ruflo** — 7-layer governance, 6 topologies, 3-tier model routing
 - **Agentic QE** — Trust tiers, reward-calibrated learning, dream cycles, PACT principles
 - **Quality Forge** — Portable brain containers, SHA-256 witness chains
+- **RVF (RuVector Format)** — Manifest catalogs, COW branching, progressive loading, data tiering (concepts adapted to JSON-native implementation)
 
 ## License
 

package/index.md

@@ -1,13 +1,13 @@
 # keysarium-core Module Index
 
-> Manifest of all modules in @dzhechkov/keysarium-core v1.0.0
+> Manifest of all modules in @dzhechkov/keysarium-core v1.1.0
 
 ## Module Registry
 
 | Module | Version | Files | Trust Tier | Description |
 |--------|---------|-------|------------|-------------|
 | governance | 1.0 | 3 | Tier 1 — Structured | Constitution, shards, checkpoints |
-| memory | 1.0 | 3 | Tier 1 — Structured | Reward-calibrated learning + dream cycles |
+| memory | 1.1 | 3 | Tier 1 — Structured | Reward-calibrated learning + dream cycles + index + tiering + COW |
 | orchestration | 1.0 | 4 | Tier 1 — Structured | Coordinator protocol + topologies + workers |
 | verification | 1.0 | 3 | Tier 1 — Structured | Hash chains + judge attestation + audit trail |
 | trust-tiers | 1.0 | 2 | Tier 1 — Structured | 4-tier classification + promotion |
@@ -27,7 +27,7 @@
 │   └── checkpoint-protocol.md         ← v1.0 — Checkpoints + promise tags
 │
 ├── memory/
-│   ├── memory-protocol.md             ← v1.0 — memory_query + memory_store
+│   ├── memory-protocol.md             ← v1.1 — memory_query + memory_store + index + tiering + COW
 │   ├── reward-tracker.md              ← v1.0 — Analytics + pattern detection
 │   └── dream-engine.md                ← v1.0 — Background insight generation
 │
@@ -59,6 +59,7 @@
 | Version | Date | Changes |
 |---------|------|---------|
 | 1.0 | 2026-03-01 | Initial extraction from Keysarium v1.1.x and lib/ protocols |
+| 1.1 | 2026-03-02 | Memory module v1.1: 2-tier index, record lifecycle (HOT/WARM/COLD/PURGE), brain container manifest, COW branching |
 
 ## Dependency Graph
 
@@ -75,7 +76,7 @@ All modules are independent. A consumer may use any subset of modules without im
 
 ## Schema Compatibility
 
-All JSON schemas defined in this package use `"version": "1.0"` as a schema version field. Consumers should check this field and warn if it does not match expected version.
+JSON schemas use `"version"` as a schema version field. Memory module uses `"1.1"`, all other modules use `"1.0"`. Consumers should check this field and warn if it does not match expected version. v1.1 readers MUST accept v1.0 files for backward compatibility.
 
 ## Terminology Mapping
 

package/memory/memory-protocol.md

@@ -1,6 +1,8 @@
 # Memory Protocol — Reward-Calibrated Learning
 
 > Core protocol for persistent memory in multi-agent pipelines. Provides `memory_query()` before stages and `memory_store()` after stages.
+>
+> **Protocol version: 1.1** — adds Index, Brain Container Schema, COW Branching, and Record Lifecycle.
 
 ## Overview
 
@@ -13,13 +15,17 @@ All memory files live in a dedicated directory:
 ```
 {memory-root}/
 ├── config.json                                    ← Global configuration
+├── index.json                                     ← 2-tier index (v1.1)
 ├── _patterns/
 │   └── domain-patterns.json                       ← Detected domain patterns
 ├── _stats/
 │   └── reward-summary.json                        ← Aggregate statistics
 ├── {domain}/                                      ← Domain-specific subdirectory
-│   └── {project-slug}/
-│       └── {stage}_{timestamp}.json               ← Individual reward records
+│   ├── {project-slug}/
+│   │   └── {stage}_{timestamp}.json               ← HOT/WARM reward records
+│   └── _archive/                                  ← COLD compressed records (v1.1)
+│       └── {project-slug}/
+│           └── {stage}_{timestamp}.json            ← Archived records
 ```
 
 The `{memory-root}` defaults to `.keysarium/memory/` but can be configured.
@@ -30,7 +36,7 @@ The `{memory-root}` defaults to `.keysarium/memory/` but can be configured.
 
 ```json
 {
-  "version": "1.0",
+  "version": "1.1",
   "retention_days": 90,
   "max_results_per_query": 10,
   "enabled": true,
@@ -40,10 +46,15 @@ The `{memory-root}` defaults to `.keysarium/memory/` but can be configured.
     "good": 0.7,
     "needs_work": 0.3,
     "failed": 0.0
-  }
+  },
+  "tier_hot_days": 30,
+  "tier_warm_days": 90,
+  "tier_cold_days": 180
 }
 ```
 
+**Tier fields (v1.1):** If `tier_hot_days` is absent, fall back to v1.0 behavior (flat `retention_days` with no tiering). When present, `retention_days` is ignored in favor of the tier thresholds.
+
 ## Protocol: memory_query(context)
 
 Call at the **start** of each pipeline stage to load relevant historical patterns.
@@ -62,12 +73,16 @@ Call at the **start** of each pipeline stage to load relevant historical pattern
 ### Algorithm
 
 1. **Check enabled:** Read `{memory-root}/config.json`. If `enabled` is false or directory does not exist, return empty result.
-2. **Resolve path:** Construct `{memory-root}/{domain}/` (scan all project slugs, not just current).
-3. **Scan records:** Read all `{stage}_*.json` files matching the requested stage across all slugs in the domain.
+2. **Read index (v1.1):** If `{memory-root}/index.json` exists, read it and filter `records` entries by `domain` + `stage`. Go to step 2a. If index does not exist, fall back to step 3 (v1.0 full scan).
+   - 2a. **Index hit:** If filtered record count ≤ `max_results_per_query`, read only those files by `file` path (Tier 2 targeted read). Go to step 4.
+   - 2b. **Index overflow:** If filtered count > limit, rank by `reward` from index entries, select top-N `file` paths, read those files. Go to step 4.
+3. **Full scan (fallback):** Construct `{memory-root}/{domain}/` (scan all project slugs). Read all `{stage}_*.json` files matching the requested stage.
 4. **Filter expired:** Exclude records where `expires_at < current_date`.
-5. **Sort:** By `reward` DESC, then `timestamp` DESC.
-6. **Limit:** Return top `max_results_per_query` records (default 10).
-7. **Enrich:** Also load `{memory-root}/_patterns/domain-patterns.json` for matching domain patterns.
+5. **Archive old records (v1.1):** If tier config is present, check record ages. Move records older than `tier_cold_days` to `{memory-root}/{domain}/_archive/{slug}/` in compressed format (see Record Lifecycle). Update `index.json`.
+6. **Purge ancient records (v1.1):** Delete records from `_archive/` older than `tier_cold_days * 2` (default 360 days). Update `index.json`. In v1.0 mode, delete records where `expires_at < current_date`.
+7. **Sort:** By `reward` DESC, then `timestamp` DESC.
+8. **Limit:** Return top `max_results_per_query` records (default 10). HOT and WARM records are preferred; COLD records are included only if result count < limit.
+9. **Enrich:** Also load `{memory-root}/_patterns/domain-patterns.json` for matching domain patterns.
 
 ### Output
 
@@ -141,17 +156,23 @@ Call at each **checkpoint** after the human responds, to persist the stage outco
 
 1. **Ensure directory exists:** Create `{memory-root}/{domain}/{slug}/` if it does not exist.
 2. **Build record:** Construct full RewardRecord JSON from input.
-3. **Compute expires_at:** `current_date + retention_days` (from config, default 90 days).
+3. **Compute expires_at:** `current_date + retention_days` (from config, default 90 days). If tier config is present, use `tier_cold_days * 2` instead.
 4. **Generate filename:** `{stage}_{ISO-timestamp}.json`.
 5. **Write file:** Write JSON to `{memory-root}/{domain}/{slug}/{filename}`.
-6. **Log:** "Stored reward {reward} ({reward_label}) for {stage} of {slug}"
+6. **Update index (v1.1):** If `{memory-root}/index.json` exists (or tier config is present), update it:
+   - Add record entry to `records` map with key = record id, value = `{domain, stage, slug, reward, skill, tier: "hot", file, expires_at}`
+   - Increment `record_count`
+   - Update `by_domain`, `by_phase`, `by_skill` aggregates (count, avg_reward)
+   - Set `last_updated` to current timestamp
+   - If `index.json` does not exist, create it with this record as the first entry
+7. **Log:** "Stored reward {reward} ({reward_label}) for {stage} of {slug}"
 
 ### RewardRecord JSON Schema
 
 ```json
 {
   "id": "{slug}_{stage}_{timestamp}",
-  "version": "1.0",
+  "version": "1.1",
   "timestamp": "ISO-8601",
   "project_slug": "{slug}",
   "domain": "{domain}",
@@ -195,3 +216,262 @@ To prevent unbounded growth:
 | Malformed JSON file | Skip that record, log warning, continue |
 | Write fails (permissions, disk) | Log error, continue pipeline without storing |
 | Domain not in `known_domains` | Use "unknown" as domain directory |
+| `index.json` corrupted | Delete and rebuild on next `memory_store()` call |
+| Archive directory write fails | Log warning, keep record in active directory |
+
+---
+
+## Index Protocol (v1.1)
+
+The index file provides a compact catalog of all records, enabling targeted file reads instead of full directory scans.
+
+### index.json Schema
+
+```json
+{
+  "version": "1.0",
+  "last_updated": "ISO-8601",
+  "record_count": 47,
+  "by_domain": {
+    "{domain}": {
+      "count": 23,
+      "slugs": ["{slug-1}", "{slug-2}"],
+      "avg_reward": 0.82
+    }
+  },
+  "by_stage": {
+    "{stage-id}": {
+      "count": 12,
+      "avg_reward": 0.65
+    }
+  },
+  "by_skill": {
+    "{skill-name}": {
+      "count": 10,
+      "avg_reward": 0.78
+    }
+  },
+  "records": {
+    "{record-id}": {
+      "domain": "{domain}",
+      "stage": "{stage-id}",
+      "slug": "{slug}",
+      "reward": 0.7,
+      "skill": "{skill-name}",
+      "tier": "hot|warm|cold",
+      "file": "{domain}/{slug}/{stage}_{timestamp}.json",
+      "expires_at": "ISO-8601"
+    }
+  }
+}
+```
+
+### Index Update Rules
+
+1. **On `memory_store()`:** Add new record entry, update aggregates.
+2. **On archive (WARM→COLD):** Update `tier` to `"cold"`, change `file` path to `_archive/` location.
+3. **On purge:** Remove record entry, decrement aggregates.
+4. **Rebuild:** If `index.json` is missing or corrupted, `memory_store()` creates a fresh index with the new record. Full rebuild requires scanning all files (expensive, avoid if possible).
+
+### Fallback Behavior
+
+If `index.json` does not exist, `memory_query()` falls back to the v1.0 full-scan algorithm. This ensures backward compatibility with existing memory directories that predate v1.1.
+
+---
+
+## Brain Container Schema (v1.1)
+
+Defines the portable brain export format with an integrity manifest.
+
+### Container Structure
+
+```json
+{
+  "version": "1.1",
+  "format": "keysarium-brain",
+  "manifest": {
+    "created_at": "ISO-8601",
+    "source_project": "{project-name}",
+    "source_branch": "{git-branch}",
+    "checksum": "sha256:{64-hex-chars}",
+    "record_count": 47,
+    "domains": ["{domain-1}", "{domain-2}"],
+    "skills": ["{skill-1}", "{skill-2}"],
+    "research_count": 5,
+    "harvest_pattern_count": 23,
+    "size_bytes": 148200,
+    "parent": null,
+    "parent_checksum": null,
+    "delta_type": null
+  },
+  "skills": { },
+  "domain_patterns": { },
+  "research_summaries": [ ],
+  "harvest_patterns": [ ],
+  "pipeline_metrics": { },
+  "reward_data": { },
+  "metadata": { }
+}
+```
+
+### Manifest Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `created_at` | string | ISO-8601 timestamp of export |
+| `source_project` | string | Project name from package.json or directory name |
+| `source_branch` | string | Current git branch at export time |
+| `checksum` | string | SHA-256 of the JSON-serialized content (all sections except `manifest`) |
+| `record_count` | integer | Total records across all sections |
+| `domains` | string[] | List of domains present in export |
+| `skills` | string[] | List of skill names present |
+| `research_count` | integer | Number of research summaries |
+| `harvest_pattern_count` | integer | Number of harvest patterns |
+| `size_bytes` | integer | Approximate size of the content payload |
+| `parent` | string\|null | Filename of parent brain container (COW mode) |
+| `parent_checksum` | string\|null | SHA-256 of parent container (COW mode) |
+| `delta_type` | string\|null | `"rfc6902"` if this is a delta container |
+
+### Checksum Computation
+
+1. Serialize all sections except `manifest` as a JSON string (sorted keys, no extra whitespace)
+2. Compute SHA-256 of the resulting string
+3. Store as `"sha256:{64-hex-chars}"`
+
+### Version Negotiation
+
+| Reader Version | File Version | Behavior |
+|---------------|-------------|----------|
+| v1.1 | v1.0 | Accept. Manifest will be absent — proceed without integrity check |
+| v1.1 | v1.1 | Accept. Read manifest, verify checksum |
+| v1.0 | v1.1 | Accept. Ignore unknown `manifest` field, read sections normally |
+| Any | Unknown | Warn, attempt best-effort import |
+
+---
+
+## COW Branching Protocol (v1.1)
+
+Copy-On-Write branching for efficient delta brain exports. A child container stores only the differences from a parent container.
+
+### Delta Container Structure
+
+```json
+{
+  "version": "1.1",
+  "format": "keysarium-brain",
+  "manifest": {
+    "created_at": "ISO-8601",
+    "source_project": "{project-name}",
+    "source_branch": "{git-branch}",
+    "checksum": "sha256:{hash-of-patch-array}",
+    "record_count": 3,
+    "domains": ["{domains-in-delta}"],
+    "skills": [],
+    "research_count": 1,
+    "harvest_pattern_count": 2,
+    "size_bytes": 4200,
+    "parent": "{parent-filename}",
+    "parent_checksum": "sha256:{parent-hash}",
+    "delta_type": "rfc6902"
+  },
+  "patch": [
+    { "op": "add", "path": "/research_summaries/-", "value": { } },
+    { "op": "replace", "path": "/skills/{name}/trust_tier", "value": 2 },
+    { "op": "add", "path": "/harvest_patterns/-", "value": { } }
+  ]
+}
+```
+
+### Export Protocol (delta mode)
+
+1. Load parent brain container from the specified path
+2. Verify parent checksum matches `manifest.checksum` of the parent file
+3. Build current full brain container (in memory, not written to disk)
+4. Compute JSON Patch (RFC 6902) between parent and current
+5. If patch is empty → log "No changes since parent" and skip export
+6. Write delta container with `manifest.parent` set to parent filename
+
+### Import Protocol (delta mode)
+
+1. Read delta container; detect `manifest.parent != null`
+2. Resolve parent file:
+   - Look in the same directory as the delta file
+   - Look in project root
+   - If not found → prompt user for path
+   - If still not found → offer partial import (self-contained `add` operations only)
+3. If parent is also a delta → resolve recursively (chain). Warn if chain depth > 3
+4. Load parent (or resolved chain base), apply JSON Patch operations in order
+5. Verify result checksum if available
+6. Proceed with standard import (selective sections, merge strategy)
+
+### JSON Patch Operations (RFC 6902)
+
+| Operation | Usage |
+|-----------|-------|
+| `add` | New records (research summaries, harvest patterns, skills) |
+| `replace` | Updated values (trust tier upgrade, new BTO score) |
+| `remove` | Deleted entries (rare, mainly cleanup) |
+| `move` | Renamed entries (rare) |
+| `copy` | Duplicated entries (rare) |
+| `test` | Pre-condition assertions (optional, for safety) |
+
+### Chain Depth Limit
+
+Delta containers may reference another delta as parent, forming a chain. Import MUST warn if chain depth exceeds 3. Recommended: periodically re-export a full container to reset the chain.
+
+---
+
+## Record Lifecycle (v1.1)
+
+Records transition through tiers based on age, trading detail for storage efficiency.
+
+### Tier Definitions
+
+| Tier | Age Range | Location | Fields | Purpose |
+|------|-----------|----------|--------|---------|
+| **HOT** | 0 – `tier_hot_days` (default 30) | `{domain}/{slug}/` | All fields | Active learning, full context |
+| **WARM** | `tier_hot_days` – `tier_warm_days` (default 90) | `{domain}/{slug}/` | All fields | Available but deprioritized in search |
+| **COLD** | `tier_warm_days` – `tier_cold_days` (default 180) | `{domain}/_archive/{slug}/` | Compressed | Long-term pattern retention |
+| **PURGE** | > `tier_cold_days` | Deleted | — | Cleanup |
+
+### COLD Record Schema (compressed)
+
+```json
+{
+  "id": "{original-id}",
+  "stage": "{stage-id}",
+  "domain": "{domain}",
+  "slug": "{slug}",
+  "reward": 0.7,
+  "reward_label": "good",
+  "skill_used": "{skill}",
+  "promise_tag": "{PROMISE_TAG}",
+  "summary": "{first 200 chars of reward_reason + checkpoint_response}",
+  "archived_at": "ISO-8601",
+  "original_timestamp": "ISO-8601"
+}
+```
+
+### Archive Procedure
+
+Triggered during `memory_query()` step 5:
+
+1. For each record in the query result set, compute age = `current_date - timestamp`
+2. If age > `tier_warm_days` and record is not already archived:
+   a. Read full record
+   b. Generate `summary` from `reward_reason` + `outcome.checkpoint_response` (truncate to 200 chars)
+   c. Write compressed record to `{memory-root}/{domain}/_archive/{slug}/{filename}`
+   d. Delete original file from active directory
+   e. Update `index.json`: set `tier` to `"cold"`, update `file` path
+
+### Purge Procedure
+
+Triggered during `memory_query()` step 6:
+
+1. Scan `{domain}/_archive/` for records where age > `tier_cold_days`
+2. Delete those files
+3. Remove entries from `index.json`
+
+### Backward Compatibility
+
+If `tier_hot_days` is NOT present in `config.json`, the entire lifecycle system is disabled. Records use the v1.0 flat `retention_days` for expiration. No archiving occurs. This ensures existing memory directories work without changes.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@dzhechkov/keysarium-core",
-  "version": "1.0.0",
+  "version": "1.1.0",
   "description": "Core framework for multi-agent pipelines: governance, memory, orchestration, verification, trust tiers, and multi-platform support",
   "main": "index.md",
   "files": [