@dzhechkov/keysarium 1.1.2 → 1.3.0

package/templates/lib/reward-tracker.md

@@ -0,0 +1,256 @@
+# Reward Tracker -- Analytics & Pattern Detection
+
+Computes aggregate statistics and detects domain patterns from accumulated reward records. Used by `/learning-stats` and by `memory_query()` for pattern enrichment.
+
+## Overview
+
+The Reward Tracker operates on the data stored by the Memory Protocol (`lib/memory-protocol.md`). It reads all RewardRecord JSON files from `.keysarium/memory/` and produces two output artifacts:
+
+1. **reward-summary.json** -- aggregate statistics per phase, domain, and skill
+2. **domain-patterns.json** -- detected patterns with confidence scores
+
+## Computation Protocol
+
+### Step 1: Load All Records
+
+1. Scan `.keysarium/memory/` recursively for all `*.json` files (excluding config.json, domain-patterns.json, reward-summary.json).
+2. Parse each JSON file as a RewardRecord.
+3. Exclude records where `expires_at < current_date`.
+4. Collect into a list sorted by `timestamp` DESC.
+
+### Step 2: Per-Phase Reward Averages
+
+Group records by `phase` and compute:
+
+```json
+{
+  "phase-0": {
+    "avg_reward": 0.85,
+    "total_runs": 12,
+    "distribution": {
+      "excellent": 8,
+      "good": 3,
+      "needs_work": 1,
+      "failed": 0
+    },
+    "trend": "stable",
+    "best_case": "retail_personalization",
+    "worst_case": "bank_kc_automation"
+  }
+}
+```
+
+**Trend Detection Algorithm:**
+1. Split records for each phase into two halves by timestamp (older half, newer half).
+2. Compute average reward for each half.
+3. If newer_avg - older_avg > 0.15 -> "improving"
+4. If older_avg - newer_avg > 0.15 -> "degrading"
+5. Otherwise -> "stable"
+6. Minimum 4 records required for trend detection; otherwise "insufficient_data".
+
+### Step 3: Per-Domain Reward Averages
+
+Group records by `domain` and compute:
+
+```json
+{
+  "banking": {
+    "avg_reward": 0.72,
+    "total_runs": 18,
+    "phase_breakdown": {
+      "phase-0": 0.85,
+      "phase-1": 0.90,
+      "phase-2": 0.55,
+      "phase-2.5": 0.70,
+      "phase-3": 0.65,
+      "phase-4": 0.80,
+      "phase-5": 0.72
+    },
+    "bottleneck_phase": "phase-2",
+    "strongest_phase": "phase-1"
+  }
+}
+```
+
+**Bottleneck Detection:**
+- Phase with the lowest average reward in a domain is flagged as `bottleneck_phase`.
+- Phase with the highest average reward is flagged as `strongest_phase`.
+- Only computed if domain has 3+ records.
+
+### Step 4: Per-Skill Effectiveness
+
+Group records by `skill_used` and cross-reference with domain:
+
+```json
+{
+  "goap-research-ed25519": {
+    "overall_avg": 0.74,
+    "total_runs": 15,
+    "by_domain": {
+      "banking": { "avg": 0.65, "runs": 8 },
+      "retail": { "avg": 0.85, "runs": 7 }
+    },
+    "best_domain": "retail",
+    "worst_domain": "banking"
+  }
+}
+```
+
+### Step 5: Domain Pattern Detection
+
+Analyze accumulated data to detect actionable patterns. Each pattern has a confidence score based on evidence count.
+
+**Pattern Detection Rules:**
+
+| Rule | Condition | Pattern Template |
+|------|-----------|-----------------|
+| Phase Bottleneck | Phase avg < 0.5 in domain, 3+ records | "{domain} cases struggle in {phase} (avg reward: {avg})" |
+| Phase Excellence | Phase avg > 0.9 in domain, 3+ records | "{domain} cases excel in {phase} (avg reward: {avg})" |
+| Skill-Domain Mismatch | Skill avg < 0.5 in domain but > 0.7 in another | "{skill} underperforms in {domain} vs {other_domain}" |
+| Improving Trend | Trend = "improving" for phase in domain | "{phase} quality is improving in {domain} domain" |
+| Degrading Trend | Trend = "degrading" for phase in domain | "{phase} quality is degrading in {domain} -- investigate" |
+| Time Overhead | Phase avg iterations > 2.0 in domain | "{domain} cases require more iterations in {phase}" |
+
+**Confidence Calculation:**
+```
+confidence = min(1.0, evidence_count / 10)
+```
+Where `evidence_count` is the number of records supporting the pattern. Confidence reaches 1.0 at 10+ supporting records.
+
+### Step 6: Write Outputs
+
+#### reward-summary.json
+
+Write to `.keysarium/memory/_stats/reward-summary.json`:
+
+```json
+{
+  "version": "1.0",
+  "generated_at": "2026-03-01T14:00:00Z",
+  "total_records": 42,
+  "total_domains": 3,
+  "total_cases": 8,
+  "phase_averages": { ... },
+  "domain_averages": { ... },
+  "skill_effectiveness": { ... },
+  "overall_average": 0.76,
+  "overall_trend": "improving"
+}
+```
+
+#### domain-patterns.json
+
+Write to `.keysarium/memory/_patterns/domain-patterns.json`:
+
+```json
+{
+  "version": "1.0",
+  "generated_at": "2026-03-01T14:00:00Z",
+  "patterns": [
+    {
+      "pattern_id": "banking-phase2-bottleneck",
+      "domain": "banking",
+      "description": "Banking cases struggle in Phase 2 (Research) with average reward 0.55",
+      "category": "phase_bottleneck",
+      "confidence": 0.80,
+      "evidence_count": 8,
+      "detected_at": "2026-03-01T14:00:00Z",
+      "examples": ["bank_kc_automation", "bank_loan_approval"],
+      "actionable_advice": "Allocate 20% extra time for Phase 2 in banking cases. Consider pre-loading regulatory templates."
+    },
+    {
+      "pattern_id": "retail-triz-effective",
+      "domain": "retail",
+      "description": "TRIZ methodology produces better results than Game Theory for retail problem-solving",
+      "category": "skill_effectiveness",
+      "confidence": 0.60,
+      "evidence_count": 6,
+      "detected_at": "2026-03-01T14:00:00Z",
+      "examples": ["ecom_personalization", "retail_inventory"],
+      "actionable_advice": "Prefer TRIZ framework over Game Theory for retail domain cases in Phase 3."
+    }
+  ]
+}
+```
+
+## Display Format for /learning-stats
+
+### Phase Averages Table
+
+```
+Phase Reward Averages (42 records, 8 cases)
+───────────────────────────────────────────
+Phase       | Avg    | Runs | Trend       |
+────────────|────────|──────|─────────────|
+Phase 0     | 0.85   | 8    | stable      |
+Phase 1     | 0.92   | 8    | improving   |
+Phase 2     | 0.65   | 7    | stable      |
+Phase 2.5   | 0.78   | 6    | improving   |
+Phase 3     | 0.71   | 5    | stable      |
+Phase 4     | 0.89   | 4    | stable      |
+Phase 5     | 0.74   | 4    | stable      |
+────────────|────────|──────|─────────────|
+Overall     | 0.76   | 42   | improving   |
+```
+
+### Domain Breakdown Table
+
+```
+Domain Breakdown
+───────────────────────────────────────
+Domain      | Avg    | Cases | Bottleneck  |
+────────────|────────|───────|─────────────|
+Banking     | 0.72   | 3     | Phase 2     |
+Retail      | 0.82   | 3     | Phase 5     |
+Enterprise  | 0.69   | 2     | Phase 3     |
+```
+
+### Detected Patterns
+
+```
+Detected Patterns (confidence > 0.5)
+──────────────────────────────────────────────────────
+1. [0.80] Banking cases struggle in Phase 2 (Research)
+   Advice: Allocate 20% extra time for Phase 2 in banking cases
+
+2. [0.60] TRIZ outperforms Game Theory for retail problem-solving
+   Advice: Prefer TRIZ framework for retail domain in Phase 3
+
+3. [0.55] Phase 1 quality is improving across all domains
+   Advice: Current exploration approach is working well
+```
+
+### Skill Effectiveness Table
+
+```
+Skill Effectiveness
+────────────────────────────────────────────────────
+Skill                          | Avg  | Best Domain |
+───────────────────────────────|──────|─────────────|
+explore                        | 0.92 | retail      |
+goap-research-ed25519          | 0.74 | retail      |
+problem-solver-enhanced        | 0.71 | enterprise  |
+reverse-engineering-unicorn    | 0.85 | banking     |
+presentation-storyteller       | 0.74 | retail      |
+```
+
+## Empty State
+
+When no records exist yet:
+
+```
+Learning Stats
+──────────────
+No reward data found in .keysarium/memory/
+
+To start collecting data:
+1. Run a case with /casarium
+2. Respond at each checkpoint
+3. Rewards are automatically tracked
+
+Memory directory: .keysarium/memory/ (will be created on first run)
+```
+
+## Modular Reuse
+
+This tracker is domain-agnostic. It works with any RewardRecord JSON that conforms to the schema defined in `lib/memory-protocol.md`. The pattern detection rules can be extended by adding entries to the Pattern Detection Rules table.

package/templates/lib/witness-chain.md

@@ -0,0 +1,372 @@
+# Witness Chain Protocol
+
+> SHA-256 hash-chain for tamper-evident artifact integrity verification.
+
+## Purpose
+
+This protocol defines how to compute SHA-256 hashes of pipeline artifacts and link them into a chain. Each artifact's hash includes the previous artifact's hash, making the chain tamper-evident: modifying any artifact breaks all downstream hashes.
+
+## Constants
+
+```
+NULL_HASH = "0000000000000000000000000000000000000000000000000000000000000000"
+CHAIN_FILE = ".witness-chain.json"
+HASH_PREFIX = "sha256:"
+```
+
+## Section 1: Hash Computation
+
+### Platform Detection
+
+Detect the available SHA-256 command:
+
+```bash
+# Try sha256sum first (Linux), then shasum (macOS)
+if command -v sha256sum &>/dev/null; then
+  SHA_CMD="sha256sum"
+elif command -v shasum &>/dev/null; then
+  SHA_CMD="shasum -a 256"
+else
+  echo "ERROR: No SHA-256 command found. Install coreutils."
+  exit 1
+fi
+```
+
+### Computing a File Hash
+
+Hash a single file's content:
+
+```bash
+HASH=$(${SHA_CMD} "path/to/file.md" | awk '{print $1}')
+```
+
+### Computing a Chained Hash
+
+Hash file content concatenated with the previous hash to create a chain link:
+
+```bash
+# Read file content and previous hash
+FILE_CONTENT=$(cat "path/to/file.md")
+PREV_HASH="<previous hash value>"
+
+# Compute chained hash
+CHAINED_HASH=$(printf '%s%s' "${FILE_CONTENT}" "${PREV_HASH}" | ${SHA_CMD} | awk '{print $1}')
+```
+
+**Important:** Use `printf '%s%s'` (not `echo`) to avoid trailing newline issues. The content is the raw file bytes followed immediately by the 64-character hex hash string.
+
+### Content Normalization
+
+Before hashing, the content should be the raw file content as stored on disk. Do NOT:
+- Trim whitespace
+- Normalize line endings
+- Strip BOM markers
+
+The hash must match the exact bytes in the file so that verification produces the same result.
+
+## Section 2: Chain Operations
+
+### Operation: Create Genesis Record
+
+Called at Phase 0 (Discovery) to initialize the chain for a new research.
+
+**Preconditions:**
+- Research directory `researches/<slug>/` exists
+- Phase 0 artifact `00_product_discovery.md` has been created
+- No `.witness-chain.json` exists yet
+
+**Procedure:**
+
+```bash
+SLUG="<research-slug>"
+DIR="researches/${SLUG}"
+ARTIFACT="00_product_discovery.md"
+PREV_HASH="${NULL_HASH}"
+
+# Compute genesis hash
+FILE_CONTENT=$(cat "${DIR}/${ARTIFACT}")
+HASH=$(printf '%s%s' "${FILE_CONTENT}" "${PREV_HASH}" | ${SHA_CMD} | awk '{print $1}')
+TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
+
+# Create .witness-chain.json
+cat > "${DIR}/.witness-chain.json" << ENDOFCHAIN
+{
+  "research_slug": "${SLUG}",
+  "created_at": "${TIMESTAMP}",
+  "last_updated": "${TIMESTAMP}",
+  "chain": [
+    {
+      "sequence": 0,
+      "phase": "phase-0",
+      "artifact": "${ARTIFACT}",
+      "hash": "${HASH_PREFIX}${HASH}",
+      "previous_hash": "${HASH_PREFIX}${PREV_HASH}",
+      "timestamp": "${TIMESTAMP}",
+      "promise_tag": "DISCOVERY_COMPLETE"
+    }
+  ]
+}
+ENDOFCHAIN
+```
+
+### Operation: Append Record
+
+Called at each subsequent phase checkpoint to add a new record to the chain.
+
+**Preconditions:**
+- `.witness-chain.json` exists in the research directory
+- The new artifact file has been created
+- The previous phase's record exists in the chain
+
+**Procedure:**
+
+1. Read `.witness-chain.json` and extract the last record's hash
+2. Compute the new chained hash: `SHA-256(new_content + last_hash)`
+3. Append the new record to the chain array
+4. Update `last_updated` timestamp
+
+```bash
+SLUG="<research-slug>"
+DIR="researches/${SLUG}"
+ARTIFACT="<artifact-filename>"
+PHASE="<phase-id>"
+PROMISE="<promise-tag>"
+
+# Get the latest hash from the chain
+PREV_HASH=$(cat "${DIR}/.witness-chain.json" | python3 -c "
+import json, sys
+chain = json.load(sys.stdin)
+last = chain['chain'][-1]
+print(last['hash'].replace('sha256:', ''))
+")
+
+# Compute new hash
+FILE_CONTENT=$(cat "${DIR}/${ARTIFACT}")
+HASH=$(printf '%s%s' "${FILE_CONTENT}" "${PREV_HASH}" | ${SHA_CMD} | awk '{print $1}')
+TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
+SEQUENCE=$(cat "${DIR}/.witness-chain.json" | python3 -c "
+import json, sys
+chain = json.load(sys.stdin)
+print(len(chain['chain']))
+")
+
+# Append to chain (using python3 for safe JSON manipulation)
+python3 -c "
+import json, sys
+with open('${DIR}/.witness-chain.json', 'r') as f:
+    chain = json.load(f)
+chain['chain'].append({
+    'sequence': ${SEQUENCE},
+    'phase': '${PHASE}',
+    'artifact': '${ARTIFACT}',
+    'hash': 'sha256:${HASH}',
+    'previous_hash': 'sha256:${PREV_HASH}',
+    'timestamp': '${TIMESTAMP}',
+    'promise_tag': '${PROMISE}'
+})
+chain['last_updated'] = '${TIMESTAMP}'
+with open('${DIR}/.witness-chain.json', 'w') as f:
+    json.dump(chain, f, indent=2)
+"
+```
+
+### Operation: Get Latest Hash
+
+Retrieve the most recent hash from the chain (needed as input for the next append).
+
+```bash
+LATEST_HASH=$(python3 -c "
+import json
+with open('${DIR}/.witness-chain.json') as f:
+    chain = json.load(f)
+print(chain['chain'][-1]['hash'].replace('sha256:', ''))
+")
+```
+
+## Section 3: Verification
+
+### Operation: Verify Chain
+
+Walk the entire chain and verify each hash matches the current file content.
+
+**Algorithm:**
+
+```
+1. Load .witness-chain.json
+2. For each record in chain (index i):
+   a. Read the artifact file
+   b. Get previous_hash:
+      - If i == 0: previous_hash = NULL_HASH
+      - Else: previous_hash = chain[i-1].hash (without prefix)
+   c. Compute expected = SHA-256(file_content + previous_hash)
+   d. Compare expected with record.hash (without prefix)
+   e. If mismatch: record broken link
+3. Produce verification report
+```
+
+**Bash implementation:**
+
+```bash
+DIR="researches/<slug>"
+
+python3 << 'VERIFY_SCRIPT'
+import json, subprocess, sys
+
+NULL_HASH = "0" * 64
+DIR = sys.argv[1] if len(sys.argv) > 1 else "."
+
+def sha256(content):
+    """Compute SHA-256 of a string."""
+    result = subprocess.run(
+        ["sha256sum"],
+        input=content.encode(),
+        capture_output=True
+    )
+    if result.returncode != 0:
+        # Fallback to shasum
+        result = subprocess.run(
+            ["shasum", "-a", "256"],
+            input=content.encode(),
+            capture_output=True
+        )
+    return result.stdout.decode().split()[0]
+
+# Load chain
+with open(f"{DIR}/.witness-chain.json") as f:
+    chain_data = json.load(f)
+
+records = chain_data["chain"]
+broken = []
+verified = 0
+
+for i, record in enumerate(records):
+    artifact_path = f"{DIR}/{record['artifact']}"
+    try:
+        with open(artifact_path) as f:
+            content = f.read()
+    except FileNotFoundError:
+        broken.append({
+            "sequence": i,
+            "artifact": record["artifact"],
+            "reason": "File not found"
+        })
+        continue
+
+    if i == 0:
+        prev_hash = NULL_HASH
+    else:
+        prev_hash = records[i-1]["hash"].replace("sha256:", "")
+
+    expected = sha256(content + prev_hash)
+    actual = record["hash"].replace("sha256:", "")
+
+    if expected == actual:
+        verified += 1
+    else:
+        broken.append({
+            "sequence": i,
+            "artifact": record["artifact"],
+            "expected_hash": f"sha256:{expected}",
+            "actual_hash": record["hash"],
+            "reason": "Hash mismatch - artifact may have been modified"
+        })
+
+# Report
+total = len(records)
+if not broken:
+    print(f"PASS: All {total} records verified successfully")
+else:
+    print(f"FAIL: {len(broken)} broken links out of {total} records")
+    for b in broken:
+        print(f"  - Record {b['sequence']} ({b['artifact']}): {b['reason']}")
+
+VERIFY_SCRIPT
+```
+
+## Section 4: File Format
+
+### .witness-chain.json Schema
+
+```json
+{
+  "research_slug": "string — research directory name",
+  "created_at": "ISO-8601 — when chain was initialized",
+  "last_updated": "ISO-8601 — when last record was added",
+  "chain": [
+    {
+      "sequence": "integer — 0-based position in chain",
+      "phase": "string — phase-0, phase-1, ..., phase-5",
+      "artifact": "string — filename relative to research dir",
+      "hash": "string — sha256:<64 hex chars>",
+      "previous_hash": "string — sha256:<64 hex chars> or sha256:<null hash>",
+      "timestamp": "ISO-8601 — when record was created",
+      "promise_tag": "string — semantic completion promise"
+    }
+  ]
+}
+```
+
+### Standard Phase-to-Artifact Mapping
+
+| Phase | Artifact | Promise Tag |
+|-------|----------|-------------|
+| phase-0 | 00_product_discovery.md | DISCOVERY_COMPLETE |
+| phase-1 | 01_case_brief.md | CASE_EXPLORED |
+| phase-2 | 02_research_findings.md | RESEARCH_PARANOID_PASSED |
+| phase-2.5 | 02.5_trend_brief.md | CJM_VALIDATED |
+| phase-3 | 03_solution_strategy.md | SOLUTION_DESIGNED |
+| phase-4 | 04_architecture.md | ARCHITECTURE_DEFINED |
+| phase-5 | 05_presentation_content.md | PRESENTATION_READY |
+
+### Example Complete Chain
+
+```json
+{
+  "research_slug": "bank_kc_automation",
+  "created_at": "2026-03-01T10:00:00Z",
+  "last_updated": "2026-03-01T14:30:00Z",
+  "chain": [
+    {
+      "sequence": 0,
+      "phase": "phase-0",
+      "artifact": "00_product_discovery.md",
+      "hash": "sha256:a1b2c3d4e5f6789012345678901234567890123456789012345678901234abcd",
+      "previous_hash": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
+      "timestamp": "2026-03-01T10:00:00Z",
+      "promise_tag": "DISCOVERY_COMPLETE"
+    },
+    {
+      "sequence": 1,
+      "phase": "phase-1",
+      "artifact": "01_case_brief.md",
+      "hash": "sha256:b2c3d4e5f67890123456789012345678901234567890123456789012345bef01",
+      "previous_hash": "sha256:a1b2c3d4e5f6789012345678901234567890123456789012345678901234abcd",
+      "timestamp": "2026-03-01T10:30:00Z",
+      "promise_tag": "CASE_EXPLORED"
+    }
+  ]
+}
+```
+
+## Section 5: Chain Repair
+
+If an artifact is legitimately modified (e.g., user requested changes during checkpoint review), the chain must be repaired:
+
+1. Re-hash the modified artifact using the previous record's hash
+2. Update the modified record's hash in `.witness-chain.json`
+3. Re-hash ALL subsequent records (cascade update)
+4. Update `last_updated` timestamp
+
+This is expected and normal. The witness chain protects against *undetected* modifications, not against all modifications.
+
+## Section 6: Platform Compatibility
+
+| Platform | Command | Notes |
+|----------|---------|-------|
+| Linux (Ubuntu, Debian, etc.) | `sha256sum` | Available by default in coreutils |
+| macOS | `shasum -a 256` | Available by default |
+| Windows (WSL) | `sha256sum` | Available in WSL |
+| Windows (Git Bash) | `sha256sum` | Available in Git for Windows |
+| Alpine Linux | `sha256sum` | Available in coreutils |
+
+The protocol should attempt `sha256sum` first, then fall back to `shasum -a 256`.