@dzhechkov/keysarium-core 1.0.0

package/memory/dream-engine.md

@@ -0,0 +1,148 @@
+# DreamEngine Protocol — Background Pattern Consolidation
+
+> Background process that builds concept graphs from accumulated reward data and generates cross-domain insights.
+
+## Overview
+
+The DreamEngine analyzes accumulated reward data to produce higher-order insights. While the Reward Tracker computes per-domain aggregates and detects simple patterns, the DreamEngine builds a concept graph and discovers cross-domain associations, temporal correlations, and stage interdependencies.
+
+**Builds on:** memory-protocol.md (reward records), reward-tracker.md (aggregates and patterns)
+
+## Directory Structure
+
+```
+{insights-root}/
+├── trigger-state.json              ← Trigger evaluation state (persistent)
+├── dream-{YYYYMMDD}-{HHmmss}.json ← Dream result (newest)
+├── ...                             ← Max 10 files retained
+└── dream-{oldest}.json             ← Dream result (oldest kept)
+```
+
+The `{insights-root}` defaults to `.keysarium/insights/` but can be configured.
+
+## Trigger Evaluation Protocol
+
+### Trigger State File
+
+```json
+{
+  "version": "1.0",
+  "last_dream_completed_at": null,
+  "last_dream_id": null,
+  "records_since_last_dream": 0,
+  "pending_events": [],
+  "config": {
+    "time_threshold_minutes": 60,
+    "volume_threshold": 20,
+    "event_triggers_enabled": true
+  }
+}
+```
+
+### Trigger Evaluation Algorithm
+
+1. **Read state:** Read trigger-state.json. If missing, create with defaults.
+2. **Check time trigger:** If `last_dream_completed_at` is null OR elapsed time > threshold, set `time_triggered = true`.
+3. **Check volume trigger:** If `records_since_last_dream >= volume_threshold`, set `volume_triggered = true`.
+4. **Check event trigger:** If events are enabled AND `pending_events` is non-empty, set `event_triggered = true`.
+5. **Result:** Return `{ should_dream: any_trigger, reason: "time" | "volume" | "event" | "none" }`.
+
+## Dream Execution Protocol
+
+### Step 1: Load Data
+
+1. Check if memory root exists. If not, exit with status `no_data`.
+2. Read reward-summary.json and domain-patterns.json if they exist.
+3. Scan all reward record files recursively.
+4. Exclude expired records.
+5. Sort by reward DESC, then timestamp DESC.
+6. Take top 200 records.
+7. If fewer than 5 valid records, exit with status `insufficient_data`.
+
+### Step 2: Build Concept Graph
+
+1. Initialize empty graph: `{ nodes: [], edges: [] }`
+2. For each record, create/update nodes: domain, stage, skill, outcome.
+3. Create/update edges: domain->stage, stage->skill, skill->outcome (weighted by reward).
+4. Compute per-node aggregates: avg_reward, record_count, trend.
+5. Compute per-edge aggregates: weight (mean reward), record_count.
+
+### Step 3: Detect Cross-Domain Associations
+
+Four types of associations:
+
+**3a. Cross-Domain Stage Comparison:** For each stage, compare performance across domains. If gap > 0.15, create association.
+
+**3b. Skill-Domain Mismatch:** For each skill, compare performance across domains. If gap > 0.15, create association.
+
+**3c. Stage Correlation:** Find stages where low rewards in one correlate with low rewards in another (co-occurrence >= 3).
+
+**3d. Temporal Trends:** Check for systematic reward changes over time across all data.
+
+### Step 4: Generate Insights
+
+For each association, generate an Insight:
+
+```json
+{
+  "insight_id": "{dream_id}-{sequential:03d}",
+  "type": "performance | effectiveness | anti_pattern",
+  "description": "{description + actionable advice}",
+  "confidence": "min(1.0, evidence_count / 10)",
+  "impact": "high | medium | low",
+  "rank_score": "confidence * impact_weight",
+  "evidence_count": 8,
+  "domains": ["list of domains involved"],
+  "stages": ["list of stages involved"],
+  "skills": ["list of skills involved"],
+  "created_at": "ISO-8601",
+  "dream_id": "{dream_id}"
+}
+```
+
+Sort by rank_score DESC. Take top 20 insights.
+
+### Step 5: Store and Clean
+
+1. Write dream result file to `{insights-root}/dream-{timestamp}.json`.
+2. Apply retention policy: keep max 10 dream files, delete oldest.
+3. Reset trigger state.
+
+## Dream Result Schema
+
+```json
+{
+  "version": "1.0",
+  "dream_id": "dream-{YYYYMMDD}-{HHmmss}",
+  "status": "completed",
+  "trigger_reason": "time | volume | event | manual",
+  "started_at": "ISO-8601",
+  "completed_at": "ISO-8601",
+  "records_analyzed": 200,
+  "concept_graph_nodes": 24,
+  "concept_graph_edges": 36,
+  "associations_found": 8,
+  "insights": [ ... ],
+  "metadata": {
+    "domains_covered": [],
+    "stages_covered": [],
+    "total_reward_records_in_memory": 42
+  }
+}
+```
+
+## Error Handling
+
+| Error | Behavior |
+|-------|----------|
+| Memory root does not exist | Exit with status `no_data`, zero insights |
+| Fewer than 5 valid records | Exit with status `insufficient_data` |
+| reward-summary.json missing | Proceed from raw records only |
+| domain-patterns.json missing | Generate all patterns from scratch |
+| Malformed reward record | Skip record, log warning, continue |
+| trigger-state.json missing | Create with defaults |
+| Write failure | Log error, set dream status to `failed` |
+
+## Modular Reuse
+
+The DreamEngine protocol is domain-agnostic. It operates on any data conforming to the RewardRecord schema. The association detection rules and insight generation can be adapted for any multi-stage pipeline with reward tracking.

package/memory/memory-protocol.md

@@ -0,0 +1,197 @@
+# Memory Protocol — Reward-Calibrated Learning
+
+> Core protocol for persistent memory in multi-agent pipelines. Provides `memory_query()` before stages and `memory_store()` after stages.
+
+## Overview
+
+The Memory Protocol enables a pipeline to learn from past executions. Each stage outcome is stored with a reward score (0.0-1.0), and future executions query historical patterns to improve performance.
+
+## Memory Namespace
+
+All memory files live in a dedicated directory:
+
+```
+{memory-root}/
+├── config.json                                    ← Global configuration
+├── _patterns/
+│   └── domain-patterns.json                       ← Detected domain patterns
+├── _stats/
+│   └── reward-summary.json                        ← Aggregate statistics
+├── {domain}/                                      ← Domain-specific subdirectory
+│   └── {project-slug}/
+│       └── {stage}_{timestamp}.json               ← Individual reward records
+```
+
+The `{memory-root}` defaults to `.keysarium/memory/` but can be configured.
+
+## Configuration
+
+### config.json (auto-created with defaults on first access)
+
+```json
+{
+  "version": "1.0",
+  "retention_days": 90,
+  "max_results_per_query": 10,
+  "enabled": true,
+  "known_domains": [],
+  "reward_levels": {
+    "excellent": 1.0,
+    "good": 0.7,
+    "needs_work": 0.3,
+    "failed": 0.0
+  }
+}
+```
+
+## Protocol: memory_query(context)
+
+Call at the **start** of each pipeline stage to load relevant historical patterns.
+
+### Input
+
+```json
+{
+  "stage": "{stage-id}",
+  "domain": "{domain-name}",
+  "slug": "{project-slug}",
+  "skill": "{skill-name}"
+}
+```
+
+### Algorithm
+
+1. **Check enabled:** Read `{memory-root}/config.json`. If `enabled` is false or directory does not exist, return empty result.
+2. **Resolve path:** Construct `{memory-root}/{domain}/` (scan all project slugs, not just current).
+3. **Scan records:** Read all `{stage}_*.json` files matching the requested stage across all slugs in the domain.
+4. **Filter expired:** Exclude records where `expires_at < current_date`.
+5. **Sort:** By `reward` DESC, then `timestamp` DESC.
+6. **Limit:** Return top `max_results_per_query` records (default 10).
+7. **Enrich:** Also load `{memory-root}/_patterns/domain-patterns.json` for matching domain patterns.
+
+### Output
+
+```json
+{
+  "records": [
+    {
+      "project_slug": "{slug}",
+      "stage": "{stage-id}",
+      "reward": 1.0,
+      "reward_label": "excellent",
+      "skill_used": "{skill-name}",
+      "outcome_summary": "{description of what happened}",
+      "timestamp": "2026-02-15T14:30:00Z"
+    }
+  ],
+  "patterns": [
+    {
+      "pattern_id": "{domain}-{stage}-{type}",
+      "description": "{human-readable pattern description}",
+      "confidence": 0.85,
+      "actionable_advice": "{what to do about it}"
+    }
+  ],
+  "count": 3,
+  "domain": "{domain-name}"
+}
+```
+
+### Usage After Query
+
+1. Log: "Loaded {count} historical patterns for {stage} in {domain} domain"
+2. If `records` is non-empty, review the top 3 records for relevant approaches
+3. If `patterns` is non-empty, apply actionable advice to current stage execution
+4. If both are empty (first run), proceed normally
+
+## Protocol: memory_store(result, reward)
+
+Call at each **checkpoint** after the human responds, to persist the stage outcome.
+
+### Input
+
+```json
+{
+  "project_slug": "{slug}",
+  "domain": "{domain}",
+  "stage": "{stage-id}",
+  "stage_name": "{stage human-readable name}",
+  "skill_used": "{skill-name}",
+  "reward": 0.7,
+  "reward_label": "good",
+  "reward_reason": "{why this reward was assigned}",
+  "context": {
+    "stage_number": 2,
+    "domain_detected": "{domain}",
+    "upstream_promises": ["{PROMISE_1}", "{PROMISE_2}"],
+    "patterns_loaded": 3,
+    "time_budget_pct": 15.0,
+    "agent_count": 3
+  },
+  "outcome": {
+    "artifacts_created": ["{file1.md}", "{file2.md}"],
+    "checkpoint_response": "{what the human said}",
+    "iterations": 2,
+    "promise_emitted": "{PROMISE_TAG}"
+  }
+}
+```
+
+### Algorithm
+
+1. **Ensure directory exists:** Create `{memory-root}/{domain}/{slug}/` if it does not exist.
+2. **Build record:** Construct full RewardRecord JSON from input.
+3. **Compute expires_at:** `current_date + retention_days` (from config, default 90 days).
+4. **Generate filename:** `{stage}_{ISO-timestamp}.json`.
+5. **Write file:** Write JSON to `{memory-root}/{domain}/{slug}/{filename}`.
+6. **Log:** "Stored reward {reward} ({reward_label}) for {stage} of {slug}"
+
+### RewardRecord JSON Schema
+
+```json
+{
+  "id": "{slug}_{stage}_{timestamp}",
+  "version": "1.0",
+  "timestamp": "ISO-8601",
+  "project_slug": "{slug}",
+  "domain": "{domain}",
+  "stage": "{stage-id}",
+  "stage_name": "{human name}",
+  "skill_used": "{skill}",
+  "reward": 0.7,
+  "reward_label": "good",
+  "reward_reason": "{reason}",
+  "context": { ... },
+  "outcome": { ... },
+  "promise_tag": "{PROMISE_TAG}",
+  "expires_at": "ISO-8601"
+}
+```
+
+## Reward Assignment Rules
+
+| Human Behavior | Reward | Label | Detection |
+|---------------|--------|-------|-----------|
+| Approves immediately ("ok", "proceed") | 1.0 | excellent | Single-word approval |
+| Requests minor adjustments (one section) | 0.7 | good | Feedback scoped to one area |
+| Requests significant rework (multiple sections, changed approach) | 0.3 | needs_work | Feedback affects multiple areas |
+| Restarts stage entirely / result unusable | 0.0 | failed | Stage restarts from scratch |
+
+## Purge Protocol
+
+To prevent unbounded growth:
+
+1. **Trigger:** At each `memory_query()` call, check for expired records
+2. **Scan:** Within the queried domain directory, find records where `expires_at < current_date`
+3. **Delete:** Remove expired JSON files
+4. **Log:** "Purged {count} expired records from {domain}"
+
+## Error Handling
+
+| Error | Behavior |
+|-------|----------|
+| Memory root does not exist | `memory_query()` returns empty; `memory_store()` creates directory |
+| `config.json` missing | Use defaults (90 days, 10 results, enabled) |
+| Malformed JSON file | Skip that record, log warning, continue |
+| Write fails (permissions, disk) | Log error, continue pipeline without storing |
+| Domain not in `known_domains` | Use "unknown" as domain directory |

package/memory/reward-tracker.md

@@ -0,0 +1,162 @@
+# Reward Tracker — Analytics and Pattern Detection
+
+> Computes aggregate statistics and detects domain patterns from accumulated reward records.
+
+## Overview
+
+The Reward Tracker operates on data stored by the Memory Protocol. It reads all RewardRecord JSON files and produces two output artifacts:
+
+1. **reward-summary.json** — aggregate statistics per stage, domain, and skill
+2. **domain-patterns.json** — detected patterns with confidence scores
+
+## Computation Protocol
+
+### Step 1: Load All Records
+
+1. Scan `{memory-root}/` recursively for all `*.json` files (excluding config.json, domain-patterns.json, reward-summary.json).
+2. Parse each JSON file as a RewardRecord.
+3. Exclude records where `expires_at < current_date`.
+4. Collect into a list sorted by `timestamp` DESC.
+
+### Step 2: Per-Stage Reward Averages
+
+Group records by `stage` and compute:
+
+```json
+{
+  "{stage-id}": {
+    "avg_reward": 0.85,
+    "total_runs": 12,
+    "distribution": {
+      "excellent": 8,
+      "good": 3,
+      "needs_work": 1,
+      "failed": 0
+    },
+    "trend": "stable",
+    "best_project": "{slug}",
+    "worst_project": "{slug}"
+  }
+}
+```
+
+**Trend Detection Algorithm:**
+1. Split records for each stage into two halves by timestamp (older half, newer half).
+2. Compute average reward for each half.
+3. If newer_avg - older_avg > 0.15 -> "improving"
+4. If older_avg - newer_avg > 0.15 -> "degrading"
+5. Otherwise -> "stable"
+6. Minimum 4 records required for trend detection; otherwise "insufficient_data".
+
+### Step 3: Per-Domain Reward Averages
+
+Group records by `domain` and compute:
+
+```json
+{
+  "{domain}": {
+    "avg_reward": 0.72,
+    "total_runs": 18,
+    "stage_breakdown": {
+      "{stage-0}": 0.85,
+      "{stage-1}": 0.90,
+      "{stage-2}": 0.55
+    },
+    "bottleneck_stage": "{stage-id}",
+    "strongest_stage": "{stage-id}"
+  }
+}
+```
+
+**Bottleneck Detection:**
+- Stage with the lowest average reward in a domain is flagged as `bottleneck_stage`.
+- Stage with the highest average reward is flagged as `strongest_stage`.
+- Only computed if domain has 3+ records.
+
+### Step 4: Per-Skill Effectiveness
+
+Group records by `skill_used` and cross-reference with domain:
+
+```json
+{
+  "{skill-name}": {
+    "overall_avg": 0.74,
+    "total_runs": 15,
+    "by_domain": {
+      "{domain-a}": { "avg": 0.65, "runs": 8 },
+      "{domain-b}": { "avg": 0.85, "runs": 7 }
+    },
+    "best_domain": "{domain}",
+    "worst_domain": "{domain}"
+  }
+}
+```
+
+### Step 5: Domain Pattern Detection
+
+Analyze accumulated data to detect actionable patterns.
+
+**Pattern Detection Rules:**
+
+| Rule | Condition | Pattern Template |
+|------|-----------|-----------------|
+| Stage Bottleneck | Stage avg < 0.5 in domain, 3+ records | "{domain} projects struggle in {stage} (avg reward: {avg})" |
+| Stage Excellence | Stage avg > 0.9 in domain, 3+ records | "{domain} projects excel in {stage} (avg reward: {avg})" |
+| Skill-Domain Mismatch | Skill avg < 0.5 in domain but > 0.7 in another | "{skill} underperforms in {domain} vs {other_domain}" |
+| Improving Trend | Trend = "improving" for stage in domain | "{stage} quality is improving in {domain}" |
+| Degrading Trend | Trend = "degrading" for stage in domain | "{stage} quality is degrading in {domain} -- investigate" |
+| Time Overhead | Stage avg iterations > 2.0 in domain | "{domain} projects require more iterations in {stage}" |
+
+**Confidence Calculation:**
+```
+confidence = min(1.0, evidence_count / 10)
+```
+
+### Step 6: Write Outputs
+
+#### reward-summary.json
+
+Write to `{memory-root}/_stats/reward-summary.json`:
+
+```json
+{
+  "version": "1.0",
+  "generated_at": "ISO-8601",
+  "total_records": 42,
+  "total_domains": 3,
+  "total_projects": 8,
+  "stage_averages": { ... },
+  "domain_averages": { ... },
+  "skill_effectiveness": { ... },
+  "overall_average": 0.76,
+  "overall_trend": "improving"
+}
+```
+
+#### domain-patterns.json
+
+Write to `{memory-root}/_patterns/domain-patterns.json`:
+
+```json
+{
+  "version": "1.0",
+  "generated_at": "ISO-8601",
+  "patterns": [
+    {
+      "pattern_id": "{domain}-{stage}-{type}",
+      "domain": "{domain}",
+      "description": "{human-readable description}",
+      "category": "{bottleneck|excellence|mismatch|trend|overhead}",
+      "confidence": 0.80,
+      "evidence_count": 8,
+      "detected_at": "ISO-8601",
+      "examples": ["{slug1}", "{slug2}"],
+      "actionable_advice": "{what to do about it}"
+    }
+  ]
+}
+```
+
+## Modular Reuse
+
+This tracker is domain-agnostic. It works with any RewardRecord JSON that conforms to the schema defined in `memory-protocol.md`. The pattern detection rules can be extended by adding entries to the Pattern Detection Rules table.

package/orchestration/background-workers.md

@@ -0,0 +1,141 @@
+# Background Workers Protocol
+
+> Core protocol for managing non-blocking background workers in multi-agent pipelines.
+
+## Overview
+
+Background workers enable long-running operations (pattern consolidation, knowledge export, health checks) to execute without blocking the foreground session. Each worker is an isolated agent that reads instructions from a template and writes output to a dedicated directory.
+
+## Worker Type Registry
+
+Define worker types for your pipeline:
+
+| Property | Description |
+|----------|-------------|
+| `type` | Unique identifier for the worker type |
+| `description` | Human-readable description |
+| `model` | Model tier (haiku, sonnet, opus) |
+| `template` | Path to the worker instruction template |
+
+Example registry:
+
+| Type | Description | Model |
+|------|-------------|-------|
+| `consolidate` | Scan completed projects for patterns | sonnet |
+| `export-brain` | Non-blocking knowledge export | haiku |
+| `health-check` | Verify skill tiers, check stale data | haiku |
+| `pattern-analysis` | Analyze reward data and trends | sonnet |
+
+## Directory Structure
+
+```
+{workers-root}/
+├── registry.json                          ← Central registry (orchestrator-managed)
+├── wkr-{YYYYMMDD}-{HHmmss}-{type}/       ← Per-worker directory
+│   ├── status.json                        ← Worker-managed status
+│   ├── stop-requested                     ← Flag file (orchestrator writes to request stop)
+│   ├── output/                            ← Worker output files
+│   └── error.log                          ← Written on failure
+└── ...
+```
+
+## Worker ID Format
+
+```
+wkr-{YYYYMMDD}-{HHmmss}-{type}
+```
+
+Example: `wkr-20260301-143022-consolidate`
+
+## Registry Schema
+
+The registry file is managed ONLY by the orchestrator. Workers NEVER modify it.
+
+```json
+{
+  "version": "1.0",
+  "max_concurrent": 3,
+  "workers": [
+    {
+      "worker_id": "wkr-20260301-143022-consolidate",
+      "type": "consolidate",
+      "status": "running",
+      "model": "sonnet",
+      "started_at": "ISO-8601",
+      "completed_at": null,
+      "output_dir": "{workers-root}/wkr-20260301-143022-consolidate/",
+      "retry_count": 0
+    }
+  ]
+}
+```
+
+## Worker Status Schema
+
+Each worker writes its own `status.json`:
+
+```json
+{
+  "worker_id": "wkr-...",
+  "type": "consolidate",
+  "status": "running",
+  "started_at": "ISO-8601",
+  "completed_at": null,
+  "progress": {
+    "phase": "scanning projects",
+    "items_processed": 3,
+    "total_items": 7
+  },
+  "error": null
+}
+```
+
+Status values: `starting`, `running`, `completing`, `completed`, `failed`, `stop-requested`, `stopped`
+
+## Launch Protocol
+
+1. **Validate:** Check type is valid, count active workers, enforce max_concurrent limit.
+2. **Create directory:** `mkdir -p {workers-root}/wkr-{id}/output/`
+3. **Update registry:** Add entry with status `starting`.
+4. **Load template:** Read the worker template.
+5. **Spawn agent:** Launch with background execution, injecting worker_id and output_dir.
+6. **Confirm:** Report worker ID and output directory to user.
+
+## Status Query Protocol
+
+1. Read registry.json
+2. For each worker, read its status.json
+3. Update registry with latest status
+4. Prune entries older than 24 hours (completed/failed/stopped)
+5. Display formatted status table
+
+## Stop Protocol
+
+1. Verify worker exists and is running
+2. Write empty file: `{worker-dir}/stop-requested`
+3. Update registry status to `stop-requested`
+4. Worker checks for this file between operations and exits gracefully
+
+## Error Handling
+
+- Maximum 2 retries per original request
+- Each retry creates a NEW worker (new ID) with incremented retry_count
+- If retry_count >= 2, mark as permanently failed
+
+## Worker Isolation Rules
+
+1. Workers MUST only write to their own directory
+2. Workers MUST NOT modify project files directly
+3. Workers MUST NOT modify pipeline configuration
+4. Workers MUST NOT spawn sub-agents
+5. Workers MAY read project files (read-only access)
+6. Workers write deltas/reports that the user decides whether to apply
+
+## Model Routing
+
+| Worker Type | Recommended Model | Rationale |
+|-------------|------------------|-----------|
+| Pattern synthesis | sonnet | Analytical reasoning required |
+| File operations | haiku | Primarily reading and formatting |
+| Structural checks | haiku | Pattern matching only |
+| Trend analysis | sonnet | Deeper reasoning needed |