@dzhechkov/keysarium 1.1.2 → 1.3.0

package/templates/lib/background-workers.md

@@ -0,0 +1,278 @@
+# Background Workers Protocol
+
+> Core protocol for managing non-blocking background workers in Keysarium.
+> Workers are Agent tool calls with `run_in_background: true` -- not OS-level daemons.
+
+## Overview
+
+Background workers enable long-running operations (pattern consolidation, brain export, health checks) to execute without blocking the foreground session. Each worker is an isolated Agent that reads its instructions from a template file and writes output to a dedicated directory.
+
+## Worker Type Registry
+
+| Type | Description | Model | Template |
+|------|-------------|-------|----------|
+| `consolidate` | Scan completed researches for patterns, update TOOLKIT_HARVEST.md | sonnet | `lib/worker-templates/consolidate.md` |
+| `export-brain` | Non-blocking brain export to `.keysarium/exports/` | haiku | `lib/worker-templates/brain-export.md` |
+| `health-check` | Verify skill trust tiers, check stale data, report system health | haiku | `lib/worker-templates/health-check.md` |
+| `pattern-analysis` | Analyze reward data and trends across completed cases | sonnet | `lib/worker-templates/consolidate.md` (extended) |
+
+## Directory Structure
+
+```
+.keysarium/
+└── workers/
+    ├── registry.json                          ← Central registry (orchestrator-managed)
+    ├── wkr-{YYYYMMDD}-{HHmmss}-{type}/       ← Per-worker directory
+    │   ├── status.json                        ← Worker-managed status
+    │   ├── stop-requested                     ← Flag file (orchestrator writes to request stop)
+    │   ├── output/                            ← Worker output files
+    │   │   ├── patterns.json                  ← (consolidate)
+    │   │   ├── harvest-delta.md               ← (consolidate)
+    │   │   ├── health-report.json             ← (health-check)
+    │   │   └── brain-export-{timestamp}.json  ← (export-brain)
+    │   └── error.log                          ← Written on failure
+    └── ...
+```
+
+## Worker ID Format
+
+```
+wkr-{YYYYMMDD}-{HHmmss}-{type}
+```
+
+Example: `wkr-20260301-143022-consolidate`
+
+Generated by the orchestrator at launch time. Guaranteed unique within a day by including seconds.
+
+## Registry Schema
+
+The registry file (`.keysarium/workers/registry.json`) is managed ONLY by the orchestrator (`/workers` command). Workers NEVER modify this file.
+
+```json
+{
+  "version": "1.0",
+  "max_concurrent": 3,
+  "workers": [
+    {
+      "worker_id": "wkr-20260301-143022-consolidate",
+      "type": "consolidate",
+      "status": "running",
+      "model": "sonnet",
+      "started_at": "2026-03-01T14:30:22Z",
+      "completed_at": null,
+      "output_dir": ".keysarium/workers/wkr-20260301-143022-consolidate/",
+      "retry_count": 0
+    }
+  ]
+}
+```
+
+## Worker Status Schema
+
+Each worker writes its own `status.json` to its output directory:
+
+```json
+{
+  "worker_id": "wkr-20260301-143022-consolidate",
+  "type": "consolidate",
+  "status": "running",
+  "started_at": "2026-03-01T14:30:22Z",
+  "completed_at": null,
+  "progress": {
+    "phase": "scanning researches",
+    "items_processed": 3,
+    "total_items": 7
+  },
+  "error": null
+}
+```
+
+Status values: `starting`, `running`, `completing`, `completed`, `failed`, `stop-requested`, `stopped`
+
+## Launch Protocol
+
+When the orchestrator receives `/workers start <type>`:
+
+### Step 1: Validate
+
+1. Check that `<type>` is a valid worker type (see registry above)
+2. Read `.keysarium/workers/registry.json` (create if not exists)
+3. Count active workers (status = `starting` or `running`)
+4. If count >= `max_concurrent` (default 3), reject with message:
+   ```
+   Cannot start worker: maximum concurrent workers (3) reached.
+   Use `/workers status` to check active workers.
+   ```
+
+### Step 2: Create Worker Directory
+
+```
+mkdir -p .keysarium/workers/wkr-{id}/output/
+```
+
+### Step 3: Update Registry
+
+Add new entry to `registry.json` with status `starting`.
+
+### Step 4: Load Template
+
+Read the worker template from `lib/worker-templates/{type}.md`.
+
+### Step 5: Spawn Agent
+
+```
+Use the Agent tool with these parameters:
+- prompt: [Contents of worker template, with worker_id and output_dir injected]
+- model: [From routing table: haiku or sonnet]
+- run_in_background: true
+```
+
+The agent prompt MUST include:
+- The full worker template instructions
+- The `worker_id` for status file naming
+- The `output_dir` path for writing results
+- Instructions to write `status.json` at start and completion
+- Instructions to check for `stop-requested` file periodically
+
+### Step 6: Confirm
+
+Report to user:
+```
+Background worker started.
+  ID:   wkr-20260301-143022-consolidate
+  Type: consolidate
+  Model: sonnet
+  Output: .keysarium/workers/wkr-20260301-143022-consolidate/
+
+Use `/workers status` to check progress.
+```
+
+## Status Query Protocol
+
+When the orchestrator receives `/workers status`:
+
+1. Read `registry.json`
+2. For each worker entry:
+   a. Read `.keysarium/workers/{id}/status.json`
+   b. Update registry entry with latest status from worker's file
+3. Prune entries where `completed_at` is older than 24 hours
+4. Write updated `registry.json`
+5. Display formatted table:
+
+```
+═══════════════════════════════════════════════════════
+Background Workers Status
+═══════════════════════════════════════════════════════
+
+ID                                    Type          Status      Progress         Started
+wkr-20260301-143022-consolidate       consolidate   running     scanning (3/7)   14:30:22
+wkr-20260301-143500-health-check      health-check  completed   6/6 checks       14:35:00
+wkr-20260301-144000-export-brain      export-brain  completed   —                14:40:00
+
+Active: 1/3 | Completed: 2 | Failed: 0
+═══════════════════════════════════════════════════════
+```
+
+## Stop Protocol
+
+When the orchestrator receives `/workers stop <id>`:
+
+1. Verify worker exists in registry and has status `running`
+2. Write an empty file: `.keysarium/workers/{id}/stop-requested`
+3. Update registry entry status to `stop-requested`
+4. Report to user:
+   ```
+   Stop requested for worker wkr-20260301-143022-consolidate.
+   Worker will stop at next safe point.
+   ```
+
+The worker is responsible for:
+- Checking for `stop-requested` file between major operations
+- Writing final `status.json` with status `stopped`
+- Exiting gracefully
+
+If the worker does not stop within the Agent tool timeout (~10 minutes), it will terminate naturally.
+
+## List Protocol
+
+When the orchestrator receives `/workers list`:
+
+Display available worker types:
+
+```
+═══════════════════════════════════════════════════════
+Available Worker Types
+═══════════════════════════════════════════════════════
+
+Type              Model    Description
+consolidate       sonnet   Scan completed researches for patterns, update TOOLKIT_HARVEST.md
+export-brain      haiku    Non-blocking brain export to .keysarium/exports/
+health-check      haiku    Verify skill trust tiers, check for stale data
+pattern-analysis  sonnet   Analyze reward data and trends across cases
+
+Usage: /workers start <type>
+═══════════════════════════════════════════════════════
+```
+
+## Error Handling
+
+### Worker Failure
+
+When a worker encounters an unrecoverable error:
+
+1. Worker writes error details to `.keysarium/workers/{id}/error.log`
+2. Worker writes `status.json` with status `failed` and error message
+3. Agent completes (failure notification sent to foreground)
+
+### Retry Logic
+
+- Maximum 2 retries per original request
+- Retry creates a NEW worker (new ID) with `retry_count` incremented
+- If `retry_count >= 2`, mark as permanently failed
+- User must manually re-launch if needed beyond 2 retries
+
+### Registry Staleness
+
+- On `/workers status`, entries older than 24 hours with status `completed`, `failed`, or `stopped` are pruned
+- Entries with status `running` but no corresponding `status.json` are marked as `unknown`
+
+## Model Routing Table
+
+| Worker Type | Model | Rationale |
+|-------------|-------|-----------|
+| `consolidate` | sonnet | Pattern synthesis requires analytical reasoning |
+| `export-brain` | haiku | Primarily file reading and JSON assembly |
+| `health-check` | haiku | Structural checks, pattern matching |
+| `pattern-analysis` | sonnet | Trend analysis requires deeper reasoning |
+
+These mappings are enforced: the orchestrator MUST specify the correct model when spawning the Agent.
+
+## Worker Isolation Rules
+
+1. Workers MUST only write to their own `.keysarium/workers/{id}/` directory
+2. Workers MUST NOT modify files in `researches/` directories
+3. Workers MUST NOT modify files in `features/` directories
+4. Workers MUST NOT modify `CLAUDE.md` or any file in `.claude/`
+5. The `consolidate` worker MAY read `TOOLKIT_HARVEST.md` but writes updates to `output/harvest-delta.md` (the user decides whether to apply)
+6. Workers MUST NOT spawn sub-agents
+
+## Integration with Existing Commands
+
+### /harvest Integration
+
+The `consolidate` worker extends `/harvest` with background execution:
+- `/harvest` = foreground, interactive, with user confirmation
+- `/workers start consolidate` = background, non-interactive, writes delta for review
+
+### /brain-export Integration
+
+The `export-brain` worker provides a non-blocking alternative:
+- `/brain-export` = foreground, with progress display
+- `/workers start export-brain` = background, writes to `.keysarium/exports/`
+
+### Auto-launch Triggers (Future)
+
+Workers can be auto-launched based on triggers:
+- After 3rd completed `/casarium` case: auto-launch `consolidate`
+- After any `/bto-test` run: auto-launch `health-check`
+- These triggers are advisory and can be disabled in rules