aether-colony 3.1.4 → 3.1.15

package/runtime/docs/codebase-review.md

@@ -0,0 +1,399 @@
+# Aether Colony System - Comprehensive Review Report
+
+**Research Date:** 2026-02-14
+**Researcher:** Oracle Ant
+**Iterations:** 9
+**Confidence Level:** 95%
+
+---
+
+## Executive Summary
+
+This report presents a comprehensive read-only review of the Aether colony system. The system is a sophisticated multi-agent framework with 34 slash commands, 4 OpenCode agents, a comprehensive CLI, and extensive utility functions. While the architecture is well-designed, several critical issues have been identified that impact functionality and consistency.
+
+### Key Findings
+
+1. **State Inconsistency**: The colony state shows "INITIALIZING" but phase 7 is marked "completed"
+2. **Path Inconsistency**: HANDOFF.md is written to `.aether/HANDOFF.md` but read from `HANDOFF.md`
+3. **Subagent Type Mismatch**: plan.md uses "general-purpose" but the system expects "general"
+4. **Missing State Fields**: References to milestone and workers fields that don't exist in COLONY_STATE.json
+
+---
+
+## Core Components Overview
+
+### 1. Slash Commands (34 commands)
+**Location:** `/Users/callumcowie/repos/Aether/.claude/commands/ant/`
+
+The slash commands form the primary user interface for the colony system:
+
+**Lifecycle Commands:**
+- `init.md` - Initialize colony with goal
+- `plan.md` - Generate project plan (50-iteration research loop)
+- `build.md` - Execute phase with workers
+- `continue.md` - Complete phase and advance (6-phase verification loop)
+- `pause-colony.md` - Save session state
+- `resume-colony.md` - Restore session state
+- `status.md` - Display colony status
+
+**Management Commands:**
+- `flags.md` - Manage blockers/issues/notes
+- `watch.md` - Real-time activity monitoring
+- `swarm.md` - Parallel bug investigation
+- `oracle.md` - Deep research (this command)
+
+### 2. OpenCode Agents (4 agents)
+**Location:** `/Users/callumcowie/repos/Aether/.opencode/agents/`
+
+- `aether-queen.md` - Orchestrator agent
+- `aether-builder.md` - Implementation agent
+- `aether-scout.md` - Research agent
+- `aether-watcher.md` - Verification agent
+
+**Note:** These agents exist but are not integrated with the slash command system.
+
+### 3. CLI (Node.js)
+**Location:** `/Users/callumcowie/repos/Aether/bin/cli.js`
+
+Comprehensive CLI with 17 library modules:
+- `errors.js` - Structured error handling
+- `file-lock.js` - Concurrent access control
+- `model-profiles.js` - Model selection and routing
+- `telemetry.js` - Usage tracking
+- `state-sync.js` - State reconciliation
+- `update-transaction.js` - Atomic updates with rollback
+
+### 4. Utility Layer (Bash)
+**Location:** `/Users/callumcowie/repos/Aether/.aether/aether-utils.sh`
+
+50+ subcommands for colony operations:
+- State management (load-state, unload-state, validate-state)
+- Error handling (error-add, error-pattern-check, error-summary)
+- Activity logging (activity-log, spawn-log, spawn-complete)
+- Flag management (flag-add, flag-check-blockers, flag-list)
+- Model profiles (model-profile, model-get, model-list)
+
+### 5. State Files
+**Location:** `/Users/callumcowie/repos/Aether/.aether/data/`
+
+- `COLONY_STATE.json` - Unified colony state (v3.0)
+- `constraints.json` - Focus and constraint signals
+- `flags.json` - Blockers, issues, and notes
+- `activity.log` - Activity stream
+- `spawn-tree.txt` - Worker spawn tracking
+- `telemetry.json` - Model performance data
+
+---
+
+## Critical Issues (Require Immediate Attention)
+
+### 1. State Inconsistency
+**Severity:** HIGH
+**Location:** `.aether/data/COLONY_STATE.json:4`
+
+**Problem:**
+The state field is set to "INITIALIZING" but the current_phase is 7 with status "completed". According to the state machine defined in status.md (line 124), valid states are: IDLE, READY, EXECUTING, PLANNING.
+
+**Current State:**
+```json
+{
+  "state": "INITIALIZING",
+  "current_phase": 7,
+  "plan": {
+    "phases": [
+      {"number": 7, "name": "Core Reliability", "status": "completed"}
+    ]
+  }
+}
+```
+
+**Fix Required:**
+Update the state field to reflect actual status:
+```json
+{
+  "state": "READY"
+}
+```
+
+### 2. HANDOFF.md Path Inconsistency
+**Severity:** HIGH
+**Location:** Multiple command files
+
+**Problem:**
+- `pause-colony.md` (line 38): Writes to `.aether/HANDOFF.md`
+- `continue.md`, `plan.md`, `status.md`: Look for `HANDOFF.md` in root
+
+This breaks the pause/resume functionality.
+
+**Files Affected:**
+- `.claude/commands/ant/pause-colony.md`
+- `.claude/commands/ant/continue.md`
+- `.claude/commands/ant/plan.md`
+- `.claude/commands/ant/status.md`
+- `.claude/commands/ant/resume-colony.md`
+
+**Fix Required:**
+Standardize all references to `.aether/HANDOFF.md`:
+
+```bash
+# In continue.md, plan.md, status.md - change:
+Read HANDOFF.md
+# To:
+Read .aether/HANDOFF.md
+
+# In resume-colony.md - change:
+rm -f .aether/HANDOFF.md
+# (This is already correct, just verify)
+```
+
+### 3. Subagent Type Mismatch
+**Severity:** HIGH
+**Location:** `.claude/commands/ant/plan.md:126`
+
+**Problem:**
+plan.md spawns scouts with `subagent_type="general-purpose"` but the OpenCode agents use `subagent_type: "general"` (see aether-queen.md line 45).
+
+**Current Code:**
+```markdown
+Spawn Research Ant (Scout) via Task tool with subagent_type="general-purpose":
+```
+
+**Fix Required:**
+Change to match the working pattern:
+```markdown
+Spawn Research Ant (Scout) via Task tool with subagent_type="general":
+```
+
+---
+
+## Medium Priority Issues
+
+### 4. Pause-Colony Step Numbering Error
+**Severity:** MEDIUM
+**Location:** `.claude/commands/ant/pause-colony.md:72-82`
+
+**Problem:**
+Step 4.6 (Set Paused Flag) appears before Step 4.5 (Commit Suggestion) in the document, causing confusion about execution order.
+
+**Fix Required:**
+Swap the step numbers:
+- Step 4.5: Set Paused Flag
+- Step 4.6: Commit Suggestion (Optional)
+
+### 5. Resume-Colony References Non-Existent Workers Field
+**Severity:** MEDIUM
+**Location:** `.claude/commands/ant/resume-colony.md:70-79`
+
+**Problem:**
+The command displays worker status from a `workers` object that doesn't exist in COLONY_STATE.json.
+
+**Current Display Logic:**
+```markdown
+WORKERS
+  If ALL workers have "idle" status, display:
+    All 6 workers idle -- colony ready
+```
+
+**Fix Required:**
+Either:
+1. Add workers tracking to COLONY_STATE.json schema, OR
+2. Remove the workers display section from resume-colony.md
+
+### 6. Status.md References Non-Existent Milestone Field
+**Severity:** MEDIUM
+**Location:** `.claude/commands/ant/status.md:127-128`
+
+**Problem:**
+The command references `milestone` and `milestone_updated_at` fields that don't exist in COLONY_STATE.json.
+
+**Fix Required:**
+Either:
+1. Add milestone tracking fields to COLONY_STATE.json:
+   ```json
+   {
+     "milestone": "First Mound",
+     "milestone_updated_at": "2026-02-14T00:00:00Z"
+   }
+   ```
+2. OR remove milestone display from status.md
+
+### 7. Hardcoded Proxy Auth Token
+**Severity:** MEDIUM
+**Location:** `.aether/model-profiles.yaml:99`
+
+**Problem:**
+The LiteLLM proxy auth token is hardcoded as 'sk-litellm-local'.
+
+**Current:**
+```yaml
+proxy:
+  endpoint: 'http://localhost:4000'
+  auth_token: sk-litellm-local
+```
+
+**Fix Required:**
+Use environment variable with fallback:
+```yaml
+proxy:
+  endpoint: 'http://localhost:4000'
+  auth_token: ${LITELLM_AUTH_TOKEN:-sk-litellm-local}
+```
+
+---
+
+## Low Priority Issues
+
+### 8. Test Flags Pollution
+**Severity:** LOW
+**Location:** `.aether/data/flags.json`
+
+**Problem:**
+8 test flags from development are still present in the production data file.
+
+**Fix Required:**
+Archive test flags or reset flags.json:
+```bash
+# Option 1: Archive
+mv .aether/data/flags.json .aether/data/flags.json.backup.$(date +%s)
+echo '{"version": 1, "flags": []}' > .aether/data/flags.json
+
+# Option 2: Keep only unresolved flags
+```
+
+### 9. Verify-Castes.md Caste List Sync
+**Severity:** LOW
+**Location:** `.claude/commands/ant/verify-castes.md`
+
+**Problem:**
+The caste list in verify-castes.md may not match the full workers.md specification.
+
+**Fix Required:**
+Compare caste list with `.aether/workers.md` and sync any differences.
+
+---
+
+## Missing Features / Gaps
+
+### Gap 1: Signal/Pheromone Management
+**Impact:** MEDIUM
+**Location:** COLONY_STATE.json has "signals" array
+
+**Problem:**
+The COLONY_STATE.json has a "signals" array for pheromone signals, but no command manages it directly.
+
+**Suggested Solution:**
+Create `/ant:signal` command for CRUD operations:
+- `/ant:signal add <type> <content>` - Add signal
+- `/ant:signal list` - List active signals
+- `/ant:signal decay <id>` - Mark signal for decay
+
+### Gap 2: Milestone Tracking
+**Impact:** MEDIUM
+
+**Problem:**
+The milestone system is defined (6 milestones) but not tracked in state.
+
+**Suggested Solution:**
+Add to COLONY_STATE.json:
+```json
+{
+  "milestone": "First Mound",
+  "milestone_updated_at": "2026-02-14T00:00:00Z"
+}
+```
+
+### Gap 3: Worker Tracking
+**Impact:** LOW
+
+**Problem:**
+resume-colony.md references workers but no tracking exists.
+
+**Suggested Solution:**
+Either add workers object to state or remove from display:
+```json
+{
+  "workers": {
+    "builder": {"status": "idle"},
+    "watcher": {"status": "idle"}
+  }
+}
+```
+
+### Gap 4: OpenCode Integration
+**Impact:** LOW
+
+**Problem:**
+OpenCode agents exist but are not integrated with slash commands.
+
+**Suggested Solution:**
+Either:
+1. Integrate agents into slash command workflow, OR
+2. Deprecate and remove OpenCode agents if not needed
+
+---
+
+## Architecture Strengths
+
+1. **State Versioning**: Auto-upgrade from v1.0/v2.0 to v3.0
+2. **Concurrency Control**: File locking with load-state/unload-state pattern
+3. **Atomic Updates**: update-transaction.js with rollback capability
+4. **Observability**: Comprehensive logging (activity.log, spawn-tree.txt, telemetry.json)
+5. **Quality Gates**: 6-phase verification loop in continue.md
+6. **Model Routing**: Task-based routing to appropriate models
+7. **Error Handling**: Structured errors with recovery suggestions
+
+---
+
+## Actionable Fix Checklist
+
+### Immediate (HIGH Priority)
+- [ ] Fix COLONY_STATE.json state field: "INITIALIZING" → "READY"
+- [ ] Standardize HANDOFF.md path in all commands to `.aether/HANDOFF.md`
+- [ ] Change plan.md subagent_type from "general-purpose" to "general"
+
+### Short-term (MEDIUM Priority)
+- [ ] Fix pause-colony.md step numbering (4.5/4.6 swap)
+- [ ] Add milestone fields to COLONY_STATE.json OR remove from status.md
+- [ ] Add workers field to COLONY_STATE.json OR remove from resume-colony.md
+- [ ] Use environment variable for proxy auth token
+
+### Long-term (LOW Priority)
+- [ ] Clean up test flags from flags.json
+- [ ] Create /ant:signal command for pheromone management
+- [ ] Sync verify-castes.md with workers.md
+- [ ] Evaluate OpenCode agent integration
+
+---
+
+## File Paths Reference
+
+### Critical Files
+- `/Users/callumcowie/repos/Aether/.aether/data/COLONY_STATE.json` - Colony state
+- `/Users/callumcowie/repos/Aether/.aether/data/flags.json` - Flags data
+- `/Users/callumcowie/repos/Aether/.aether/model-profiles.yaml` - Model configuration
+
+### Command Files
+- `/Users/callumcowie/repos/Aether/.claude/commands/ant/init.md`
+- `/Users/callumcowie/repos/Aether/.claude/commands/ant/plan.md`
+- `/Users/callumcowie/repos/Aether/.claude/commands/ant/build.md`
+- `/Users/callumcowie/repos/Aether/.claude/commands/ant/continue.md`
+- `/Users/callumcowie/repos/Aether/.claude/commands/ant/pause-colony.md`
+- `/Users/callumcowie/repos/Aether/.claude/commands/ant/resume-colony.md`
+- `/Users/callumcowie/repos/Aether/.claude/commands/ant/status.md`
+
+### System Files
+- `/Users/callumcowie/repos/Aether/.aether/aether-utils.sh` - Utility layer
+- `/Users/callumcowie/repos/Aether/bin/cli.js` - CLI entry point
+- `/Users/callumcowie/repos/Aether/.aether/workers.md` - Worker specifications
+
+---
+
+## Conclusion
+
+The Aether colony system is a well-architected framework with sophisticated state management, quality controls, and observability. The identified issues are primarily consistency problems rather than fundamental design flaws. With the fixes outlined in this report, the system should operate reliably and maintain data integrity across all components.
+
+**Estimated Fix Time:** 2-4 hours for all HIGH and MEDIUM priority issues.
+
+---
+
+*Report generated by Oracle Ant - Aether Colony Research System*

package/runtime/docs/command-sync.md

@@ -0,0 +1,164 @@
+# Command Sync Strategy
+
+This document describes how Aether slash commands are distributed to Claude Code and OpenCode, and the bulletproof sync mechanisms that ensure consistency.
+
+---
+
+## Distribution Model
+
+### Claude Code: Global Sync
+
+Aether commands are synced to the global Claude Code config directory:
+
+```
+~/.claude/commands/ant/
+```
+
+This is done via `aether install` which copies commands from the npm package to the global location. The sync uses **hash-based comparison** to only copy files that have changed.
+
+### OpenCode: Repo-Local Only
+
+OpenCode **does not** have a global discovery mechanism for slash commands. Commands must exist in the repo-local directory:
+
+```
+.opencode/commands/ant/
+```
+
+This is why Aether maintains parallel command directories:
+- `.claude/commands/ant/` — Claude Code commands
+- `.opencode/commands/ant/` — OpenCode commands (repo-local)
+
+---
+
+## Why the Strategies Differ
+
+| Feature | Claude Code | OpenCode |
+|---------|-------------|----------|
+| Global command discovery | Yes (`~/.claude/commands/`) | No |
+| Repo-local commands | Supported | Required |
+| Namespace isolation | `/ant:` prefix | `/ant:` prefix |
+
+**Key insight:** OpenCode's architecture requires repo-local commands. There is no equivalent to `~/.claude/commands/` that provides automatic slash command discovery. This is a fundamental platform difference.
+
+---
+
+## Hash-Based Idempotent Sync
+
+The sync system in `bin/cli.js` uses **hash comparison before copying**:
+
+```javascript
+// Hash comparison: only copy if file doesn't exist or hash differs
+let shouldCopy = true;
+if (fs.existsSync(destPath)) {
+  const srcHash = hashFileSync(srcPath);
+  const destHash = hashFileSync(destPath);
+  if (srcHash === destHash) {
+    shouldCopy = false;
+    skipped++;
+  }
+}
+
+if (shouldCopy) {
+  fs.copyFileSync(srcPath, destPath);
+}
+```
+
+**Why this matters:**
+- **Idempotent:** Running `aether install` multiple times produces the same result
+- **Efficient:** Unchanged files are skipped (visible in logs as "skipped")
+- **Bulletproof:** No unnecessary writes reduce the risk of corruption
+
+---
+
+## Environment Variable Validation
+
+Before constructing any paths that use the user's home directory, the code validates the HOME environment variable:
+
+```javascript
+const HOME = process.env.HOME || process.env.USERPROFILE;
+if (!HOME) {
+  console.error('Error: HOME environment variable is not set');
+  console.error('Please ensure HOME or USERPROFILE is defined');
+  process.exit(1);
+}
+
+// Now safe to use
+const COMMANDS_DEST = path.join(HOME, '.claude', 'commands', 'ant');
+```
+
+**Why this matters:**
+- Prevents crashes on systems where HOME is not set
+- Provides clear error message instead of cryptic path errors
+- Supports both Unix-like systems (HOME) and Windows (USERPROFILE)
+
+---
+
+## Verification Commands
+
+### Check Sync Status
+
+To verify commands are in sync between Claude Code and OpenCode within the repo:
+
+```bash
+./bin/generate-commands.sh check
+```
+
+This performs two passes:
+1. **Pass 1:** File count and name comparison
+2. **Pass 2:** SHA-1 hash comparison for content-level drift detection
+
+### Verify Global Installation
+
+To verify global Claude Code commands are installed:
+
+```bash
+ls ~/.claude/commands/ant/
+```
+
+### Verify Global OpenCode Commands
+
+To verify global OpenCode commands (if previously synced):
+
+```bash
+ls ~/.config/opencode/commands/ant/
+```
+
+---
+
+## Sync Workflow
+
+```
+┌─────────────────┐
+│  Write commands │
+│  in .claude/    │
+└────────┬────────┘
+         │
+         ▼
+┌─────────────────────────────────────┐
+│  aether install                     │
+│  (syncs to ~/.claude/commands/ant/)│
+└────────┬────────────────────────────┘
+         │
+         ▼
+┌─────────────────────────────────────┐
+│  Hash-based copy                    │
+│  (only copies if hash differs)      │
+└─────────────────────────────────────┘
+```
+
+---
+
+## Anti-Patterns to Avoid
+
+1. **Unconditional copy:** Never copy files without checking if they changed
+2. **Skip hash comparison:** Always use hash comparison for idempotency
+3. **Ignore HOME validation:** Always validate HOME before path construction
+4. **Assume OpenCode global:** OpenCode does not support global command discovery
+
+---
+
+## See Also
+
+- `bin/cli.js` — Implementation of hash-based sync
+- `bin/generate-commands.sh` — Repo-level sync verification
+- `.aether/docs/namespace.md` — Namespace isolation strategy

package/runtime/docs/implementation-learnings.md

@@ -0,0 +1,89 @@
+# Implementation Learnings
+
+Valuable findings from Aether v3.1 development that apply to other projects.
+
+## Workflow Patterns
+
+1. **Claude Code global sync** works by copying commands from `.claude/commands/` to `~/.claude/commands/`
+   - This enables sharing commands across all projects
+   - Source of truth is the repo, hub is the distribution mechanism
+
+2. **OpenCode requires repo-local setup**
+   - Each repo that wants ant commands must set them up locally
+   - Unlike Claude Code, there's no global command directory
+
+3. **Hash comparison prevents unnecessary file writes**
+   - Comparing SHA-1 hashes before writes preserves file timestamps
+   - Reduces git noise from unchanged files
+
+4. **Namespace isolation via 'ant:' prefix**
+   - Prevents collisions with other agents
+   - Creates clear command ownership
+
+5. **CLI sync verification catches content drift**
+   - `generate-commands.sh check` uses SHA-1 checksums
+   - Detects when .claude/ and .opencode/ mirrors diverge
+
+## Error Handling Patterns
+
+- Use `json_err "$E_*" "message"` consistently (not bare strings)
+- Always release locks in error paths (use trap or explicit release)
+- Validate JSON before atomic writes
+- Use fallback json_err for backward compatibility
+
+## File Operation Patterns
+
+- Use `atomic_write()` for all state modifications
+- Acquire locks before reading/writing shared state
+- Create backups BEFORE validation for true atomicity
+- Use temp file + mv pattern for atomic operations
+
+## Model Routing Patterns
+
+- Validate LiteLLM proxy health before spawning
+- Support CLI --model override for one-time changes
+- Log actual model used per spawn for telemetry
+- Use task-based routing keywords for automatic selection
+
+## Codebase Patterns Discovered
+
+### Pattern 1: JSON Error Response Standard
+All commands output JSON with `{"ok": true/false, "result": ...}` or `{"ok": false, "error": ...}`
+
+### Pattern 2: Feature Degradation Pattern
+```bash
+if type feature_enabled &>/dev/null && ! feature_enabled "file_locking"; then
+  json_warn "W_DEGRADED" "File locking disabled - proceeding without lock"
+else
+  acquire_lock ...
+fi
+```
+
+### Pattern 3: Atomic Write Pattern
+All state modifications use `atomic_write()` from atomic-write.sh (temp file + mv)
+
+### Pattern 4: Trap-based Cleanup
+file-lock.sh uses `trap cleanup_locks EXIT TERM INT` for lock cleanup
+
+### Pattern 5: Inconsistent Error Code Evolution
+Commands added early use hardcoded strings; commands added later use `$E_VALIDATION_FAILED` constant. This creates inconsistency that should be standardized.
+
+## Connections Between Issues
+
+1. **BUG-005** (flag-auto-resolve lock leak) and **BUG-002** (flag-add lock leak) share the same root cause: inconsistent error handling patterns
+2. **BUG-004** and **ISSUE-001** are the same issue: missing error code constants
+3. **ISSUE-004** (template path) affects **GAP-006** (missing docs) - both relate to queen-* command usability
+4. **BUG-001** (awk apostrophes) affects the same lines that use Pattern 2 (feature degradation)
+
+## Fix Priority Matrix
+
+| Priority | Issues | Effort | Impact |
+|----------|--------|--------|--------|
+| Critical | BUG-005, BUG-011 | Low | Deadlock prevention |
+| High | BUG-002, BUG-008 | Low | Lock safety |
+| Medium | BUG-003, BUG-006, BUG-007 | Medium | Code quality |
+| Low | ISSUE-001 through ISSUE-007 | Medium | Consistency |
+
+---
+
+*Preserved from Phase 0 cleanup - 2026-02-15*