aether-colony 3.1.17 → 5.1.0

package/CHANGELOG.md

@@ -5,6 +5,185 @@ All notable changes to the Aether Colony project will be documented in this file
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [2.1.0] - 2026-03-24
+
+Six phases of production hardening (Phases 9-14) targeting reliability, maintainability, and developer experience.
+
+### Added
+- **State API facade** (`state-api.sh`) -- centralized COLONY_STATE.json access with lock/validate/migrate pattern
+- **Builder output verification** (`verify-claims`) -- cross-references builder file claims against filesystem to catch fabrication
+- **Error handling infrastructure** (`_aether_log_error`) -- structured error logging across all modules with `[error]` prefix
+- **SUPPRESS:OK convention** -- intentional error suppressions annotated for auditability (cleanup, read-default, existence-test, cross-platform, idempotent, validation)
+- **Per-phase research** -- scouts investigate domain knowledge before task decomposition (`/ant:plan` Step 3.6)
+- **Research context injection** -- builder and watcher prompts receive domain research during builds (16K character budget)
+- **Deprecation warning system** (`_deprecation_warning`) -- 18 dead subcommands emit stderr warnings with `[deprecated]` prefix
+- **Rolling state checkpoints** -- COLONY_STATE.json backed up before every build-wave (3 max retained)
+- **Context trimming notification** -- colony-prime emits visible notice when context is trimmed to stay within budget
+
+### Changed
+- **Monolith modularization** -- aether-utils.sh reduced from ~11,600 to ~5,200 lines (55% reduction)
+  - 9 domain modules extracted: flag, spawn, session, suggest, queen, swarm, learning, pheromone, state-api
+  - ~72 subcommands moved to domain modules, sourced on demand
+  - All 580+ tests passing with modularized structure
+- **Error handling overhaul** -- ~110 lazy error suppressions replaced with proper fallbacks, ~48 dangerous suppressions on state-mutation paths fixed with explicit error handling
+- **Continue-advance state writes** -- now go through locked subcommand (prevents concurrent corruption)
+- **Help JSON** -- updated with deprecation markers and Deprecated section
+- **Documentation accuracy** -- all docs/ files swept for stale counts, line numbers, and dates; comprehensive v2.1 changelog added
+
+### Fixed
+- **Hive wisdom type coercion** -- retrieval works regardless of string/number confidence format
+- **Midden race condition** -- PID-scoped temp files prevent concurrent write data loss
+- **Learning recovery** -- corrupted learning-observations.json auto-recovers from template
+- **Date-sensitive test failures** -- dynamic dates (futureISO) prevent recurring expiration failures
+- **Context-continuity test** -- pre-existing failure fixed (QUAL-09)
+
+## [2.0.0] - 2026-03-21
+
+### Added
+- **Hive Brain** — cross-colony wisdom sharing with domain-scoped retrieval, 200-entry LRU, multi-repo confidence boosting
+- **Autopilot (`/ant:run`)** — automated build-verify-advance loop with smart pausing
+- **User Preferences** — colony adapts to user communication style via QUEEN.md
+- **Quality gate agents** — Probe (coverage), Auditor (quality), Gatekeeper (security), Measurer (performance)
+- **`instinct-apply` subcommand** — tracks instinct usage with confidence adjustment
+- **`midden-review` subcommand** — lists unacknowledged failures grouped by category
+- **`midden-acknowledge` subcommand** — marks midden entries as addressed
+- **Pheromone content deduplication** — SHA-256 hashing prevents duplicate signals
+- **Pheromone prompt injection sanitization** — blocks LLM instruction override attempts
+- **Colony-prime token budget** — 8000/4000 char budget with priority-based truncation
+- **`/ant:patrol`** — comprehensive pre-seal audit of work against plan
+- **Colony pheromone exchange** — XML export/import for cross-colony signal sharing
+
+### Changed
+- **Monolith decomposition** — extracted hive-* (561 lines) and midden-* (260 lines) from aether-utils.sh to .aether/utils/
+- **Instinct trigger format** — triggers stored without "When" prefix; display/promotion adds it
+- **Learning-promote-auto** — recurrence-calibrated confidence formula
+
+### Fixed
+- **"When when" stutter** — fixed across 4 locations (seal.md, prompt-print, learning-promote-auto, oracle)
+- **Seal hive promotion** — jq path `.instincts[]` corrected to `.memory.instincts[]`
+- **instinct-create locking** — added trap-based lock to prevent concurrent write corruption
+- **midden-recent-failures limit** — positional parameter fix after extraction
+
+## [1.1.11] - 2026-02-26
+
+### Fixed
+- `midden-recent-failures` now reads `.entries[]` from midden.json instead of querying non-existent `.signals[]` — builders can finally see past failures
+
+### Added
+- `instinct-create` subcommand with deduplication and 30-instinct cap — programmatic instinct management replaces manual JSON manipulation
+- Midden context injected into builder prompts during build waves — workers avoid repeating past mistakes
+- Decisions auto-emit FEEDBACK pheromones (strength 0.65, TTL 30d) so builders see architectural choices
+- `context-update constraint` now handles `feedback` type alongside `redirect` and `focus`
+- Context-update calls added to Claude Code pheromone commands (focus, redirect, feedback)
+- OpenCode pheromone commands brought to parity with pheromone-write + context-update calls
+
+### Changed
+- `continue-advance.md` instinct extraction simplified to use `instinct-create` instead of inline JSON
+- `learning-promote-auto` now also creates instincts from promoted learnings
+
+## [1.1.10] - 2026-02-26
+
+### Changed
+- `lay-eggs` is now a pure bootstrap command — sets up `.aether/` in a repo from the global hub without starting a colony
+- `init` now assumes Aether is already set up and focuses only on starting a colony with a goal
+- All help files, rules, and workflow documentation updated to reflect the lay-eggs/init separation
+
+## [1.1.5] - 2026-02-23
+
+### Added
+- `validate-worker-response` command with caste-aware schema checks for builder/watcher/probe/scout payloads, wired into build playbooks and OpenCode build flow.
+- `queen-thresholds`, `spawn-efficiency`, `entropy-score`, `incident-rule-add`, and `eternal-store` commands.
+- Incident/self-evolution artifacts: `.aether/docs/INCIDENT_TEMPLATE.md`, `.aether/scripts/weekly-audit.sh`, `.aether/scripts/incident-test-add.sh`.
+- Additional regression coverage for spawn enforcement, state lock failure handling, threshold output, worker validation, spawn efficiency, eternal memory promotion, and entropy scoring.
+
+### Changed
+- Spawn guard now supports hard enforcement via `spawn-can-spawn [depth] --enforce`; worker and playbook guidance updated to use enforced mode.
+- Build orchestration guidance now refreshes `colony-prime --compact` before each worker dispatch so new pheromones/memory are visible mid-build.
+- Promotion thresholds consolidated behind a single source of truth and reused by learning promotion/proposal/memory metric paths.
+- `midden.template.json` extended with `spawn_metrics`.
+
+### Fixed
+- Locked + atomic state writes for critical mutation paths in `COLONY_STATE.json` handling (`error-add`, failure event append in `spawn-complete`, `grave-add`, schema migration, and lock-aware state-loader validation handoff).
+- Pheromone write path now sanitizes content, validates strength bounds, and uses lock + atomic write semantics.
+- `pheromone-expire` now promotes high-strength expired signals (`>0.8`) into eternal memory instead of silently decaying.
+- `detectDirtyRepo` now preserves porcelain status columns for the first line (prevents staged/unstaged misclassification when parsing git status output).
+
+---
+
+## v5.0.0 — Worker Emergence (2026-02-20)
+
+**Major milestone:** Every ant caste is now a real Claude Code subagent. 22 agents ship through the hub, ready for resolution via the Task tool.
+
+### Added
+- **22 Claude Code subagents**: Every ant caste is now a first-class subagent resolvable via the Task tool
+  - Core: Builder, Watcher
+  - Orchestration: Queen, Scout, Route-Setter, 4 Surveyors (nest, disciplines, pathogens, provisions)
+  - Specialists: Keeper, Tracker, Probe, Weaver, Auditor
+  - Niche: Chaos, Archaeologist, Gatekeeper, Includer, Measurer, Sage, Ambassador, Chronicler
+- **Agent distribution pipeline**: `npm install -g .` syncs agents to hub at `~/.aether/system/agents-claude/`, `aether update` delivers to target repos
+- **6 AVA tests for agent quality**: Frontmatter validation, tool restrictions, naming conventions, content standards
+- **repo-structure.md**: Quick re-orientation guide for the codebase
+
+### Fixed
+- **Bash line wrapping bug**: Fixed 58 instances across 7 command files where description text was inside code blocks causing "with: command not found" errors
+- **Lint regression test**: CLEAN-03 test scans both Claude and OpenCode command directories
+
+### Changed
+- **.aether/docs/ curated**: 8 keepers at root, 6 archived for reference
+- **README.md updated**: Action-oriented tone, v5.0 agent capabilities featured, caste table by tier
+- **ROADMAP.md and STATE.md**: v5.0 marked shipped, all 31 phases complete at 100%
+
+### Phases Shipped
+- **Phase 27**: Distribution Infrastructure + First Core Agents (4 plans)
+- **Phase 28**: Orchestration Layer + Surveyor Variants (3 plans)
+- **Phase 29**: Specialist Agents + Agent Tests (3 plans)
+- **Phase 30**: Niche Agents (3 plans)
+- **Phase 31**: Integration Verification + Cleanup (3 plans)
+
+---
+
+### Removed
+- Old planning phases 10-19 archived to docs/plans/ (completed phases)
+- Orphaned worktree salvage files moved to docs/worktree-salvage/
+
+---
+
+## v4.0.0 -- Distribution Simplification
+
+**Breaking change:** The `runtime/` staging directory has been removed. The npm package now reads directly from `.aether/`, with private directories (data/, dreams/, oracle/, etc.) excluded by `.aether/.npmignore`.
+
+### Changed
+- `.aether/` is now directly included in the npm package (private dirs excluded by `.aether/.npmignore`)
+- `bin/validate-package.sh` replaces `bin/sync-to-runtime.sh` — validates required files, no copying
+- Hub sync uses exclude-based approach instead of triplicate 59-62 file allowlists
+- Pre-commit hook repurposed for validation (no more runtime/ sync)
+- `aether update` uses `syncAetherToRepo` (exclude-based) for all system file distribution
+- All three distribution paths (system files, commands, agents) unified in `setupHub()`
+
+### Removed
+- `runtime/` staging directory — eliminated entirely
+- `bin/sync-to-runtime.sh` — replaced by validation-only script
+- `SYSTEM_FILES` allowlist arrays in cli.js and update-transaction.js
+- `copySystemFiles()` and `syncSystemFilesWithCleanup()` functions
+
+### Added
+- `bin/validate-package.sh` — pre-packaging validation with `--dry-run` mode
+- Private data exposure guard — blocks packaging if .npmignore doesn't cover private dirs
+- Migration message for users upgrading from v3.x
+- `npm pack --dry-run` recommended for verifying package contents
+
+### Fixed
+- ISSUE-004: Template path hardcoded to runtime/ — resolved by eliminating runtime/ entirely
+
+### Migration
+- Run `npm install -g aether-colony` to get v4.0
+- Your colony state and data are unaffected
+- The only change is how the package is built — distributed content is identical
+
+---
+
 ## [3.1.5] - 2026-02-15
 
 ### Fixed
@@ -218,3 +397,180 @@ Development releases (versions 2.0.0-2.4.2) building toward stable release. Key
 - 2026-02-12: CHANGELOG.md — added repo-local path migration entry
 - 2026-02-12: README.md — Updated to describe repo-local .aether/ architecture; removed global ~/.aether/ runtime references, restructured File Structure section with repo-local paths primary
 - 2026-02-13: bin/cli.js, update.md — Added orphan cleanup (syncDirWithCleanup), git dirty-file detection with --force stash, --dry-run preview, hub manifest generation
+
+---
+
+## Colony Work Log
+
+The following entries are automatically generated by the colony during work phases.
+
+
+## 2026-02-21
+
+### Phase 37 — Plan 02
+
+- **Files:** `aether-utils.sh`, `CHANGELOG.md`
+- **Decisions:** Created changelog-append function; Added changelog-collect-plan-data helper
+- **What Worked:** Function works correctly
+- **Requirements:** LOG-01 addressed
+
+### Phase 37 — Plan 99
+
+- **Files:** `test.md`
+- **Decisions:** Test
+- **What Worked:** Works
+- **Requirements:** TEST-01 addressed
+
+### Phase 0 — Plan 
+
+## 2026-03-20
+
+### Phase 0 — Plan 01
+
+- **Files:** `.aether/aether-utils.sh`, `tests/bash/test-hive-init.sh`, `tests/bash/test-hive-read.sh`, `tests/bash/test-hive-integration.sh`, `tests/integration/hive-store.test.js`, `tests/unit/colony-state.test.js`
+- **Decisions:** Hub-level lock isolation for shared files; Reuse pheromone-write sanitization pattern for user input
+- **What Worked:** hive-init creates wisdom.json; hive-store adds/merges with dedup; hive-read filters by domain; Cross-repo lock fix
+
+### Phase 0 — Plan 01
+
+- **Files:** `.aether/aether-utils.sh`, `tests/bash/test-hive-abstract.sh`, `tests/bash/test-hive-promote.sh`, `tests/bash/test-hive-abstraction-pipeline.sh`
+- **Decisions:** Orchestrator delegation via bash $0; Path stripping regex for abstraction
+- **What Worked:** hive-abstract generalizes text; hive-promote orchestrates pipeline; Multi-repo merge works
+
+### Phase 04 — Plan 01
+
+- **Files:** `.claude/commands/ant/seal.md`, `.opencode/commands/ant/seal.md`, `tests/bash/test-seal-hive-promotion.sh`
+- **Decisions:** Use --text and --source-repo for hive-promote API (not --instinct)
+- **What Worked:** Archaeology pre-build scan catches API mismatches; Hive promotion non-blocking in seal ceremony
+
+### Phase 05 — Plan 01
+
+- **Files:** `.aether/aether-utils.sh`, `tests/bash/test-hive-confidence-boost.sh`
+- **Decisions:** Confidence boosts at 2/3/4+ repo thresholds using max() for never-downgrade
+- **What Worked:** Atomic jq pipelines preserve lock safety; awk for float math follows codebase precedent
+
+### Phase 06 — Plan 01
+
+- **Files:** `CLAUDE.md`
+- **Decisions:** Document all Hive Brain features in CLAUDE.md
+- **What Worked:** Chaos testing docs against code reveals accuracy gaps
+
+### Phase 0 — Plan 00
+
+- **Files:** `.aether/aether-utils.sh`, `CLAUDE.md`, `.claude/commands/ant/seal.md`, `.opencode/commands/ant/seal.md`
+- **Decisions:** Colony sealed at Crowned Anthill; Build the Hive Brain — cross-colony wisdom intelligence layer
+- **What Worked:** 6 phases completed; 16 wisdom proposals promoted to QUEEN.md; Hive Brain fully operational with domain-scoped retrieval and multi-repo confidence boosting
+
+## 2026-03-21
+
+### Phase 0 — Plan 01
+
+- **Files:** `.aether/aether-utils.sh`, `.claude/commands/ant/seal.md`, `.opencode/commands/ant/seal.md`
+- **Decisions:** Fix when-when stutter across 4 locations; Fix seal.md jq path .instincts to .memory.instincts; Add instinct-apply subcommand; Fix instinct-create locking bug
+- **What Worked:** Archaeologist pre-scan catches latent bugs; Trap-based locking prevents deadlocks; 549 tests pass
+
+### Phase 0 — Plan 01
+
+- **Files:** `.aether/aether-utils.sh`
+- **Decisions:** Add midden-review subcommand; Add midden-acknowledge subcommand
+- **What Worked:** Midden feedback loop closed; 6 tests pass; Trap-based locking for acknowledge
+
+### Phase 0 — Plan 01
+
+- **Files:** `.aether/aether-utils.sh`, `.aether/utils/hive.sh`, `.aether/utils/midden.sh`
+- **Decisions:** Extract hive-* subcommands to hive.sh; Extract midden-* subcommands to midden.sh; Fix limit arg bug in midden-recent-failures
+- **What Worked:** 12004 to 11221 lines (-783); Source-and-delegate pattern proven; 576 tests pass
+
+### Phase 0 — Plan 01
+
+- **Files:** `TO-DOS.md`, `README.md`, `CHANGELOG.md`, `CLAUDE.md`
+- **Decisions:** Audit TO-DOS (11 completed, 10 pending); Update README to v2.0.0; Write CHANGELOG [2.0.0]; Update CLAUDE.md counts
+- **What Worked:** All docs reflect v2.0.0 reality; No stale version references
+
+### Phase 0 — Plan 01
+
+- **Files:** `package.json`, `.aether/version.json`
+- **Decisions:** Bump version to 2.0.0; Push to hub
+- **What Worked:** 542 tests pass; Package validated; Hub install verified
+
+### Phase 0 — Plan 00
+
+- **Files:** `.aether/aether-utils.sh`, `.aether/utils/hive.sh`, `.aether/utils/midden.sh`, `CLAUDE.md`, `README.md`, `package.json`
+- **Decisions:** Colony sealed at Crowned Anthill; Ship-ready Aether v2
+- **What Worked:** 5 phases completed; 22 wisdom proposals promoted; v2.0.0 on hub
+
+## 2026-03-22
+
+### Phase 0 — Plan 00
+
+- **Files:** `.aether/utils/skills.sh`, `.aether/skills/`, `bin/cli.js`, `.claude/commands/ant/skill-create.md`
+- **Decisions:** Colony sealed at Crowned Anthill; Implement Aether Skills Layer — smart-matched colony and domain skills for workers
+- **What Worked:** 5 phases completed; 28 skills created (10 colony + 18 domain); skills.sh engine (502 lines); Build pipeline skill injection; Hub distribution with manifest protection; Colony wisdom promoted to QUEEN.md
+
+## 2026-03-23
+
+### Phase 0 — Plan 00
+
+- **Files:** `.aether/utils/oracle/oracle.sh`, `.aether/.npmignore`, `bin/cli.js`, `.claude/commands/ant/oracle.md`
+- **Decisions:** Colony sealed at Crowned Anthill; Fix oracle.sh distribution to installed repos
+- **What Worked:** 4 phases completed; Oracle tmux launch path fixed; Hub state leak prevented
+
+## 2026-03-27
+
+### Phase 0 — Plan 00
+
+- **Files:** `colony-state.template.json`, `colony-state-reset.jq.template`, `seal.md`, `entomb.md`, `emoji-audit.sh`, `colony-visuals/SKILL.md`, `aether-utils.sh`, `CLAUDE.md`
+- **Decisions:** Colony sealed at Crowned Anthill v1; Add versioning to seal/entomb lifecycle and enforce consistent emoji usage
+- **What Worked:** 5 phases completed; Colony wisdom promoted to QUEEN.md; 24 new tests added
+
+### Phase 0 — Plan 00
+
+- **Files:** `queen.sh`, `aether-utils.sh`, `midden.sh`, `aether-sage.md`, `test-queen-charter.test.sh`, `test-midden-bridge.sh`, `test-aether-utils.sh`
+- **Decisions:** Colony sealed at Crowned Anthill v2; Fix seal ceremony audit issues
+- **What Worked:** 5 phases completed; 4 instincts promoted to hive; Colony wisdom promoted to QUEEN.md
+
+### Phase 0 — Plan 00
+
+- **Files:** `init.md`, `seal.md`, `build-prep.md`, `build-verify.md`, `build-wave.md`, `build-context.md`, `continue-gates.md`, `continue-verify.md`, `colony-visuals/SKILL.md`
+- **Decisions:** Colony sealed at Crowned Anthill; Enforce consistent visual styling across all Aether commands
+- **What Worked:** 5 phases completed; Banners standardized; Spawn announcements unified; Emoji map expanded; Progress bars added
+
+## 2026-03-28
+
+### Phase 0 — Plan 01
+
+- **Files:** `tests/bash/test-aether-utils.sh`
+- **Decisions:** Replace hardcoded dates with dynamic cross-platform computation
+- **What Worked:** Dynamic dates prevent time-based test degradation
+
+### Phase 2 — Plan 01
+
+- **Files:** `queen.sh`
+- **Decisions:** Fix trap composition, JSON escaping, local declarations in queen.sh
+- **What Worked:** Verification passed with 616 tests; Auditor found 3 HIGH pre-existing issues fixed
+
+### Phase 0 — Plan 01
+
+- **Files:** `queen.sh`
+- **Decisions:** Replace sed c-command with head/tail; Add empty-file safety guards; Fix sources and priming flags
+- **What Worked:** head/tail proven safer than sed c on macOS; empty-file guard prevents data destruction
+
+## 2026-03-29
+
+### Phase 4 — Plan 01
+
+- **Files:** `init.md`, `seal.md`, `entomb.md`
+- **Decisions:** Lifecycle commands handle colony_version via template system; Command parity maintained across Claude and OpenCode
+- **What Worked:** Verification of init/seal/entomb colony_version handling; Cross-platform seal commit synthesis and push prompts confirmed
+
+### Phase 0 — Plan 06
+
+- **Files:** `spawn-tree.sh`, `spawn.sh`, `spawn-tree.test.js`
+- **Decisions:** Replace O(n^2) bash loops with single-pass awk; Add jq validation to wrapper functions; Update test fixtures to 7-field format
+- **What Worked:** awk single-pass parsing eliminates 4000+ subprocess forks; Test fixtures now match production format
+- **Requirements:** spawn-tree.sh, spawn.sh, spawn-tree.test.js addressed
+
+### Phase 7 — Plan 01
+
+- **Files:** `.aether/data/AUDIT-REPORT.md`
+- **Decisions:** audit-report-corrected