groove-dev 0.27.175 → 0.27.179

package/AUTONOMOUS_AGENT_FRAMEWORK.md

@@ -0,0 +1,654 @@
+# Autonomous Claude Agent Framework — Build Plan
+
+A general-purpose framework for deploying Claude Code as an autonomous employee that
+runs 24/7, iterates on complex multi-step tasks, learns from failures, and communicates
+progress — without human intervention between cycles.
+
+Built and validated on the GRAFT MoE training pipeline (10+ autonomous cycles, code
+modifications, experiment design, failure analysis, and creative problem-solving).
+
+---
+
+## Architecture Overview
+
+Five components working together:
+
+```
+┌─────────────┐    cron     ┌──────────────┐   spawns    ┌───────────┐
+│   Crontab   │ ──────────> │ Orchestrator │ ──────────> │ Claude CLI│
+│  (heartbeat)│             │  (Python)    │             │ (decision │
+└─────────────┘             └──────┬───────┘             │  maker)   │
+                                   │                     └─────┬─────┘
+                            reads/writes                       │
+                                   │                    reads/writes/
+                                   v                    executes
+                            ┌──────────────┐                   │
+                            │  State Layer │ <─────────────────┘
+                            │              │
+                            │ - state.json │    ┌───────────────┐
+                            │ - journal    │    │  Work Layer   │
+                            │ - roadmap    │    │               │
+                            │ - prompt     │    │ - scripts     │
+                            │ - CLAUDE.md  │    │ - outputs     │
+                            └──────────────┘    │ - logs        │
+                                                │ - email       │
+                                                └───────────────┘
+```
+
+**Human communicates IN** by editing state files (prompt, roadmap, state.json).
+**Agent communicates OUT** via email, logs, and state file updates.
+No blocking. No polling. No awaiting approval. Fire and forget.
+
+---
+
+## Component 1: The Orchestrator
+
+**What it is:** A Python script triggered by cron. It does NOT make decisions — it checks
+status, detects transitions, and hands off to Claude when something needs a decision.
+
+**Core logic (pseudocode):**
+
+```python
+def main():
+    state = load_state()
+    phase = state["phase"]
+
+    # 1. Is something actively running?
+    if is_anything_running():
+        log_path = get_active_log(state, phase)
+        report_progress(phase, log_path)
+
+        # Stale process detection
+        if log_is_stale(log_path, max_age_hours=4):
+            handoff_to_claude("Process stuck — log stale for 4+ hours")
+        return
+
+    # 2. Nothing running — a transition happened. Hand off to Claude.
+    situation = detect_situation(state, phase)  # completed / failed / idle
+    handoff_to_claude(situation)
+```
+
+**Key design decisions:**
+
+- **Orchestrator is dumb, Claude is smart.** The orchestrator is a ~200-line state machine
+  that detects "something changed" and calls Claude. All intelligence lives in the prompt.
+  This means you can change Claude's behavior entirely by editing the prompt file — no code
+  deploys needed.
+
+- **Stale process detection.** Long-running jobs (training, builds, deploys) write to log
+  files. If the log hasn't been updated in N hours, the process is stuck. The orchestrator
+  detects this and hands off to Claude to diagnose and fix it.
+
+- **Graceful degradation.** If Claude CLI fails (timeout, API error), the orchestrator can
+  fall back to rule-based handlers for known phases. Optional but useful for critical paths.
+
+### The Handoff Function
+
+The most important function in the orchestrator:
+
+```python
+def handoff_to_claude(situation):
+    state = load_state()
+    journal = load_journal()
+
+    prompt = f"""
+    {read_file("system_prompt.md")}       # Identity, tools, rules, philosophy
+    
+    ## Current Situation
+    {situation}                             # What just happened
+    
+    ## Full Context
+    {build_context_summary(journal, state)} # History, findings, current settings
+    
+    ## Recent Log Output
+    {tail_log(get_active_log(state), 60)}   # Last 60 lines of relevant log
+    
+    Take ownership. Analyze, decide, and execute.
+    Update state and journal with your decisions.
+    """
+
+    result = subprocess.run(
+        ["claude", "-p", "--dangerously-skip-permissions", "--output-format", "text"],
+        input=prompt,
+        capture_output=True, text=True,
+        timeout=600,       # 10 min max per handoff
+        cwd=PROJECT_DIR,
+    )
+```
+
+**Critical:** `--dangerously-skip-permissions` is required for autonomous operation.
+Claude needs to run commands, edit files, and launch processes without approval prompts.
+
+### Orchestrator Robustness
+
+Things Claude WILL do that break naive orchestrators:
+
+- **Write non-dict values to state fields.** Claude might set `current_run` to a string
+  instead of a dict, or to `null`. Always use defensive accessors:
+  ```python
+  def safe_dict(state, key):
+      v = state.get(key)
+      return v if isinstance(v, dict) else {}
+  ```
+
+- **Set unexpected phase names.** Claude will invent phases like `"awaiting_decision"` or
+  `"levers_exhausted"`. Your orchestrator should handle unknown phases gracefully (hand off
+  to Claude again rather than crash).
+
+- **Write verbose essays into state fields.** Phase notes may be 500+ characters. Don't
+  assume they're short. Don't use them as dict keys.
+
+---
+
+## Component 2: The State Layer
+
+Five files that together give Claude full context every invocation. Claude has no memory
+between cron ticks — these files ARE its memory.
+
+### 2a. State File (state.json)
+
+The real-time status of the agent. Updated every invocation by both orchestrator and Claude.
+
+```json
+{
+  "phase": "training-cycle-3",
+  "phase_note": "Training on mixed dataset, lr=5e-5, 5000 steps",
+  "cycle": 3,
+  "current_run": {
+    "pid": 12345,
+    "settings": {"lr": "5e-5", "steps": 5000, "data": "mixed_316k.jsonl"},
+    "started": "2026-06-27T14:00:00"
+  },
+  "champion": {
+    "tag": "cycle2-best",
+    "score": 85.3,
+    "backups": ["model.safetensors.cycle2_best"]
+  },
+  "history": [
+    {"tag": "cycle1", "score": 78.2, "note": "baseline"},
+    {"tag": "cycle2", "score": 85.3, "note": "new best, lr=3e-5"}
+  ]
+}
+```
+
+**Design principles:**
+- One file, flat structure, human-readable
+- History array preserves every result — Claude uses this to avoid repeating failures
+- Champion section tracks the current best and where backups live
+- Phase + phase_note tell Claude exactly where things stand
+
+### 2b. System Prompt (prompt.md)
+
+Claude's identity, tools, rules, and philosophy. This is the most important file in the
+framework — it determines how Claude behaves. Sections:
+
+1. **Identity** — "You are the owner, not a helper. You decide and execute."
+2. **Philosophy** — How to think about failure, when to iterate vs pivot
+3. **Communication model** — How human communicates in (file edits), how agent communicates out (email)
+4. **Available tools** — Exact commands Claude can run, with examples
+5. **Rules/guardrails** — Hard limits that must never be violated
+6. **What to do now** — Decision tree for each invocation
+
+### 2c. Roadmap (roadmap.md)
+
+Multi-stage plan with graduation criteria per stage. Gives Claude long-term direction
+without micromanaging individual decisions.
+
+```markdown
+## Stage 1: Foundation ✅ COMPLETE
+**Graduation criteria (ALL must be met):**
+- [x] Metric A >= threshold
+- [x] Metric B >= threshold
+
+## Stage 2: Optimization ← CURRENT
+**Key activities:** (suggestions, not prescriptions)
+**Graduation criteria:**
+- [ ] Metric A >= higher_threshold
+- [ ] At least N experiments completed
+
+## Stage 3: Scale
+(future stages give Claude something to work toward)
+```
+
+**Why this matters:** Without a roadmap, Claude declares "goal reached" after hitting the
+first milestone. The roadmap gives it permanent forward momentum — there's always a next
+stage.
+
+### 2d. Project Journal (journal.json)
+
+Accumulated knowledge across all cycles. Every experiment, finding, and lesson. Claude
+reads this every invocation to avoid repeating mistakes.
+
+```json
+{
+  "entries": [
+    {
+      "date": "2026-06-25",
+      "event": "Cycle 2 completed — new best score",
+      "details": "lr=3e-5 with 3000 steps. Score improved 78.2 → 85.3.",
+      "lesson": "Lower LR works better than higher for this task."
+    }
+  ],
+  "key_findings": [
+    "lr > 1e-4 causes catastrophic regression",
+    "Data source A is saturated after 2 cycles"
+  ]
+}
+```
+
+### 2e. CLAUDE.md
+
+Standard Claude Code project context file. Quick reference for paths, commands, and
+constraints. Claude reads this automatically at the start of every invocation.
+
+---
+
+## Component 3: The Philosophy Prompt
+
+The most underrated component. Without the right framing, Claude will:
+- Declare "all options exhausted" after 2-3 failures
+- Park itself waiting for human approval
+- Treat each regression as a dead end instead of a data point
+
+### The Failure-as-Data Mindset
+
+This section in the prompt is what keeps Claude iterating:
+
+```markdown
+## How to Think About Failure
+
+A regression is not a reason to stop — it's a data point that tells you something.
+Your job is to extract the lesson and design a better experiment.
+
+When an experiment regresses:
+1. Ask WHY it failed. What specifically went wrong?
+2. Identify what VARIABLE you can change.
+3. Design the next experiment to test a DIFFERENT hypothesis.
+4. Revert to champion, launch the new experiment, keep moving.
+
+The pattern you must NEVER fall into: run one configuration → regression →
+declare the entire approach "closed with evidence" → park and wait.
+
+An approach is only exhausted when you've varied every independent variable
+and can explain WHY none of them can work based on first principles.
+```
+
+### The Never-Park Directive
+
+```markdown
+NEVER block on "awaiting approval." You are autonomous. If you reach a decision point:
+1. Email your analysis and recommendation
+2. Proceed with the safest promising option (you can always revert)
+3. Keep iterating — momentum matters more than permission
+```
+
+### Creative Freedom with Guardrails
+
+Don't prescribe what Claude should try — describe the search space and give it freedom
+to explore. Instead of:
+```
+# BAD: Prescriptive checklist
+Try experiment A, then B, then C, then D in order.
+```
+
+Do:
+```
+# GOOD: Context + freedom
+Here's what hasn't been explored yet and why it matters:
+- You've never tried X with Y combined
+- Every prior attempt used the same value for Z
+- The search space includes A, B, C dimensions
+
+You have creative freedom to try anything within the guardrails.
+Modify code, invent new strategies, combine approaches. The only
+requirement is that you keep moving forward.
+```
+
+---
+
+## Component 4: Communication Model
+
+### Human → Agent (Inbound)
+
+The human NEVER interacts with Claude directly during autonomous operation. Instead:
+
+1. Edit the **system prompt** to change behavior or add new directives
+2. Edit the **state file** to redirect Claude's focus or reset a phase
+3. Edit the **roadmap** to adjust goals or graduation criteria
+4. Edit **CLAUDE.md** to update project context
+
+Claude checks all these files every invocation. Changes take effect on the next cron tick.
+
+### Agent → Human (Outbound)
+
+1. **Email** — Progress reports after each cycle, alerts on failures, recommendations
+   at decision points. Claude sends these via a utility script.
+2. **Orchestrator log** — Timestamped record of every check-in, handoff, and Claude response.
+   The human monitors this for real-time status.
+3. **State file** — Claude updates phase, notes, and history after every action.
+4. **Journal** — Detailed write-up of what was tried, what was learned, what's next.
+
+### The "Monitor" Pattern (for the human)
+
+The human's workflow:
+```bash
+# Quick status check
+tail -20 output/orchestrator.log
+
+# What's Claude working on?
+cat output/state.json | jq '.phase, .phase_note'
+
+# Full history
+cat output/state.json | jq '.history'
+
+# Redirect Claude (edit a file, next cron tick picks it up)
+vim output/prompt.md
+```
+
+---
+
+## Component 5: Cron Configuration
+
+### Cadence Selection
+
+- **Too fast (< 30 min):** Multiple unnecessary check-ins per job. False stale-process
+  alarms. Wastes API calls. Claude gets confused by overlapping invocations.
+- **Too slow (> 2 hr):** Long idle gaps between cycle completion and next action.
+  Momentum loss.
+- **Sweet spot: 60 minutes.** Long enough for most jobs to complete between ticks.
+  Short enough to keep momentum. One hour is the natural unit.
+
+```crontab
+0 * * * * cd /path/to/project && python3 orchestrator.py >> output/orchestrator.log 2>&1
+```
+
+### Anti-Patterns
+
+- **Don't run the orchestrator AND Claude simultaneously.** The cron job should be the ONLY
+  entry point. If you manually trigger Claude while cron is running, they'll fight over
+  the state file.
+- **Don't let cron stack.** If a handoff takes 10 minutes and cron fires again, you get
+  two Claudes. Use a lockfile:
+  ```python
+  LOCKFILE = Path("/tmp/orchestrator.lock")
+  if LOCKFILE.exists():
+      pid = int(LOCKFILE.read_text())
+      if psutil.pid_exists(pid):
+          log("Previous run still active, skipping")
+          return
+  LOCKFILE.write_text(str(os.getpid()))
+  try:
+      main()
+  finally:
+      LOCKFILE.unlink(missing_ok=True)
+  ```
+
+---
+
+## Deployment Checklist (General)
+
+### 1. Define the Task
+
+- What is the agent trying to achieve? (measurable goal)
+- What are the stages/milestones? (roadmap)
+- What tools does the agent need? (commands, scripts, APIs)
+- What are the hard limits? (cost caps, rate limits, destructive actions to prohibit)
+- How do you measure success? (metrics, benchmarks, KPIs)
+
+### 2. Build the State Layer
+
+```
+project/
+  output/
+    state.json          # Current status, phase, history
+    journal.json        # Accumulated knowledge
+    orchestrator.log    # Timestamped activity log
+    prompt.md           # System prompt (identity + tools + rules + philosophy)
+    roadmap.md          # Staged plan with graduation criteria
+  CLAUDE.md             # Project context for Claude Code
+  orchestrator.py       # Cron-triggered orchestrator
+  utils/
+    send_report.py      # Email utility (or Slack, webhook, etc.)
+```
+
+### 3. Write the System Prompt
+
+Must include:
+- [ ] Identity ("you are the owner, not a helper")
+- [ ] Failure philosophy ("regression = data point, not dead end")
+- [ ] Communication model ("human edits files, you send email")
+- [ ] Available tools (exact commands with examples)
+- [ ] Guardrails (hard limits, cost caps, things that must never happen)
+- [ ] Decision tree ("what to do now" section)
+- [ ] Never-park directive ("never set phase to awaiting/idle/blocked")
+
+### 4. Write the Orchestrator
+
+Must handle:
+- [ ] State loading with defensive accessors
+- [ ] Process detection (is anything running?)
+- [ ] Progress reporting (parse logs for status)
+- [ ] Stale process detection (log age check)
+- [ ] Transition detection (completed / failed / idle)
+- [ ] Claude handoff (full context prompt → CLI → log response)
+- [ ] Timeout handling (Claude CLI can hang)
+- [ ] Unknown phase handling (don't crash on unexpected state)
+- [ ] Lockfile to prevent cron stacking
+
+### 5. Set Up Cron
+
+```bash
+# Install
+crontab -e
+# Add
+0 * * * * cd /path/to/project && python3 orchestrator.py >> output/orchestrator.log 2>&1
+```
+
+### 6. Test the Loop
+
+1. Set state to a known phase
+2. Run orchestrator manually: `python3 orchestrator.py`
+3. Verify Claude received context and took action
+4. Check state file was updated
+5. Check journal was updated
+6. Check email/notification was sent
+7. Run orchestrator again — verify it detects the new state correctly
+
+---
+
+## Example: Marketing Campaign Agent
+
+Applying this framework to a marketing use case:
+
+**Task:** Autonomous A/B testing and optimization of ad campaigns.
+
+**Roadmap stages:**
+1. Baseline — run initial campaigns, establish control metrics
+2. Hypothesis generation — analyze performance data, identify underperforming segments
+3. Test design — create variants, set up A/B splits
+4. Iteration — run tests, analyze results, promote winners, design next test
+5. Scale — expand winning strategies across channels
+
+**Tools:**
+```
+# Ad platform CLI / API calls
+python3 tools/create_variant.py --campaign X --variant "copy text" --audience Y
+python3 tools/get_metrics.py --campaign X --since 24h
+python3 tools/promote_winner.py --campaign X --variant Z
+python3 tools/pause_underperformer.py --campaign X --threshold 0.5
+```
+
+**Guardrails:**
+- Daily spend cap: never exceed $X per campaign
+- Never pause ALL variants simultaneously
+- Always keep the control running
+- Never modify campaigns flagged as "locked" in state
+- Minimum test duration: 48 hours before drawing conclusions
+
+**State:**
+```json
+{
+  "phase": "testing-round-3",
+  "active_tests": [
+    {"campaign": "summer-2026", "variant": "urgency-copy", "started": "2026-06-25", "ctr": 3.2}
+  ],
+  "champion": {"variant": "social-proof-v2", "ctr": 4.1, "cpa": 12.50},
+  "history": [...]
+}
+```
+
+**Philosophy prompt addition:**
+```
+A test that underperforms the control is not a failure — it eliminates a hypothesis.
+Your job is to generate better hypotheses. Analyze WHY the variant underperformed
+(wrong audience? wrong channel? wrong time of day? copy too long?) and design the
+next test to isolate that variable.
+```
+
+---
+
+## Groove Integration Plan
+
+To make this deployable from Groove as a first-class feature:
+
+### Core Abstraction
+
+```yaml
+# groove_agent.yaml — agent definition
+name: "campaign-optimizer"
+description: "Autonomous A/B testing for ad campaigns"
+cadence: "0 * * * *"  # hourly
+timeout: 600           # max 10 min per handoff
+
+state:
+  file: output/state.json
+  schema: schemas/campaign_state.json  # optional validation
+
+prompt:
+  file: prompt.md
+  
+roadmap:
+  file: roadmap.md
+
+tools:
+  - name: "get_metrics"
+    command: "python3 tools/get_metrics.py"
+  - name: "create_variant"
+    command: "python3 tools/create_variant.py"
+    guardrail: "spend_cap_check"
+
+guardrails:
+  spend_cap:
+    type: "pre_execution_check"
+    command: "python3 tools/check_spend.py --max 500"
+  
+notifications:
+  - type: email
+    to: ryan@axomlabs.ai
+  - type: slack
+    channel: "#agent-alerts"
+
+logs:
+  orchestrator: output/orchestrator.log
+  agent: output/agent_diagnosis.log
+```
+
+### Groove CLI
+
+```bash
+# Deploy a new agent
+groove agent create --config groove_agent.yaml
+
+# Check status
+groove agent status campaign-optimizer
+
+# View recent activity
+groove agent logs campaign-optimizer --tail 50
+
+# Redirect the agent (edit prompt, next tick picks it up)
+groove agent edit-prompt campaign-optimizer
+
+# Pause/resume
+groove agent pause campaign-optimizer
+groove agent resume campaign-optimizer
+
+# One-shot manual trigger (outside cron)
+groove agent trigger campaign-optimizer
+```
+
+### Groove Dashboard
+
+- **Agent list** — all deployed agents, current phase, last activity, health
+- **Activity feed** — real-time log of what each agent is doing
+- **Metrics** — per-agent: cycles completed, success rate, time between actions
+- **Controls** — pause, resume, trigger, edit prompt, view state
+- **Alerts** — agent stuck, cost threshold hit, consecutive failures
+
+### Template Library
+
+Pre-built agent templates for common use cases:
+
+| Template | Use Case | Cadence |
+|----------|----------|---------|
+| `ml-trainer` | Autonomous model training/eval loop | 60 min |
+| `campaign-optimizer` | Ad A/B testing and optimization | 60 min |
+| `content-pipeline` | Content generation, review, publish | 30 min |
+| `data-monitor` | Data quality monitoring and alerting | 15 min |
+| `code-reviewer` | Continuous PR review and feedback | 30 min |
+| `seo-optimizer` | Keyword research, content optimization | 120 min |
+| `report-generator` | Periodic business intelligence reports | daily |
+| `competitor-tracker` | Monitor competitor pricing/features | 120 min |
+
+Each template ships with: orchestrator.py, prompt.md, roadmap.md, state schema,
+example tools, and guardrails appropriate to the domain.
+
+---
+
+## Lessons Learned (from GRAFT deployment)
+
+### What Worked
+1. **File-based communication.** Editing a prompt file is the most natural way to redirect
+   an agent. No API calls, no webhooks, no auth — just edit a file and wait for the next tick.
+
+2. **Journal as memory.** Claude has no memory between invocations. The journal IS its
+   memory. Every lesson, every failure, every finding persists across cycles.
+
+3. **Champion/backup pattern.** Always preserve the best result before experimenting.
+   This makes Claude fearless — any experiment can be safely reverted.
+
+4. **Dumb orchestrator, smart agent.** Keeping decision logic in the prompt (not code)
+   means you can change behavior without deploying. Edit a markdown file and the agent
+   adapts on the next tick.
+
+5. **Email for outbound communication.** Simple, async, doesn't require the human to be
+   watching a dashboard. Claude emails when something important happens.
+
+### What Failed (and fixes)
+
+1. **Claude parks itself.** Without explicit never-park directives AND a failure philosophy,
+   Claude will declare "all options exhausted" after 2-3 regressions. Fix: the "How to Think
+   About Failure" section in the prompt. Frame the search space, not the checklist.
+
+2. **Prescriptive experiment lists.** Telling Claude exactly what to try (experiment A, B, C)
+   defeats the purpose. It follows the list mechanically instead of thinking creatively.
+   Fix: describe the unexplored dimensions and give creative freedom.
+
+3. **Stale log false alarms.** The orchestrator checked the training log during eval phases,
+   triggering stale process warnings that killed healthy eval runs. Fix: use the correct
+   log for the current phase.
+
+4. **Claude writes garbage to state.** It will write strings where you expect dicts, null
+   where you expect objects, 500-word essays in phase names. Fix: defensive accessors
+   everywhere. Never trust the shape of state values.
+
+5. **Cron too fast (30 min).** Caused 4-10 unnecessary check-ins per job, false alarms
+   during long-running tasks, and confusion from overlapping handoffs. Fix: 60 min cadence
+   for most tasks.
+
+6. **Claude blocks on approval.** Despite saying "you're autonomous," Claude would email
+   recommendations and set phase to "awaiting_decision." Fix: explicit directive + make
+   clear that human communicates via file edits, not email replies.
+
+7. **Zombie processes.** Long-running evaluation processes that hang or get orphaned. After
+   a week, had 42 zombie processes consuming 21GB RAM. Fix: stale process detection +
+   periodic cleanup.

package/node_modules/@groove-dev/cli/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@groove-dev/cli",
-  "version": "0.27.175",
+  "version": "0.27.179",
   "description": "GROOVE CLI — manage AI coding agents from your terminal",
   "license": "FSL-1.1-Apache-2.0",
   "type": "module",

package/node_modules/@groove-dev/daemon/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@groove-dev/daemon",
-  "version": "0.27.175",
+  "version": "0.27.179",
   "description": "GROOVE daemon — agent orchestration engine",
   "license": "FSL-1.1-Apache-2.0",
   "type": "module",

package/node_modules/@groove-dev/daemon/src/api.js

@@ -28,6 +28,7 @@ import { registerFileRoutes, resetEditorRoot } from './routes/files.js';
 import { registerNetworkRoutes } from './routes/network.js';
 import { registerScheduleRoutes } from './routes/schedules.js';
 import { registerInnerChatRoutes } from './routes/innerchat.js';
+import { registerAutoAgentRoutes } from './routes/auto-agents.js';
 
 const __dirname = dirname(fileURLToPath(import.meta.url));
 const pkgVersion = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version;
@@ -127,7 +128,7 @@ export function createApi(app, daemon) {
     next();
   });
 
-  app.use(express.json({ limit: '6mb' }));
+  app.use(express.json({ limit: '100mb' }));
 
   // Health check
   app.get('/api/health', (req, res) => {
@@ -177,6 +178,7 @@ export function createApi(app, daemon) {
   registerNetworkRoutes(app, daemon);
   registerScheduleRoutes(app, daemon);
   registerInnerChatRoutes(app, daemon);
+  registerAutoAgentRoutes(app, daemon);
 
 
   // Token usage