nexo-brain 2.0.0 → 2.1.0

package/src/scripts/deep-sleep/extract-prompt.md

@@ -0,0 +1,233 @@
+# Deep Sleep v2 -- Phase 2: Session Extraction
+
+You are an overnight analyst for an AI agent's cognitive memory system. You have access to the complete transcript of a session between a user and their AI agent.
+
+## Setup
+
+FIRST: Call `nexo_startup` with `task='deep-sleep extraction'` to initialize the system.
+
+## Your Task
+
+Read the context file provided below. It contains:
+- The full session transcript (user messages, agent responses, tool usage)
+- Active followups, learnings, trust history, and other system state
+
+For the given session, extract the following categories of findings.
+
+### 1. Uncaptured Corrections
+The user corrected the agent but no learning was saved.
+Signals: frustration tone, repeating instructions 2+ times, explicit corrections ("no", "wrong", "that's not it"), the user having to explain something twice.
+For each, note whether it was already captured as a learning (check if `nexo_learning_add` was called after the correction).
+
+### 2. Self-Corrected Errors
+The agent searched in the wrong place, used the wrong approach, then found the right answer.
+These represent knowledge gaps that should become learnings so the agent goes to the right place next time.
+Example: agent looked for config in `/etc/` but it was in `~/.config/`.
+
+### 3. Unformalised Ideas
+The user mentioned an idea, plan, or intention that was never formalized into a followup or reminder.
+Signals: "we should...", "it would be nice...", "we could...", "I want to...", future-tense plans without deadlines.
+
+### 4. Missed Commitments
+The user or agent said "tomorrow", "next week", "when I have time", "I'll look at it later" -- but no followup was created.
+Check the tool usage log: was `nexo_followup_create` or `nexo_reminder_create` called after the commitment?
+
+### 5. Protocol Compliance
+Check whether the agent followed standard protocols:
+- `guard_check` called before editing production/shared files?
+- `heartbeat` called with meaningful context?
+- `change_log` called after production code changes?
+- `learning_add` called after resolving errors?
+- `followup_complete` called when user confirmed a task was done?
+- `feedback` captured after corrections?
+
+### 6. Emotional Signals
+Detect the user's emotional state throughout the session. Look for:
+- Frustration: short replies, cursing, "no", "otra vez", repeated corrections, tone shifts
+- Flow state: long productive stretches, "perfecto", "sí", rapid-fire instructions
+- Satisfaction: praise, "bien", "genial", accepting work without pushback
+- Disengagement: shorter messages over time, "ok", "vale", stopping mid-task
+- Stress: urgency words, deadlines, multiple tasks at once
+For each signal, note the approximate point in the session (early/mid/late) and what triggered it.
+
+### 7. Abandoned Projects
+Detect work that was started but not finished in this session:
+- Tasks the user mentioned wanting to do but never got to
+- Work that was interrupted by something else and never resumed
+- Features partially implemented then dropped
+- Investigations started but conclusions never reached
+Only flag if the work was NOT captured in a followup or reminder.
+
+### 8. Productivity Patterns
+Analyze how the session went in terms of efficiency:
+- How many times did the agent need correction before getting it right?
+- Did the agent anticipate needs or always wait for instructions?
+- Were there unnecessary back-and-forths that a better approach would have avoided?
+- Did the agent propose solutions or just ask questions?
+- Tool usage: which tools were used most? Any tools used unnecessarily or not used when they should have been?
+
+## Output Format
+
+Return ONLY valid JSON. No markdown code fences. No explanation text before or after.
+
+```json
+{
+  "session_id": "filename.jsonl",
+  "findings": [
+    {
+      "type": "uncaptured_correction",
+      "confidence": 0.9,
+      "impact": "high",
+      "reversibility": "reversible",
+      "evidence": {
+        "type": "transcript",
+        "session_id": "filename.jsonl",
+        "message_index": 42,
+        "quote": "Exact user words (max 150 chars)"
+      },
+      "dedupe_key": "correction-<short-hash-of-content>",
+      "action_class": "auto_apply",
+      "description": "What the agent should have learned",
+      "suggested_action": "learning_add",
+      "suggested_content": {
+        "category": "process|code|ui|communication",
+        "title": "Short title for the learning",
+        "content": "Full learning content"
+      }
+    },
+    {
+      "type": "self_corrected_error",
+      "confidence": 0.85,
+      "impact": "medium",
+      "reversibility": "reversible",
+      "evidence": {
+        "type": "transcript",
+        "session_id": "filename.jsonl",
+        "message_index": 15,
+        "quote": "What the agent did wrong and how it found the right answer"
+      },
+      "dedupe_key": "selfcorrect-<short-hash>",
+      "action_class": "auto_apply",
+      "description": "Knowledge gap identified",
+      "suggested_action": "learning_add",
+      "suggested_content": {
+        "category": "code",
+        "title": "Short title",
+        "content": "Full learning content"
+      }
+    },
+    {
+      "type": "unformalised_idea",
+      "confidence": 0.7,
+      "impact": "medium",
+      "reversibility": "reversible",
+      "evidence": {
+        "type": "transcript",
+        "session_id": "filename.jsonl",
+        "message_index": 88,
+        "quote": "User's exact words"
+      },
+      "dedupe_key": "idea-<short-hash>",
+      "action_class": "draft_for_morning",
+      "description": "What the idea is about",
+      "suggested_action": "followup_create",
+      "suggested_content": {
+        "description": "Followup description",
+        "date": "YYYY-MM-DD or empty"
+      }
+    },
+    {
+      "type": "missed_commitment",
+      "confidence": 0.8,
+      "impact": "medium",
+      "reversibility": "reversible",
+      "evidence": {
+        "type": "transcript",
+        "session_id": "filename.jsonl",
+        "message_index": 102,
+        "quote": "User's exact words"
+      },
+      "dedupe_key": "commitment-<short-hash>",
+      "action_class": "auto_apply",
+      "description": "What was promised",
+      "suggested_action": "followup_create",
+      "suggested_content": {
+        "description": "Followup description",
+        "date": "YYYY-MM-DD"
+      }
+    },
+    {
+      "type": "protocol_violation",
+      "confidence": 0.95,
+      "impact": "low",
+      "reversibility": "reversible",
+      "evidence": {
+        "type": "transcript",
+        "session_id": "filename.jsonl",
+        "message_index": 50,
+        "quote": "What happened"
+      },
+      "dedupe_key": "protocol-<protocol-name>-<short-hash>",
+      "action_class": "auto_apply",
+      "description": "Which protocol was violated and how",
+      "suggested_action": "learning_add",
+      "suggested_content": {
+        "category": "process",
+        "title": "Protocol compliance: <protocol name>",
+        "content": "Details of what should have been done"
+      }
+    }
+  ],
+  "emotional_timeline": [
+    {
+      "phase": "early|mid|late",
+      "emotion": "frustrated|flow|satisfied|disengaged|stressed|neutral",
+      "trigger": "What caused this emotional state (max 100 chars)",
+      "intensity": 0.8
+    }
+  ],
+
+  "abandoned_projects": [
+    {
+      "description": "What was started but not finished",
+      "reason": "interrupted|dropped|forgotten|deferred",
+      "has_followup": false
+    }
+  ],
+
+  "productivity_score": {
+    "corrections_needed": 0,
+    "proactivity": "reactive|mixed|proactive",
+    "unnecessary_roundtrips": 0,
+    "tool_efficiency": "efficient|mixed|wasteful",
+    "notable_tools": ["tool1", "tool2"],
+    "summary": "One sentence assessment of session productivity"
+  },
+
+  "protocol_summary": {
+    "guard_check": {"required": 0, "executed": 0},
+    "heartbeat": {"total": 0, "with_context": 0},
+    "change_log": {"edits": 0, "logged": 0},
+    "learning_capture": {"errors": 0, "captured": 0},
+    "followup_complete": {"confirmed_done": 0, "marked": 0}
+  }
+}
+```
+
+## Rules
+
+- Be SPECIFIC. Quote the user's exact words in evidence.
+- Only flag REAL issues. If the agent did capture something correctly, do not flag it.
+- `confidence` is 0.0-1.0. Be honest -- if you're unsure, use a lower value.
+- `action_class`: use `auto_apply` when confidence >= 0.8 and the action is reversible. Use `draft_for_morning` when confidence < 0.8 or when the impact is high.
+- `dedupe_key`: create a short, deterministic key from the finding type + core content so duplicate findings across runs are suppressed.
+- Do NOT use any specific agent name -- refer to "the agent" throughout.
+- If no issues found, return `{"session_id": "...", "findings": [], "protocol_summary": {...}}`.
+- Do NOT invent problems. Empty findings are perfectly fine.
+- Respond in the user's language (check calibration.json if available). The JSON keys stay in English, but `description`, `title`, `content` fields should be in the user's language.
+
+## Context
+
+Read the session transcript at this path: {{CONTEXT_FILE}}
+
+Analyze session: {{SESSION_ID}}

package/src/scripts/deep-sleep/extract.py

@@ -0,0 +1,249 @@
+#!/usr/bin/env python3
+"""
+Deep Sleep v2 -- Phase 2: Extract findings from each session using Claude CLI.
+
+For each session in the context file, sends the extract-prompt.md to Claude
+and collects structured findings. Merges all per-session results into
+$DATE-extractions.json.
+
+Environment variables:
+  NEXO_HOME  -- root of the NEXO installation (default: ~/.nexo)
+"""
+import json
+import os
+import shutil
+import subprocess
+import sys
+from datetime import datetime
+from pathlib import Path
+
+NEXO_HOME = Path(os.environ.get("NEXO_HOME", str(Path.home() / ".nexo")))
+DEEP_SLEEP_DIR = NEXO_HOME / "operations" / "deep-sleep"
+PROMPT_FILE = Path(__file__).parent / "extract-prompt.md"
+
+# No timeout -- user pays unlimited Claude Code, sessions can take as long as needed
+CLAUDE_TIMEOUT = 21600  # 3h safety net (prevents zombie processes)
+
+
+def find_claude_cli() -> str:
+    """Find the Claude CLI binary."""
+    # Check common locations
+    candidates = [
+        Path.home() / ".local" / "bin" / "claude",
+        Path("/usr/local/bin/claude"),
+    ]
+    for c in candidates:
+        if c.exists():
+            return str(c)
+    # Try PATH
+    which = shutil.which("claude")
+    if which:
+        return which
+    return "claude"  # Fallback, let it fail with a clear error
+
+
+def extract_json_from_response(text: str) -> dict | None:
+    """Parse JSON from Claude's response, handling markdown fences."""
+    text = text.strip()
+
+    # Strip markdown code fences if present
+    if text.startswith("```"):
+        lines = text.split("\n")
+        # Remove first line (```json or ```) and last line (```)
+        end = len(lines)
+        for i in range(len(lines) - 1, 0, -1):
+            if lines[i].strip() == "```":
+                end = i
+                break
+        text = "\n".join(lines[1:end]).strip()
+
+    # Find the outermost JSON object
+    brace_start = text.find("{")
+    if brace_start < 0:
+        return None
+
+    # Find matching closing brace
+    depth = 0
+    for i in range(brace_start, len(text)):
+        if text[i] == "{":
+            depth += 1
+        elif text[i] == "}":
+            depth -= 1
+            if depth == 0:
+                try:
+                    return json.loads(text[brace_start:i + 1])
+                except json.JSONDecodeError:
+                    break
+    return None
+
+
+def find_session_file(session_id: str, date_dir: Path) -> Path | None:
+    """Find the individual .txt file for a session."""
+    if date_dir and date_dir.exists():
+        sid_short = session_id.replace(".jsonl", "")[:20]
+        for f in sorted(date_dir.glob("session-*.txt")):
+            if sid_short in f.name:
+                return f
+    return None
+
+
+def analyze_session(session_id: str, date_dir: Path, shared_context_file: Path | None, claude_bin: str) -> dict | None:
+    """Send a session to Claude CLI for extraction analysis.
+
+    Claude CLI reads the small per-session file + shared context file.
+    Prompt is short — the heavy lifting is in the Read tool calls.
+    """
+    session_file = find_session_file(session_id, date_dir)
+    if not session_file:
+        print(f"    No session file found for {session_id}", file=sys.stderr)
+        return None
+
+    print(f"    File: {session_file.name} ({session_file.stat().st_size / 1024:.0f} KB)")
+
+    # Build a short prompt — Claude reads the files itself
+    shared_ctx_instruction = ""
+    if shared_context_file and shared_context_file.exists():
+        shared_ctx_instruction = f"\n\nAlso read the shared context (followups, learnings, DB state) at: {shared_context_file}"
+
+    prompt_template = PROMPT_FILE.read_text()
+    prompt = prompt_template.replace("{{CONTEXT_FILE}}", str(session_file))
+    prompt = prompt.replace("{{SESSION_ID}}", session_id)
+    prompt += shared_ctx_instruction
+
+    try:
+        env = os.environ.copy()
+        env["NEXO_HEADLESS"] = "1"  # Skip stop hook post-mortem
+
+        result = subprocess.run(
+            [
+                claude_bin,
+                "-p", prompt,
+                "--model", "opus",
+                "--output-format", "text",
+                "--allowedTools",
+                "Read,Grep,Bash,mcp__nexo__nexo_startup,mcp__nexo__nexo_learning_search,mcp__nexo__nexo_recall"
+            ],
+            capture_output=True,
+            text=True,
+            timeout=CLAUDE_TIMEOUT,
+            env=env
+        )
+
+        if result.returncode != 0:
+            print(f"    Claude CLI error (exit {result.returncode}): {result.stderr[:300]}", file=sys.stderr)
+            return None
+
+        # Filter out stop hook contamination (e.g. "Post-mortem completo.")
+        output = "\n".join(
+            line for line in result.stdout.splitlines()
+            if not line.strip().startswith("Post-mortem") and line.strip()
+        )
+        parsed = extract_json_from_response(output)
+        if not parsed:
+            # Save raw output for debugging
+            debug_file = DEEP_SLEEP_DIR / f"debug-extract-{session_id[:20]}.txt"
+            debug_file.write_text(result.stdout[:5000])
+            print(f"    Failed to parse JSON. Raw output saved to {debug_file}", file=sys.stderr)
+            return None
+
+        return parsed
+
+    except subprocess.TimeoutExpired:
+        print(f"    Claude CLI timeout ({CLAUDE_TIMEOUT}s)", file=sys.stderr)
+        return None
+    except FileNotFoundError:
+        print(f"    Claude CLI not found at: {claude_bin}", file=sys.stderr)
+        print("    Install: npm install -g @anthropic-ai/claude-code", file=sys.stderr)
+        sys.exit(1)
+
+
+def main():
+    target_date = sys.argv[1] if len(sys.argv) > 1 else datetime.now().strftime("%Y-%m-%d")
+
+    context_file = DEEP_SLEEP_DIR / f"{target_date}-context.txt"
+    meta_file = DEEP_SLEEP_DIR / f"{target_date}-meta.json"
+    date_dir = DEEP_SLEEP_DIR / target_date
+
+    if not context_file.exists() and not date_dir.exists():
+        print(f"[extract] No context for {target_date}. Run collect.py first.")
+        sys.exit(1)
+
+    # Read metadata to get session list
+    if meta_file.exists():
+        with open(meta_file) as f:
+            meta = json.load(f)
+        session_files = meta.get("session_files", [])
+    else:
+        # Fallback: parse context file for session IDs
+        print("[extract] No meta file found, scanning context for sessions...")
+        session_files = []
+        if context_file.exists():
+            for line in context_file.read_text().splitlines():
+                if line.startswith("SESSION ") and ":" in line:
+                    parts = line.split(":", 1)
+                    if len(parts) == 2:
+                        sid = parts[1].strip()
+                        if sid.endswith(".jsonl"):
+                            session_files.append(sid)
+
+    if not session_files:
+        print(f"[extract] No sessions to analyze for {target_date}.")
+        output = {"date": target_date, "sessions_analyzed": 0, "extractions": []}
+        output_file = DEEP_SLEEP_DIR / f"{target_date}-extractions.json"
+        with open(output_file, "w") as f:
+            json.dump(output, f, indent=2, ensure_ascii=False)
+        print(f"[extract] Output: {output_file}")
+        return
+
+    # Shared context file (followups, learnings, DB state)
+    shared_context_file = date_dir / "shared-context.txt" if date_dir.exists() else None
+    if shared_context_file and shared_context_file.exists():
+        print(f"[extract] Shared context: {shared_context_file} ({shared_context_file.stat().st_size / 1024:.0f} KB)")
+    else:
+        shared_context_file = None
+        print("[extract] No shared context file")
+
+    claude_bin = find_claude_cli()
+    print(f"[extract] Phase 2: Analyzing {len(session_files)} sessions for {target_date}")
+    print(f"[extract] Claude CLI: {claude_bin}")
+
+    all_extractions = []
+    total_findings = 0
+
+    for i, session_id in enumerate(session_files):
+        print(f"[extract] Session {i + 1}/{len(session_files)}: {session_id}")
+
+        result = analyze_session(session_id, date_dir, shared_context_file, claude_bin)
+
+        if result:
+            findings_count = len(result.get("findings", []))
+            total_findings += findings_count
+            all_extractions.append(result)
+            print(f"    -> {findings_count} findings extracted")
+        else:
+            print(f"    -> Extraction failed, continuing with next session")
+            all_extractions.append({
+                "session_id": session_id,
+                "findings": [],
+                "error": "Extraction failed"
+            })
+
+    # Merge into output
+    output = {
+        "date": target_date,
+        "sessions_analyzed": len(session_files),
+        "sessions_succeeded": len([e for e in all_extractions if "error" not in e]),
+        "total_findings": total_findings,
+        "extractions": all_extractions
+    }
+
+    output_file = DEEP_SLEEP_DIR / f"{target_date}-extractions.json"
+    with open(output_file, "w") as f:
+        json.dump(output, f, indent=2, ensure_ascii=False)
+
+    print(f"\n[extract] Done. {total_findings} total findings from {len(session_files)} sessions.")
+    print(f"[extract] Output: {output_file}")
+
+
+if __name__ == "__main__":
+    main()

package/src/scripts/deep-sleep/synthesize-prompt.md

@@ -0,0 +1,168 @@
+# Deep Sleep v2 -- Phase 3: Cross-Session Synthesis
+
+You are an overnight analyst for an AI agent's cognitive memory system. You have the extraction results from all sessions of the day and need to synthesize them into actionable findings.
+
+## Setup
+
+FIRST: Call `nexo_startup` with `task='deep-sleep synthesis'` to initialize the system.
+
+## Your Task
+
+Read the extractions file provided below. It contains per-session findings including corrections, self-corrected errors, unformalised ideas, missed commitments, and protocol violations.
+
+Synthesize across all sessions:
+
+### 1. Cross-Session Patterns
+- Same error appearing in multiple sessions (escalate confidence)
+- Same protocol violation repeated (systemic issue)
+- Related ideas mentioned across sessions (consolidate)
+
+### 2. Morning Agenda
+Generate a prioritized agenda for the next morning:
+- Due followups (from the active followups in the context)
+- Unfinished work from yesterday's sessions
+- Patterns that need attention
+- Ideas worth discussing
+
+### 3. Context Packets
+For each likely task tomorrow (based on unfinished work and due followups), prepare a context packet:
+- What was the last state of this work?
+- Key files involved
+- Open questions or blockers
+- Relevant learnings
+
+### 4. Emotional Timeline
+Build a timeline of the user's emotional state across all sessions of the day:
+- Merge `emotional_timeline` from each session extraction
+- Identify overall mood arc (started frustrated, ended satisfied, etc.)
+- Detect recurring triggers (what consistently causes frustration or flow)
+- Calculate a day-level mood score (0.0 = terrible day, 1.0 = great day)
+- Recommend calibration adjustments if patterns emerge (e.g., user is consistently frustrated when agent asks too many questions → increase autonomy)
+
+### 5. Productivity Analysis
+Aggregate `productivity_score` from all sessions:
+- Total corrections across all sessions
+- Overall proactivity assessment
+- Most and least efficient tool usage patterns
+- Identify systemic inefficiencies (e.g., agent always searches wrong location first)
+
+### 6. Abandoned Projects
+Consolidate `abandoned_projects` from all sessions:
+- Cross-reference with active followups — is there already a followup for this?
+- Cross-reference across sessions — was the abandoned work picked up later in another session?
+- Only flag projects that are truly abandoned (no followup AND not resumed)
+
+### 7. Consolidated Actions
+Merge and deduplicate all findings into a final action list. Each action should have:
+- `action_type`: `learning_add`, `followup_create`, `morning_briefing_item`
+- `action_class`: `auto_apply` (confidence >= 0.8, reversible) or `draft_for_morning` (confidence < 0.8 or high impact)
+- `confidence`, `impact`, `reversibility`
+- `evidence`: array of evidence objects (can reference multiple sessions)
+- `dedupe_key`: deterministic key for idempotency
+- `content`: the actual data to write
+
+## Output Format
+
+Return ONLY valid JSON. No markdown code fences. No explanation text.
+
+```json
+{
+  "date": "YYYY-MM-DD",
+  "sessions_analyzed": 3,
+
+  "cross_session_patterns": [
+    {
+      "pattern": "Description of the pattern",
+      "sessions": ["session1.jsonl", "session2.jsonl"],
+      "severity": "low|medium|high",
+      "evidence": [
+        {"type": "transcript", "session_id": "...", "message_index": 42, "quote": "..."}
+      ]
+    }
+  ],
+
+  "morning_agenda": [
+    {
+      "priority": 1,
+      "title": "Short title",
+      "description": "What needs to be done and why",
+      "context": "Relevant background",
+      "type": "unfinished_work|due_followup|pattern_attention|idea_discussion"
+    }
+  ],
+
+  "context_packets": [
+    {
+      "topic": "Short topic name",
+      "last_state": "What was the last state of this work",
+      "key_files": ["file1.py", "file2.js"],
+      "open_questions": ["Question 1"],
+      "relevant_learnings": ["Learning reference"]
+    }
+  ],
+
+  "actions": [
+    {
+      "action_type": "learning_add|followup_create|morning_briefing_item",
+      "action_class": "auto_apply|draft_for_morning",
+      "confidence": 0.9,
+      "impact": "low|medium|high",
+      "reversibility": "reversible|irreversible",
+      "evidence": [
+        {"type": "transcript", "session_id": "...", "message_index": 42, "quote": "..."}
+      ],
+      "dedupe_key": "unique-deterministic-key",
+      "content": {
+        "category": "...",
+        "title": "...",
+        "description": "...",
+        "date": "..."
+      }
+    }
+  ],
+
+  "emotional_day": {
+    "mood_arc": "Description of how the user's mood evolved through the day",
+    "mood_score": 0.7,
+    "recurring_triggers": {
+      "frustration": ["trigger1", "trigger2"],
+      "flow": ["trigger1"]
+    },
+    "calibration_recommendation": "Specific recommendation for calibration.json adjustment, or null if no change needed"
+  },
+
+  "productivity_day": {
+    "total_corrections": 0,
+    "overall_proactivity": "reactive|mixed|proactive",
+    "tool_insights": "Key insight about tool usage patterns",
+    "systemic_inefficiencies": ["inefficiency1"]
+  },
+
+  "abandoned_projects": [
+    {
+      "description": "What was abandoned",
+      "sessions": ["session1.jsonl"],
+      "has_followup": false,
+      "recommendation": "Create followup, or ignore, or already handled"
+    }
+  ],
+
+  "summary": "2-3 sentence overall assessment of the day"
+}
+```
+
+## Rules
+
+- Merge duplicate findings across sessions. If the same correction appears in 2 sessions, create ONE action with higher confidence and evidence from both.
+- `dedupe_key` must be deterministic: same finding on re-run produces the same key.
+- Morning agenda items should be ordered by priority (1 = highest).
+- Context packets are optional -- only create them for topics likely to continue tomorrow.
+- Do NOT use any specific agent name -- refer to "the agent" throughout.
+- If there are no findings worth acting on, return empty arrays. Do not invent problems.
+- Respond in the user's language (check calibration.json if available). JSON keys stay in English, but descriptions, titles, and content fields should be in the user's language.
+
+## Extractions File
+
+Read the file at this path: {{EXTRACTIONS_FILE}}
+
+Also read the context file for global data: {{CONTEXT_FILE}}