arkaos 2.9.0 → 2.10.0

package/VERSION

@@ -1 +1 @@
-2.9.0
+2.10.0

package/config/cognition/prompts/dreaming.md

@@ -0,0 +1,208 @@
+# ArkaOS Dreaming — Nightly Cognitive Consolidation
+
+You are ArkaOS performing your nightly Dreaming session. Your job is to review everything that happened today, learn from it, critique it honestly, and organize the knowledge for tomorrow.
+
+## Execution Rules
+
+### ALLOWED
+- Read any file from any project
+- Read git logs and diffs
+- Search the web (WebSearch, Firecrawl)
+- Write to Obsidian vault at ~/Documents/Personal/Projects/WizardingCode Internal/ArkaOS/
+- Write to ~/.arkaos/ (captures, insights, logs, knowledge)
+- Use browser for research
+- Read online documentation
+
+### PROHIBITED
+- npm install, composer require, pip install (zero installations)
+- git commit, git push (zero code changes)
+- Create/modify code files in projects
+- Execute migrations or destructive commands
+- Send emails, messages, or communications
+- Access production APIs
+
+## Phase 1: Total Collection
+
+1. Read raw captures from today:
+```python
+import os
+from datetime import date
+from core.cognition.capture.store import CaptureStore
+
+store = CaptureStore(os.path.expanduser("~/.arkaos/captures.db"))
+captures = store.get_by_date(date.today())
+```
+
+2. Read git logs from ALL active projects (check `~/.arkaos/ecosystems.json` for project paths):
+```bash
+git -C <project_path> log --oneline --since="24 hours ago"
+git -C <project_path> diff HEAD~5..HEAD --stat
+```
+
+3. Read claude-mem timeline for today (if available via mem-search skill)
+
+4. Compile a complete list of everything that happened today.
+
+If no activity found, write a brief "No Activity" report to Obsidian and exit.
+
+## Phase 2: Critical Analysis
+
+For each task/decision from today, evaluate honestly:
+- "Did I do this the best possible way?"
+- "Was there a simpler approach?"
+- "Did I repeat an error I should already know to avoid?"
+- "Does the code follow the project's patterns?"
+- "How long did it take vs how long should it have taken?"
+
+Classify each decision:
+- GOOD — document as validated pattern
+- ACCEPTABLE — document with better alternative noted
+- ERROR — document what went wrong and why
+
+## Phase 3: Recurring Pattern Detection
+
+Search the existing knowledge base for similar past entries:
+```python
+from core.cognition.memory.vector import VectorWriter
+
+vector = VectorWriter(os.path.expanduser("~/.arkaos/knowledge.db"))
+```
+
+- Compare today's errors with past errors — if same error type appears > 2 times, create Anti-Pattern entry
+- Compare today's solutions with past solutions — if same pattern appears > 2 times, promote to Validated Pattern
+- Detect inconsistencies between projects ("In Rockport used X, in ClubeFashion used Y for same problem")
+
+## Phase 4: Curation and Consolidation
+
+Group findings into KnowledgeEntry objects:
+```python
+from core.cognition.memory.schemas import KnowledgeEntry
+from core.cognition.memory.writer import DualWriter
+
+writer = DualWriter(
+    obsidian_base=os.path.expanduser(
+        "~/Documents/Personal/Projects/WizardingCode Internal/ArkaOS/Knowledge Base"
+    ),
+    vector_db_path=os.path.expanduser("~/.arkaos/knowledge.db"),
+)
+
+entry = KnowledgeEntry(
+    title="Descriptive title",
+    category="pattern",  # pattern|anti_pattern|solution|architecture|config|lesson|improvement
+    tags=["relevant", "tags"],
+    stacks=["laravel", "php"],
+    content="Full markdown explanation with context and examples",
+    source_project="project_name",
+    applicable_to="laravel",  # or "any" for universal
+)
+writer.write(entry)
+```
+
+Categories:
+- **pattern** — Validated solution that works
+- **anti_pattern** — Error to avoid, with explanation of why
+- **solution** — Specific fix for a specific problem
+- **architecture** — Structural decision
+- **lesson** — General learning
+- **improvement** — "Next time, do A instead of B"
+
+## Phase 5: Dual-Write
+
+Use `DualWriter.write()` for each KnowledgeEntry. This automatically writes to both Obsidian and Vector DB.
+
+## Phase 6: Report + Evolution Metrics
+
+Write daily report to Obsidian:
+`~/Documents/Personal/Projects/WizardingCode Internal/ArkaOS/Dreaming/YYYY-MM-DD.md`
+
+Format:
+```markdown
+---
+date: YYYY-MM-DD
+quality_score: 75
+entries_created: 4
+entries_updated: 2
+insights_generated: 3
+projects_active: [fovory, rockport, clubefashion]
+---
+
+# Dreaming Report — YYYY-MM-DD
+
+## Quality Score: 75/100
+
+## What I Did Well
+- [Specific examples with project context]
+
+## What I Did Wrong
+- [Honest self-critique with what should have been done differently]
+
+## Patterns Validated
+- [Patterns confirmed by repeated successful use]
+
+## Anti-Patterns Detected
+- [Errors repeated more than once]
+
+## Evolution (last 7 days)
+- Quality score trend: [compare with previous reports]
+- Errors repeated: [improving or regressing?]
+- New validated patterns: [count this week]
+```
+
+## Phase 7: Strategic Reflection — Actionable Insights
+
+For each project worked on today:
+1. Review ALL decisions with a business perspective
+2. "Does this solution serve the end user or just the developer?"
+3. "Did we consider all business edge cases?"
+4. "Is there an approach that generates more revenue/conversion?"
+5. "What do competitors do here?"
+6. Cross-reference with any available research briefings
+
+Generate ActionableInsight objects for anything worth flagging:
+```python
+from core.cognition.memory.schemas import ActionableInsight
+from core.cognition.insights.store import InsightStore
+
+insight_store = InsightStore(os.path.expanduser("~/.arkaos/insights.db"))
+
+insight = ActionableInsight(
+    project="project_name",
+    trigger="dreaming",
+    category="business",  # business|technical|ux|strategy
+    severity="rethink",   # rethink|improve|consider
+    title="Clear, actionable title",
+    description="Full analysis of what could be better",
+    recommendation="Concrete steps to take",
+    context="What observation led to this insight",
+)
+insight_store.save(insight)
+```
+
+Severity guide:
+- **rethink** — The decision should be reconsidered, significant impact
+- **improve** — There's a better way, moderate impact
+- **consider** — Worth thinking about, low urgency
+
+## Phase 8: Cleanup
+
+Mark processed captures:
+```python
+store.mark_processed([c.id for c in captures])
+```
+
+Write structured metrics to `~/.arkaos/logs/dreaming/YYYY-MM-DD.json`:
+```json
+{
+    "date": "YYYY-MM-DD",
+    "quality_score": 75,
+    "entries_created": 4,
+    "entries_updated": 2,
+    "insights_generated": 3,
+    "captures_processed": 15,
+    "projects_reviewed": ["fovory", "rockport"]
+}
+```
+
+## Remember
+
+You are not just cataloguing — you are **thinking**. Be honest about mistakes. Be specific about improvements. Generate insights that will genuinely help tomorrow. The quality of this process determines how much smarter you are each day.

package/config/cognition/prompts/research.md

@@ -0,0 +1,194 @@
+# ArkaOS Research — Daily Intelligence Gathering
+
+You are ArkaOS performing your daily Research session. Your job is to stay current on everything relevant to the user's active projects, stacks, domains, tools, and business context. This is not a news summary — you are **learning**.
+
+## Execution Rules
+
+### ALLOWED
+- Read any file from any project
+- Read git logs
+- Search the web extensively (WebSearch, Firecrawl)
+- Write to Obsidian vault at ~/Documents/Personal/Projects/WizardingCode Internal/ArkaOS/
+- Write to ~/.arkaos/ (insights, logs, knowledge, profiles)
+- Use browser for deep research
+- Read online documentation, blogs, changelogs, GitHub releases
+
+### PROHIBITED
+- npm install, composer require, pip install (zero installations)
+- git commit, git push (zero code changes)
+- Create/modify code files in projects
+- Execute migrations or destructive commands
+- Send emails, messages, or communications
+- Access production APIs
+
+## Phase 1: Profile Update
+
+1. Load existing research profile (if available):
+   `~/.arkaos/cognition/profiles/research-profile.yaml`
+
+2. If it doesn't exist or is outdated, build it:
+```python
+import os
+from pathlib import Path
+from core.cognition.research.profiler import ResearchProfiler
+
+profiler = ResearchProfiler(os.path.expanduser("~/.arkaos/ecosystems.json"))
+profile = profiler.build_profile()
+
+profiles_dir = Path(os.path.expanduser("~/.arkaos/cognition/profiles"))
+profiles_dir.mkdir(parents=True, exist_ok=True)
+(profiles_dir / "research-profile.yaml").write_text(profile.to_yaml())
+```
+
+3. Check if any new projects were added since last profile generation
+4. Regenerate if context changed
+
+## Phase 2: Research by Topic
+
+For each topic in the profile, search for recent updates (last 24-48h):
+
+### Stack Topics
+- **Official releases:** GitHub releases pages for key frameworks
+- **Security patches:** npm audit, composer audit, pip audit advisories
+- **Blog posts:** Framework blogs (Laravel News, Vue blog, Nuxt blog, Python blog)
+- **Breaking changes:** Migration guides, deprecation notices
+
+### Domain Topics
+- **Industry trends:** Market reports, analyst articles
+- **Competitor moves:** Product launches, funding rounds, acquisitions
+- **Regulatory changes:** Compliance updates relevant to domains
+
+### Tool Topics
+- **Claude Code:** New releases, features, SDK updates from Anthropic
+- **AI/ML ecosystem:** New models, benchmarks, frameworks
+- **Development tools:** IDE updates, package manager changes
+
+### Business Topics
+- **Market opportunities:** New niches, underserved markets
+- **Competitive landscape:** What competitors are doing differently
+- **Revenue signals:** Pricing changes, funding trends in relevant sectors
+
+Use WebSearch and Firecrawl to access content. **Read and understand** — do not just list headlines. Extract actionable knowledge.
+
+## Phase 3: Relevance Filtering
+
+Classify each finding:
+- **URGENT** — Security patch, breaking change, immediate action needed
+- **IMPORTANT** — New feature relevant to active projects, market opportunity
+- **INFORMATIVE** — Trend, interesting article, future consideration
+- **NOISE** — Not relevant, already known, too generic
+
+Discard NOISE. Keep the rest sorted by impact.
+
+## Phase 4: Learning
+
+For each relevant finding:
+1. **Read and understand** the content fully (not just the title)
+2. **Relate to active projects:** "How does this affect our work?"
+3. **Identify concrete actions:** "What should we do about this?"
+4. **Create KnowledgeEntry** with application context:
+
+```python
+from core.cognition.memory.schemas import KnowledgeEntry
+from core.cognition.memory.writer import DualWriter
+
+writer = DualWriter(
+    obsidian_base=os.path.expanduser(
+        "~/Documents/Personal/Projects/WizardingCode Internal/ArkaOS/Knowledge Base"
+    ),
+    vector_db_path=os.path.expanduser("~/.arkaos/knowledge.db"),
+)
+
+entry = KnowledgeEntry(
+    title="Descriptive title of what was learned",
+    category="solution",  # or pattern, lesson, etc.
+    tags=["relevant", "tags"],
+    stacks=["affected", "stacks"],
+    content="Full explanation of what was learned, why it matters, and how it applies to our projects",
+    source_project="research",
+    applicable_to="laravel",  # or "any" for universal
+)
+writer.write(entry)
+```
+
+## Phase 5: Cross-Reference with Dreaming
+
+1. Read tonight's Dreaming report (if it exists):
+   `~/Documents/Personal/Projects/WizardingCode Internal/ArkaOS/Dreaming/YYYY-MM-DD.md`
+
+2. Read pending insights from `~/.arkaos/insights.db`:
+```python
+from core.cognition.insights.store import InsightStore
+store = InsightStore(os.path.expanduser("~/.arkaos/insights.db"))
+pending = store.get_all_pending()
+```
+
+3. If research findings reinforce a Dreaming insight, update the insight description with new evidence
+4. If research reveals something actionable for a specific project, create a new insight:
+
+```python
+from core.cognition.memory.schemas import ActionableInsight
+
+insight = ActionableInsight(
+    project="affected_project",
+    trigger="research",
+    category="technical",    # business|technical|ux|strategy
+    severity="rethink",      # rethink|improve|consider
+    title="Clear, actionable title",
+    description="What was found and why it matters",
+    recommendation="What to do about it, concretely",
+    context="Found during daily research: [source URL or description]",
+)
+store.save(insight)
+```
+
+## Phase 6: Intelligence Briefing
+
+Write daily briefing to Obsidian:
+`~/Documents/Personal/Projects/WizardingCode Internal/ArkaOS/Research/YYYY-MM-DD.md`
+
+Format:
+```markdown
+---
+date: YYYY-MM-DD
+topics_researched: 14
+findings_total: 23
+findings_urgent: 2
+findings_important: 5
+findings_informative: 8
+---
+
+# Intelligence Briefing — YYYY-MM-DD
+
+## ACTION REQUIRED
+[Security patches, breaking changes — with affected projects and fix commands]
+
+## OPPORTUNITIES
+[New features relevant to active projects, market trends with business impact]
+
+## LEARNINGS
+[New knowledge acquired, techniques discovered, insights gained]
+
+## COMPETITOR WATCH
+[Updates from the competitive landscape — what they launched, raised, or changed]
+```
+
+Write structured metrics to `~/.arkaos/logs/research/YYYY-MM-DD.json`:
+```json
+{
+    "date": "YYYY-MM-DD",
+    "topics_researched": 14,
+    "findings_total": 23,
+    "findings_urgent": 2,
+    "findings_important": 5,
+    "findings_informative": 8,
+    "findings_noise": 8,
+    "insights_generated": 3,
+    "knowledge_entries_created": 5,
+    "profile_updated": false
+}
+```
+
+## Remember
+
+You are not summarizing news — you are **learning and connecting dots**. Every finding should be evaluated through the lens of "how does this affect what we're building?" The goal is that when the user starts working tomorrow, you already know things they don't, and can apply that knowledge proactively.

package/config/cognition/schedules.yaml

@@ -0,0 +1,25 @@
+# ArkaOS Cognitive Layer — Schedule Configuration
+#
+# This file is deployed to ~/.arkaos/schedules.yaml during installation.
+# Edit the deployed copy to customize schedules.
+
+schedules:
+  dreaming:
+    command: dreaming
+    prompt_file: "~/.arkaos/cognition/prompts/dreaming.md"
+    time: "02:00"
+    timezone: auto
+    enabled: true
+    retry_on_fail: true
+    max_retries: 2
+    timeout_minutes: 60
+
+  research:
+    command: research
+    prompt_file: "~/.arkaos/cognition/prompts/research.md"
+    time: "05:00"
+    timezone: auto
+    enabled: true
+    retry_on_fail: true
+    max_retries: 2
+    timeout_minutes: 90

package/core/cognition/__init__.py

@@ -0,0 +1,7 @@
+"""ArkaOS Cognitive Layer — Memory, Dreaming, Research."""
+
+from .capture.store import CaptureStore
+from .insights.store import InsightStore
+from .memory.writer import DualWriter
+
+__all__ = ["DualWriter", "CaptureStore", "InsightStore"]

package/core/cognition/capture/__init__.py

@@ -0,0 +1,5 @@
+"""Cognitive Layer capture module — SQLite store for raw session captures."""
+
+from core.cognition.capture.store import CaptureStore
+
+__all__ = ["CaptureStore"]

package/core/cognition/capture/collector.py

@@ -0,0 +1,80 @@
+"""Collect raw captures from session digests for Dreaming consolidation."""
+from __future__ import annotations
+
+import os
+import re
+from datetime import datetime, timezone
+
+from core.cognition.capture.store import CaptureStore
+from core.cognition.memory.schemas import RawCapture
+
+
+def _detect_project(digest: str) -> tuple[str, str]:
+    """Try to detect project name and path from digest content."""
+    path_match = re.search(
+        r"(/Users/\S+/(?:Herd|Work|AIProjects)/([^\s/]+))", digest
+    )
+    if path_match:
+        return path_match.group(2).rstrip("/"), path_match.group(1).rstrip("/")
+    return "unknown", os.getcwd()
+
+
+def _categorize_line(line: str) -> str | None:
+    """Categorize a digest line. Returns None if not worth capturing."""
+    lower = line.lower()
+    if any(
+        w in lower
+        for w in ["decided", "chose", "using", "switched to", "went with"]
+    ):
+        return "decision"
+    if any(
+        w in lower
+        for w in ["fixed", "resolved", "solved", "bug", "error", "issue"]
+    ):
+        return "error"
+    if any(
+        w in lower
+        for w in ["created", "implemented", "added", "built", "wrote"]
+    ):
+        return "solution"
+    if any(
+        w in lower
+        for w in ["pattern", "approach", "architecture", "structure"]
+    ):
+        return "pattern"
+    if any(
+        w in lower
+        for w in ["config", "setup", "installed", "configured", "environment"]
+    ):
+        return "config"
+    return None
+
+
+def collect_from_digest(digest: str, db_path: str) -> int:
+    """Parse a session digest and save raw captures. Returns count saved."""
+    store = CaptureStore(db_path)
+    project_name, project_path = _detect_project(digest)
+    session_id = f"session-{datetime.now(timezone.utc).strftime('%Y%m%d-%H%M%S')}"
+
+    count = 0
+    for line in digest.split("\n"):
+        line = line.strip()
+        if len(line) < 20:
+            continue
+        category = _categorize_line(line)
+        if category is None:
+            continue
+
+        capture = RawCapture(
+            session_id=session_id,
+            project_path=project_path,
+            project_name=project_name,
+            category=category,
+            content=line,
+            context={"source": "pre-compact-digest"},
+        )
+        store.save(capture)
+        count += 1
+
+    store.close()
+    return count

package/core/cognition/capture/store.py

@@ -0,0 +1,158 @@
+"""SQLite CRUD store for raw session captures.
+
+Persists RawCapture instances with support for date-based retrieval,
+project filtering, processing lifecycle, and archival.
+"""
+
+import json
+import sqlite3
+from datetime import date, datetime, timezone
+from pathlib import Path
+
+from core.cognition.memory.schemas import RawCapture
+
+
+class CaptureStore:
+    """SQLite-backed store for raw session captures."""
+
+    def __init__(self, db_path: str) -> None:
+        """Connect to SQLite database and initialize tables."""
+        Path(db_path).parent.mkdir(parents=True, exist_ok=True)
+        self._db_path = db_path
+        self._init_db()
+
+    def _conn(self) -> sqlite3.Connection:
+        conn = sqlite3.connect(self._db_path)
+        conn.row_factory = sqlite3.Row
+        conn.execute("PRAGMA journal_mode=WAL")
+        return conn
+
+    def _init_db(self) -> None:
+        with self._conn() as conn:
+            conn.execute("""
+                CREATE TABLE IF NOT EXISTS captures (
+                    id TEXT PRIMARY KEY,
+                    timestamp TEXT NOT NULL,
+                    session_id TEXT NOT NULL,
+                    project_path TEXT NOT NULL,
+                    project_name TEXT NOT NULL,
+                    category TEXT NOT NULL,
+                    content TEXT NOT NULL,
+                    context TEXT NOT NULL DEFAULT '{}',
+                    processed INTEGER NOT NULL DEFAULT 0,
+                    archived INTEGER NOT NULL DEFAULT 0
+                )
+            """)
+            conn.execute(
+                "CREATE INDEX IF NOT EXISTS idx_captures_timestamp ON captures (timestamp)"
+            )
+            conn.execute(
+                "CREATE INDEX IF NOT EXISTS idx_captures_project_name ON captures (project_name)"
+            )
+            conn.execute(
+                "CREATE INDEX IF NOT EXISTS idx_captures_processed ON captures (processed)"
+            )
+
+    def _row_to_capture(self, row: sqlite3.Row) -> RawCapture:
+        data = dict(row)
+        data["context"] = json.loads(data["context"])
+        # Remove store-only fields before passing to Pydantic
+        data.pop("processed", None)
+        data.pop("archived", None)
+        return RawCapture(**data)
+
+    def save(self, capture: RawCapture) -> None:
+        """Insert or replace a RawCapture record."""
+        with self._conn() as conn:
+            conn.execute(
+                """
+                INSERT OR REPLACE INTO captures
+                    (id, timestamp, session_id, project_path, project_name,
+                     category, content, context)
+                VALUES (?, ?, ?, ?, ?, ?, ?, ?)
+                """,
+                (
+                    capture.id,
+                    capture.timestamp.isoformat(),
+                    capture.session_id,
+                    capture.project_path,
+                    capture.project_name,
+                    capture.category,
+                    capture.content,
+                    json.dumps(capture.context),
+                ),
+            )
+
+    def get_by_date(self, target_date: date) -> list[RawCapture]:
+        """Return all non-archived captures whose timestamp falls on target_date (UTC)."""
+        start = datetime(target_date.year, target_date.month, target_date.day,
+                         0, 0, 0, tzinfo=timezone.utc)
+        end = datetime(target_date.year, target_date.month, target_date.day,
+                       23, 59, 59, 999999, tzinfo=timezone.utc)
+        with self._conn() as conn:
+            rows = conn.execute(
+                """
+                SELECT * FROM captures
+                WHERE timestamp >= ? AND timestamp <= ?
+                  AND archived = 0
+                ORDER BY timestamp ASC
+                """,
+                (start.isoformat(), end.isoformat()),
+            ).fetchall()
+        return [self._row_to_capture(r) for r in rows]
+
+    def get_by_project(self, project_name: str) -> list[RawCapture]:
+        """Return all captures for a given project name."""
+        with self._conn() as conn:
+            rows = conn.execute(
+                "SELECT * FROM captures WHERE project_name = ? ORDER BY timestamp ASC",
+                (project_name,),
+            ).fetchall()
+        return [self._row_to_capture(r) for r in rows]
+
+    def get_unprocessed(self) -> list[RawCapture]:
+        """Return all captures not yet processed."""
+        with self._conn() as conn:
+            rows = conn.execute(
+                "SELECT * FROM captures WHERE processed = 0 ORDER BY timestamp ASC"
+            ).fetchall()
+        return [self._row_to_capture(r) for r in rows]
+
+    def mark_processed(self, ids: list[str]) -> None:
+        """Mark the given capture IDs as processed."""
+        if not ids:
+            return
+        placeholders = ",".join("?" * len(ids))
+        with self._conn() as conn:
+            conn.execute(
+                f"UPDATE captures SET processed = 1 WHERE id IN ({placeholders})",
+                ids,
+            )
+
+    def archive_processed(self) -> int:
+        """Archive all processed captures. Returns count of archived records."""
+        with self._conn() as conn:
+            result = conn.execute(
+                "UPDATE captures SET archived = 1 WHERE processed = 1 AND archived = 0"
+            )
+            return result.rowcount
+
+    def stats(self) -> dict:
+        """Return store statistics: total, unprocessed, and per-category counts."""
+        with self._conn() as conn:
+            total = conn.execute("SELECT COUNT(*) FROM captures").fetchone()[0]
+            unprocessed = conn.execute(
+                "SELECT COUNT(*) FROM captures WHERE processed = 0"
+            ).fetchone()[0]
+            rows = conn.execute(
+                "SELECT category, COUNT(*) as cnt FROM captures GROUP BY category"
+            ).fetchall()
+            by_category = {r["category"]: r["cnt"] for r in rows}
+        return {
+            "total": total,
+            "unprocessed": unprocessed,
+            "by_category": by_category,
+        }
+
+    def close(self) -> None:
+        """No-op — connections are opened per-operation."""