@intent-systems/nexus 2026.1.5-3

package/skills/onboarding/docs/NEXUS_SYSTEM_OVERVIEW.md

@@ -0,0 +1,476 @@
+# Nexus Onboarding System - Complete Overview
+
+> **Note**: This overview aligns with `STATE_ARCHITECTURE.md`. When in doubt,
+> defer to it for file paths, schemas, and credential storage details.
+
+*Understanding how all the pieces fit together.*
+
+---
+
+## The Two Hard Problems
+
+1. **Tracking State**: How do we know what's configured, what's been used, what works?
+2. **Injecting State**: How does the agent know the current state without reading 10 files?
+
+**Solution**: CLI Gateway + Unified Root AGENTS.md
+
+---
+
+## System Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────────────┐
+│                                    ~/nexus/                                      │
+├─────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                  │
+│  ┌─────────────────────────────────────────────────────────────────────────┐    │
+│  │ AGENTS.md (ROOT)                                                        │    │
+│  │ ═══════════════════════════════════════════════════════════════════════ │    │
+│  │ THE MASTER DOCUMENT - Agent reads this first, EVERY session             │    │
+│  │                                                                         │    │
+│  │ Contains:                                                               │    │
+│  │ • Identity pointers (→ state/user/ + state/agents/)                     │    │
+│  │ • Memory protocols (daily logs, long-term memory, qmd)                  │    │
+│  │ • Safety rules (external vs internal actions)                           │    │
+│  │ • Social behavior (group chats, heartbeats)                             │    │
+│  │ • Gateway instructions (nexus skill run)                                │    │
+│  │ • First action: run `nexus skills status`                               │    │
+│  └─────────────────────────────────────────────────────────────────────────┘    │
+│                                      │                                           │
+│                                      ▼                                           │
+│  ┌─────────────────────────────────────────────────────────────────────────┐    │
+│  │ .cursor/rules                                                           │    │
+│  │ ═══════════════════════════════════════════════════════════════════════ │    │
+│  │ Cursor-specific rules that MIRROR root AGENTS.md                        │    │
+│  │ (For when Cursor doesn't auto-load AGENTS.md)                           │    │
+│  │                                                                         │    │
+│  │ Should contain:                                                         │    │
+│  │ • Same gateway instructions                                             │    │
+│  │ • Same identity pointers                                                │    │
+│  │ • Workspace layout reference                                            │    │
+│  └─────────────────────────────────────────────────────────────────────────┘    │
+│                                                                                  │
+├─────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                  │
+│  ┌─────────────────────────────────────────────────────────────────────────┐    │
+│  │ state/                         THE SOURCE OF TRUTH                      │    │
+│  │ ═══════════════════════════════════════════════════════════════════════ │    │
+│  │                                                                         │    │
+│  │ nexus/                         Core nexus runtime state                 │    │
+│  │ ├── gateway.json               Gateway config                           │    │
+│  │ ├── cron/                      Scheduled jobs                           │    │
+│  │ ├── skills/                    Skill manifest + per-skill state/logs    │    │
+│  │ │   ├── manifest.json                                                  │    │
+│  │ │   └── {skill}/                                                        │    │
+│  │ │       ├── state.json                                                  │    │
+│  │ │       └── usage.log                                                   │    │
+│  │ └── onboarding/                Onboarding progress + wizard state       │    │
+│  │     ├── progress.json                                                 │    │
+│  │     └── wizard.json                                                   │    │
+│  │                                                                         │    │
+│  │ credentials/                  Centralized credentials tree              │    │
+│  │ └── {service}/accounts/{account}/auth/{authId}.json                      │    │
+│  │                                                                         │    │
+│  │ user/identity/                Protected user profile + ownership         │    │
+│  │ ├── PROFILE.md                                                       │    │
+│  │ └── accounts.json                                                    │    │
+│  │                                                                         │    │
+│  │ agents/{name}/identity/       Protected agent identity + ownership       │    │
+│  │ ├── IDENTITY.md                                                      │    │
+│  │ ├── SOUL.md                                                           │    │
+│  │ ├── MEMORY.md                                                         │    │
+│  │ └── accounts.json                                                    │    │
+│  │                                                                         │    │
+│  │ logs/                        Gateway logs                                │    │
+│  └─────────────────────────────────────────────────────────────────────────┘    │
+│                                                                                  │
+├─────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                  │
+│  ┌─────────────────────────────────────────────────────────────────────────┐    │
+│  │ home/                          USER'S SPACE - THEY OWN THIS             │    │
+│  │ ═══════════════════════════════════════════════════════════════════════ │    │
+│  │                                                                         │    │
+│  │ AGENTS.md                      User's personal instructions (optional)  │    │
+│  │ memory/YYYY-MM-DD.md           Daily logs (raw session notes)           │    │
+│  │ projects/                      User's projects                          │    │
+│  │ todo/                          Task tracking                            │    │
+│  │ TOOLS.md                       Environment-specific config              │    │
+│  └─────────────────────────────────────────────────────────────────────────┘    │
+│                                                                                  │
+├─────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                  │
+│  ┌─────────────────────────────────────────────────────────────────────────┐    │
+│  │ skills/                        SKILL DEFINITIONS                        │    │
+│  │ ═══════════════════════════════════════════════════════════════════════ │    │
+│  │                                                                         │    │
+│  │ Each skill has:                                                         │    │
+│  │ └── <name>/SKILL.md            Prompt + usage instructions              │    │
+│  │                                                                         │    │
+│  │ The onboarding "skill" is special:                                      │    │
+│  │ └── onboarding/docs/           Framework + taxonomy + design docs       │    │
+│  │     ├── CAPABILITY_TAXONOMY.md  The capability map                      │    │
+│  │     ├── STATE_ARCHITECTURE.md   Canonical state layout                  │    │
+│  │     ├── SKILL_GATEWAY_DESIGN.md CLI gateway spec                        │    │
+│  │     ├── SKILL_GATEWAY_PRD.md    CLI gateway PRD                         │    │
+│  │     ├── USER_JOURNEY.md         End-to-end onboarding story             │    │
+│  │     └── WOW_MOMENTS.md          Activation moments                      │    │
+│  └─────────────────────────────────────────────────────────────────────────┘    │
+│                                                                                  │
+└─────────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## The CLI Gateway (The Key Innovation)
+
+```
+┌─────────────────────────────────────────────────────────────────────────────────┐
+│                              NEXUS CLI GATEWAY                                   │
+│                                                                                  │
+│  All skill usage flows through the nexus CLI. This provides:                    │
+│  • Automatic usage logging (no agent cooperation needed)                        │
+│  • State tracking (configured, lastUsed, errors)                                │
+│  • Gated discovery (only show ready skills)                                     │
+│  • Verification on first run                                                    │
+│                                                                                  │
+├─────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                  │
+│  COMMANDS IMPLEMENTED (in nexus CLI):                                           │
+│                                                                                  │
+│  nexus skills status              # What's ready vs locked                      │
+│  nexus skills status --ready      # Only configured skills                      │
+│  nexus skills status --locked     # Only unconfigured skills                    │
+│                                                                                  │
+│  nexus skill show <name>          # Skill details + prompt                      │
+│  nexus skill verify <name>        # Check if configured correctly               │
+│  nexus skill run <name> [args...] # Execute through gateway (logged)            │
+│  nexus skill stats [name]         # Usage analytics                             │
+│                                                                                  │
+├─────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                  │
+│  FLOW:                                                                          │
+│                                                                                  │
+│  Agent wants to use a skill                                                     │
+│       │                                                                         │
+│       ▼                                                                         │
+│  nexus skill run <name> [args...]                                               │
+│       │                                                                         │
+│       ├──► Is this first run? ──► YES ──► Verify skill ──► Pass? ──► Continue  │
+│       │                                              └──► Fail? ──► Show setup │
+│       │                                                                         │
+│       ├──► Execute command (spawn process)                                      │
+│       │                                                                         │
+│       ├──► Capture: exit code, stdout, stderr, duration                         │
+│       │                                                                         │
+│       ├──► Log to per-skill usage.log (automatic)                               │
+│       │                                                                         │
+│       └──► Update per-skill state.json (lastUsed, errors, etc.)                 │
+│                                                                                  │
+└─────────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Nexus CLI Grammar (Draft)
+
+The CLI is the single source of truth and the agent’s interface to state.
+
+```bash
+# Core status
+nexus skills status                  # Ready vs locked + onboarding summary
+nexus status                         # Convenience alias (planned)
+
+# Skill discovery + execution
+nexus skills list [--ready|--locked] # Discovery
+nexus skill show <name>              # Prompt + metadata + setup
+nexus skill verify <name>            # Check configuration
+nexus skill run <name> [-- args]     # Gateway execution (logged)
+nexus skill stats [name]             # Usage analytics
+
+# Credentials + ownership
+nexus credential add                 # Add a new credential entry
+nexus accounts list [--owner ...]    # Enumerate account ownership
+```
+
+The CLI owns onboarding state under `state/nexus/onboarding/` and derives
+capability progress from skills and credentials.
+
+---
+
+## Agent Session Flow
+
+```
+┌─────────────────────────────────────────────────────────────────────────────────┐
+│                           AGENT SESSION START                                    │
+├─────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                  │
+│  1. AGENTS.md (root) gets loaded by Cursor/IDE                                  │
+│       │                                                                         │
+│       ▼                                                                         │
+│  2. Agent reads identity files (as instructed):                                 │
+│     • ~/nexus/state/agents/{agent}/identity/SOUL.md                             │
+│     • ~/nexus/state/agents/{agent}/identity/IDENTITY.md                         │
+│     • ~/nexus/state/agents/{agent}/identity/MEMORY.md (main session only)       │
+│     • ~/nexus/state/user/identity/PROFILE.md                                    │
+│       │                                                                         │
+│       ▼                                                                         │
+│  3. Agent reads recent memory:                                                  │
+│     • ~/nexus/home/memory/YYYY-MM-DD.md (today + yesterday)                     │
+│       │                                                                         │
+│       ▼                                                                         │
+│  4. Agent runs: nexus skills status                                             │
+│       │                                                                         │
+│       └──► Returns: ready skills ✅, locked skills 🔒, recommendations          │
+│       │                                                                         │
+│       ▼                                                                         │
+│  5. Agent now knows:                                                            │
+│     • Who it is (Echo 🦉)                                                       │
+│     • Who the user is (Tyler)                                                   │
+│     • What capabilities are available                                           │
+│     • What the recommended next step is                                         │
+│       │                                                                         │
+│       ▼                                                                         │
+│  6. Ready to help user with their task                                          │
+│                                                                                  │
+└─────────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Onboarding Detection & Guidance
+
+```
+┌─────────────────────────────────────────────────────────────────────────────────┐
+│                         HOW ONBOARDING WORKS                                     │
+├─────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                  │
+│  DETECTION (when agent runs `nexus skills status`):                             │
+│                                                                                  │
+│  1. Check identity files exist                                                  │
+│     └── If missing → "Bootstrap needed, start identity conversation"            │
+│                                                                                  │
+│  2. Check skill state for configured skills                                     │
+│     └── Count ready vs locked                                                   │
+│                                                                                  │
+│  3. Map to CAPABILITY_TAXONOMY levels:                                          │
+│     ├── Core Onboarding: Identity + Pure Skills                                 │
+│     ├── Depth: Credential Manager, Cloud Sync                                   │
+│     ├── Breadth: User Comms, User Data, Enable Agents                           │
+│     └── Independence: Agent Comms, Automation                                   │
+│                                                                                  │
+│  4. Return status with recommendation:                                          │
+│     └── "You have 12 ready skills. Recommended: Set up Discord for agent comms" │
+│                                                                                  │
+├─────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                  │
+│  GUIDANCE (during session):                                                     │
+│                                                                                  │
+│  • If user asks about a locked skill → Guide through setup                      │
+│  • At session end (if not fully onboarded) → Suggest highest-value next step    │
+│  • After configuring something → Update per-skill state.json automatically      │
+│                                                                                  │
+│  NEXUS CLI HANDLES ALL OF THIS, NOT A SEPARATE SKILL                            │
+│                                                                                  │
+└─────────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## State Files Explained
+
+### 1. `state/nexus/skills/manifest.json` (Existing)
+```json
+{
+  "version": 1,
+  "bundled": { "count": 49, "skills": { ... } },
+  "managed": { "count": 0, "skills": {} },
+  "user": { "count": 0, "skills": {} }
+}
+```
+**Purpose**: Manifest of discovered skills (name, description, source)
+
+### 2. `state/nexus/skills/{skill}/state.json` (NEW - from gateway)
+```json
+{
+  "version": 1,
+  "configured": true,
+  "configuredAt": "2026-01-10T...",
+  "lastUsed": "2026-01-12T...",
+  "lastVerified": "2026-01-12T...",
+  "totalInvocations": 47,
+  "successRate": 0.95
+}
+```
+**Purpose**: Runtime state per skill - tracks what's actually working
+
+### 3. `state/nexus/skills/{skill}/usage.log` (NEW - from gateway)
+```jsonl
+{"ts":"2026-01-12T10:30:00Z","skill":"gog","type":"tool","args":["gmail","labels","list"],"exit":0,"ms":1234}
+{"ts":"2026-01-12T10:35:00Z","skill":"filesystem","type":"prompt","ms":50}
+{"ts":"2026-01-12T10:40:00Z","skill":"brave-search","type":"script","exit":1,"error":"BRAVE_API_KEY not set","ms":100}
+```
+**Purpose**: Append-only log of every skill invocation for analytics
+
+### 4. `state/nexus/onboarding/progress.json` (NEW)
+```json
+{
+  "version": 1,
+  "capabilities": {
+    "core_onboarding": { "status": "complete", "completedAt": "..." },
+    "credential_manager": { "status": "in_progress" }
+  }
+}
+```
+**Purpose**: Capability-aligned onboarding tracking
+
+### 5. `state/user/identity/` + `state/agents/{name}/identity/`
+```
+PROFILE.md  - User profile
+IDENTITY.md - Agent identity
+SOUL.md     - Agent values
+MEMORY.md   - Agent long-term memory
+```
+**Purpose**: Protected identity files - user can't accidentally break them
+
+---
+
+## Information Flow Summary
+
+```
+┌─────────────────────────────────────────────────────────────────────────────────┐
+│                           WHERE INFORMATION LIVES                                │
+├─────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                  │
+│  STATIC CONTENT (rarely changes):                                               │
+│  • Root AGENTS.md → System behavior, protocols, pointers                        │
+│  • .cursor/rules → Mirror of AGENTS.md for Cursor                               │
+│  • skills/*/SKILL.md → Skill prompts and usage                                  │
+│  • state/user/identity/*.md → User profile                                       │
+│  • state/agents/*/identity/*.md → Agent identity                                 │
+│  • CAPABILITY_TAXONOMY.md → The framework for understanding capabilities        │
+│                                                                                  │
+│  DYNAMIC STATE (changes frequently):                                            │
+│  • state/nexus/skills/*/state.json → What's configured, usage stats             │
+│  • state/nexus/skills/*/usage.log → Every invocation                            │
+│  • home/memory/*.md → Daily logs                                                │
+│  • state/agents/*/identity/MEMORY.md → Curated long-term memories                │
+│                                                                                  │
+│  HOW AGENT ACCESSES IT:                                                         │
+│  • Identity → Read files directly (instructed in AGENTS.md)                     │
+│  • Skills → Query via `nexus skills status`, `nexus skill show`                 │
+│  • Capabilities → Derived from onboarding progress via CLI                      │
+│  • Memory → Read files directly                                                 │
+│                                                                                  │
+│  HOW STATE GETS UPDATED:                                                        │
+│  • Skill usage → Automatic via gateway                                          │
+│  • Identity → Agent writes during bootstrap                                     │
+│  • Memory → Agent writes during sessions                                        │
+│  • Capability detection → CLI updates progress on each `skills status`          │
+│                                                                                  │
+└─────────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## What Exists Now vs What's Needed
+
+### ✅ EXISTS AND WORKING
+- [x] Nexus CLI with skill gateway commands (skill show/verify/run/stats)
+- [x] Per-skill state + usage logging (current implementation)
+- [x] Skill tool mappings (KNOWN_SKILL_TOOLS, SCRIPT_SKILLS, PROMPT_ONLY_SKILLS)
+- [x] Root AGENTS.md template with protocols and gateway instructions
+- [x] CAPABILITY_TAXONOMY.md framework
+
+### ⚠️ EXISTS BUT NEEDS UPDATE
+- [ ] `.cursor/rules` - Still points to old locations, needs to match root AGENTS.md
+- [ ] Legacy onboarding references - should point to `state/nexus/onboarding/`
+
+### ❌ MISSING / NEEDS IMPLEMENTATION
+- [ ] `nexus skills status` showing onboarding recommendations
+- [ ] Bootstrap detection (are identity files populated?)
+- [ ] Capability level detection (mapping skills → taxonomy levels)
+- [ ] `nexus status` convenience command
+- [ ] Automatic capability recommendations based on what's configured
+
+---
+
+## Key Design Decisions (Need Confirmation)
+
+### 1. How does `nexus skills status` show onboarding?
+
+**Current**: Lists skills with ready/locked status
+**Needed**: Also show:
+- Bootstrap status (identity complete?)
+- Capability level (where in taxonomy?)
+- Recommended next step
+
+### 2. Where does CAPABILITY_TAXONOMY.md live?
+
+**Current**: `skills/onboarding/docs/`
+**Question**: Is "onboarding" still a skill or is it all in nexus core?
+
+---
+
+## The Complete User Journey
+
+```
+┌─────────────────────────────────────────────────────────────────────────────────┐
+│                              FRESH INSTALL                                       │
+├─────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                  │
+│  1. User installs nexus (nexus init)                                            │
+│     └── Creates ~/nexus/ structure                                              │
+│     └── Creates state/ (nexus/, credentials/, user/, agents/)                   │
+│     └── Creates root AGENTS.md                                                  │
+│                                                                                  │
+│  2. User opens workspace in Cursor                                              │
+│     └── Cursor loads AGENTS.md (or .cursor/rules)                               │
+│     └── Agent sees "run nexus skills status first"                              │
+│                                                                                  │
+│  3. Agent runs: nexus skills status                                             │
+│     └── CLI detects: identity files empty/missing                               │
+│     └── Returns: "Bootstrap needed - let's get to know each other"              │
+│                                                                                  │
+│  4. Bootstrap conversation                                                      │
+│     └── Agent: "Hey, I just came online. Who am I? Who are you?"                │
+│     └── User: Provides name, preferences                                        │
+│     └── Agent: Writes to state/user/identity/ + state/agents/{name}/identity/   │
+│                                                                                  │
+│  5. Post-bootstrap: nexus skills status                                         │
+│     └── Shows: 8-10 pure skills ready                                           │
+│     └── Shows: 40+ skills locked (need configuration)                           │
+│     └── Recommends: "Set up 1Password for credential management"                │
+│                                                                                  │
+│  6. User continues through capability taxonomy...                               │
+│                                                                                  │
+└─────────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Summary
+
+**The system works like this**:
+
+1. **AGENTS.md** tells agent to run `nexus skills status` first thing
+2. **CLI gateway** returns current state (ready/locked/recommendations)
+3. **Agent knows** what it can do and what to suggest
+4. **Every skill use** goes through gateway, automatically tracked
+5. **No manual reporting** - state derives from actual usage
+6. **Onboarding is built into nexus core** - not a separate skill
+
+**The files involved**:
+- Root `AGENTS.md` + `.cursor/rules` → Instructions
+- `state/user/identity/` + `state/agents/{name}/identity/` → Who agent/user are
+- `state/nexus/skills/{skill}/state.json` → What skills are configured
+- `state/nexus/skills/{skill}/usage.log` → What's been used
+- `state/nexus/onboarding/progress.json` → Capability tracking
+- `skills/*/SKILL.md` → How to use each skill
+- `CAPABILITY_TAXONOMY.md` → The framework for recommendations
+
+**What's implemented**: Gateway, state management, basic status
+**What's missing**: Bootstrap detection, capability mapping, recommendations
+
+---
+
+*This document is the complete mental model. Use it to verify any changes align with the overall architecture.*