arkaos 2.15.0 → 2.16.0

package/departments/kb/skills/knowledge/SKILL.md

@@ -0,0 +1,94 @@
+---
+name: arka-knowledge
+description: >
+  Dynamic knowledge base powered by Obsidian. Async background processing: queues YouTube
+  downloads, transcribes with Whisper (local or API), and pauses for interactive analysis.
+  Runs 5 parallel analysis agents (Frameworks, Strategy, Voice & Style, Principles, Topics)
+  to build expert personas. Learns from articles and URLs. Creates searchable persona profiles,
+  topic cross-references, and source catalogs. Generates content in any learned persona's voice
+  and style. All output organized in Obsidian vault with MOC pages.
+  Use when user says "kb", "learn", "persona", "knowledge", "youtube", "transcribe", "article",
+  "research", "analyze", "source", "topic", "search knowledge", "write as", "queue", "process",
+  "capabilities", or wants to learn from any content source or use a persona's expertise.
+---
+
+# Knowledge Base — ARKA OS Department
+
+Dynamic knowledge acquisition and management system. Learn from YouTube videos, articles, books, and any content source. Build expert personas and a searchable knowledge base.
+
+**ALL output goes to the Obsidian vault.**
+
+**Background processing:** Downloads and transcriptions run as background jobs. Queue 100 URLs and keep working. Process results interactively when ready.
+
+## Obsidian Configuration
+
+- **Vault:** `/Users/andreagroferreira/Documents/Personal`
+- **Config:** Read `knowledge/obsidian-config.json` for full path/convention details
+- **MCP:** Use Obsidian MCP when available, fallback to direct file Write
+- **Conventions:** YAML frontmatter, wikilinks `[[]]`, MOC references, kebab-case tags
+
+## Commands
+
+| Command | Description |
+|---------|-------------|
+| `/kb learn <url> [url2 ...] [--persona "Name"]` | Queue download + transcription (async, non-blocking) |
+| `/kb learn-text <file/url> --persona "Name"` | Learn from text/article content (synchronous) |
+| `/kb queue` | Show all queued/running/ready jobs |
+| `/kb status [job-id]` | Detailed status of a specific job |
+| `/kb process <job-id>` | Analyze a ready transcription (interactive choices) |
+| `/kb process --all` | Process all ready jobs |
+| `/kb capabilities` | Show available tools and API keys |
+| `/kb cleanup [--older-than 90d]` | Remove old media files |
+| `/kb persona <name>` | View/manage a persona profile |
+| `/kb personas` | List all personas and their stats |
+| `/kb search <query>` | Search knowledge base by topic |
+| `/kb write --persona "Name" --type <type>` | Generate content using a persona's style |
+| `/kb topics` | List all knowledge topics |
+| `/kb update <persona> <youtube-url>` | Add more content to existing persona |
+
+## Scripts Location
+
+All KB scripts are in the `scripts/` subdirectory of this skill:
+- `scripts/kb-check-capabilities.sh` — System capability probe
+- `scripts/kb-queue.sh` — Queue dispatcher
+- `scripts/kb-worker.sh` — Background worker
+- `scripts/kb-status.sh` — Status checker
+- `scripts/kb-cleanup.sh` — Media cleanup
+
+When installed, scripts live at `~/.claude/skills/arka-knowledge/scripts/`.
+
+## Squad
+
+- **Lead:** Clara (KB department)
+- **Specialists:** Persona Analyst, Topic Cataloger, Voice Profiler
+- **Quality Gate:** Marta → Eduardo (copy) + Francisca (tech). Mandatory.
+
+## Job Status Flow
+
+```
+queued → downloading → transcribing → ready → analyzing → completed
+              ↓              ↓
+           failed          failed
+```
+
+The `ready` → `analyzing` transition happens when user runs `/kb process <job-id>`. Analysis requires Claude Code's LLM (5 parallel agents); cannot run in background bash.
+
+## Obsidian Output Paths
+
+| Content | Vault Path |
+|---------|-----------|
+| Personas | `Personas/<Name>.md` |
+| Video Sources | `Sources/Videos/<date> <title>.md` |
+| Article Sources | `Sources/Articles/<date> <title>.md` |
+| Topics | `Topics/<Topic Name>.md` |
+| Frameworks | `🧠 Knowledge Base/Frameworks/<name>.md` |
+| Raw Transcripts | `🧠 Knowledge Base/Raw Transcripts/<name>.txt` |
+| MOC Pages | `Personas MOC.md`, `Topics MOC.md`, `Sources MOC.md` |
+
+## References
+
+Read only when needed for the task at hand:
+
+- **Ingestion pipeline** (`/kb learn`, `/kb learn-text`, `/kb process`, `/kb process --all`, `/kb cleanup`, Obsidian page formats, 5+1 analysis agents): `references/ingestion-pipeline.md`
+- **Async processing** (`/kb queue`, `/kb status`, `/kb capabilities`, job state file, worker mechanics, media storage, status transitions): `references/async-processing.md`
+- **Persona utilities** (`/kb personas`, `/kb persona`, `/kb topics`, `/kb search`, `/kb write`, `/kb update`): `references/ingestion-pipeline.md` (persona output section)

package/departments/kb/skills/knowledge/references/async-processing.md

@@ -0,0 +1,89 @@
+# arka-knowledge — async-processing
+
+Referenced from SKILL.md. Read only when needed.
+
+## /kb capabilities
+
+Check what tools and API keys are available for KB processing.
+
+**Steps:**
+1. Run `bash <scripts-dir>/kb-check-capabilities.sh`
+2. Read `~/.arka-os/capabilities.json`
+3. Display results to the user in a formatted table
+
+Shows: binary availability (whisper, yt-dlp, ffmpeg, jq, python3), API keys (OpenAI, Gemini, OpenRouter), and the selected transcription method.
+
+## /kb queue
+
+Show all jobs and their current status.
+
+**Steps:**
+1. Run `bash <scripts-dir>/kb-status.sh`
+2. Or read `~/.arka-os/kb-jobs.json` directly and format as a table
+3. Show: job ID, status, title, transcription method
+4. Status colors: queued (yellow), downloading/transcribing (blue), ready (green), completed (green), failed (red)
+
+## /kb status [job-id]
+
+Detailed status of a specific job.
+
+**Steps:**
+1. Run `bash <scripts-dir>/kb-status.sh <job-id>`
+2. Or read the job from `~/.arka-os/kb-jobs.json` and display all fields
+3. If `--json` flag: output raw JSON
+
+## Worker / Queue Mechanics
+
+- **Queue dispatcher:** `kb-queue.sh` writes a new job record to `~/.arka-os/kb-jobs.json` (status=`queued`), then detaches `kb-worker.sh` in background.
+- **Worker:** `kb-worker.sh` handles a single job — runs yt-dlp, writes `audio.wav`, transcribes (Whisper local or API), updates status transitions (`downloading` → `transcribing` → `ready` or `failed`).
+- **Concurrency:** all writes to `kb-jobs.json` must use `flock` on `~/.arka-os/kb-jobs.lock` to avoid corruption.
+- **Logs:** per-job — `download.log`, `transcribe.log`, `worker.log` in the job output directory.
+- **LLM boundary:** worker NEVER calls the LLM. The `ready` → `analyzing` → `completed` transitions happen only when `/kb process <job-id>` runs in Claude Code.
+
+## Job Status Flow
+
+```
+queued → downloading → transcribing → ready → analyzing → completed
+              ↓              ↓
+           failed          failed
+```
+
+## Media Storage
+
+```
+~/.arka-os/
+├── media/                          # Permanent, organized media storage
+│   ├── 2026-03-15/                 # Date-based grouping
+│   │   ├── a1b2c3d4/               # Job ID directory
+│   │   │   ├── metadata.json       # yt-dlp output (title, duration)
+│   │   │   ├── audio.wav           # Downloaded audio file
+│   │   │   ├── audio.txt           # Transcription output
+│   │   │   ├── download.log        # yt-dlp log
+│   │   │   ├── transcribe.log      # Whisper log
+│   │   │   └── worker.log          # Background process log
+├── kb-jobs.json                    # Job state file
+├── capabilities.json               # System capabilities
+└── .env                            # API keys
+```
+
+## Job State File Shape (`~/.arka-os/kb-jobs.json`)
+
+```json
+{
+  "jobs": [
+    {
+      "id": "a1b2c3d4",
+      "url": "<youtube-url>",
+      "persona": "<Name or null>",
+      "status": "queued|downloading|transcribing|ready|analyzing|completed|failed",
+      "method": "whisper-local|whisper-api|gemini|openrouter",
+      "output_dir": "~/.arka-os/media/<date>/<id>/",
+      "title": "<video title>",
+      "duration": "<hh:mm:ss>",
+      "created_at": "<ISO8601>",
+      "updated_at": "<ISO8601>",
+      "error": null
+    }
+  ]
+}
+```

package/departments/kb/skills/knowledge/references/ingestion-pipeline.md

@@ -0,0 +1,308 @@
+# arka-knowledge — ingestion-pipeline
+
+Referenced from SKILL.md. Read only when needed.
+
+## /kb learn <url> [url2 ...] [--persona "Name"]
+
+**This command is NON-BLOCKING.** It queues jobs and returns immediately.
+
+### Step 1: Check Capabilities
+```bash
+bash <scripts-dir>/kb-check-capabilities.sh
+```
+Read `~/.arka-os/capabilities.json`. If `yt-dlp` is not available, tell the user to install it and stop. If no transcription method is available, warn the user (download-only mode).
+
+### Step 2: Queue Each URL
+For each URL provided, run:
+```bash
+bash <scripts-dir>/kb-queue.sh "<url>" --persona "<Name>"
+```
+Returns a job ID (8 chars) immediately. Download + transcription runs in the background.
+
+### Step 3: Display Summary
+```
+═══ ARKA KB — Jobs Queued ═══
+  Job a1b2c3d4 → <url1>
+  Job e5f6g7h8 → <url2>
+  ...
+Transcription: <method>
+Media: ~/.arka-os/media/<date>/
+
+Run /kb queue to check progress.
+Run /kb process <job-id> when jobs are ready.
+═════════════════════════════
+```
+
+**IMPORTANT:** Do NOT wait for downloads to complete. Return to the user immediately after queuing.
+
+## /kb process <job-id>
+
+Analyze a ready transcription. This is the INTERACTIVE step that requires Claude Code's LLM.
+
+### Step 1: Validate Job
+Read `~/.arka-os/kb-jobs.json`. Find job by ID. Verify status is `ready`.
+
+### Step 2: Read Transcript
+Read `<job-output-dir>/audio.txt` for the transcription.
+Read `<job-output-dir>/metadata.json` for video title, duration, etc.
+
+### Step 3: Ask User What To Do
+
+Use AskUserQuestion:
+
+1. **Full analysis** — Run all 5 agents, create/update persona + source + topics + MOC pages
+2. **Create/update persona only**
+3. **Extract frameworks only**
+4. **Save transcript to Obsidian only**
+5. **Custom analysis**
+
+### Step 4: Update Job Status
+Set status to `analyzing` using flock:
+```bash
+(flock -x 200; jq --arg id "<job-id>" '(.jobs[] | select(.id == $id)).status = "analyzing"' ~/.arka-os/kb-jobs.json > /tmp/kb-tmp.$$.json && mv /tmp/kb-tmp.$$.json ~/.arka-os/kb-jobs.json) 200>~/.arka-os/kb-jobs.lock
+```
+
+### Step 5: Execute Analysis (5 parallel agents + DISC)
+
+Launch simultaneously via Task tool:
+
+**Agent 1: Frameworks Extractor** — frameworks, models, methodologies, step-by-step processes, acronyms.
+
+**Agent 2: Strategy Analyzer** — strategies, tactics, specific advice, results/numbers/case studies.
+
+**Agent 3: Voice & Style Profiler** — speaking style (formal/casual), repeated phrases, opening/closing patterns, metaphors.
+
+**Agent 4: Principles Extractor** — core beliefs, what they argue against, driving philosophy.
+
+**Agent 5: Topic Cataloger** — topics covered, relation to existing KB topics, keywords/categories.
+
+**Agent 6: DISC Behavioral Profiler**
+- Likely DISC profile from the content
+- Pace: fast (D/I) vs deliberate (S/C)
+- Focus: tasks/results (D/C) vs people (I/S)
+- Handling disagreement: confront=D, persuade=I, avoid=S, analyze=C
+- Linguistic patterns: imperative=D, enthusiasm=I, caution=S, data-heavy=C
+- Secondary profile + confidence (high/medium/low) with evidence
+
+### Step 6: Write to Obsidian
+
+**Create/Update Persona** — `Personas/<Name>.md`:
+
+```markdown
+---
+type: persona
+name: <Full Name>
+expertise:
+  - "<primary expertise>"
+  - "<secondary expertise>"
+date_updated: <YYYY-MM-DD>
+tags:
+  - "persona"
+  - "<expertise-kebab-case>"
+---
+
+# <Full Name>
+
+> [One-line description]
+
+## Voice & Style
+
+[From Agent 3]
+
+## Behavioral Profile (DISC)
+
+> **Primary:** {X} ({name}) | **Secondary:** {Y} ({name}) | **Confidence:** {level}
+
+[From Agent 6]
+
+### Communication Patterns
+- **Pace:** {observed}
+- **Focus:** {task vs people}
+- **Decision speed:** {how they advocate for action}
+
+### How to Create Content as This Persona
+- **Tone calibration:** {DISC-based}
+- **Opening pattern:** {DISC-informed}
+- **Argument structure:** {how they build arguments}
+- **Call-to-action style:** {how they close}
+
+### DISC Evidence
+- "{quote for primary}"
+- "{quote for secondary}"
+
+## Core Philosophy
+
+[From Agent 4]
+
+## Key Frameworks
+
+[From Agent 1]
+
+### Framework: <Name>
+- Step 1: ...
+- Step 2: ...
+
+## Strategies & Tactics
+
+[From Agent 2]
+
+## Notable Quotes
+
+> "Exact quote"
+> "Another memorable phrase"
+
+## Sources
+
+- [[<YYYY-MM-DD> <Video Title>]]
+
+---
+*Part of the [[Personas MOC]]*
+```
+
+**If existing persona — UPDATE:**
+1. Read existing file
+2. MERGE new info (don't replace)
+3. Add new frameworks, strategies, quotes
+4. Add new source link
+5. Update `date_updated`
+6. Note contradictions if position changed
+
+**Create Source File** — `Sources/Videos/<YYYY-MM-DD> <Video Title>.md`:
+
+```markdown
+---
+type: source
+source_type: video
+title: "<Video Title>"
+url: "<youtube-url>"
+persona: "[[<Name>]]"
+date_processed: <YYYY-MM-DD>
+duration: "<duration>"
+tags:
+  - "source"
+  - "video"
+  - "<topic-kebab-case>"
+---
+
+# <Video Title>
+
+> Source video for [[<Name>]] persona
+
+## Key Takeaways
+
+[Top 5-10 insights]
+
+## Frameworks Found
+
+[List — link to persona]
+
+## Full Analysis
+
+[Combined output from 5 agents]
+
+## Raw Transcript
+
+<details>
+<summary>Click to expand full transcript</summary>
+
+[Full transcription text]
+
+</details>
+
+---
+*Part of the [[Sources MOC]]*
+```
+
+**Catalog by Topic** — `Topics/<Topic Name>.md`:
+
+```markdown
+---
+type: topic
+name: <Topic Name>
+related_personas:
+  - "[[<Name>]]"
+date_updated: <YYYY-MM-DD>
+tags:
+  - "topic"
+  - "<topic-kebab-case>"
+---
+
+# <Topic Name>
+
+## Perspectives
+
+### [[<Persona Name>]]
+[What this persona says about this topic]
+
+---
+*Part of the [[Topics MOC]]*
+```
+
+**Update MOC Pages:** `Personas MOC.md`, `Sources MOC.md`, `Topics MOC.md`. Create if absent.
+
+### Step 7: Update Job Status
+Set status to `completed` in `~/.arka-os/kb-jobs.json`.
+
+### Step 8: Report
+```
+═══ ARKA KB — Processing Complete ═══
+Job:        <job-id>
+Persona:    <Name> (new/updated)
+Source:     "<Video Title>" (YouTube)
+Duration:   <duration>
+Vault:      Personas/<Name>.md
+Source:     Sources/Videos/<date> <title>.md
+New frameworks found: <count>
+New strategies found: <count>
+Topics tagged: <list>
+Media:      <output-dir>
+══════════════════════════════════════
+```
+
+## /kb process --all
+
+Process all jobs with status `ready`:
+1. Read `~/.arka-os/kb-jobs.json`
+2. Filter `status == ready`
+3. Run the `/kb process <job-id>` workflow for each
+4. Ask once what analysis type to use (or per-job)
+
+## /kb learn-text <file/url> --persona "Name"
+
+Same workflow as `/kb process` full analysis but skip download/transcribe.
+- If URL: use WebFetch
+- If file: read directly
+- Source goes to `Sources/Articles/`
+- Synchronous (no background needed)
+
+## /kb write --persona "Name" --type <type>
+
+1. Read `Personas/<Name>.md` for voice and style
+2. Read linked source files for frameworks and strategies
+3. Generate content in that persona's approach
+
+Supported types: `landing-page`, `email`, `ad`, `social-post`, `blog`, `pitch`, `script`.
+
+With `--personas` (plural) → blend: primary's voice, frameworks from all, annotate origin.
+
+## /kb search <query>
+
+1. Grep across `Personas/`, `Topics/`, `Sources/`
+2. Return results organized by relevance
+3. Show which personas have insights
+4. Include quotes and framework references
+
+## /kb personas
+
+List all files in `Personas/`. Show: name, expertise, source count, last updated.
+
+## /kb topics
+
+List all files in `Topics/`. Show: topic, related personas, last updated.
+
+## /kb cleanup [--older-than 90d]
+
+```bash
+bash <scripts-dir>/kb-cleanup.sh --older-than <days>
+```
+Use `--dry-run` first. Confirm before deletion. Report space freed.

package/departments/kb/skills/knowledge/scripts/kb-check-capabilities.sh

@@ -0,0 +1,143 @@
+#!/bin/bash
+# ============================================================================
+# ARKA OS — KB Capability Detection
+# Probes local binaries and API keys, writes ~/.arka-os/capabilities.json
+# ============================================================================
+set -euo pipefail
+
+ARKA_DIR="$HOME/.arka-os"
+CAPS_FILE="$ARKA_DIR/capabilities.json"
+ENV_FILE="$ARKA_DIR/.env"
+
+mkdir -p "$ARKA_DIR"
+
+# Load env if present
+[ -f "$ENV_FILE" ] && source "$ENV_FILE"
+
+# ─── Binary checks ──────────────────────────────────────────────────────────
+
+check_binary() {
+    local name="$1"
+    local path=""
+    local version=""
+
+    if path=$(command -v "$name" 2>/dev/null); then
+        case "$name" in
+            whisper)  version=$(pip3 show openai-whisper 2>/dev/null | grep "^Version:" | cut -d' ' -f2 || echo "installed") ;;
+            yt-dlp)   version=$($path --version 2>&1 | head -1 || echo "unknown") ;;
+            ffmpeg)   version=$(ffmpeg -version 2>&1 | head -1 | sed 's/ffmpeg version //' | cut -d' ' -f1 || echo "unknown") ;;
+            jq)       version=$($path --version 2>&1 || echo "unknown") ;;
+            python3)  version=$($path --version 2>&1 | cut -d' ' -f2 || echo "unknown") ;;
+        esac
+        version=$(printf '%s' "$version" | tr -d '\n\r' | tr -cd '[:print:]')
+        printf '{"available":true,"path":"%s","version":"%s"}' "$path" "$version"
+    else
+        printf '{"available":false,"path":null,"version":null}'
+    fi
+}
+
+# ─── API key checks ─────────────────────────────────────────────────────────
+
+check_key() {
+    local var_name="$1"
+    local value="${!var_name:-}"
+    if [ -n "$value" ] && [ ${#value} -ge 8 ]; then
+        printf '{"configured":true,"key_prefix":"%s***"}' "${value:0:6}"
+    else
+        printf '{"configured":false,"key_prefix":null}'
+    fi
+}
+
+# ─── Determine transcription method ─────────────────────────────────────────
+
+whisper_available=false
+openai_available=false
+
+if command -v whisper &>/dev/null; then
+    whisper_available=true
+fi
+if [ -n "${OPENAI_API_KEY:-}" ] && [ ${#OPENAI_API_KEY} -ge 8 ]; then
+    openai_available=true
+fi
+
+if $whisper_available; then
+    transcription_method="local_whisper"
+    transcription_note="Using local Whisper (free, private)"
+elif $openai_available; then
+    transcription_method="openai_api"
+    transcription_note="Using OpenAI Whisper API (requires credits)"
+else
+    transcription_method="none"
+    transcription_note="No transcription available. Install whisper (pip install openai-whisper) or set OPENAI_API_KEY."
+fi
+
+# ─── Build JSON ──────────────────────────────────────────────────────────────
+
+cat > "$CAPS_FILE" << JSONEOF
+{
+  "generated_at": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
+  "binaries": {
+    "whisper": $(check_binary whisper),
+    "yt-dlp": $(check_binary yt-dlp),
+    "ffmpeg": $(check_binary ffmpeg),
+    "jq": $(check_binary jq),
+    "python3": $(check_binary python3)
+  },
+  "api_keys": {
+    "OPENAI_API_KEY": $(check_key OPENAI_API_KEY),
+    "GEMINI_API_KEY": $(check_key GEMINI_API_KEY),
+    "OPENROUTER_API_KEY": $(check_key OPENROUTER_API_KEY),
+    "REPLICATE_API_TOKEN": $(check_key REPLICATE_API_TOKEN),
+    "FAL_KEY": $(check_key FAL_KEY)
+  },
+  "transcription": {
+    "method": "$transcription_method",
+    "note": "$transcription_note"
+  }
+}
+JSONEOF
+
+# ─── Pretty output if running interactively ──────────────────────────────────
+
+if [ -t 1 ]; then
+    CYAN='\033[0;36m'
+    GREEN='\033[0;32m'
+    YELLOW='\033[1;33m'
+    RED='\033[0;31m'
+    NC='\033[0m'
+
+    echo ""
+    echo -e "${CYAN}═══ ARKA OS — Capabilities ═══${NC}"
+    echo ""
+    echo -e "${CYAN}Binaries:${NC}"
+    for bin in whisper yt-dlp ffmpeg jq python3; do
+        if command -v "$bin" &>/dev/null; then
+            echo -e "  ${GREEN}✓${NC} $bin ($(command -v "$bin"))"
+        else
+            echo -e "  ${RED}✗${NC} $bin — not found"
+        fi
+    done
+
+    echo ""
+    echo -e "${CYAN}API Keys:${NC}"
+    for key in OPENAI_API_KEY GEMINI_API_KEY OPENROUTER_API_KEY REPLICATE_API_TOKEN FAL_KEY; do
+        value="${!key:-}"
+        if [ -n "$value" ] && [ ${#value} -ge 8 ]; then
+            echo -e "  ${GREEN}✓${NC} $key (${value:0:6}***)"
+        else
+            echo -e "  ${YELLOW}⚠${NC} $key — not set"
+        fi
+    done
+
+    echo ""
+    echo -e "${CYAN}Transcription:${NC}"
+    case "$transcription_method" in
+        local_whisper) echo -e "  ${GREEN}✓${NC} $transcription_note" ;;
+        openai_api)    echo -e "  ${GREEN}✓${NC} $transcription_note" ;;
+        none)          echo -e "  ${RED}✗${NC} $transcription_note" ;;
+    esac
+
+    echo ""
+    echo -e "Written to: ${CYAN}$CAPS_FILE${NC}"
+    echo -e "${CYAN}═══════════════════════════════${NC}"
+fi