mega-brain-ai 1.1.0

package/system/docs/architecture/04-integrations.md

@@ -0,0 +1,283 @@
+# Mega Brain - Integration Architecture
+
+## Integration Overview
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                         MEGA BRAIN INTEGRATIONS                          │
+└─────────────────────────────────────────────────────────────────────────┘
+
+                              ┌───────────────┐
+                              │  MEGA BRAIN   │
+                              └───────┬───────┘
+                                      │
+            ┌─────────────────────────┼─────────────────────────┐
+            │                         │                         │
+            ▼                         ▼                         ▼
+    ┌───────────────┐        ┌───────────────┐        ┌───────────────┐
+    │   YouTube     │        │  AssemblyAI   │        │  Filesystem   │
+    │ Transcript API│        │     API       │        │     MCP       │
+    └───────────────┘        └───────────────┘        └───────────────┘
+```
+
+---
+
+## API Integrations
+
+### 1. YouTube Transcript API
+
+**Purpose:** Fetch transcripts from YouTube videos without downloading the video
+
+**Package:** `youtube-transcript-api`
+
+**Version:** 1.2.3+
+
+| Property | Value |
+|----------|-------|
+| **Protocol** | HTTPS |
+| **Auth** | None required |
+| **Rate Limit** | Fair use (no official limit) |
+| **Data Format** | JSON array |
+
+**Usage:**
+```python
+from youtube_transcript_api import YouTubeTranscriptApi
+
+# Initialize (new API in v1.2.3+)
+api = YouTubeTranscriptApi()
+
+# Fetch transcript
+transcript = api.fetch("VIDEO_ID")
+
+# Response format
+[
+    {"text": "Hello everyone", "start": 0.0, "duration": 2.5},
+    {"text": "Today we're going to talk about", "start": 2.5, "duration": 3.0},
+    ...
+]
+```
+
+**Error Handling:**
+| Error | Cause | Action |
+|-------|-------|--------|
+| `TranscriptsDisabled` | Video has no captions | Use AssemblyAI |
+| `NoTranscriptFound` | Captions not in target language | Try auto-generated |
+| `VideoUnavailable` | Private or deleted video | Inform user |
+
+---
+
+### 2. AssemblyAI API
+
+**Purpose:** Transcribe local audio/video files with advanced features
+
+**Package:** `assemblyai`
+
+| Property | Value |
+|----------|-------|
+| **Protocol** | HTTPS REST |
+| **Auth** | API Key (header) |
+| **Rate Limit** | Based on plan |
+| **Data Format** | JSON |
+
+**Configuration:**
+```python
+import assemblyai as aai
+
+# Set API key
+aai.settings.api_key = os.getenv("ASSEMBLYAI_API_KEY")
+
+# Configure transcription
+config = aai.TranscriptionConfig(
+    language_code="en",
+    punctuate=True,
+    format_text=True,
+    auto_chapters=True,
+    speaker_labels=True,
+    entity_detection=True,
+)
+
+# Transcribe
+transcriber = aai.Transcriber()
+transcript = transcriber.transcribe(file_path, config=config)
+```
+
+**Features Used:**
+| Feature | Description | Use Case |
+|---------|-------------|----------|
+| `punctuate` | Add punctuation | Readability |
+| `format_text` | Format output | Clean text |
+| `auto_chapters` | Detect topics | Navigation |
+| `speaker_labels` | Identify speakers | Multi-speaker content |
+| `entity_detection` | Extract entities | Names, companies |
+
+**Environment Variable:**
+```bash
+ASSEMBLYAI_API_KEY=af1dcbc4367d4224bf879b12c1995921
+```
+
+---
+
+## MCP Integrations
+
+### Filesystem MCP
+
+**Purpose:** Provide Claude Code access to local filesystem
+
+**Package:** `@anthropic-ai/mcp-server-filesystem`
+
+**Configuration (settings.local.json):**
+```json
+{
+  "mcpServers": {
+    "filesystem": {
+      "command": "npx",
+      "args": [
+        "-y",
+        "@anthropic-ai/mcp-server-filesystem",
+        "/path/to/your/mega-brain"
+      ],
+      "env": {}
+    }
+  }
+}
+```
+
+**Capabilities:**
+- Read files within project directory
+- Write files within project directory
+- List directory contents
+- File metadata access
+
+**Security:**
+- Scoped to project directory only
+- No access outside defined path
+
+---
+
+## Hook Integrations
+
+### PreToolUse Hooks
+
+**WebSearch Enhancement:**
+```json
+{
+  "matcher": "WebSearch",
+  "hooks": [{
+    "type": "command",
+    "command": "echo 'Adding current year to search query if not present'"
+  }]
+}
+```
+
+### PostToolUse Hooks
+
+**INBOX Monitoring:**
+```json
+{
+  "matcher": "Write",
+  "hooks": [{
+    "type": "command",
+    "command": "if [[ \"$TOOL_INPUT\" == *\"inbox\"* ]] && [[ \"$TOOL_INPUT\" == *\".txt\"* ]]; then echo '[INBOX] New transcript detected'; fi"
+  }]
+}
+```
+
+---
+
+## Permission Configuration
+
+### Allowed Operations
+
+**Bash Commands:**
+```json
+[
+  "Bash(mkdir:*)",
+  "Bash(cp:*)",
+  "Bash(mv:*)",
+  "Bash(ls:*)",
+  "Bash(python:*)",
+  "Bash(python3:*)",
+  "Bash(pip:*)",
+  "Bash(npm:*)",
+  "Bash(node:*)",
+  "Bash(git:*)",
+  "Bash(yt-dlp:*)"
+]
+```
+
+**File Operations:**
+```json
+[
+  "Read(*)",
+  "Write(*)",
+  "Edit(*)"
+]
+```
+
+**Web Access:**
+```json
+[
+  "WebFetch(domain:youtube.com)",
+  "WebFetch(domain:www.youtube.com)",
+  "WebFetch(domain:youtu.be)",
+  "WebFetch(domain:api.assemblyai.com)",
+  "WebSearch"
+]
+```
+
+---
+
+## Environment Variables
+
+| Variable | Purpose | Required |
+|----------|---------|----------|
+| `ASSEMBLYAI_API_KEY` | AssemblyAI authentication | For local transcription |
+| `MEGA_BRAIN_ROOT` | Project root path | Optional (default) |
+| `KNOWLEDGE_PATH` | Knowledge folder path | Optional (default) |
+| `INBOX_PATH` | INBOX folder path | Optional (default) |
+
+**Default Paths (settings.local.json):**
+```json
+{
+  "env": {
+    "MEGA_BRAIN_ROOT": "/path/to/your/mega-brain",
+    "KNOWLEDGE_PATH": "/path/to/your/mega-brain/knowledge",
+    "INBOX_PATH": "/path/to/your/mega-brain/inbox"
+  }
+}
+```
+
+---
+
+## Integration Dependencies
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                         DEPENDENCY GRAPH                                 │
+└─────────────────────────────────────────────────────────────────────────┘
+
+  Python Packages              Node Packages              System
+  ═══════════════              ═════════════              ══════
+
+  youtube-transcript-api       @anthropic-ai/             Claude Code
+        │                      mcp-server-filesystem           │
+        │                            │                         │
+        ▼                            ▼                         ▼
+  assemblyai                    npx (npm)               Bash shell
+        │                            │                         │
+        └────────────────────────────┴─────────────────────────┘
+                                     │
+                               MEGA BRAIN
+```
+
+---
+
+## Future Integration Opportunities
+
+| Integration | Purpose | Priority |
+|-------------|---------|----------|
+| **Notion API** | Export knowledge to Notion | Medium |
+| **Obsidian** | Sync with Obsidian vault | Medium |
+| **Whisper API** | Alternative transcription | Low |
+| **Deepgram** | Alternative transcription | Low |
+| **PostgreSQL MCP** | Structured knowledge storage | Low |

package/system/docs/architecture/README.md

@@ -0,0 +1,71 @@
+# Mega Brain - Architecture Documentation
+
+## Overview
+
+This folder contains comprehensive architecture documentation for the Mega Brain knowledge management system.
+
+## Documentation Structure
+
+```
+docs/architecture/
+├── README.md                    # This file
+├── 01-system-context.md         # High-level system view (C4 Level 1)
+├── 02-components.md             # Component breakdown (C4 Level 2)
+├── 03-data-flow.md              # Data transformation pipelines
+├── 04-integrations.md           # External APIs and MCPs
+└── diagrams/
+    ├── system-overview.mmd      # Main system diagram
+    ├── component-diagram.mmd    # Component relationships
+    └── data-flow.mmd            # Sequence diagram
+```
+
+## Quick Navigation
+
+| Document | Description | Audience |
+|----------|-------------|----------|
+| [System Context](01-system-context.md) | What the system does and who uses it | All stakeholders |
+| [Components](02-components.md) | Internal structure and configuration | Developers |
+| [Data Flow](03-data-flow.md) | How data moves through the system | Developers |
+| [Integrations](04-integrations.md) | External dependencies and APIs | DevOps, Developers |
+
+## Diagrams
+
+All diagrams are in Mermaid format (`.mmd`). To view them:
+
+### Option 1: VS Code
+Install the "Markdown Preview Mermaid Support" extension
+
+### Option 2: Online
+Paste content into [Mermaid Live Editor](https://mermaid.live/)
+
+### Option 3: CLI
+```bash
+npm install -g @mermaid-js/mermaid-cli
+mmdc -i diagrams/system-overview.mmd -o system-overview.png
+```
+
+## Key Architectural Decisions
+
+1. **Local Processing**: All content processed locally for privacy
+2. **AI-Powered Extraction**: Using Claude Code for intelligent analysis
+3. **Theme-Based Organization**: 10 predefined knowledge themes
+4. **Source Attribution**: Every insight traced back to source
+
+## Technology Stack
+
+| Layer | Technology |
+|-------|------------|
+| AI Runtime | Claude Code CLI |
+| Configuration | JSON, Markdown |
+| Scripts | Python 3.x |
+| Transcription | youtube-transcript-api, AssemblyAI |
+| MCP | @anthropic-ai/mcp-server-filesystem |
+
+## Generated
+
+This documentation was auto-generated using:
+```
+/documentation/create-architecture-documentation
+```
+
+Last updated: December 2024

package/system/docs/architecture/diagrams/component-diagram.mmd

@@ -0,0 +1,50 @@
+%%{init: {'theme': 'base'}}%%
+
+graph TB
+    subgraph Claude[".claude/ Configuration"]
+        direction TB
+        subgraph Agents["agents/"]
+            A1[b2b-sales-expert.md]
+        end
+        subgraph Commands["commands/"]
+            C1[process-video.md]
+            C2[extract-knowledge.md]
+            C3[scan-inbox.md]
+            C4[documentation/...]
+        end
+        subgraph Skills["skills/"]
+            S1[video-processing/]
+            S2[knowledge-extraction/]
+        end
+        SETTINGS[settings.local.json]
+    end
+
+    subgraph Scripts["scripts/"]
+        PY1[transcribe.py]
+        PY2[auto_organize_inbox.py]
+    end
+
+    subgraph Data["Data Folders"]
+        INBOX[(00-INBOX)]
+        KNOWLEDGE[(KNOWLEDGE/)]
+    end
+
+    %% Dependencies
+    C1 --> S1
+    C2 --> S2
+    C2 --> A1
+    C3 --> PY2
+
+    S1 --> PY1
+
+    C1 --> INBOX
+    C2 --> KNOWLEDGE
+    C3 --> INBOX
+
+    SETTINGS -.->|configures| Commands
+    SETTINGS -.->|configures| Skills
+
+    %% Styling
+    style Claude fill:#e3f2fd,stroke:#1976d2
+    style Scripts fill:#fff3e0,stroke:#f57c00
+    style Data fill:#e8f5e9,stroke:#388e3c

package/system/docs/architecture/diagrams/data-flow.mmd

@@ -0,0 +1,39 @@
+%%{init: {'theme': 'base'}}%%
+
+sequenceDiagram
+    participant U as User
+    participant CMD as Claude Commands
+    participant YT as YouTube API
+    participant AAI as AssemblyAI
+    participant INBOX as 00-INBOX
+    participant AGENT as B2B Sales Agent
+    participant KB as KNOWLEDGE/
+
+    Note over U,KB: Pipeline 1: YouTube Video Processing
+    U->>CMD: /process-video [YouTube URL]
+    CMD->>YT: Fetch transcript
+    YT-->>CMD: Return transcript data
+    CMD->>INBOX: Save transcript.txt
+    CMD-->>U: Transcript saved
+
+    Note over U,KB: Pipeline 2: Local File Processing
+    U->>CMD: /process-video [local file]
+    CMD->>AAI: Upload & transcribe
+    AAI-->>CMD: Return transcript
+    CMD->>INBOX: Save transcript.txt
+    CMD-->>U: Transcript saved
+
+    Note over U,KB: Pipeline 3: Knowledge Extraction
+    U->>CMD: /extract-knowledge [transcript]
+    CMD->>INBOX: Read transcript
+    CMD->>AGENT: Analyze content
+    AGENT->>AGENT: Detect themes
+    AGENT->>AGENT: Extract insights
+    AGENT->>KB: Create/update knowledge files
+    AGENT-->>U: Extraction complete
+
+    Note over U,KB: Utility: Scan INBOX
+    U->>CMD: /scan-inbox
+    CMD->>INBOX: List files
+    CMD->>CMD: Detect themes
+    CMD-->>U: Suggest actions

package/system/docs/architecture/diagrams/system-overview.mmd

@@ -0,0 +1,68 @@
+%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#4a90a4', 'primaryTextColor': '#fff', 'primaryBorderColor': '#2d5a6b', 'lineColor': '#5a5a5a', 'secondaryColor': '#f4f4f4', 'tertiaryColor': '#e8e8e8'}}}%%
+
+graph TD
+    subgraph External["External Sources"]
+        YT[YouTube Video]
+        LV[Local Video/Audio]
+        DOC[Documents]
+    end
+
+    subgraph Input["00-INBOX"]
+        INBOX[(Drop Zone)]
+    end
+
+    subgraph Processing["Processing Layer"]
+        YTAPI[YouTube Transcript API]
+        AAI[AssemblyAI API]
+        AGENT[b2b-sales-expert Agent]
+    end
+
+    subgraph Commands["Claude Commands"]
+        CMD1["/process-video"]
+        CMD2["/extract-knowledge"]
+        CMD3["/scan-inbox"]
+    end
+
+    subgraph Output["KNOWLEDGE Base"]
+        K01[01-estrutura-time]
+        K02[02-processo-vendas]
+        K03[03-contratacao]
+        K04[04-comissionamento]
+        K05[05-metricas]
+        K99[99-secundario]
+    end
+
+    %% Flow
+    YT -->|URL| CMD1
+    LV -->|File path| CMD1
+    DOC --> INBOX
+
+    CMD1 -->|YouTube| YTAPI
+    CMD1 -->|Local| AAI
+
+    YTAPI --> INBOX
+    AAI --> INBOX
+
+    CMD3 -->|Scans| INBOX
+    INBOX -->|Transcript| CMD2
+
+    CMD2 --> AGENT
+    AGENT -->|Classify| K01
+    AGENT -->|Classify| K02
+    AGENT -->|Classify| K03
+    AGENT -->|Classify| K04
+    AGENT -->|Classify| K05
+    AGENT -->|Classify| K99
+
+    %% Styling
+    classDef external fill:#e3f2fd,stroke:#1976d2
+    classDef input fill:#fff3e0,stroke:#f57c00
+    classDef processing fill:#e8f5e9,stroke:#388e3c
+    classDef command fill:#fce4ec,stroke:#c2185b
+    classDef output fill:#f3e5f5,stroke:#7b1fa2
+
+    class YT,LV,DOC external
+    class INBOX input
+    class YTAPI,AAI,AGENT processing
+    class CMD1,CMD2,CMD3 command
+    class K01,K02,K03,K04,K05,K99 output