groove-dev 0.27.77 → 0.27.78

package/CLAUDE.md

@@ -263,10 +263,3 @@ Audit-driven release. Multi-agent orchestration system with 7 coordination layer
 - Dashboard: routing donut, cache panel, context health gauges
 - Monitor/QC agent mode (stay active, loop)
 - Distribution: demo video, HN launch, Twitter content
-
-<!-- GROOVE:START -->
-## GROOVE Orchestration (auto-injected)
-Active agents: 0
-See AGENTS_REGISTRY.md for full agent state.
-**Memory policy:** GROOVE manages project memory automatically. Do not read or write MEMORY.md or .groove/memory/ files directly.
-<!-- GROOVE:END -->

package/MOE_TRAINING_PIPELINE.md

@@ -706,15 +706,219 @@ the network team to add patterns to `scrubber.py`.
 
 ---
 
-## 10. File Reference
-
-| File | Description |
-|------|-------------|
-| `moe-team/src/training/__init__.py` | Package exports: CaptureSession, ConsentManager, ConsentRecord, CorpusStats, DomainTagger, PIIScrubber, TrainingCorpus, TrainingRecord |
-| `moe-team/src/training/intake.py` | TrainingDataIntake — simple submit/batch/delete API |
-| `moe-team/src/training/consent.py` | ConsentManager — SQLite consent storage, versioned consent |
-| `moe-team/src/training/scrubber.py` | PIIScrubber — 13 compiled regex patterns for PII removal |
-| `moe-team/src/training/domain_tagger.py` | DomainTagger — keyword-based domain classification |
-| `moe-team/src/training/corpus.py` | TrainingCorpus — JSONL storage with daily rotation |
-| `moe-team/src/training/capture.py` | CaptureSession — session-oriented capture with lifecycle |
-| `moe-team/src/training/stats.py` | CorpusStats — summary, daily growth, domain breakdown |
+## 10. Build plan
+
+1.
+TrajectoryCapture is a standalone component, runs in parallel, behind the opt-in toggle. Not embedded in the Journalist. When the user hasn't opted in, the module doesn't even load — zero overhead. When they flip the switch, it hooks into the same stdout stream the Journalist reads but operates independently with pure pattern matching. No AI cost on the user's machine.
+1.
+AI enrichment (cognitive_target classification, quality scoring, model fingerprinting via LLM-as-a-Judge) happens at Central Command on your AWS instance. The user's machine sends raw structured trajectories. Central Command does the expensive analysis. This means you control the enrichment model, can upgrade it without shipping app updates, and users pay nothing extra.
+1.
+Both sides of every coordination exchange get captured. When agent A knocks on agent B's lock, you capture agent A's request envelope AND agent B's response envelope, each with a shared coordination_id that links them. This gives the MoE training data the full picture: how to ask for help AND how to respond to requests. This is the data that teaches Groove agents to be team players. No other dataset on earth has this.
+1.
+Streaming chunked envelopes. Send sub-envelopes of up to 200 steps during the session, each signed with the ECDH session HMAC and incrementing sequence numbers. When the session ends, send a final SESSION_CLOSE envelope with the outcome data (success/failure, user interventions count, total tokens, duration). Central Command stitches the chunks into the full trajectory using session_id + sequence numbers. This way you don't lose data on crashes, and you can do real-time quality monitoring on incoming data.
+THE FULL ARCHITECTURE — TrajectoryCapture System:
+
+Component: TrajectoryCapture (new file in packages/daemon/src/)
+
+Lifecycle:
+
+-
+Daemon starts: checks opt-in flag (is_capture_enabled). If false, does nothing. Zero imports, zero overhead.
+-
+User opts in: daemon loads TrajectoryCapture, it registers as a listener on ProcessManager's stdout stream
+-
+Agent spawns: TrajectoryCapture opens a new capture context for that agent, performs ECDH handshake with Central Command, gets session attestation
+-
+During session: parses stdout into typed steps, buffers until 200 steps or 5 minutes (whichever comes first), signs and transmits chunk
+-
+Agent completes/crashes/killed: sends SESSION_CLOSE with outcome data
+-
+User opts out: TrajectoryCapture detaches from stdout stream, flushes any buffered data, tears down
+Step Classification (deterministic, pattern-based):
+
+Each provider's parseOutput() already knows the output format. TrajectoryCapture extends this with step type detection:
+
+-
+"thought": Lines inside thinking/reasoning blocks. Claude Code wraps these clearly. Codex has internal reasoning markers. Gemini has thinking blocks.
+-
+"action": Tool calls — Read, Edit, Write, Bash, Grep, Glob. Parse the tool name and arguments. For Bash, capture the command string.
+-
+"observation": Tool results — the output that comes back after an action. Truncate intelligently: first 50 + last 20 lines for long outputs, full content for short ones.
+-
+"correction": User message that arrives AFTER the agent has taken at least one action. This is the RLHF gold. The ProcessManager can signal when user input arrives mid-session.
+-
+"resolution": The agent's final summary or commit message. Detect by position (last substantive output before session end) and content patterns (commit, done, implemented, fixed).
+-
+"error": Build failures, test failures, permission errors, crashes. Pattern match on common error signatures (Error:, FAIL, exit code, stack traces).
+-
+"coordination": Lock requests, QC submissions, knock protocol messages. These come through LockManager and Supervisor events, not stdout. TrajectoryCapture also listens to these EventEmitters.
+Envelope Structure (refined):
+
+{
+  "envelope_id": "env_<uuid>",
+  "session_id": "sess_<uuid>",
+  "chunk_sequence": 3,
+  "contributor_id": "<anonymous_install_uuid>",
+  "attestation": {
+    "session_hmac": "<HMAC-SHA256>",
+    "sequence": 47,
+    "app_version_hash": "<sha256 of groove binary>"
+  },
+  "metadata": {
+    "model_engine": "claude-opus-4.6",
+    "provider": "claude-code",
+    "agent_role": "frontend",
+    "agent_id": "frontend-1",
+    "task_complexity": "heavy",
+    "team_size": 4,
+    "session_quality": 82,
+    "groove_version": "0.27.77"
+  },
+  "trajectory_log": [
+    {
+      "step": 1,
+      "type": "thought",
+      "timestamp": 1745625600.0,
+      "content": "I need to fix the WebSocket reconnection logic...",
+      "token_count": 142
+    },
+    {
+      "step": 2,
+      "type": "action",
+      "timestamp": 1745625602.3,
+      "tool": "Grep",
+      "arguments": {"pattern": "WebSocket", "path": "src/"},
+      "token_count": 28
+    },
+    {
+      "step": 3,
+      "type": "observation",
+      "timestamp": 1745625603.1,
+      "content": "Found 4 matches in src/ws-client.js...",
+      "token_count": 89,
+      "truncated": false
+    },
+    {
+      "step": 4,
+      "type": "correction",
+      "timestamp": 1745625680.0,
+      "content": "No, use exponential backoff instead of fixed retry",
+      "source": "user",
+      "token_count": 12
+    },
+    {
+      "step": 5,
+      "type": "coordination",
+      "timestamp": 1745625700.0,
+      "coordination_id": "coord_<uuid>",
+      "direction": "outbound",
+      "target_agent": "backend-1",
+      "protocol": "knock",
+      "content": "Requesting lock on src/api/ws-handler.js"
+    }
+  ]
+}
+The SESSION_CLOSE envelope:
+
+{
+  "envelope_id": "env_<uuid>",
+  "session_id": "sess_<uuid>",
+  "type": "SESSION_CLOSE",
+  "attestation": { ... },
+  "outcome": {
+    "status": "SUCCESS",
+    "user_interventions": 1,
+    "total_steps": 847,
+    "total_chunks": 5,
+    "total_tokens": 48200,
+    "duration_seconds": 1820,
+    "files_modified": 6,
+    "errors_encountered": 2,
+    "errors_recovered": 2,
+    "coordination_events": 3
+  }
+}
+CENTRAL COMMAND (AWS) — what it receives and does:
+
+Endpoint: POST https://api.groovedev.ai/v1/training/ingest
+
+Receives chunked envelopes in real time. For each:
+
+1.
+Verify HMAC against the session's shared secret
+2.
+Verify sequence number is monotonically increasing
+3.
+Store raw envelope in S3 (or local JSONL as staging)
+4.
+On SESSION_CLOSE: stitch all chunks into full trajectory, run AI enrichment:
+-
+Cognitive target classification (syntactic/semantic/strategic/corrective/coordinative)
+-
+Model fingerprint verification (LLM-as-a-Judge)
+-
+Latency profile analysis
+-
+Quality score
+-
+Multiplier calculation (model tier x correction count x complexity x coordination uniqueness)
+1.
+Update the off-chain ledger with contributor points
+2.
+Every 24h: compute Merkle root, publish to Base L2 (future)
+WHAT MAKES THIS DATASET UNIQUE:
+
+The five data types no one else has:
+
+1.
+Multi-agent coordination trajectories — how agents divide work, negotiate locks, hand off context
+2.
+Role-specific reasoning chains — planner thinks differently than QC thinks differently than frontend builder
+3.
+Context rotation recovery — how agent N+1 picks up where agent N left off using the handoff brief
+4.
+User corrections in context — not just "this is wrong," but the full trajectory showing what the agent tried, where it went wrong, and how the human redirected
+5.
+Error-recovery arcs — agent hits wall, tries alternatives, eventually succeeds (or doesn't) — the full struggle
+This is training data for agents that work in teams, recover from mistakes, and learn from human feedback. That's the product.
+
+THE BUILD PLAN:
+
+This breaks into two workstreams:
+
+Workstream 1 — Client side (in the Groove daemon):
+
+-
+TrajectoryCapture component (new daemon module)
+-
+Opt-in toggle wiring (consent flow from the earlier spec)
+-
+ECDH session handshake with Central Command
+-
+Provider-specific stdout parsers for step classification
+-
+Chunked envelope packaging and HMAC signing
+-
+Transmission queue (non-blocking, fail-silent)
+-
+EventEmitter hooks for coordination events
+Workstream 2 — Server side (on AWS/Central Command):
+
+-
+Ingest endpoint (POST /v1/training/ingest)
+-
+Session registry (ECDH key management)
+-
+HMAC verification + sequence validation
+-
+Raw envelope storage
+-
+Stitcher (chunks to full trajectory on SESSION_CLOSE)
+-
+AI enrichment pipeline (runs after stitching)
+-
+Off-chain ledger
+-
+Contributor trust scoring
+-
+Stats/monitoring dashboard

package/moe-training/DEPLOY_CENTRAL_COMMAND.md

@@ -0,0 +1,413 @@
+# Central Command — AWS Deployment Guide
+
+Build guide for deploying the MoE Training Data ingest server on the groovedev.ai AWS instance.
+
+Last updated: 2026-04-23
+
+---
+
+## 1. What This Server Does
+
+Central Command receives Trajectory Envelopes from Groove desktop clients that have opted into training data sharing. It verifies cryptographic attestation (ECDH + HMAC), stores verified envelopes, stitches multi-chunk sessions, scores trajectories using a multiplier matrix, and credits contributors with points.
+
+All source code is in: `moe-training/server/`
+Shared utilities: `moe-training/shared/`
+
+---
+
+## 2. System Requirements
+
+- Node.js 20+ LTS (ES modules, native fetch)
+- build-essential + python3 (for compiling better-sqlite3 native bindings)
+- nginx (reverse proxy, SSL termination)
+- certbot (Let's Encrypt SSL for api.groovedev.ai)
+- PM2 (process manager) or systemd
+- 1GB+ RAM, 20GB+ disk (envelopes grow over time)
+
+---
+
+## 3. Environment Variables
+
+| Variable | Default | Description |
+|---|---|---|
+| GROOVE_CENTRAL_PORT | 8443 | Port the Express server listens on (behind nginx) |
+| NODE_ENV | production | Set to production for security defaults |
+
+The server does NOT need any API keys. It only receives data — it does not call any external APIs. The enrichment pipeline (LLM-as-a-Judge) is currently a stub and will need API keys when activated post-launch.
+
+---
+
+## 4. Directory Structure on Server
+
+```
+/opt/groove-central/
+  moe-training/
+    package.json
+    package-lock.json
+    server/            <-- the server code
+    shared/            <-- shared crypto/schema/constants
+    client/            <-- NOT needed on server, but harmless to include
+    data/              <-- created automatically on first run
+      sessions.db      <-- SQLite: ECDH session state, rate limiting
+      ledger.db        <-- SQLite: contributor points and balances
+      envelopes/       <-- JSONL envelope storage (daily rotation)
+        2026-04-26.jsonl
+        2026-04-27.jsonl
+```
+
+The data/ directory is created automatically by the server components on first request. SQLite databases use WAL mode for crash safety. File permissions: directories 0o700, files 0o600.
+
+---
+
+## 5. Dependencies
+
+```json
+{
+  "better-sqlite3": "^11.0.0",
+  "uuid": "^9.0.0",
+  "express": "^4.18.0"
+}
+```
+
+better-sqlite3 is a native C++ addon — it needs compilation tools:
+
+```bash
+sudo apt update
+sudo apt install -y build-essential python3 git
+```
+
+---
+
+## 6. Deployment Steps
+
+### 6a. Clone and Install
+
+```bash
+# Clone the repo (or scp the moe-training directory)
+cd /opt/groove-central
+git clone https://github.com/grooveai-dev/groove.git
+cd groove/moe-training
+
+# Install dependencies
+npm install --production
+```
+
+### 6b. Test the Server Locally
+
+```bash
+# Start the server
+GROOVE_CENTRAL_PORT=8443 node server/index.js
+
+# In another terminal, verify health
+curl http://localhost:8443/health
+# Expected: {"status":"ok","uptime":...}
+
+# Verify session endpoint
+curl -X POST http://localhost:8443/v1/sessions/open \
+  -H "Content-Type: application/json" \
+  -d '{"session_id":"test-1","public_key":"dGVzdA==","provider":"claude-code","model":"claude-opus-4-6","machine_fingerprint":"test","app_version_hash":"abc","groove_version":"0.27.77"}'
+# Expected: {"server_public_key":"..."}
+
+# Verify stats endpoint
+curl http://localhost:8443/v1/stats/summary
+# Expected: {"totalEnvelopes":0,...}
+
+# Kill the test server (Ctrl+C)
+```
+
+### 6c. PM2 Process Manager
+
+```bash
+# Install PM2 globally
+sudo npm install -g pm2
+
+# Create ecosystem config
+cat > /opt/groove-central/groove/moe-training/ecosystem.config.cjs << 'EOF'
+module.exports = {
+  apps: [{
+    name: 'groove-central',
+    script: 'server/index.js',
+    cwd: '/opt/groove-central/groove/moe-training',
+    env: {
+      NODE_ENV: 'production',
+      GROOVE_CENTRAL_PORT: 8443,
+    },
+    instances: 1,
+    max_memory_restart: '500M',
+    log_date_format: 'YYYY-MM-DD HH:mm:ss',
+    error_file: '/var/log/groove-central/error.log',
+    out_file: '/var/log/groove-central/access.log',
+    merge_logs: true,
+  }],
+};
+EOF
+
+# Create log directory
+sudo mkdir -p /var/log/groove-central
+sudo chown $USER:$USER /var/log/groove-central
+
+# Start with PM2
+pm2 start ecosystem.config.cjs
+
+# Save PM2 config for auto-restart on reboot
+pm2 save
+pm2 startup
+# (follow the instructions PM2 prints to enable startup hook)
+```
+
+### 6d. Nginx Reverse Proxy + SSL
+
+```nginx
+# /etc/nginx/sites-available/groove-central
+
+server {
+    listen 80;
+    server_name api.groovedev.ai;
+
+    # Certbot will add the redirect after SSL is set up
+    location / {
+        return 301 https://$host$request_uri;
+    }
+}
+
+server {
+    listen 443 ssl http2;
+    server_name api.groovedev.ai;
+
+    # SSL certs (managed by certbot)
+    ssl_certificate /etc/letsencrypt/live/api.groovedev.ai/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/api.groovedev.ai/privkey.pem;
+    ssl_protocols TLSv1.2 TLSv1.3;
+    ssl_ciphers HIGH:!aNULL:!MD5;
+
+    # Proxy to Node.js server
+    location /v1/ {
+        proxy_pass http://127.0.0.1:8443;
+        proxy_http_version 1.1;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+
+        # Large envelopes (up to 5MB)
+        client_max_body_size 5m;
+
+        # Timeouts for stitching/scoring on SESSION_CLOSE
+        proxy_read_timeout 30s;
+        proxy_send_timeout 30s;
+    }
+
+    location /health {
+        proxy_pass http://127.0.0.1:8443;
+        proxy_http_version 1.1;
+    }
+
+    # Block everything else
+    location / {
+        return 404;
+    }
+}
+```
+
+```bash
+# Enable the site
+sudo ln -s /etc/nginx/sites-available/groove-central /etc/nginx/sites-enabled/
+sudo nginx -t
+sudo systemctl reload nginx
+
+# Get SSL certificate
+sudo certbot --nginx -d api.groovedev.ai
+```
+
+### 6e. DNS
+
+Point api.groovedev.ai to the AWS instance's public IP:
+- Type: A record
+- Name: api
+- Value: <AWS instance public IP>
+- TTL: 300
+
+If api.groovedev.ai already points to this instance for other services, add the /v1/ location block to the existing nginx config instead of creating a new server block.
+
+---
+
+## 7. API Endpoints
+
+All endpoints are under /v1/ (proxied through nginx).
+
+### Session Management
+
+POST /v1/sessions/open
+  Body: { session_id, public_key (base64), provider, model, machine_fingerprint, app_version_hash, groove_version }
+  Returns: { server_public_key (base64) }
+  Errors: 400 (missing fields), 429 (rate limited: 20 sessions/hour per fingerprint)
+
+POST /v1/sessions/close
+  Body: { session_id }
+  Returns: { closed: true }
+  Errors: 404 (unknown session)
+
+### Data Ingestion
+
+POST /v1/training/ingest
+  Body: Full Trajectory Envelope JSON (up to 5MB)
+  Returns: { accepted: true, envelope_id }
+  Errors: { accepted: false, reason: "..." }
+  Special: SESSION_CLOSE envelopes trigger stitching, scoring, and ledger credit
+
+### Statistics
+
+GET /v1/stats/summary
+  Returns: { totalEnvelopes, totalSteps, totalSessions, activeSessions, uniqueContributors, storageSizeMb, totalPointsAwarded }
+
+GET /v1/stats/daily?days=7
+  Returns: [{ date, envelopes, steps, sessions, points }, ...]
+
+GET /v1/stats/models
+  Returns: { "claude-opus-4-6": { sessions, steps, points, percentage }, ... }
+
+GET /v1/stats/providers
+  Returns: { "claude-code": { sessions, steps, points }, ... }
+
+GET /v1/stats/leaderboard?limit=10
+  Returns: [{ contributor_id (truncated), total_points, total_sessions }, ...]
+
+### Health
+
+GET /health
+  Returns: { status: "ok", uptime: seconds }
+
+---
+
+## 8. ECDH Handshake Flow
+
+This is how client (Groove daemon) and server authenticate:
+
+1. Client generates ephemeral ECDH keypair (prime256v1 curve)
+2. Client POSTs public key to /v1/sessions/open
+3. Server generates its own ECDH keypair, derives shared secret, stores session
+4. Server returns its public key
+5. Client derives the same shared secret (ECDH math)
+6. Every envelope is HMAC-SHA256 signed: HMAC(shared_secret, JSON(envelope) + sequence_number)
+7. Server verifies HMAC and sequence number on each ingest
+8. Sequence numbers are monotonically increasing — prevents replay attacks
+
+The shared secret NEVER crosses the wire. Both sides derive it independently from the key exchange. An attacker would need the server's ephemeral private key to forge envelopes.
+
+---
+
+## 9. Data Storage
+
+### SQLite Databases
+
+sessions.db — Active and closed session records
+  Columns: session_id, server_private_key, server_public_key, shared_secret, client_public_key, provider, model, machine_fingerprint, app_version_hash, groove_version, expected_sequence, status, created_at, closed_at
+  WAL mode enabled for concurrent reads
+
+ledger.db — Contributor points
+  Table credits: id, contributor_id, session_id, points, base_points, multiplier_breakdown (JSON), created_at
+  Table balances: contributor_id (PK), total_points, total_sessions, last_credit_at, trust_score
+  WAL mode enabled
+
+### JSONL Envelope Storage
+
+Location: ./data/envelopes/YYYY-MM-DD.jsonl
+One JSON object per line, daily rotation.
+Each line is a complete verified envelope (chunk or SESSION_CLOSE).
+
+Storage will grow linearly with usage. Estimate: ~1KB per envelope chunk, 5-10 chunks per session.
+At 100 sessions/day = ~1MB/day. At 10,000 sessions/day = ~100MB/day.
+
+---
+
+## 10. Scoring / Multiplier Matrix
+
+When a SESSION_CLOSE envelope arrives, the server stitches all chunks and scores:
+
+Base points: 1 per trajectory step
+
+Model multiplier (applied to all steps):
+  claude-opus-4-6, claude-opus-4-7: 5x
+  claude-sonnet-4-6: 3x
+  gpt-4.5, o3: 5x
+  o4-mini: 2x
+  gemini-2.5-pro: 3x
+  gemini-2.5-flash: 1.5x
+
+Quality multipliers (applied to relevant steps):
+  User corrections present: 10x on correction steps
+  Coordination events: 5x on coordination steps
+  Error recovery arcs: 3x on error+resolution steps
+  Heavy task complexity: 2x base
+  Session quality >= 80: 1.5x total
+
+---
+
+## 11. Monitoring
+
+Check server health:
+  curl https://api.groovedev.ai/health
+
+Check corpus stats:
+  curl https://api.groovedev.ai/v1/stats/summary
+
+Check daily growth:
+  curl https://api.groovedev.ai/v1/stats/daily?days=7
+
+Check active sessions:
+  curl https://api.groovedev.ai/v1/stats/summary | jq .activeSessions
+
+PM2 monitoring:
+  pm2 status
+  pm2 logs groove-central
+  pm2 monit
+
+---
+
+## 12. Backup Strategy
+
+Critical data to back up:
+  ./data/sessions.db — session keys (active sessions need this)
+  ./data/ledger.db — contributor points (this is money)
+  ./data/envelopes/ — raw training data (the product)
+
+Recommended: daily cron to rsync data/ to S3 or another volume.
+
+```bash
+# Example cron (add to crontab -e)
+0 3 * * * aws s3 sync /opt/groove-central/groove/moe-training/data/ s3://groove-training-backup/$(date +\%Y-\%m-\%d)/ --quiet
+```
+
+---
+
+## 13. Security Notes
+
+- The server accepts connections from any origin (CORS: *) because Groove clients connect from user machines worldwide
+- Rate limiting: max 20 sessions per machine fingerprint per hour (prevents session spam)
+- HMAC verification on every envelope prevents forged data
+- Sequence numbers prevent replay attacks
+- SQLite databases contain ECDH private keys — protect with filesystem permissions (0o600)
+- nginx should only expose /v1/ and /health paths — everything else returns 404
+- No authentication on stats endpoints (they show aggregate data only, no PII)
+
+---
+
+## 14. Future Additions (Not in This Build)
+
+- Enrichment pipeline: LLM-as-a-Judge for model fingerprinting and cognitive target classification (server/enrichment.js is a stub)
+- S3 migration: move envelope storage from local JSONL to S3 for durability
+- PostgreSQL migration: move from SQLite to PostgreSQL if concurrent write pressure increases
+- Merkle tree: daily hash of ledger for on-chain publication (Base L2)
+- Admin dashboard: web UI for monitoring corpus health, contributor activity, data quality
+
+---
+
+## 15. Quick Reference
+
+Start: pm2 start ecosystem.config.cjs
+Stop: pm2 stop groove-central
+Restart: pm2 restart groove-central
+Logs: pm2 logs groove-central
+Status: pm2 status
+Health: curl https://api.groovedev.ai/health
+Stats: curl https://api.groovedev.ai/v1/stats/summary