@mmmbuto/nexuscli 0.8.8 → 0.8.9

package/lib/server/docs/PIPELINE_INTEGRATION.md

@@ -0,0 +1,636 @@
+# NexusCLI Pipeline Integration
+
+**Version**: 0.1.0
+**Created**: 2025-11-17
+**Based on**: NexusChat development workflow (CLAUDE.md)
+
+---
+
+## 🔄 Pipeline Overview
+
+NexusCLI follows a **5-phase pipeline** for distributed CLI orchestration, adapted from the NexusChat development workflow:
+
+```
+Analysis → Coordination → Execution → Test → Deploy
+```
+
+Each phase has specific responsibilities and integrates with the Control Plane and Node Wrappers.
+
+---
+
+## 📋 Phase Breakdown
+
+### Phase 1: Analysis
+
+**Purpose**: Understand what needs to be done, where, and validate feasibility.
+
+**Control Plane Responsibilities**:
+
+1. **Request Validation**
+   - Parse job specification (tool, command, target nodes)
+   - Validate command syntax
+   - Check tool availability on target nodes
+   - Verify user permissions
+
+2. **Resource Assessment**
+   - Query Node Registry for available nodes
+   - Check node capacity (active jobs vs max concurrent)
+   - Estimate execution time and resource requirements
+
+3. **Risk Analysis**
+   - Identify destructive commands (rm, mkfs, dd, etc.)
+   - Flag production nodes for approval workflows
+   - Check command whitelists/blacklists
+
+**API Endpoints**:
+```
+POST /api/v1/jobs/validate
+```
+
+**Example**:
+```json
+// Request
+{
+  "tool": "bash",
+  "command": "rm -rf /var/www/*",
+  "nodeIds": ["prod-001"]
+}
+
+// Response
+{
+  "valid": false,
+  "warnings": [
+    {
+      "type": "destructive_command",
+      "message": "Command contains 'rm -rf' - requires approval",
+      "severity": "critical"
+    }
+  ],
+  "suggestions": [
+    "Use 'rm -rf /var/www/tmp/*' for safer scoped deletion",
+    "Add --dry-run flag to test first"
+  ]
+}
+```
+
+---
+
+### Phase 2: Coordination
+
+**Purpose**: Plan execution strategy, select nodes, queue jobs.
+
+**Control Plane Responsibilities**:
+
+1. **Node Selection**
+   - Choose optimal node(s) based on:
+     - Current load
+     - Tool availability
+     - Geographic location (latency)
+     - Cost (if WellaNet integration)
+
+2. **Job Scheduling**
+   - Queue jobs if nodes are busy
+   - Prioritize based on user roles
+   - Schedule batch jobs with dependencies
+
+3. **Session Initialization**
+   - Create job records in storage
+   - Generate unique jobIds
+   - Set up SSE stream endpoints
+
+**Algorithm** (Node Selection):
+```javascript
+function selectNode(jobSpec, availableNodes) {
+  // 1. Filter by tool availability
+  const capableNodes = availableNodes.filter(node =>
+    node.capabilities.tools.includes(jobSpec.tool)
+  );
+
+  // 2. Filter by load threshold
+  const readyNodes = capableNodes.filter(node =>
+    node.activeJobs < node.capabilities.maxConcurrentJobs
+  );
+
+  // 3. Sort by load (ascending)
+  readyNodes.sort((a, b) => a.activeJobs - b.activeJobs);
+
+  // 4. Return least loaded node
+  return readyNodes[0] || null;
+}
+```
+
+**API Endpoints**:
+```
+POST /api/v1/jobs (creates and coordinates job)
+GET /api/v1/nodes/optimal?tool=bash (suggest best node)
+```
+
+---
+
+### Phase 3: Execution
+
+**Purpose**: Execute command on remote node(s), stream real-time output.
+
+**Workflow**:
+
+```
+Control Plane                    Wrapper (Node)
+     │                               │
+     │ 1. POST /jobs                 │
+     ├──────────────────────────────▶│
+     │   {jobId, tool, command}      │
+     │                               │
+     │ 2. Accept job                 │
+     │◀──────────────────────────────┤
+     │   {status: "accepted"}        │
+     │                               │
+     │                               │ 3. Spawn PTY
+     │                               │    pty.spawn(bash, ['-c', cmd])
+     │                               │
+     │ 4. GET /jobs/:id/stream (SSE) │
+     ├──────────────────────────────▶│
+     │                               │
+     │ 5. Stream events              │
+     │◀──────────────────────────────┤
+     │   data: {type:"status",...}   │
+     │   data: {type:"output_chunk"} │
+     │   ...                         │
+     │   data: {type:"done"}         │
+     │                               │
+     │ 6. Close stream               │
+     │◀──────────────────────────────┤
+```
+
+**Wrapper Execution Flow** (based on NexusChat claude-wrapper.js):
+
+```javascript
+async function executeJob({ jobId, command, workingDir, timeout }) {
+  // 1. Spawn PTY process
+  const ptyProcess = pty.spawn('/bin/bash', ['-c', command], {
+    cwd: workingDir,
+    env: process.env,
+    cols: 80,
+    rows: 30,
+  });
+
+  // 2. Attach output parser
+  const parser = new OutputParser();
+  let stdout = '';
+  let stderr = '';
+
+  ptyProcess.onData((data) => {
+    stdout += data;
+
+    // Parse and emit events
+    const events = parser.parse(data);
+    events.forEach(event => {
+      emitSSE(jobId, event); // Send to Control Plane
+    });
+  });
+
+  // 3. Handle timeout
+  const timer = setTimeout(() => {
+    ptyProcess.kill('SIGTERM');
+    emitSSE(jobId, {
+      type: 'error',
+      error: 'Job timeout exceeded',
+      timeout: timeout,
+    });
+  }, timeout);
+
+  // 4. Wait for completion
+  ptyProcess.onExit(({ exitCode }) => {
+    clearTimeout(timer);
+
+    // Save result
+    saveJobResult(jobId, {
+      exitCode,
+      stdout: cleanAnsi(stdout),
+      stderr: cleanAnsi(stderr),
+      duration: Date.now() - startTime,
+    });
+
+    // Emit done event
+    emitSSE(jobId, {
+      type: 'done',
+      exitCode,
+    });
+  });
+}
+```
+
+**Output Parser** (adapted from NexusChat output-parser.js):
+
+```javascript
+class OutputParser {
+  constructor() {
+    this.state = 'idle';
+    this.buffer = '';
+  }
+
+  parse(chunk) {
+    const events = [];
+    this.buffer += chunk;
+
+    // Parse line by line
+    const lines = this.buffer.split('\n');
+    this.buffer = lines.pop(); // Keep incomplete line
+
+    for (const line of lines) {
+      // Detect tool execution
+      if (/^Running (bash|git|docker):/.test(line)) {
+        events.push({
+          type: 'status',
+          category: 'tool',
+          message: line,
+          icon: '🔧',
+        });
+      }
+
+      // Detect errors
+      if (/^Error:|FATAL:/.test(line)) {
+        events.push({
+          type: 'status',
+          category: 'warning',
+          message: line,
+          icon: '⚠️',
+        });
+      }
+
+      // Stream output
+      events.push({
+        type: 'output_chunk',
+        stream: 'stdout',
+        text: line + '\n',
+      });
+    }
+
+    return events;
+  }
+}
+```
+
+---
+
+### Phase 4: Test
+
+**Purpose**: Validate execution results, assert expected outcomes.
+
+**Validation Types**:
+
+1. **Exit Code Validation**
+   ```javascript
+   if (job.result.exitCode !== 0) {
+     throw new Error(`Job failed with exit code ${job.result.exitCode}`);
+   }
+   ```
+
+2. **Output Validation** (regex patterns)
+   ```javascript
+   const expectedPattern = /Successfully deployed to production/;
+   if (!expectedPattern.test(job.result.stdout)) {
+     throw new Error('Expected success message not found in output');
+   }
+   ```
+
+3. **Side-Effect Validation** (chained jobs)
+   ```javascript
+   // Example: After "deploy to prod", verify service is running
+   const verifyJob = await JobManager.create({
+     nodeId: 'prod-001',
+     tool: 'bash',
+     command: 'systemctl status myapp',
+     parent: job.jobId, // Link to parent job
+   });
+
+   if (verifyJob.result.exitCode !== 0) {
+     // Rollback parent job
+     await JobManager.rollback(job.jobId);
+   }
+   ```
+
+**Test Configuration** (per job):
+```json
+{
+  "jobId": "job-123",
+  "tool": "bash",
+  "command": "npm run build",
+  "tests": [
+    {
+      "type": "exit_code",
+      "expected": 0
+    },
+    {
+      "type": "output_contains",
+      "pattern": "Build successful",
+      "stream": "stdout"
+    },
+    {
+      "type": "file_exists",
+      "path": "/var/www/dist/index.html"
+    }
+  ]
+}
+```
+
+**API Endpoints**:
+```
+POST /api/v1/jobs/:id/test (run tests on completed job)
+GET /api/v1/jobs/:id/test-results
+```
+
+---
+
+### Phase 5: Deploy
+
+**Purpose**: Execute follow-up actions, trigger dependent jobs, update state.
+
+**Deployment Patterns**:
+
+1. **Sequential Chain**
+   ```javascript
+   // Build → Test → Deploy → Verify
+   const pipeline = [
+     { tool: 'bash', command: 'npm run build' },
+     { tool: 'bash', command: 'npm test' },
+     { tool: 'bash', command: 'rsync -av dist/ /var/www/prod/' },
+     { tool: 'bash', command: 'systemctl restart nginx' },
+   ];
+
+   for (const step of pipeline) {
+     const job = await JobManager.create(step);
+     await JobManager.waitForCompletion(job.jobId);
+
+     if (job.result.exitCode !== 0) {
+       // Rollback previous steps
+       await rollbackPipeline(pipeline.slice(0, pipeline.indexOf(step)));
+       break;
+     }
+   }
+   ```
+
+2. **Parallel Fanout**
+   ```javascript
+   // Deploy to multiple nodes simultaneously
+   const deployJobs = await Promise.all([
+     JobManager.create({ nodeId: 'web-001', command: 'deploy.sh' }),
+     JobManager.create({ nodeId: 'web-002', command: 'deploy.sh' }),
+     JobManager.create({ nodeId: 'web-003', command: 'deploy.sh' }),
+   ]);
+
+   // Wait for all to complete
+   await Promise.all(deployJobs.map(j => JobManager.waitForCompletion(j.jobId)));
+   ```
+
+3. **Conditional Branching**
+   ```javascript
+   const buildJob = await JobManager.create({ command: 'npm run build' });
+   await JobManager.waitForCompletion(buildJob.jobId);
+
+   if (buildJob.result.exitCode === 0) {
+     // Success: deploy to production
+     await JobManager.create({
+       nodeId: 'prod-001',
+       command: 'deploy-prod.sh',
+       parent: buildJob.jobId,
+     });
+   } else {
+     // Failure: notify team
+     await JobManager.create({
+       tool: 'bash',
+       command: 'curl -X POST slack-webhook -d "Build failed"',
+     });
+   }
+   ```
+
+**API Endpoints**:
+```
+POST /api/v1/pipelines (create multi-job pipeline)
+GET /api/v1/pipelines/:id (get pipeline status)
+POST /api/v1/jobs/:id/rollback (rollback deployed job)
+```
+
+---
+
+## 🔗 Pipeline Example: Full Deployment Workflow
+
+### Scenario: Deploy web application to production
+
+```javascript
+// Phase 1: ANALYSIS
+const analysisResult = await fetch('/api/v1/jobs/validate', {
+  method: 'POST',
+  body: JSON.stringify({
+    tool: 'bash',
+    command: 'cd /var/www/myapp && git pull && npm run build',
+    nodeId: 'prod-web-001',
+  }),
+});
+// → { valid: true, estimatedDuration: 120000, warnings: [] }
+
+// Phase 2: COORDINATION
+const buildJob = await fetch('/api/v1/jobs', {
+  method: 'POST',
+  body: JSON.stringify({
+    nodeId: 'build-server',
+    tool: 'bash',
+    command: 'cd /var/www/myapp && npm run build',
+    metadata: { pipeline: 'production-deploy' },
+  }),
+});
+// → { jobId: "job-001", streamEndpoint: "/jobs/job-001/stream" }
+
+// Phase 3: EXECUTION (real-time streaming)
+const eventSource = new EventSource('/api/v1/jobs/job-001/stream');
+eventSource.onmessage = (event) => {
+  const data = JSON.parse(event.data);
+
+  if (data.type === 'status') {
+    console.log(`[${data.category}] ${data.message}`);
+    // → [tool] Bash: npm run build
+  }
+
+  if (data.type === 'output_chunk') {
+    console.log(data.text);
+    // → > webpack compiled successfully
+  }
+
+  if (data.type === 'done') {
+    console.log('✅ Build completed');
+    eventSource.close();
+  }
+};
+
+// Phase 4: TEST
+const testResult = await fetch('/api/v1/jobs/job-001/test', {
+  method: 'POST',
+  body: JSON.stringify({
+    tests: [
+      { type: 'exit_code', expected: 0 },
+      { type: 'file_exists', path: '/var/www/myapp/dist/index.html' },
+    ],
+  }),
+});
+// → { passed: true, results: [...] }
+
+// Phase 5: DEPLOY
+if (testResult.passed) {
+  const deployJob = await fetch('/api/v1/jobs', {
+    method: 'POST',
+    body: JSON.stringify({
+      nodeId: 'prod-web-001',
+      tool: 'bash',
+      command: 'rsync -av /var/www/myapp/dist/ /var/www/production/',
+      parent: 'job-001', // Link to build job
+    }),
+  });
+
+  // Verify deployment
+  const verifyJob = await fetch('/api/v1/jobs', {
+    method: 'POST',
+    body: JSON.stringify({
+      nodeId: 'prod-web-001',
+      tool: 'bash',
+      command: 'systemctl status nginx && curl -I http://localhost',
+      parent: deployJob.jobId,
+    }),
+  });
+}
+```
+
+---
+
+## 🔄 Relationship to NexusChat Workflow
+
+**NexusChat CLAUDE.md Pipeline** (adapted):
+
+```
+1. MODIFICA       → Analysis (validate changes)
+2. TEST           → Coordination (select test environment)
+3. DOCUMENTA      → Execution (run tests, generate docs)
+4. COMMIT ATOMICO → Test (verify all tests pass)
+5. PUSH           → Deploy (push to git, deploy to production)
+```
+
+**NexusCLI Equivalent**:
+
+```
+1. ANALYSIS       → What needs to be done? Where? How?
+2. COORDINATION   → Which nodes? In what order? When?
+3. EXECUTION      → Run commands, stream output
+4. TEST           → Did it work? Assert expectations
+5. DEPLOY         → Apply changes, rollback if needed
+```
+
+---
+
+## 📊 Monitoring & Observability
+
+### Metrics (per phase)
+
+**Analysis**:
+- Validation failures per hour
+- Average validation time
+- Most common validation errors
+
+**Coordination**:
+- Node selection time
+- Queue depth
+- Jobs pending vs executing
+
+**Execution**:
+- Average job duration (per tool)
+- Success rate (exit code 0)
+- Timeout rate
+
+**Test**:
+- Test pass rate
+- Failed assertion types
+- Rollback frequency
+
+**Deploy**:
+- Deployment success rate
+- Rollback time
+- Pipeline completion time
+
+**API Endpoint**:
+```
+GET /api/v1/metrics/pipeline
+```
+
+**Response**:
+```json
+{
+  "analysis": {
+    "totalValidations": 1523,
+    "validationFailures": 45,
+    "avgValidationTime": 12
+  },
+  "execution": {
+    "totalJobs": 8421,
+    "successfulJobs": 8103,
+    "failedJobs": 318,
+    "avgDuration": 3421
+  },
+  "deploy": {
+    "totalDeployments": 234,
+    "successRate": 0.963,
+    "avgRollbackTime": 8234
+  }
+}
+```
+
+---
+
+## 🚀 Future Enhancements
+
+### Pipeline Templates
+
+**Predefined Workflows**:
+```yaml
+# templates/web-deploy.yaml
+name: "Production Web Deployment"
+phases:
+  - analysis:
+      validate: true
+      approvers: ["admin@example.com"]
+
+  - coordination:
+      nodes: ["build-server"]
+
+  - execution:
+      steps:
+        - { tool: "bash", command: "git pull" }
+        - { tool: "bash", command: "npm install" }
+        - { tool: "bash", command: "npm run build" }
+
+  - test:
+      tests:
+        - { type: "exit_code", expected: 0 }
+        - { type: "file_exists", path: "dist/index.html" }
+
+  - deploy:
+      strategy: "rolling"
+      nodes: ["prod-web-001", "prod-web-002", "prod-web-003"]
+      healthCheck: "curl -f http://localhost/health"
+```
+
+### AI-Assisted Analysis
+
+**Future Integration**:
+- Use Claude/GPT to analyze commands for risks
+- Suggest safer alternatives
+- Auto-generate test assertions
+- Predict optimal node selection
+
+---
+
+## 📚 Related Documents
+
+- [API & Wrapper Contract](./API_WRAPPER_CONTRACT.md) - API specification
+- [Architecture Overview](./ARCHITECTURE.md) - System architecture
+- [NexusChat CLAUDE.md](../../CLAUDE.md) - Original workflow
+
+---
+
+_Generated by Claude Code (Sonnet 4.5) - 2025-11-17_