@boshu2/vibe-check 2.2.1 → 2.4.0

package/.agents/plans/2025-12-29-complexity-drivers-plan.md

@@ -0,0 +1,253 @@
+---
+date: 2025-12-29
+type: Plan
+topic: "Add complexity drivers for Rust, PHP, and Java"
+research: ".agents/research/2025-12-29-complexity-drivers.md"
+tags: [plan, drivers, complexity, rust, php, java]
+status: COMPLETED
+---
+
+# Plan: Additional Complexity Drivers
+
+**Created:** 2025-12-29
+**Research:** .agents/research/2025-12-29-complexity-drivers.md
+**Vibe Level:** L4 (high confidence - following established driver pattern)
+
+---
+
+## Overview
+
+Add complexity drivers for **Rust**, **PHP**, and **Java** to expand vibe-check's language coverage. These are the highest-value additions based on language popularity (TIOBE/Stack Overflow 2025) and tool quality. All three have tools with JSON output, making implementation straightforward.
+
+---
+
+## Approach
+
+Follow existing driver pattern established by `python.sh`, `javascript.sh`, and `go.sh`:
+
+1. Shell script wrapper around language-specific tool
+2. Transform tool output to `ComplexityReport` JSON schema
+3. Update CLI error messages to include new driver
+4. Update documentation
+
+**Why this approach:**
+- Proven pattern (3 working drivers)
+- Language-agnostic kernel stays clean
+- Shell scripts are portable and don't add npm dependencies
+
+---
+
+## Features
+
+### Feature 1: Rust Driver
+
+**Priority:** P1
+**Type:** feature
+**Depends On:** None
+
+**Acceptance Criteria:**
+- [ ] `drivers/rust.sh` created wrapping `rust-code-analysis-cli`
+- [ ] Outputs valid `ComplexityReport` JSON
+- [ ] Handles missing tool with helpful error message
+- [ ] Handles empty directories gracefully
+- [ ] CLI updated to list `rust` as available driver
+- [ ] Documentation updated (drivers/README.md, README.md)
+
+**Files Affected:**
+- `drivers/rust.sh` - New driver script
+- `drivers/README.md` - Add Rust section
+- `README.md` - Add Rust to Available Drivers list
+- `src/commands/driver.ts` - Update available drivers message
+- `src/commands/modularity.ts` - Update available drivers message
+
+**Test Strategy:**
+1. Create test Rust file with known complexity
+2. Run `./drivers/rust.sh /path/to/test`
+3. Verify JSON output matches schema
+4. Run `vibe-check driver rust /path/to/test`
+5. Run `vibe-check modularity --with-complexity rust`
+
+**Tool Details:**
+```bash
+# Install
+cargo install rust-code-analysis-cli
+
+# Usage (native JSON)
+rust-code-analysis-cli -m -O json -p /path/to/src
+```
+
+---
+
+### Feature 2: PHP Driver
+
+**Priority:** P2
+**Type:** feature
+**Depends On:** None (can be done in parallel with Rust)
+
+**Acceptance Criteria:**
+- [ ] `drivers/php.sh` created wrapping `phpmd`
+- [ ] Outputs valid `ComplexityReport` JSON
+- [ ] Handles missing tool with helpful error message
+- [ ] Handles empty directories gracefully
+- [ ] CLI updated to list `php` as available driver
+- [ ] Documentation updated
+
+**Files Affected:**
+- `drivers/php.sh` - New driver script
+- `drivers/README.md` - Add PHP section
+- `README.md` - Add PHP to Available Drivers list
+- `src/commands/driver.ts` - Update available drivers message
+- `src/commands/modularity.ts` - Update available drivers message
+
+**Test Strategy:**
+1. Create test PHP file with known complexity
+2. Run `./drivers/php.sh /path/to/test`
+3. Verify JSON output matches schema
+4. Run `vibe-check driver php /path/to/test`
+
+**Tool Details:**
+```bash
+# Install
+composer global require phpmd/phpmd
+
+# Usage (native JSON)
+phpmd /path/to/src json codesize
+```
+
+---
+
+### Feature 3: Java Driver
+
+**Priority:** P2
+**Type:** feature
+**Depends On:** None (can be done in parallel)
+
+**Acceptance Criteria:**
+- [ ] `drivers/java.sh` created wrapping PMD
+- [ ] Outputs valid `ComplexityReport` JSON
+- [ ] Handles missing tool with helpful error message
+- [ ] Handles empty directories gracefully
+- [ ] CLI updated to list `java` as available driver
+- [ ] Documentation updated
+
+**Files Affected:**
+- `drivers/java.sh` - New driver script
+- `drivers/README.md` - Add Java section
+- `README.md` - Add Java to Available Drivers list
+- `src/commands/driver.ts` - Update available drivers message
+- `src/commands/modularity.ts` - Update available drivers message
+
+**Test Strategy:**
+1. Create test Java file with known complexity
+2. Run `./drivers/java.sh /path/to/test`
+3. Verify JSON output matches schema
+4. Run `vibe-check driver java /path/to/test`
+
+**Tool Details:**
+```bash
+# Install (download PMD)
+# https://pmd.github.io/
+
+# Usage (JSON via flag)
+pmd check -d /path/to/src -R category/java/design.xml -f json
+```
+
+**Note:** PMD requires JRE. Document this clearly.
+
+---
+
+## Implementation Order
+
+| Step | Feature | Depends On | Validation |
+|------|---------|------------|------------|
+| 1 | Rust Driver | - | `vibe-check driver rust ./test` produces valid JSON |
+| 2 | PHP Driver | - | `vibe-check driver php ./test` produces valid JSON |
+| 3 | Java Driver | - | `vibe-check driver java ./test` produces valid JSON |
+
+**Note:** All three can be implemented in parallel - no dependencies between them.
+
+---
+
+## Beads Issues to Create
+
+After approval, these issues will be created:
+
+| ID | Title | Type | Priority | Depends On |
+|----|-------|------|----------|------------|
+| TBD | Epic: Additional Complexity Drivers | epic | P1 | - |
+| TBD | Create Rust complexity driver | feature | P1 | Epic |
+| TBD | Create PHP complexity driver | feature | P2 | Epic |
+| TBD | Create Java complexity driver | feature | P2 | Epic |
+
+---
+
+## Output Format Reference
+
+All drivers must output JSON matching this schema:
+
+```typescript
+{
+  tool: string;           // "rust-code-analysis", "phpmd", "pmd"
+  language: string;       // "rust", "php", "java"
+  generatedAt: string;    // ISO timestamp
+  files: {
+    [filepath: string]: {
+      functions: Array<{
+        name: string;
+        complexity: number;
+        grade: 'A' | 'B' | 'C' | 'D' | 'E' | 'F';
+        line: number;
+        endLine?: number;
+      }>;
+      avgComplexity: number;
+      maxComplexity: number;
+      grade: 'A' | 'B' | 'C' | 'D' | 'E' | 'F';
+    };
+  };
+  summary: {
+    totalFiles: number;
+    totalFunctions: number;
+    avgComplexity: number;
+    gradeDistribution: Record<'A'|'B'|'C'|'D'|'E'|'F', number>;
+  };
+}
+```
+
+**Grade Thresholds:**
+- A: 1-5 (simple)
+- B: 6-10 (acceptable)
+- C: 11-20 (consider refactoring)
+- D: 21-30 (refactor)
+- E: 31-40 (high risk)
+- F: 41+ (unmaintainable)
+
+---
+
+## Rollback Procedure
+
+Each driver is independent. To rollback:
+1. `git revert <commit>` for the specific driver
+2. Remove the `drivers/<lang>.sh` file
+3. Update CLI error messages to remove language
+
+---
+
+## Not In Scope
+
+- Ruby driver (P3 - requires text parsing)
+- C# driver (P3 - requires XML parsing)
+- Automated tool installation
+- Windows-specific handling
+
+---
+
+## Next Steps
+
+1. Review and approve this plan
+2. Run beads issue creation (below)
+3. `bd ready` to see unblocked issues
+4. `/implement` to execute
+
+---
+
+**Output:** .agents/plans/2025-12-29-complexity-drivers-plan.md

package/.agents/research/2025-12-28-ai-platform-security-integration.md

@@ -0,0 +1,295 @@
+---
+date: 2025-12-28
+type: Research
+topic: "Integrating ai-platform security and LLM validation into vibe-check"
+tags: [research, security, llm-validation, integration, ai-platform, vibe-check]
+status: COMPLETE
+---
+
+# Research: Integrating AI-Platform Security & LLM Validation into Vibe-Check
+
+**Created:** 2025-12-28
+**Goal:** Understand how to integrate the security and LLM validation components from ai-platform into vibe-check for detecting AI agent misbehavior patterns.
+
+---
+
+## Executive Summary
+
+The **ai-platform** repository (`/Users/fullerbt/workspaces/work/ai-platform`) contains a comprehensive security and LLM validation stack built for production use in classified environments. Key components include: **RBAC access control**, **audit logging** (SIEM-ready), **agent execution tracking** (Prometheus metrics), **rate limiting with token budgets**, and **contract-driven agent testing**.
+
+**Recommendation:** Integrate these concepts into vibe-check as a new `ai-safety/` or `agent-validation/` module, extending the existing inner-loop failure pattern detection to include LLM-specific antipatterns like **prompt injection attempts**, **hallucination markers**, **scope violations**, and **contract drift**.
+
+---
+
+## Current State
+
+### What Exists in ai-platform
+
+| Component | Location | Purpose |
+|-----------|----------|---------|
+| **Access Control** | `services/gateway/access_control.py` | RBAC for agents/tools via OIDC groups |
+| **Audit Logging** | `services/gateway/audit.py` | JSON request/response logging for SIEM |
+| **Agent Audit** | `services/gateway/agent_audit.py` | Agent execution tracking + Prometheus metrics |
+| **Rate Limiting** | `services/gateway/rate_limit.py` | Per-user rate limits + token budgets |
+| **Config Validator** | `services/etl/app/config_validator.py` | Startup config health checks |
+| **Security Tests** | `tests/agents/test_agent_security.py` | Boundary violation detection |
+| **Contract Tests** | `tests/agents/test_agent_contract_validation.py` | Output contract compliance |
+| **Platform Validator** | `services/etl/scripts/validate-platform.py` | End-to-end LLM platform checks |
+
+### Key Patterns Worth Porting to vibe-check
+
+1. **Agent Scope Boundary Detection** (`test_agent_security.py`)
+   - Detects when agents attempt actions outside their declared scope
+   - Pattern: Check if commit changes files/APIs not in the agent's domain
+
+2. **Hallucination Markers** (`test_agent_security.py:50-70`)
+   - Detects file path hallucination (made-up paths)
+   - Pattern: Commits referencing non-existent files or phantom dependencies
+
+3. **Secret Leakage Detection** (`test_agent_security.py:120-150`)
+   - Regex patterns for API keys, tokens, credentials
+   - Pattern: Commits accidentally exposing secrets
+
+4. **Contract Drift Detection** (`test_agent_contract_validation.py`)
+   - Validates agent responses match expected structure
+   - Pattern: Detect when AI outputs stop conforming to expected formats
+
+5. **Token Budget Tracking** (`rate_limit.py`)
+   - Tracks token consumption per user/session
+   - Pattern: Detect runaway token usage (context explosion)
+
+### Key Files
+
+| File | Purpose | Relevance to vibe-check |
+|------|---------|-------------------------|
+| `services/gateway/access_control.py` | RBAC via OIDC groups | Could map to commit author scope detection |
+| `services/gateway/audit.py` | JSON audit logging | Structure for storing AI interaction events |
+| `services/gateway/agent_audit.py` | Prometheus metrics + Langfuse traces | Pattern for tracking AI agent behavior over time |
+| `tests/agents/test_agent_security.py` | Security test patterns | Regex patterns and violation detection logic |
+| `services/gateway/rate_limit.py` | Token budget tracking | Pattern for detecting context window abuse |
+
+### Existing Patterns in vibe-check
+
+vibe-check already has:
+- **Inner-loop failure detection** (`src/inner-loop/`) with 4 detectors
+- **Spiral detection** with pattern regexes in `watch.ts`
+- **Session tracking** with baseline comparison
+- **NDJSON storage** for historical pattern analysis
+- **Prometheus-like metrics** (not exposed, but structured similarly)
+
+---
+
+## Findings
+
+### Finding 1: Security Validation Patterns Are Git-Analyzable
+
+**Evidence:** The ai-platform security tests detect violations by analyzing:
+- Commit messages for intent signals
+- File changes for scope violations
+- Code content for secret patterns
+
+**Implication:** These patterns can be adapted for vibe-check's commit-based analysis:
+```
+ai-platform pattern → vibe-check integration
+-------------------------------------------
+Scope violation test → Detect commits touching files outside declared domain
+Secret leakage test → Detect commits adding API keys/tokens
+Hallucination test → Detect commits referencing non-existent paths
+Contract drift test → Detect commits breaking expected output formats
+```
+
+### Finding 2: Agent Audit Structure Is Session-Compatible
+
+**Evidence:** `agent_audit.py` tracks events in a structure similar to vibe-check sessions:
+```python
+audit_record = {
+    "correlation_id": str,       # → session_id
+    "agent_name": str,           # → (new field)
+    "user_identity": str,        # → (already tracked)
+    "tokens_used": {             # → (new metric)
+        "input": int,
+        "output": int,
+        "total": int
+    },
+    "tool_invocations": [...],   # → (map to commit file changes)
+    "duration_ms": int           # → (session duration)
+}
+```
+
+**Implication:** Can extend `active-session.json` with agent/LLM metadata.
+
+### Finding 3: Rate Limiting = Token Spiral Detection
+
+**Evidence:** `rate_limit.py` implements token budget tracking:
+```python
+@dataclass
+class UserRateLimit:
+    max_tokens_per_minute: int = 100_000
+    max_tokens_per_hour: int = 1_000_000
+    max_tokens_per_day: int = 10_000_000
+```
+
+**Implication:** vibe-check could detect "token spirals" - sessions where token consumption explodes, indicating:
+- Context window abuse
+- Prompt stuffing
+- Repeated failed attempts (AI trying same thing repeatedly)
+
+### Finding 4: Secret Detection Regex Is Ready to Use
+
+**Evidence:** `test_agent_security.py:120-150`:
+```python
+SECRET_PATTERNS = [
+    r'sk-[a-zA-Z0-9]{48}',           # OpenAI
+    r'ghp_[a-zA-Z0-9]{36}',          # GitHub PAT
+    r'glpat-[a-zA-Z0-9]{20}',        # GitLab PAT
+    r'AKIA[0-9A-Z]{16}',             # AWS Access Key
+    r'xox[baprs]-[a-zA-Z0-9-]+',     # Slack tokens
+]
+```
+
+**Implication:** Can add a `secret-leakage.ts` detector to inner-loop.
+
+### Finding 5: Contract Validation Is Output-Pattern Detection
+
+**Evidence:** `test_agent_contract_validation.py` validates agent outputs against YAML-defined contracts:
+```yaml
+agents:
+  - name: code-review
+    output_contract:
+      required_sections: ["summary", "issues", "suggestions"]
+      max_length: 5000
+```
+
+**Implication:** vibe-check could detect "contract drift" - when AI commits start deviating from expected patterns:
+- Commit messages becoming less structured
+- PR descriptions missing required sections
+- Code comments becoming vague or absent
+
+---
+
+## Constraints
+
+| Constraint | Impact | Mitigation |
+|------------|--------|------------|
+| ai-platform is Python, vibe-check is TypeScript | Can't share code directly | Port patterns/logic, not code |
+| ai-platform runs in Kubernetes, vibe-check is CLI | Different runtime context | Adapt to git-based detection |
+| ai-platform has Prometheus/Langfuse dependencies | Heavy deps for CLI tool | Use file-based storage, optionally export |
+| Token tracking requires LLM API integration | vibe-check only sees git | Estimate tokens from commit size/complexity |
+
+---
+
+## Risks
+
+| Risk | Likelihood | Impact | Mitigation |
+|------|------------|--------|------------|
+| Over-engineering simple CLI tool | Medium | High | Start with 2-3 highest-value patterns |
+| False positives on secret detection | Medium | Medium | Require confidence threshold, allow suppression |
+| Token estimation inaccuracy | High | Low | Use as relative signal, not absolute |
+| Breaking existing inner-loop interface | Low | High | Extend, don't replace |
+
+---
+
+## Recommendation
+
+**Approach:** Add a new `src/ai-safety/` module to vibe-check with 4 new detectors:
+
+### Phase 1: High-Value Ports (Recommended First)
+
+1. **Secret Leakage Detector** (`src/ai-safety/secret-leakage.ts`)
+   - Port regex patterns from `test_agent_security.py`
+   - Scan commit diffs for exposed secrets
+   - Integrate into `session end` output
+
+2. **Scope Violation Detector** (`src/ai-safety/scope-violation.ts`)
+   - Detect commits touching files outside declared domain
+   - Requires domain configuration (similar to access_control.py mappings)
+   - Warning when agent strays from its lane
+
+### Phase 2: Medium-Value Ports
+
+3. **Contract Drift Detector** (`src/ai-safety/contract-drift.ts`)
+   - Detect when commit message patterns degrade
+   - Track deviation from conventional commit format
+   - Alert when AI stops following established patterns
+
+4. **Token Spiral Estimator** (`src/ai-safety/token-spiral.ts`)
+   - Estimate token usage from commit size/complexity
+   - Detect sessions with exploding context
+   - Use as relative metric (not absolute)
+
+### Integration Points
+
+```typescript
+// src/ai-safety/index.ts
+export interface AISafetyAnalysis {
+  secretsDetected: SecretLeakageResult;
+  scopeViolations: ScopeViolationResult;
+  contractDrift: ContractDriftResult;
+  tokenSpiral: TokenSpiralResult;
+  summary: {
+    totalIssues: number;
+    criticalIssues: number;
+    warningIssues: number;
+    overallHealth: 'healthy' | 'warning' | 'critical';
+  };
+  recommendations: string[];
+}
+
+export function analyzeAISafety(
+  commits: Commit[],
+  filesPerCommit: Map<string, string[]>,
+  config?: AISafetyConfig
+): AISafetyAnalysis;
+```
+
+**Integration into existing commands:**
+- `session end` → Add `ai_safety` section to JSON output
+- `watch` → Add real-time alerts for secret detection
+- `insights` → Add AI safety pattern history
+
+### Why This Approach
+
+1. **Non-breaking:** Extends existing architecture, doesn't replace
+2. **High-value first:** Secret leakage is immediately useful
+3. **Familiar patterns:** Same structure as existing inner-loop detectors
+4. **Low deps:** No new dependencies required (pure TypeScript)
+5. **Git-native:** Works with existing commit-based analysis
+
+---
+
+## Alternatives Considered
+
+1. **Import ai-platform as dependency** - Rejected: Different language, heavy deps
+2. **Create shared npm/pip packages** - Rejected: Over-engineering for 4 patterns
+3. **Keep validation in ai-platform only** - Rejected: vibe-check users need these patterns
+4. **Port entire audit system** - Rejected: Too complex, different runtime
+
+---
+
+## Next Steps
+
+1. Run `/plan` to create implementation plan from this research
+2. Plan will create beads issues for each detector
+3. Implement in priority order: secrets → scope → contract → token
+
+---
+
+## Appendix: Key Source File References
+
+### ai-platform Security Components
+- `/Users/fullerbt/workspaces/work/ai-platform/services/gateway/access_control.py` - RBAC implementation
+- `/Users/fullerbt/workspaces/work/ai-platform/services/gateway/audit.py` - Audit logging
+- `/Users/fullerbt/workspaces/work/ai-platform/services/gateway/agent_audit.py` - Agent execution tracking
+- `/Users/fullerbt/workspaces/work/ai-platform/services/gateway/rate_limit.py` - Rate limiting
+- `/Users/fullerbt/workspaces/work/ai-platform/tests/agents/test_agent_security.py` - Security tests
+- `/Users/fullerbt/workspaces/work/ai-platform/tests/agents/test_agent_contract_validation.py` - Contract tests
+
+### vibe-check Integration Points
+- `/Users/fullerbt/workspaces/personal/vibe-check/src/inner-loop/index.ts` - Existing detector orchestrator
+- `/Users/fullerbt/workspaces/personal/vibe-check/src/commands/session.ts` - Session end output
+- `/Users/fullerbt/workspaces/personal/vibe-check/src/commands/watch.ts` - Real-time monitoring
+- `/Users/fullerbt/workspaces/personal/vibe-check/src/types.ts` - Type definitions
+
+---
+
+**Output:** .agents/research/2025-12-28-ai-platform-security-integration.md