claude-flow-novice 2.0.0 → 2.0.1

package/docs/PRE_COMMIT_HOOK.md

@@ -0,0 +1,294 @@
+# Pre-commit Hook: Database Secret Scanning
+
+## Overview
+
+The pre-commit hook automatically scans SQLite database files for secrets, API keys, passwords, and other sensitive data before allowing commits. This prevents accidental leakage of credentials in version control.
+
+**Status**: Production Ready
+**Version**: 1.0
+**Last Updated**: 2025-10-11
+
+---
+
+## Installation
+
+### Quick Install
+
+```bash
+bash scripts/install-pre-commit-hook.sh
+```
+
+### Manual Installation
+
+```bash
+# Copy hook to git hooks directory
+cp config/hooks/pre-commit-db-scan .git/hooks/pre-commit
+chmod +x .git/hooks/pre-commit
+```
+
+---
+
+## What It Checks
+
+The hook scans staged database files for the following secret patterns:
+
+### API Keys
+- `api_key`, `api-key`, `apiKey`
+- `ZAI_API_KEY`
+- `ANTHROPIC_API_KEY`
+- Anthropic API keys: `sk-ant-*`
+
+### Authentication
+- `password`
+- `Bearer` tokens
+- `auth_key`, `auth-key`
+- Session tokens: `sess-*`
+
+### Credentials
+- `secret`
+- `token`
+- `private_key`, `private-key`
+- `credential`
+
+---
+
+## How It Works
+
+1. **Git Commit Triggered**: When you run `git commit`, the hook executes automatically
+2. **Find Staged Databases**: Scans for `*.db` and `*.sqlite` files in the commit
+3. **Dump Database Content**: Uses `sqlite3 .dump` to extract all data
+4. **Pattern Matching**: Checks content against secret patterns (case-insensitive)
+5. **Block or Allow**:
+   - ✅ No secrets found → Commit proceeds
+   - ❌ Secrets detected → Commit blocked with detailed report
+
+---
+
+## Example Output
+
+### Clean Database (Allowed)
+
+```bash
+🔍 Scanning database files for secrets...
+Found database files:
+.artifacts/database/swarm-memory.db
+
+Scanning: .artifacts/database/swarm-memory.db
+  ✓ Scan complete for .artifacts/database/swarm-memory.db
+
+✅ Database secret scan passed - no secrets detected
+```
+
+### Secrets Detected (Blocked)
+
+```bash
+🔍 Scanning database files for secrets...
+Found database files:
+memory/coordination.db
+
+Scanning: memory/coordination.db
+  ❌ ERROR: Found potential secret matching pattern: api[_-]?key
+    INSERT INTO config VALUES ('ZAI_API_KEY', 'sk-ant-1234567890abc***');
+
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+❌ COMMIT BLOCKED: 1 potential secret(s) found in database files
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+🔒 Security Issue: Database files contain potential secrets or API keys
+
+To fix:
+  1. Remove secrets from database files before committing
+  2. Use environment variables for API keys instead
+  3. Sanitize data before storing in SQLite (see sanitizeForStorage())
+  4. If false positive, add pattern to whitelist in this hook
+```
+
+---
+
+## Bypassing the Hook (NOT RECOMMENDED)
+
+If you absolutely must commit a database with detected patterns:
+
+```bash
+git commit --no-verify
+```
+
+**⚠️ Warning**: Only use `--no-verify` if you're certain the detected patterns are false positives. Committing secrets can lead to:
+- Security breaches
+- Unauthorized API usage
+- Data leaks
+- Compliance violations
+
+---
+
+## False Positives
+
+If the hook detects legitimate data (not actual secrets), you can:
+
+### Option 1: Whitelist Pattern (Recommended)
+
+Edit `.git/hooks/pre-commit` and add exception logic:
+
+```bash
+# Example: Allow "api_key" in documentation
+if echo "$db" | grep -q "docs/"; then
+  echo "  ℹ️  Skipping docs database (whitelisted)"
+  continue
+fi
+```
+
+### Option 2: Sanitize Before Storage
+
+Use the `sanitizeForStorage()` function before storing data:
+
+```typescript
+// Before storing in SQLite
+function sanitizeForStorage(data: any): any {
+  const secrets = [
+    /api[_-]?key/i,
+    /password/i,
+    /secret/i,
+    /token/i
+  ];
+
+  const serialized = JSON.stringify(data);
+
+  for (const pattern of secrets) {
+    if (pattern.test(serialized)) {
+      throw new Error(`Cannot store data containing secrets: ${pattern}`);
+    }
+  }
+
+  return data;
+}
+```
+
+---
+
+## Integration with .gitignore
+
+The hook works alongside `.gitignore` allowlist patterns:
+
+```gitignore
+# Block all databases by default
+*.db
+*.sqlite
+*.db-journal
+*.db-wal
+
+# Allow specific project databases (protected by pre-commit hook)
+!.artifacts/database/swarm-memory.db
+!.artifacts/database/coordination-state.db
+!memory/project-context.db
+
+# Still block temporary/cache databases
+**/*-temp.db
+**/*-cache.db
+```
+
+---
+
+## Uninstallation
+
+```bash
+# Remove the hook
+rm .git/hooks/pre-commit
+
+# Restore backup if created
+mv .git/hooks/pre-commit.backup-<timestamp> .git/hooks/pre-commit
+```
+
+---
+
+## Troubleshooting
+
+### Hook Not Running
+
+**Problem**: Commits proceed without scanning
+
+**Solutions**:
+1. Check hook is executable: `ls -la .git/hooks/pre-commit`
+2. Make executable: `chmod +x .git/hooks/pre-commit`
+3. Verify git config: `git config --get core.hooksPath`
+
+### sqlite3 Command Not Found
+
+**Problem**: Hook fails with "sqlite3: command not found"
+
+**Solutions**:
+- Ubuntu/Debian: `sudo apt-get install sqlite3`
+- macOS: `brew install sqlite`
+- Windows (WSL): `sudo apt-get install sqlite3`
+
+### Hook Runs Too Slow
+
+**Problem**: Scanning large databases takes too long
+
+**Solutions**:
+1. Add database to `.gitignore` (don't commit it)
+2. Split large databases into smaller files
+3. Use temporary databases for development (blocked by `*-temp.db` pattern)
+
+---
+
+## Security Best Practices
+
+### 1. Environment Variables
+
+Store secrets in environment variables, not databases:
+
+```typescript
+// ❌ BAD: Storing API key in database
+await db.run('INSERT INTO config VALUES (?, ?)', ['api_key', 'sk-ant-1234']);
+
+// ✅ GOOD: Use environment variable
+const apiKey = process.env.ANTHROPIC_API_KEY;
+```
+
+### 2. Encrypted Storage
+
+If you must store secrets in databases:
+
+```typescript
+import crypto from 'crypto';
+
+function encryptSecret(secret: string, key: string): string {
+  const cipher = crypto.createCipher('aes-256-cbc', key);
+  return cipher.update(secret, 'utf8', 'hex') + cipher.final('hex');
+}
+
+// Store encrypted (but still avoid if possible)
+const encrypted = encryptSecret(apiKey, process.env.ENCRYPTION_KEY);
+await db.run('INSERT INTO config VALUES (?, ?)', ['api_key', encrypted]);
+```
+
+### 3. Separate Development/Production Databases
+
+- Development: Use mock/test API keys (can commit)
+- Production: Use environment variables (never commit)
+
+---
+
+## Related Documentation
+
+- [COORDINATOR_COMMUNICATION_REQUIREMENTS.md](../planning/COORDINATOR_COMMUNICATION_REQUIREMENTS.md) - Memory database requirements
+- [Post-Edit Pipeline](../config/hooks/post-edit-pipeline.js) - Hook integration
+- [CLAUDE.md](../CLAUDE.md) - Project security standards
+
+---
+
+## Support
+
+**Issues**: File a GitHub issue with:
+- Hook output (sanitized to remove actual secrets)
+- Database file size
+- Git version: `git --version`
+- SQLite version: `sqlite3 --version`
+
+**Questions**: See [COORDINATOR_COMMUNICATION_REQUIREMENTS.md](../planning/COORDINATOR_COMMUNICATION_REQUIREMENTS.md) Section "Memory Database Requirements"
+
+---
+
+**Version**: 1.0
+**Maintainer**: Claude Flow DevOps Team
+**License**: Same as claude-flow-novice project

package/docs/README.md

@@ -1,162 +1,139 @@
-# Claude Flow Documentation Hub
-
-## 🚀 Welcome to Claude Flow v2.0.0-alpha.88
-
-This is the comprehensive documentation hub for Claude Flow, the ultimate AI agent orchestration platform with 54+ specialized agents, 112 MCP tools, and swarm intelligence capabilities.
-
-> **🎯 Post-Phase 14 Organization**: All 2,557+ documentation files have been organized into logical categories for optimal navigation and usability.
-
-## 📋 Quick Navigation
-
-### 🏃‍♂️ **Getting Started**
-- **[Project Overview](../README.md)** - Quick start and project introduction
-- **[Installation Guide](operations/DEPLOYMENT.md)** - Complete setup instructions
-- **[User Guide](user/USER_GUIDE.md)** - User-friendly documentation
-- **[Tutorial](user/tutorial.md)** - Step-by-step learning path
-
-### 🏗️ **For Developers**
-- **[Development Workflow](development/DEVELOPMENT_WORKFLOW.md)** - Development best practices
-- **[SPARC Methodology](development/SPARC.md)** - Structured development approach
-- **[Technical Implementation](development/technical-implementation-guide.md)** - Implementation details
-- **[Troubleshooting](development/troubleshooting.md)** - Common issues and solutions
-
-### 🏛️ **System Architecture**
-- **[Architecture Overview](architecture/ARCHITECTURE.md)** - System design and patterns
-- **[Swarm Intelligence](architecture/SWARM.md)** - Distributed coordination
-- **[Agent System](architecture/AGENTS.md)** - Agent architecture and capabilities
-- **[Full-Stack Specification](architecture/full-stack-swarm-team-specification.md)** - Complete team coordination
-
-### 📡 **API & Integration**
-- **[API Documentation](api/API_DOCUMENTATION.md)** - Complete API reference
-- **[MCP Tools](api/MCP_TOOLS.md)** - MCP tool integration guide
-- **[Swarm Integration API](api/mcp-swarm-integration-api.md)** - Swarm coordination APIs
-- **[Custom Framework API](api/custom-framework-api.md)** - Custom integrations
-
-### 👥 **User Guides**
-- **[User Guide](user/USER_GUIDE.md)** - Comprehensive user documentation
-- **[Novice Guide](user/novice-user-guide.md)** - Beginner-friendly introduction
-- **[Enterprise Guide](user/enterprise-stakeholder-guide.md)** - Enterprise deployment
-- **[Preference System](user/PREFERENCE_SYSTEM_GUIDE.md)** - Customization options
-
-### ⚙️ **Operations & Deployment**
-- **[Deployment Guide](operations/DEPLOYMENT.md)** - Production deployment
-- **[Security Audit](operations/SECURITY_AUDIT_REPORT.md)** - Security best practices
-- **[Performance Analysis](operations/performance-analysis-report.md)** - Performance optimization
-- **[Cleanup Phases](operations/cleanup-phases/)** - Documentation cleanup process
-
-## 📚 **Organized Documentation Structure**
-
-### 📁 **API Documentation** (`/api/`)
-Complete API reference, MCP tools, and integration guides
-- REST API endpoints and examples
-- MCP tool documentation (112 tools)
-- Integration patterns and best practices
-- OpenAPI specifications
-
-### 📁 **Development Guides** (`/development/`)
-Development workflows, methodologies, and technical guides
-- SPARC development methodology (17 modes)
-- Development best practices
-- Technical implementation guides
-- Troubleshooting and debugging
-
-### 📁 **Architecture Documentation** (`/architecture/`)
-System design, patterns, and architectural decisions
-- High-level system architecture
-- Swarm coordination patterns (54+ agents)
-- Agent system design
-- Scalability and performance patterns
-
-### 📁 **User Documentation** (`/user/`)
-User guides, tutorials, and getting started information
-- Getting started guides
-- User tutorials and walkthroughs
-- Enterprise user documentation
-- Configuration and preferences
-
-### 📁 **Operations Documentation** (`/operations/`)
-Deployment, monitoring, and operational procedures
-- Production deployment guides
-- Security and compliance documentation
-- Performance monitoring and optimization
-- Operational procedures and checklists
-
-## 🗺️ **Navigation & Cross-References**
-
-### 🎯 **By User Journey**
-| Journey | Start Here | Next Steps | Advanced |
-|---------|------------|------------|----------|
-| **🚀 Quick Start** | [INDEX.md](INDEX.md) → [USER_GUIDE.md](USER_GUIDE.md) | [AGENTS.md](AGENTS.md) | [SPARC.md](SPARC.md) |
-| **🏗️ Development** | [DEVELOPMENT_WORKFLOW.md](DEVELOPMENT_WORKFLOW.md) | [API_DOCUMENTATION.md](API_DOCUMENTATION.md) | [ARCHITECTURE.md](ARCHITECTURE.md) |
-| **🚀 Deployment** | [DEPLOYMENT.md](DEPLOYMENT.md) | [SECURITY_AUDIT_REPORT.md](SECURITY_AUDIT_REPORT.md) | [architecture/](architecture/) |
-| **🤖 Agents** | [AGENTS.md](AGENTS.md) → [SWARM.md](SWARM.md) | [agent-analysis-report.md](agent-analysis-report.md) | [agent-scope-creep-prevention-guide.md](agent-scope-creep-prevention-guide.md) |
-
-### 📚 **Specialized Documentation**
-- **📁 [architecture/](architecture/)** - Architecture specifications and designs
-- **📁 [automation/](automation/)** - Test automation and CI/CD strategies
-- **📁 [implementation/](implementation/)** - Implementation guides and specifications
-- **📁 [ux-design/](ux-design/)** - User experience and interface design
-- **📁 [validation/](validation/)** - Validation reports and testing strategies
-- **📁 [performance/](performance/)** - Performance analysis and optimization
-- **📁 [research/](research/)** - Research findings and analysis reports
-
-### 🔗 **Cross-Reference Matrix**
-| If you're reading... | You might also need... |
-|---------------------|------------------------|
-| **[AGENTS.md](AGENTS.md)** | [SWARM.md](SWARM.md), [agent-analysis-report.md](agent-analysis-report.md) |
-| **[SWARM.md](SWARM.md)** | [ARCHITECTURE.md](ARCHITECTURE.md), [full-stack-swarm-team-specification.md](full-stack-swarm-team-specification.md) |
-| **[DEPLOYMENT.md](DEPLOYMENT.md)** | [SECURITY_AUDIT_REPORT.md](SECURITY_AUDIT_REPORT.md), [deployment-checklist.md](deployment-checklist.md) |
-| **[ARCHITECTURE.md](ARCHITECTURE.md)** | [architecture/](architecture/), [technical-implementation-guide.md](technical-implementation-guide.md) |
-| **[SPARC.md](SPARC.md)** | [DEVELOPMENT_WORKFLOW.md](DEVELOPMENT_WORKFLOW.md), [consolidated-cli-implementation.md](consolidated-cli-implementation.md) |
-
-## 🚀 Quick Access
-
-### 🎯 **Essential References**
-- **[🗺️ Complete Site Map](SITE_MAP.md)** - Find any documentation quickly
-- **[📖 Documentation Hub](INDEX.md)** - Guided navigation with learning paths
-- **[🎯 User Guide](USER_GUIDE.md)** - Step-by-step tutorials
-- **[🔧 API Reference](API_DOCUMENTATION.md)** - Complete API documentation
-
-### 🆘 **Need Help?**
-- **[📋 Troubleshooting Guide](troubleshooting.md)** - Common issues and solutions
-- **[🎯 CLI Validation](cli-validation-report.md)** - Command reference and validation
-- **[📊 Performance Tips](performance-analysis-report.md)** - Optimization strategies
-- **[📝 Validation Summary](validation-executive-summary.md)** - System validation status
-
-### 🔍 **Search Strategy**
-1. **Browse by topic** using the [Site Map](SITE_MAP.md)
-2. **Follow learning paths** in the [Documentation Hub](INDEX.md)
-3. **Check cross-references** within each document
-4. **Use specialized directories** for deep-dive topics
-
-## 📞 Support & Community
-
-### 🆘 **Get Help**
-- **🐛 [GitHub Issues](https://github.com/ruvnet/claude-flow/issues)** - Bug reports and feature requests
-- **💬 [Discord Community](https://discord.gg/claude-flow)** - Real-time community support
-- **📚 [Documentation](SITE_MAP.md)** - Comprehensive guides and references
-- **📧 Support**: support@claude-flow.ai
-
-### 🤝 **Contribute**
-- **📝 Documentation Updates** - Submit PRs to improve guides
-- **🔗 Cross-Reference Improvements** - Help enhance navigation
-- **🐛 Bug Reports** - Report issues via GitHub
-- **💡 Feature Suggestions** - Share ideas with the community
-
-### 🏷️ **Documentation Tags**
-`#core-docs` `#agents` `#swarm` `#sparc` `#deployment` `#architecture` `#api` `#mcp` `#validation` `#performance` `#ux` `#research`
+# Claude Flow Novice Documentation
+
+**AI Agent Orchestration Made Simple**
+
+## Quick Start Paths
+
+### 🚀 New Users
+- [Quick Start Guide](./guides/QUICK_START.md) - Get running in 5 minutes
+- [Installation](./guides/INSTALLATION.md) - Setup instructions
+- [User Guide](./guides/user/USER_GUIDE.md) - Complete user documentation
+
+### 💻 Developers
+- [Development Setup](./guides/DEVELOPMENT_SETUP.md)
+- [API Reference](./api/API.md)
+- [Examples](./guides/EXAMPLES.md)
+
+### 🏗️ Architecture & Advanced
+- [System Architecture](./architecture/ARCHITECTURE.md)
+- [CFN Loop](./cfn-loop/CFN_LOOP_PHASE_ORCHESTRATION.md)
+- [Fleet Manager](./architecture/fleet-manager-design.md)
+
+---
+
+## Documentation Categories
+
+### 📚 Core Documentation
+
+**[guides/](./guides/)** - User guides and tutorials
+- Quick start, installation, troubleshooting
+- User workflows and examples
+- Contributing guidelines
+
+**[api/](./api/)** - API documentation
+- API reference and configuration
+- Function catalog and routing
+- Validation guides
+
+**[reference/](./reference/)** - Reference materials
+- Changelogs and version history
+- Wiki and templates
+- General reference docs
+
+### 🔧 Development
+
+**[development/](./development/)** - Development workflows
+- SDK testing and integration
+- Development guides
+- Implementation references
+
+**[integration/](./integration/)** - Integration & hooks
+- Slash commands
+- Workflow automation
+- Hook pipeline documentation
+
+**[testing/](./testing/)** - Testing & validation
+- Test guides and results
+- Validation reports
+- Testing frameworks
+
+### 🏛️ Architecture & Operations
+
+**[architecture/](./architecture/)** - System design
+- System architecture
+- Agent coordination patterns
+- Fleet management
+- WASM acceleration
+
+**[cfn-loop/](./cfn-loop/)** - CFN Loop methodology
+- 4-loop architecture
+- Phase orchestration
+- Product Owner GOAP
+
+**[operations/](./operations/)** - Production operations
+- Deployment guides
+- Monitoring & APM
+- Runbooks and procedures
+
+### 🔐 Security & Performance
+
+**[security/](./security/)** - Security documentation
+- Security practices
+- Authentication guides
+- Security audits
+
+**[performance/](./performance/)** - Performance optimization
+- WASM deliverables
+- Optimization guides
+- Performance tuning
+
+**[migration/](./migration/)** - Migration guides
+- Version migration paths
+- MCP deprecation notices
+- Upgrade guides
 
 ---
 
-<div align="center">
+## Common Tasks
+
+### Setting Up
+1. [Install Claude Flow Novice](./guides/INSTALLATION.md)
+2. [Run Setup Wizard](./guides/SETUP_WIZARD.md)
+3. [Your First Swarm](./guides/QUICK_START.md)
+
+### Building Applications
+1. [API Development](./api/API.md)
+2. [Using CFN Loop](./cfn-loop/CFN_LOOP_PHASE_ORCHESTRATION.md)
+3. [Testing Your Code](./testing/README.md)
 
-## 🎯 **Find What You Need**
+### Production Deployment
+1. [Deployment Checklist](./operations/DEPLOYMENT.md)
+2. [Security Hardening](./security/DEPLOYMENT_CHECKLIST.md)
+3. [Production Operations](./operations/PRODUCTION_OPERATIONS.md)
 
-**[🗺️ Site Map](SITE_MAP.md)** • **[📖 Documentation Hub](INDEX.md)** • **[🎯 User Guide](USER_GUIDE.md)** • **[🔧 API Docs](API_DOCUMENTATION.md)**
+### Troubleshooting
+- [Troubleshooting Guide](./guides/TROUBLESHOOTING.md)
+- [Error Messages](./guides/EXAMPLES.md)
+- [Common Issues](./operations/README.md)
 
 ---
 
-*This documentation system is maintained by the Cross-Reference Manager*
-*Last Updated: September 26, 2025 | Version: 2.0.0-alpha.88*
+## Version Information
+
+**Current Version**: v1.7
+- WASM 40x acceleration (Sprints 1.2-1.4 COMPLETE)
+- Event Bus: 398,373 events/sec
+- Fleet Manager: 1000+ concurrent agents
+- Redis persistence and coordination
+
+**See Also**: [Migration Guide](./migration/V2_MIGRATION_GUIDE.md) for upgrading
+
+---
+
+## Archive
+
+Historical documentation has been moved to [archive/](./archive/) for reference purposes.
+
+---
 
-</div>
+**Need Help?** Start with the [Quick Start Guide](./guides/QUICK_START.md) or [User Guide](./guides/user/USER_GUIDE.md)