musubix 1.8.0 → 2.0.0

package/docs/USER-GUIDE.md

@@ -22,10 +22,17 @@
 14. [KGPR - Knowledge Graph Pull Request](#kgpr---knowledge-graph-pull-request) *(v1.6.4)*
 15. [YATA Platform Enhancements](#yata-platform-enhancements) *(v1.7.0)*
 16. [Formal Verification](#formal-verification) *(v1.7.5)*
-17. [MCP Server Integration](#mcp-server-integration)
-17. [YATA Integration](#yata-integration)
-18. [Best Practices](#best-practices)
-19. [Troubleshooting](#troubleshooting)
+17. [Security Analysis](#security-analysis) *(v1.8.0)*
+18. [DFG/CFG Extraction](#dfgcfg-extraction) *(v2.0.0)*
+19. [Lean 4 Integration](#lean-4-integration) *(v2.0.0)*
+20. [YATA Scale](#yata-scale) *(v2.0.0)*
+21. [Library Learning](#library-learning) *(v2.0.0 NEW!)*
+22. [Neural Search](#neural-search) *(v2.0.0 NEW!)*
+23. [Program Synthesis](#program-synthesis) *(v2.0.0 NEW!)*
+24. [MCP Server Integration](#mcp-server-integration)
+25. [YATA Integration](#yata-integration)
+26. [Best Practices](#best-practices)
+27. [Troubleshooting](#troubleshooting)
 
 ---
 
@@ -1207,6 +1214,829 @@ console.log(`Impacted nodes: ${impact.totalImpacted}`);
 
 ---
 
+## Security Analysis
+
+*(v1.8.0)*
+
+The `@nahisaho/musubix-security` package provides comprehensive security analysis capabilities for TypeScript/JavaScript projects.
+
+### Installation
+
+```bash
+npm install @nahisaho/musubix-security
+```
+
+### Vulnerability Scanning
+
+Detects OWASP Top 10 and CWE Top 25 vulnerabilities using AST analysis:
+
+```typescript
+import { VulnerabilityScanner, createSecurityService } from '@nahisaho/musubix-security';
+
+// Scan a single file
+const scanner = new VulnerabilityScanner();
+const vulnerabilities = scanner.scanFile('src/api.ts');
+
+// Scan a directory
+const result = await scanner.scanDirectory('./src');
+console.log(`Found ${result.vulnerabilities.length} vulnerabilities`);
+console.log(`Scanned ${result.scannedFiles} files`);
+```
+
+### Detected Vulnerabilities
+
+| Category | CWE | Severity |
+|----------|-----|----------|
+| SQL Injection | CWE-89 | Critical |
+| Command Injection | CWE-78 | Critical |
+| XSS | CWE-79 | High |
+| Path Traversal | CWE-22 | High |
+| Code Injection | CWE-94 | Critical |
+| NoSQL Injection | CWE-943 | High |
+
+### Secret Detection
+
+Detects hardcoded credentials and sensitive information:
+
+```typescript
+import { SecretDetector } from '@nahisaho/musubix-security';
+
+const detector = new SecretDetector();
+const secrets = detector.scanContent(content, 'config.ts');
+const result = await detector.scan('./src');
+
+console.log(`Found ${result.summary.total} secrets`);
+```
+
+### Detected Secret Types
+
+| Type | Pattern |
+|------|--------|
+| AWS Access Key | `AKIA...` |
+| AWS Secret Key | 40-char base64 |
+| GitHub Token | `ghp_*`, `gho_*`, `ghu_*` |
+| Private Key | PEM format |
+| Database URL | `postgres://`, `mongodb://` |
+| JWT Secret | JWT signing secrets |
+| Stripe Key | `sk_live_*`, `sk_test_*` |
+
+### Taint Analysis
+
+Tracks data flow from user input (sources) to dangerous functions (sinks):
+
+```typescript
+import { TaintAnalyzer } from '@nahisaho/musubix-security';
+
+const analyzer = new TaintAnalyzer();
+const result = analyzer.analyze('./src');
+
+console.log(`Sources: ${result.sources.length}`);
+console.log(`Sinks: ${result.sinks.length}`);
+console.log(`Taint paths: ${result.paths.length}`);
+```
+
+### Dependency Auditing
+
+Integrates with npm audit to detect vulnerable dependencies:
+
+```typescript
+import { DependencyAuditor } from '@nahisaho/musubix-security';
+
+const auditor = new DependencyAuditor();
+const result = await auditor.audit('./project');
+
+console.log(`Critical: ${result.summary.critical}`);
+console.log(`High: ${result.summary.high}`);
+```
+
+### Integrated Security Service
+
+Combines all security analysis features:
+
+```typescript
+import { createSecurityService } from '@nahisaho/musubix-security';
+
+const service = createSecurityService();
+
+// Full security scan
+const result = await service.scan({
+  target: './src',
+  vulnerabilities: true,
+  taint: true,
+  secrets: true,
+  dependencies: true,
+  generateFixes: true,
+});
+
+console.log(`Total vulnerabilities: ${result.summary.totalVulnerabilities}`);
+console.log(`Total secrets: ${result.summary.totalSecrets}`);
+console.log(`Fixes generated: ${result.summary.fixesGenerated}`);
+```
+
+### Report Generation
+
+Generate reports in multiple formats:
+
+```typescript
+// SARIF format (GitHub Code Scanning compatible)
+const sarifReport = await service.generateReport(result, 'sarif');
+
+// Markdown format
+const mdReport = await service.generateReport(result, 'markdown');
+
+// HTML format
+const htmlReport = await service.generateReport(result, 'html');
+```
+
+### CLI Usage
+
+```bash
+# Full security scan
+npx musubix-security scan ./src
+
+# Vulnerability scan only
+npx musubix-security scan ./src --vulnerabilities-only
+
+# Secret detection
+npx musubix-security secrets ./src
+
+# Taint analysis
+npx musubix-security taint ./src
+
+# Dependency audit
+npx musubix-security audit ./project
+
+# Generate SARIF report
+npx musubix-security scan ./src --format sarif --output report.sarif
+```
+
+### v1.8.0 Package Summary
+
+| Package | Description |
+|---------|-------------|
+| `@nahisaho/musubix-security` | Vulnerability scanning, secret detection, taint analysis, dependency auditing |
+
+---
+
+## DFG/CFG Extraction
+
+*(v2.0.0)*
+
+The `@nahisaho/musubix-dfg` package provides Data Flow Graph (DFG) and Control Flow Graph (CFG) extraction for TypeScript and JavaScript code analysis.
+
+### Installation
+
+```bash
+npm install @nahisaho/musubix-dfg
+```
+
+### DFG Extraction
+
+Extract data flow graphs from source code:
+
+```typescript
+import { extractDFG, DFGExtractor } from '@nahisaho/musubix-dfg';
+
+// Simple extraction
+const dfg = extractDFG(sourceCode, 'typescript');
+
+// With configuration
+const extractor = new DFGExtractor({
+  includeImplicitFlows: true,
+  trackConstants: true,
+});
+const result = extractor.extract(sourceCode);
+
+console.log(`Nodes: ${result.nodes.length}`);
+console.log(`Edges: ${result.edges.length}`);
+```
+
+### CFG Extraction
+
+Extract control flow graphs:
+
+```typescript
+import { extractCFG, CFGExtractor } from '@nahisaho/musubix-dfg';
+
+const cfg = extractCFG(sourceCode);
+
+// Traverse CFG
+for (const block of cfg.blocks) {
+  console.log(`Block ${block.id}: ${block.statements.length} statements`);
+  console.log(`Successors: ${block.successors.join(', ')}`);
+}
+```
+
+### Def-Use Chain Analysis
+
+Build definition-use chains for variable tracking:
+
+```typescript
+import { analyzeDefUse } from '@nahisaho/musubix-dfg';
+
+const chains = analyzeDefUse(dfg);
+
+for (const chain of chains) {
+  console.log(`Variable: ${chain.variable}`);
+  console.log(`Defined at: line ${chain.definition.line}`);
+  console.log(`Used at: ${chain.uses.map(u => u.line).join(', ')}`);
+}
+```
+
+### Data Dependency Analysis
+
+Analyze data dependencies between statements:
+
+```typescript
+import { analyzeDataDependencies } from '@nahisaho/musubix-dfg';
+
+const deps = analyzeDataDependencies(dfg);
+
+for (const dep of deps) {
+  console.log(`${dep.from} -> ${dep.to}: ${dep.type}`);
+}
+```
+
+---
+
+## Lean 4 Integration
+
+*(v2.0.0)*
+
+The `@nahisaho/musubix-lean` package provides integration with the Lean 4 theorem prover for formal verification of requirements.
+
+### Installation
+
+```bash
+npm install @nahisaho/musubix-lean
+```
+
+### EARS to Lean Conversion
+
+Convert EARS requirements to Lean 4 theorems:
+
+```typescript
+import { EarsToLeanConverter } from '@nahisaho/musubix-lean';
+
+const converter = new EarsToLeanConverter();
+
+// Convert EARS requirement
+const earsRequirement = {
+  id: 'REQ-001',
+  type: 'event-driven',
+  trigger: 'user clicks submit',
+  response: 'system saves form data',
+};
+
+const theorem = converter.convert(earsRequirement);
+console.log(theorem.leanCode);
+// theorem REQ_001 : ∀ (s : State), 
+//   user_clicks_submit s → saves_form_data (next s)
+```
+
+### Lean AST Parsing
+
+Parse and manipulate Lean 4 code:
+
+```typescript
+import { LeanParser, LeanAST } from '@nahisaho/musubix-lean';
+
+const parser = new LeanParser();
+const ast = parser.parse(leanCode);
+
+// Traverse AST
+for (const decl of ast.declarations) {
+  if (decl.type === 'theorem') {
+    console.log(`Theorem: ${decl.name}`);
+    console.log(`Statement: ${decl.statement}`);
+  }
+}
+```
+
+### Proof Engine
+
+Execute proofs using Lean 4:
+
+```typescript
+import { LeanProofEngine } from '@nahisaho/musubix-lean';
+
+const engine = new LeanProofEngine({
+  leanPath: '/usr/bin/lean',
+  timeout: 30000,
+});
+
+const result = await engine.prove(theorem);
+
+if (result.success) {
+  console.log('Proof verified!');
+  console.log(`Proof: ${result.proof}`);
+} else {
+  console.log(`Failed: ${result.error}`);
+}
+```
+
+### ReProver Integration
+
+Use ReProver for automated proof search:
+
+```typescript
+import { ReProverClient } from '@nahisaho/musubix-lean';
+
+const reprover = new ReProverClient({
+  endpoint: 'http://localhost:8080',
+  maxDepth: 10,
+});
+
+const proof = await reprover.searchProof(theorem);
+
+if (proof.found) {
+  console.log('Proof found!');
+  console.log(proof.tactics);
+}
+```
+
+---
+
+## YATA Scale
+
+*(v2.0.0)*
+
+The `@nahisaho/yata-scale` package provides distributed knowledge graph capabilities with sharding, caching, and synchronization.
+
+### Installation
+
+```bash
+npm install @nahisaho/yata-scale
+```
+
+### High-Level API
+
+Use `YataScaleManager` for simplified access:
+
+```typescript
+import { YataScaleManager } from '@nahisaho/yata-scale';
+
+const yata = new YataScaleManager({
+  shards: 16,
+  cacheConfig: {
+    l1MaxSize: 1000,
+    l2MaxSize: 10000,
+    l3Path: './cache',
+  },
+});
+
+// Store entity
+await yata.putEntity({
+  id: 'entity-1',
+  type: 'Document',
+  properties: { title: 'Example' },
+});
+
+// Query
+const results = await yata.query('SELECT ?s WHERE { ?s rdf:type Document }');
+```
+
+### Sharding
+
+Distribute data across shards using consistent hashing:
+
+```typescript
+import { ShardManager, HashPartitionStrategy } from '@nahisaho/yata-scale';
+
+const shardManager = new ShardManager({
+  shardCount: 16,
+  virtualNodes: 150,
+  strategy: new HashPartitionStrategy(),
+});
+
+// Get shard for entity
+const shard = shardManager.getShardForEntity('entity-id');
+console.log(`Entity assigned to shard ${shard.id}`);
+
+// Rebalance on node changes
+await shardManager.addNode('node-5');
+```
+
+### Multi-Tier Caching
+
+Three-tier caching for optimal performance:
+
+```typescript
+import { CacheManager, L1Cache, L2Cache, L3Cache } from '@nahisaho/yata-scale';
+
+const cache = new CacheManager({
+  l1: new L1Cache({ maxSize: 1000 }),    // LRU cache
+  l2: new L2Cache({ maxSize: 10000 }),   // LFU cache
+  l3: new L3Cache({ path: './cache' }),  // Disk cache
+});
+
+// Cache operations
+await cache.set('key', value, { ttl: 3600 });
+const result = await cache.get('key');
+
+// Invalidation
+await cache.invalidate('key');
+await cache.invalidatePattern('user:*');
+```
+
+### Index Management
+
+Multiple index types for efficient queries:
+
+```typescript
+import { IndexManager, BPlusTreeIndex, FullTextIndex, GraphIndex } from '@nahisaho/yata-scale';
+
+const indexManager = new IndexManager();
+
+// B+ Tree for range queries
+indexManager.addIndex('timestamp', new BPlusTreeIndex());
+
+// Full-text for search
+indexManager.addIndex('content', new FullTextIndex());
+
+// Graph for traversal
+indexManager.addIndex('relationships', new GraphIndex());
+```
+
+### Vector Clock Synchronization
+
+Distributed synchronization with conflict resolution:
+
+```typescript
+import { SyncController, VectorClock, ConflictResolver } from '@nahisaho/yata-scale';
+
+const sync = new SyncController({
+  nodeId: 'node-1',
+  conflictStrategy: 'last-write-wins',
+});
+
+// Synchronize with peers
+await sync.synchronize();
+
+// Manual conflict resolution
+sync.onConflict((conflict) => {
+  console.log(`Conflict on ${conflict.key}`);
+  return conflict.local; // or conflict.remote
+});
+```
+
+### v2.0.0 Package Summary
+
+| Package | Tests | Description |
+|---------|-------|-------------|
+| `@nahisaho/musubix-dfg` | 30 | DFG/CFG extraction, Def-Use analysis |
+| `@nahisaho/musubix-lean` | 151 | Lean 4 theorem proving, EARS conversion |
+| `@nahisaho/yata-scale` | 57 | Distributed sharding, caching, sync |
+| **Total** | **238** | **Phase 1: Deep Symbolic Integration** |
+
+---
+
+## Library Learning
+
+*(v2.0.0 NEW!)*
+
+The `@nahisaho/musubix-library-learner` package provides intelligent API pattern extraction and documentation from TypeScript libraries, enabling the system to learn and suggest library usage patterns.
+
+### Installation
+
+```bash
+npm install @nahisaho/musubix-library-learner
+```
+
+### Library Analysis
+
+Analyze and learn patterns from TypeScript libraries:
+
+```typescript
+import { LibraryLearner, LibraryAnalyzer } from '@nahisaho/musubix-library-learner';
+
+const learner = new LibraryLearner();
+
+// Analyze a library
+const patterns = await learner.analyze({
+  libraryName: 'lodash',
+  sourceDir: './node_modules/lodash',
+  includeExamples: true,
+});
+
+console.log(`Found ${patterns.length} API patterns`);
+for (const pattern of patterns) {
+  console.log(`- ${pattern.name}: ${pattern.description}`);
+}
+```
+
+### Pattern Extraction
+
+Extract usage patterns from code examples:
+
+```typescript
+import { PatternExtractor } from '@nahisaho/musubix-library-learner';
+
+const extractor = new PatternExtractor();
+
+const patterns = await extractor.extract({
+  code: sourceCode,
+  libraryContext: 'express',
+});
+
+// Get common patterns
+for (const pattern of patterns) {
+  console.log(`Pattern: ${pattern.type}`);
+  console.log(`Example: ${pattern.example}`);
+  console.log(`Frequency: ${pattern.frequency}`);
+}
+```
+
+### API Documentation Generation
+
+Generate documentation from learned patterns:
+
+```typescript
+import { DocGenerator } from '@nahisaho/musubix-library-learner';
+
+const generator = new DocGenerator();
+
+const docs = await generator.generate({
+  patterns: learnedPatterns,
+  format: 'markdown',
+  includeExamples: true,
+});
+
+await fs.writeFile('API-PATTERNS.md', docs);
+```
+
+### TypeScript Analysis
+
+Deep analysis of TypeScript declarations:
+
+```typescript
+import { TypeScriptAnalyzer } from '@nahisaho/musubix-library-learner';
+
+const analyzer = new TypeScriptAnalyzer();
+
+const analysis = await analyzer.analyze('./src/index.ts');
+
+// Get exported types
+for (const type of analysis.exports) {
+  console.log(`${type.name}: ${type.kind}`);
+  if (type.kind === 'function') {
+    console.log(`  Params: ${type.parameters.join(', ')}`);
+    console.log(`  Returns: ${type.returnType}`);
+  }
+}
+```
+
+---
+
+## Neural Search
+
+*(v2.0.0 NEW!)*
+
+The `@nahisaho/musubix-neural-search` package provides semantic code search using embeddings and neural ranking, enabling context-aware code discovery across your codebase.
+
+### Installation
+
+```bash
+npm install @nahisaho/musubix-neural-search
+```
+
+### Semantic Search
+
+Search code by meaning, not just keywords:
+
+```typescript
+import { NeuralSearchEngine } from '@nahisaho/musubix-neural-search';
+
+const search = new NeuralSearchEngine({
+  embeddingModel: 'code-bert',
+  indexPath: './search-index',
+});
+
+// Index your codebase
+await search.index('./src');
+
+// Semantic search
+const results = await search.search({
+  query: 'function that validates email addresses',
+  limit: 10,
+});
+
+for (const result of results) {
+  console.log(`${result.file}:${result.line} (score: ${result.score})`);
+  console.log(`  ${result.snippet}`);
+}
+```
+
+### Code Embeddings
+
+Generate embeddings for code snippets:
+
+```typescript
+import { CodeEmbedder } from '@nahisaho/musubix-neural-search';
+
+const embedder = new CodeEmbedder();
+
+const embedding = await embedder.embed({
+  code: 'function add(a: number, b: number) { return a + b; }',
+  language: 'typescript',
+});
+
+// Compare similarity
+const similarity = embedder.cosineSimilarity(embedding1, embedding2);
+console.log(`Similarity: ${similarity}`);
+```
+
+### Hybrid Search
+
+Combine lexical and semantic search:
+
+```typescript
+import { HybridSearchEngine } from '@nahisaho/musubix-neural-search';
+
+const hybrid = new HybridSearchEngine({
+  lexicalWeight: 0.3,
+  semanticWeight: 0.7,
+});
+
+const results = await hybrid.search({
+  query: 'authentication middleware express',
+  filters: {
+    language: 'typescript',
+    path: 'src/middleware/**',
+  },
+});
+```
+
+### Code Ranking
+
+Neural ranking for search results:
+
+```typescript
+import { CodeRanker } from '@nahisaho/musubix-neural-search';
+
+const ranker = new CodeRanker();
+
+const ranked = await ranker.rank({
+  query: 'parse JSON safely',
+  candidates: searchResults,
+  context: {
+    currentFile: './src/utils.ts',
+    recentFiles: ['./src/api.ts', './src/config.ts'],
+  },
+});
+```
+
+---
+
+## Program Synthesis
+
+*(v2.0.0 NEW!)*
+
+The `@nahisaho/musubix-synthesis` package provides neural-guided program synthesis, generating code from specifications using symbolic constraints and neural networks.
+
+### Installation
+
+```bash
+npm install @nahisaho/musubix-synthesis
+```
+
+### Specification-Based Synthesis
+
+Generate code from formal specifications:
+
+```typescript
+import { ProgramSynthesizer } from '@nahisaho/musubix-synthesis';
+
+const synthesizer = new ProgramSynthesizer();
+
+const result = await synthesizer.synthesize({
+  specification: {
+    input: 'array of numbers',
+    output: 'array of numbers (sorted ascending)',
+    constraints: ['stable sort', 'O(n log n) complexity'],
+  },
+  language: 'typescript',
+});
+
+console.log(result.code);
+// function sort(arr: number[]): number[] {
+//   return [...arr].sort((a, b) => a - b);
+// }
+```
+
+### Example-Based Synthesis
+
+Generate code from input/output examples:
+
+```typescript
+import { ExampleBasedSynthesizer } from '@nahisaho/musubix-synthesis';
+
+const synthesizer = new ExampleBasedSynthesizer();
+
+const result = await synthesizer.synthesize({
+  examples: [
+    { input: ['hello', 'world'], output: 'hello world' },
+    { input: ['foo', 'bar', 'baz'], output: 'foo bar baz' },
+  ],
+  language: 'typescript',
+});
+
+console.log(result.code);
+// function join(arr: string[]): string {
+//   return arr.join(' ');
+// }
+```
+
+### Sketch-Based Synthesis
+
+Fill in holes in partial programs:
+
+```typescript
+import { SketchSynthesizer } from '@nahisaho/musubix-synthesis';
+
+const synthesizer = new SketchSynthesizer();
+
+const result = await synthesizer.complete({
+  sketch: `
+    function validate(email: string): boolean {
+      const pattern = ??; // hole to fill
+      return pattern.test(email);
+    }
+  `,
+  constraints: ['must validate common email formats'],
+});
+
+console.log(result.code);
+// const pattern = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
+```
+
+### Neural-Guided Search
+
+Use neural networks to guide synthesis search:
+
+```typescript
+import { NeuralGuidedSynthesizer } from '@nahisaho/musubix-synthesis';
+
+const synthesizer = new NeuralGuidedSynthesizer({
+  model: 'codegen-2B',
+  beamWidth: 5,
+  maxDepth: 10,
+});
+
+const result = await synthesizer.synthesize({
+  description: 'Parse a CSV string into array of objects with headers as keys',
+  tests: [
+    {
+      input: 'name,age\\nAlice,30\\nBob,25',
+      output: [{ name: 'Alice', age: '30' }, { name: 'Bob', age: '25' }],
+    },
+  ],
+});
+
+if (result.verified) {
+  console.log('All tests pass!');
+  console.log(result.code);
+}
+```
+
+### Synthesis Verification
+
+Verify synthesized code against specifications:
+
+```typescript
+import { SynthesisVerifier } from '@nahisaho/musubix-synthesis';
+
+const verifier = new SynthesisVerifier();
+
+const verification = await verifier.verify({
+  code: synthesizedCode,
+  specification: originalSpec,
+  tests: testCases,
+});
+
+console.log(`Verified: ${verification.passed}`);
+console.log(`Coverage: ${verification.testCoverage}%`);
+for (const issue of verification.issues) {
+  console.log(`Issue: ${issue.message}`);
+}
+```
+
+### v2.0.0 Package Summary
+
+| Package | Tests | Description |
+|---------|-------|-------------|
+| **Phase 1: Deep Symbolic Integration** | | |
+| `@nahisaho/musubix-dfg` | 30 | DFG/CFG extraction, Def-Use analysis |
+| `@nahisaho/musubix-lean` | 151 | Lean 4 theorem proving, EARS conversion |
+| `@nahisaho/yata-scale` | 57 | Distributed sharding, caching, sync |
+| **Phase 2: Advanced Learning** | | |
+| `@nahisaho/musubix-library-learner` | 132 | API pattern extraction, documentation |
+| `@nahisaho/musubix-neural-search` | 144 | Semantic code search, embeddings |
+| `@nahisaho/musubix-synthesis` | 146 | Neural-guided program synthesis |
+| **Total v2.0.0 New** | **660** | **6 new packages** |
+
+---
+
 ## MCP Server Integration
 
 ### CLI Startup
@@ -1817,6 +2647,6 @@ const client = createYATAClient({
 
 ---
 
-**Version**: 1.6.0  
-**Last Updated**: 2026-01-06  
+**Version**: 2.0.0  
+**Last Updated**: 2026-01-08  
 **MUSUBIX Project**