claude-flow-novice 2.18.12 → 2.18.14

package/.claude/skills/cfn-local-ruvector-accelerator/docs/PHASE_4_QUERY_API.md

@@ -0,0 +1,210 @@
+# Phase 4: Agent Query API Implementation
+
+## Overview
+
+Phase 4 of the AST-Aware RuVector Accelerator implements a high-level query interface for agent use cases, replacing complex grep patterns with structured SQL queries. The implementation provides fast (<50ms) queries on 10k+ entity indexes with support for both Rust and TypeScript codebases.
+
+## Features Implemented
+
+### 1. Query API (`src/query_api.rs`)
+
+The QueryApi struct provides six core query methods:
+
+- **`find_functions_using_type(type_name)`**: Find all functions that use a specific type
+- **`find_callers_of_function(function_name, exclude_module)`**: Find all callers of a function
+- **`find_types_used_elsewhere(file_path)`**: Find types defined in a file that are used elsewhere
+- **`find_implementations(trait_name)`**: Find all implementations of a trait/interface
+- **`find_public_api(module_path)`**: Get the public API surface of a module
+- **`find_references_to_path(path)`**: Find all references to a path for refactoring
+
+### 2. CLI Commands
+
+#### `ruvector find` Command
+Structured search with multiple filter options:
+
+```bash
+# Find functions using a type
+ruvector find --uses-type Album
+
+# Find callers of a function, excluding certain module
+ruvector find --called-by create_album --exclude tests
+
+# Find types from a file used elsewhere
+ruvector find --types-from src/models.rs
+
+# Find implementations of a trait
+ruvector find --implements Display
+
+# Find public API of a module
+ruvector find --public-api auth
+
+# Filter by entity kind
+ruvector find --uses-type String --kind function
+
+# Output in different formats
+ruvector find --uses-type Album --format json
+ruvector find --uses-type Album --format csv --output results.csv
+```
+
+#### `ruvector refs` Command
+Find all references to an entity:
+
+```bash
+# Simple reference search
+ruvector refs create_album
+
+# With specific entity kind
+ruvector refs Album --kind struct
+
+# Group results by file
+ruvector refs Album --group-by-file
+
+# Tree format for visual hierarchy
+ruvector refs Album --format tree
+
+# Filter to specific file
+ruvector refs Album --file src/main.rs
+
+# Inbound/outbound reference filtering
+ruvector refs create_album --inbound
+ruvector refs create_album --outbound
+```
+
+### 3. Performance Optimizations
+
+- Database indexes optimized for common query patterns
+- Composite indexes for entity+kind, file+kind, and type+usage patterns
+- Query result caching in SQLite WAL mode
+- Memory-mapped I/O for large databases (256MB default)
+
+### 4. Output Formats
+
+All queries support multiple output formats:
+
+- **Simple**: One line per result (default)
+- **Detailed**: Full entity information with context
+- **JSON**: Machine-readable format for agents
+- **CSV**: Spreadsheet-friendly format
+- **Tree**: Hierarchical view for references
+
+## Usage Examples
+
+### Agent Integration
+
+The query API is designed for easy integration with AI agents:
+
+```rust
+use crate::query_api::QueryApi;
+use crate::store_v2::StoreV2;
+
+let store = StoreV2::new(&db_path)?;
+let query_api = QueryApi::new(store);
+
+// Find all functions using the Album type
+let results = query_api.find_functions_using_type("Album")?;
+
+// JSON output for agent consumption
+let json_output = results.format_json()?;
+```
+
+### Refactoring Support
+
+The references query enables safe refactoring:
+
+```bash
+# Before renaming a function
+ruvector refs old_function_name --format tree
+
+# Check all uses of a type before major changes
+ruvector find --uses-type MyStruct --format detailed
+```
+
+### Code Analysis
+
+Understand codebase structure and dependencies:
+
+```bash
+# Public API surface
+ruvector find --public-api . --format json
+
+# Type usage statistics
+ruvector find --types-from src/lib.rs --format detailed
+```
+
+## Database Schema
+
+The query API uses the v2 schema with optimized tables:
+
+- `entities`: Core entity definitions (functions, types, etc.)
+- `refs`: Cross-references between entities
+- `type_usage`: Type usage tracking for dependency analysis
+- `entity_embeddings`: Vector embeddings for semantic search
+
+## Performance Metrics
+
+Target performance for all queries:
+- **< 50ms** for databases with 10,000+ entities
+- **< 200ms** for databases with 100,000+ entities
+- **Concurrent query support** through SQLite connection pooling
+
+## Testing
+
+Run the test suite:
+
+```bash
+# Basic functionality test
+./test_query_api.sh
+
+# Performance benchmarks
+cargo test --release query_performance
+
+# Integration tests
+cargo test query_api_integration
+```
+
+## Future Enhancements
+
+Planned improvements for Sprint 4-2:
+
+1. **Natural Language Query Parsing**
+   - Parse "find all functions that return Result"
+   - Support "show me callers of main in tests only"
+
+2. **Advanced Filters**
+   - Filter by visibility (pub, pub(crate), private)
+   - Filter by attributes (#[test], #[async_trait])
+   - Date range filters for recently changed code
+
+3. **Query Composition**
+   - Chain multiple queries
+   - Set operations (union, intersection, difference)
+
+4. **Performance Optimizations**
+   - Query result pagination
+   - Incremental result streaming
+   - Parallel query execution
+
+## Integration with CFN Agents
+
+The query API is designed to work seamlessly with CFN agents:
+
+- Agents can query code structure without parsing files
+- Fast lookups enable real-time code analysis
+- JSON output format easy for agent consumption
+- Structured queries replace brittle grep patterns
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Slow queries**: Check database indexes with `EXPLAIN QUERY PLAN`
+2. **No results**: Verify entities are indexed with `ruvector stats`
+3. **JSON errors**: Ensure entity names are properly quoted
+
+### Debug Queries
+
+Enable debug logging to see actual SQL queries:
+
+```bash
+RUST_LOG=debug ruvector find --uses-type Album
+```

package/.claude/skills/cfn-local-ruvector-accelerator/docs/RUST_AST_EXTRACTOR_IMPLEMENTATION.md

@@ -0,0 +1,222 @@
+# Rust AST Extractor Implementation
+
+## Overview
+
+The Rust AST Extractor is implemented as part of Phase 2 of the AST-Aware RuVector Accelerator. It provides entity and reference extraction from Rust source code using tree-sitter.
+
+## Architecture
+
+### Core Components
+
+1. **Extractor Trait** (`src/extractors/mod.rs`)
+   - Common interface for all language extractors
+   - Defines `Entity`, `Reference`, and `ExtractionResult` types
+   - Provides utility functions for node traversal and metadata extraction
+
+2. **RustExtractor** (`src/extractors/rust.rs`)
+   - Concrete implementation for Rust language
+   - Uses tree-sitter-rust for parsing
+   - Extracts functions, structs, impls, traits, enums, and references
+
+3. **Entity Types**
+   - `Function`: Functions with parameters, return type, visibility
+   - `Struct`: Struct definitions with fields and generics
+   - `Impl`: Implementation blocks (trait impls and inherent impls)
+   - `Trait`: Trait definitions with methods and bounds
+   - `Enum`: Enum definitions with variants
+   - `TypeAlias`: Type aliases
+
+4. **Reference Types**
+   - `Calls`: Function calls
+   - `Uses`: Type usage
+   - `Imports`: Use declarations
+   - `Implements`: Trait implementations
+
+## Current Status
+
+### ✅ Completed
+
+1. **Core Infrastructure**
+   - Extractor trait definition
+   - Entity and reference data structures
+   - Utility functions for tree-sitter operations
+
+2. **Rust Extractor Skeleton**
+   - Full extractor implementation structure
+   - Methods for all entity types
+   - Reference extraction for calls, types, and imports
+   - Comprehensive test suite
+
+3. **Dependencies Added**
+   - tree-sitter = "0.20"
+   - tree-sitter-rust = "0.20"
+   - tree-sitter-typescript = "0.20"
+
+### ⚠️ Pending Integration
+
+The tree-sitter-rust grammar needs to be properly integrated. Currently, the extractor returns an error:
+```
+tree-sitter-rust not yet integrated
+```
+
+## Next Steps for Full Implementation
+
+### 1. Tree-sitter Grammar Integration
+
+To complete the implementation:
+
+1. **Build tree-sitter parsers as a build script**:
+   ```rust
+   // build.rs
+   fn main() {
+       tree_sitter_cli::generate::generate_parser_in(
+           "src/grammar",
+           "src/grammar.rs",
+           false
+       ).unwrap();
+   }
+   ```
+
+2. **Initialize tree-sitter-rust properly**:
+   ```rust
+   extern "C" {
+       fn tree_sitter_rust() -> Language;
+   }
+
+   let language = unsafe { tree_sitter_rust() };
+   ```
+
+3. **Alternative: Use tree-sitter-loader**:
+   ```rust
+   use tree_sitter::Loader;
+
+   let mut loader = Loader::new();
+   loader.configure_language("rust", tree_sitter_rust::language())?;
+   ```
+
+### 2. Enhanced Entity Extraction
+
+Once tree-sitter is integrated:
+
+1. **Generic Parameter Extraction**
+   - Extract type parameters with bounds
+   - Handle lifetime parameters
+   - Track where clauses
+
+2. **Method Extraction in Impl Blocks**
+   - Extract methods from impl blocks
+   - Associate methods with their types
+   - Handle trait method implementations
+
+3. **Attribute Extraction**
+   - Parse derive macros
+   - Extract custom attributes
+   - Handle conditionals (`#[cfg(...)]`)
+
+4. **Module System**
+   - Track module declarations
+   - Handle module visibility
+   - Extract mod hierarchy
+
+### 3. Reference Resolution
+
+1. **Cross-file References**
+   - Track imports and exports
+   - Resolve fully qualified paths
+   - Handle crate references
+
+2. **Type Resolution**
+   - Build type maps
+   - Track type aliases
+   - Resolve generic instantiations
+
+3. **Call Graph Construction**
+   - Map function calls to definitions
+   - Handle trait method calls
+   - Track closures and async functions
+
+## Usage Example
+
+```rust
+use local_ruvector::extractors::create_rust_extractor;
+
+let mut extractor = create_rust_extractor()?;
+let source = r#"
+    pub fn add(a: i32, b: i32) -> i32 {
+        a + b
+    }
+"#;
+
+let result = extractor.extract("example.rs", source)?;
+println!("Found {} entities", result.entities.len());
+println!("Found {} references", result.references.len());
+```
+
+## Testing
+
+The implementation includes comprehensive tests:
+
+1. **Unit Tests** (`tests/test_rust_extractor.rs`)
+   - Basic extraction functionality
+   - Complex code patterns
+   - Error handling
+
+2. **Example** (`examples/rust_ast_extractor_demo.rs`)
+   - Demonstrates full extraction workflow
+   - Shows entity and reference output
+
+## Integration with RuVector
+
+Once complete, the extractor will integrate with:
+
+1. **Indexing Pipeline**
+   - Extract entities during file scanning
+   - Store in SQLite database
+   - Generate embeddings for semantic search
+
+2. **Search Engine**
+   - Enable entity-based queries
+   - Support reference traversal
+   - Provide code intelligence features
+
+3. **CLI Commands**
+   - `index`: Extract and store entities
+   - `query`: Search extracted entities
+   - `analyze`: Generate code metrics
+
+## Performance Considerations
+
+1. **Incremental Parsing**
+   - Only reparse changed files
+   - Cache parse trees
+   - Parallel processing
+
+2. **Memory Optimization**
+   - Stream large files
+   - Limit node traversal depth
+   - Reuse tree-sitter parsers
+
+3. **Database Indexing**
+   - Index entity fields
+   - Optimize reference lookups
+   - Compress metadata
+
+## File Locations
+
+- Core extractor: `/mnt/c/Users/masha/Documents/claude-flow-novice/.claude/skills/cfn-local-ruvector-accelerator/src/extractors/rust.rs`
+- Module definitions: `/mnt/c/Users/masha/Documents/claude-flow-novice/.claude/skills/cfn-local-ruvector-accelerator/src/extractors/mod.rs`
+- Tests: `/mnt/c/Users/masha/Documents/claude-flow-novice/.claude/skills/cfn-local-ruvector-accelerator/tests/test_rust_extractor.rs`
+- Example: `/mnt/c/Users/masha/Documents/claude-flow-novice/.claude/skills/cfn-local-ruvector-accelerator/examples/rust_ast_extractor_demo.rs`
+
+## Confidence Score
+
+**Current Implementation: 0.75**
+
+- ✅ Complete structure and scaffolding
+- ✅ All entity types defined
+- ✅ Reference extraction logic
+- ⚠️ Tree-sitter integration pending
+- ⚠️ Generic handling needs refinement
+- ⚠️ Cross-file reference resolution TBD
+
+Once tree-sitter-rust is properly integrated and tests pass, confidence score will be **0.95**.

package/.claude/skills/cfn-local-ruvector-accelerator/docs/TRANSACTION_MANAGEMENT.md

@@ -0,0 +1,241 @@
+# Transaction Management in AST-Aware RuVector Accelerator
+
+## Overview
+
+This document describes the transaction management improvements implemented to ensure data integrity during multi-step database operations.
+
+## Problem Statement
+
+The original code lacked proper transaction management for:
+- Batch entity insertions
+- File indexing operations
+- Schema migrations
+- Multi-table updates
+
+This could lead to:
+- Partial updates and data corruption
+- Inconsistent database state
+- Difficulty recovering from errors
+
+## Solution Architecture
+
+### 1. Transaction-Aware Storage Layer (`store_v2_tx.rs`)
+
+Created a new transaction-aware storage layer that provides:
+- Batch operations with automatic transaction wrapping
+- Atomic file indexing
+- Schema migration support with rollback
+- Proper error handling with context
+
+#### Key Features:
+
+```rust
+// Batch insert with automatic transaction
+let ids = store.insert_entities_batch(&entities)?;
+
+// Atomic file indexing - all or nothing
+store.index_file_atomic(file_path, file_hash, |tx| {
+    // All operations within this closure are atomic
+    // If an error occurs, everything rolls back
+    insert_entities(tx)?;
+    insert_references(tx)?;
+    insert_embeddings(tx)?;
+    Ok(())
+})?;
+
+// Atomic schema migration
+store.migrate_schema_atomic("migration_name", |tx| {
+    // Migration logic
+    Ok(())
+})?;
+```
+
+### 2. Enhanced File Indexing (`index_ast.rs`)
+
+Modified the AST indexing process to use transactions:
+
+- All entities, references, and embeddings for a file are inserted atomically
+- Previous data for a file is cleared before inserting new data
+- File hash tracking ensures incremental indexing works correctly
+
+```rust
+// Original approach - vulnerable to partial updates
+for entity in entities {
+    store.insert_entity(entity)?; // Each call is separate
+}
+
+// New approach - atomic
+store.index_file_atomic(&file_path, &file_hash, |tx| {
+    // All operations in a single transaction
+    for entity in entities {
+        insert_entity_in_tx(tx, entity)?;
+    }
+    Ok(())
+})?;
+```
+
+### 3. Robust Migration System (`migration_tx.rs`)
+
+Implemented a new migration system with:
+
+- Savepoint-based transactions for nested operations
+- Batch processing to avoid long-running transactions
+- Progress tracking and error reporting
+- Automatic rollback on failures
+- Migration validation
+
+```rust
+// Atomic migration with validation
+MigrationWithTx::migrate_v1_to_v2_atomic(&conn)?;
+MigrationWithTx::validate_migration(&conn)?;
+```
+
+## Implementation Details
+
+### Transaction Patterns
+
+1. **Simple Transaction**
+```rust
+let tx = conn.transaction()?;
+// ... operations ...
+tx.commit()?;
+```
+
+2. **Savepoint for Nested Operations**
+```rust
+let sp = conn.savepoint()?;
+// ... operations ...
+sp.commit()?; // or sp.rollback()?
+```
+
+3. **Atomic Operation Pattern**
+```rust
+fn atomic_operation<F>(&self, f: F) -> Result<()>
+where F: FnOnce(&Transaction) -> Result<()>
+{
+    let tx = self.transaction()?;
+    f(&tx)?; // May fail
+    tx.commit()?; // Only commits if f succeeds
+    Ok(())
+}
+```
+
+### Error Handling Strategy
+
+1. **Context Preservation**
+   - All errors include context about what operation failed
+   - Uses `anyhow::Context` for chainable error messages
+
+2. **Automatic Rollback**
+   - Dropping a transaction without commit triggers rollback
+   - Savepoints can be explicitly rolled back
+
+3. **Batch Processing**
+   - Large operations are split into batches
+   - Each batch is a separate transaction
+   - Partial progress is preserved
+
+### Performance Considerations
+
+1. **Statement Preparation**
+   - Prepared statements are reused within transactions
+   - Reduces parsing overhead
+
+2. **Batch Size**
+   - Configurable batch sizes prevent long-running transactions
+   - Default batch size: 1000 for migrations, 100 for file indexing
+
+3. **Connection Pooling**
+   - WAL mode enables concurrent readers
+   - Memory-mapped I/O for better performance
+
+## Usage Examples
+
+### Indexing a File with Transactions
+
+```rust
+let store = StoreV2WithTx::new(&db_path)?;
+
+// All operations for the file are atomic
+let result = store.index_file_atomic("src/main.rs", "hash123", |tx| {
+    // Clear existing data for this file
+    tx.execute("DELETE FROM entities WHERE file_path = ?", ["src/main.rs"])?;
+
+    // Insert new entities
+    for entity in entities {
+        insert_entity(tx, entity)?;
+    }
+
+    // Insert references
+    for reference in references {
+        insert_reference(tx, reference)?;
+    }
+
+    // Update file hash
+    tx.execute(
+        "INSERT OR REPLACE INTO file_hashes (file_path, file_hash) VALUES (?, ?)",
+        ["src/main.rs", "hash123"]
+    )?;
+
+    Ok(())
+})?;
+
+// If any operation failed, everything rolls back
+```
+
+### Migration with Validation
+
+```rust
+// Migrate with automatic rollback on errors
+match MigrationWithTx::migrate_v1_to_v2_atomic(&conn) {
+    Ok(()) => {
+        // Validate migration was successful
+        MigrationWithTx::validate_migration(&conn)?;
+        println!("Migration completed successfully");
+    }
+    Err(e) => {
+        eprintln!("Migration failed: {}", e);
+        // Database remains in original state
+    }
+}
+```
+
+## Testing
+
+Comprehensive tests verify:
+- Atomic operations rollback on errors
+- Transaction isolation works correctly
+- Batch operations maintain consistency
+- Migration rollback preserves original data
+- Concurrent access patterns
+
+Run tests with:
+```bash
+cargo test transaction
+```
+
+## Best Practices
+
+1. **Always use transactions for multi-step operations**
+2. **Keep transactions as short as possible**
+3. **Use savepoints for nested operations**
+4. **Provide meaningful error context**
+5. **Validate after complex operations**
+
+## Migration Checklist
+
+When updating existing code to use transactions:
+
+1. Identify multi-step operations
+2. Wrap them in transactions or atomic operations
+3. Test rollback behavior
+4. Add error context
+5. Consider batch processing for large operations
+
+## Future Improvements
+
+1. **Distributed Transactions**: Support for multiple database files
+2. **Transaction Logging**: Detailed logging of transaction boundaries
+3. **Performance Monitoring**: Track transaction durations
+4. **Deadlock Detection**: Automatic detection and resolution
+5. **Async Support**: Async transaction operations for better concurrency