clusterkit 0.1.0.pre.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: fd025da9b7f5c97e370d05fb1062484cb99b0aaaa4a7c310eb27df78336c91b4
+  data.tar.gz: 7665cf847930bc47cb04adc1f466ba09ea33c14da5f242434b08493047945f91
+SHA512:
+  metadata.gz: a8db6d4738ad99a20887aef90398d4163bdd8bc47bbdb8dda74496adc105602051999e50d2bc6003b4981d16b8121151d71cf55618691262b4a092f3c46a2545
+  data.tar.gz: 1a6e43a00d19d7fdaf35be6deffa5215e994a1f013b140aa23ac9bbb9d982facde94d710a894cba39e1fbd39bccd7c020b86a99e9d32d2aa256f717844623e5e

data/.rspec

@@ -0,0 +1,3 @@
+--require spec_helper
+--format documentation
+--color

data/.simplecov

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+SimpleCov.configure do
+  # Add custom groups
+  add_group 'Core', 'lib/annembed/embedder'
+  add_group 'UMAP', 'lib/annembed/umap'
+  add_group 'Utils', 'lib/annembed/utils'
+  add_group 'Configuration', 'lib/annembed/config'
+  
+  # Track branches as well as lines
+  enable_coverage :branch
+  
+  # Set thresholds (temporarily disabled to diagnose issues)
+  # minimum_coverage line: 50, branch: 40
+  
+  # Don't refuse to run tests if coverage drops (during development)
+  # refuse_coverage_drop
+  
+  # Maximum coverage drop allowed
+  maximum_coverage_drop 5
+  
+  # Configure output directory
+  coverage_dir 'coverage'
+  
+  # Track test files separately
+  track_files 'lib/**/*.rb'
+  
+  # Custom filters
+  add_filter do |source_file|
+    # Skip version file
+    source_file.filename.include?('version.rb')
+  end
+  
+  # Include timestamp in coverage report
+  SimpleCov.formatter = SimpleCov::Formatter::MultiFormatter.new([
+    SimpleCov::Formatter::HTMLFormatter,
+  ])
+  
+  # Set project name
+  command_name 'RSpec'
+  
+  # Merge results from multiple test runs
+  use_merging true
+  
+  # Set result cache timeout (in seconds)
+  merge_timeout 3600
+end

data/CHANGELOG.md

@@ -0,0 +1,35 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+- Clean, scikit-learn-like API for UMAP
+  - `fit(data)` - Train the model
+  - `transform(data)` - Transform new data
+  - `fit_transform(data)` - Train and transform in one step
+  - `fitted?` - Check if model is trained
+  - `save(path)` - Save trained model
+  - `load(path)` - Load trained model
+- Model persistence with save/load functionality
+- Data export/import utilities for caching results
+- Comprehensive test suite for UMAP interface
+- Detailed README with practical examples
+
+### Changed
+- Complete API redesign to follow ML library conventions
+- Removed confusing `save_embeddings`/`load_embeddings` methods
+- Separated model operations from data caching concerns
+
+### Fixed
+- Intermittent test failures with boundary assertions
+- Data normalization issues with extreme values
+
+## [0.1.0] - TBD
+
+### Added
+- Initial release with basic embedding functionality

data/CLAUDE.md

@@ -0,0 +1,226 @@
+# CLAUDE.md - clusterkit Project Guide
+
+## Project Vision
+clusterkit brings high-performance dimensionality reduction and embedding algorithms to Ruby by wrapping the annembed Rust crate. This gem is part of the ruby-nlp ecosystem, which aims to provide Ruby developers with native machine learning and NLP capabilities through best-in-breed Rust implementations.
+
+## Core Principles
+
+### 1. Ruby-First Design
+- Provide an idiomatic Ruby API that feels natural to Ruby developers
+- Follow Ruby naming conventions (snake_case methods, proper use of symbols)
+- Support Ruby's duck typing while maintaining type safety at the Rust boundary
+- Integrate seamlessly with Ruby's data science ecosystem
+
+### 2. Performance Without Compromise
+- Leverage Rust's performance for compute-intensive operations
+- Use Magnus for zero-copy data transfer where possible
+- Enable parallelization by default
+- Provide progress feedback for long-running operations
+
+### 3. Ecosystem Integration
+- Primary support for Numo::NArray (the NumPy of Ruby)
+- Work well with other ruby-nlp gems (lancelot, red-candle)
+- Support common Ruby data formats and visualization tools
+- Play nice with Jupyter notebooks (iruby)
+
+## Technical Guidelines
+
+### Magnus Best Practices
+
+1. **Memory Management**
+   ```rust
+   // Good: Let Magnus handle Ruby object lifecycle
+   let array: RArray = data.try_convert()?;
+   
+   // Avoid: Manual memory management
+   // Don't try to manually free Ruby objects
+   ```
+
+2. **Error Handling**
+   ```rust
+   // Always wrap errors properly
+   use magnus::Error;
+   
+   fn risky_operation() -> Result<RArray, Error> {
+       annembed_call()
+           .map_err(|e| Error::new(exception::runtime_error(), e.to_string()))?
+   }
+   ```
+
+3. **Type Conversions**
+   ```rust
+   // Define clear conversion traits
+   impl TryFrom<Value> for EmbedConfig {
+       type Error = Error;
+       // Robust conversion with good error messages
+   }
+   ```
+
+### Ruby API Design
+
+1. **Method Naming**
+   - Use Ruby conventions: `fit_transform`, not `fitTransform`
+   - Predicates end with `?`: `converged?`, `fitted?`
+   - Dangerous methods end with `!`: `normalize!`
+
+2. **Parameter Handling**
+   ```ruby
+   # Good: Use keyword arguments with defaults
+   def initialize(method: :umap, n_components: 2, **options)
+   
+   # Avoid: Positional arguments for configuration
+   def initialize(method, n_components, min_dist, spread, ...)
+   ```
+
+3. **Return Values**
+   - Return Ruby arrays for small results
+   - Return Numo::NArray for large matrices
+   - Support multiple return formats via options
+
+### Performance Considerations
+
+1. **Data Transfer**
+   - Minimize copying between Ruby and Rust
+   - Use view/slice operations when possible
+   - Support streaming for large datasets
+
+2. **Threading**
+   - Respect Ruby's GVL (Global VM Lock)
+   - Release GVL for long-running Rust operations
+   - Use Rust's parallelization, not Ruby threads
+
+3. **Memory Usage**
+   - Provide memory estimates for large operations
+   - Support out-of-core processing for huge datasets
+   - Clear progress indication for long operations
+
+## Code Style Guidelines
+
+### Rust Side
+- Follow Rust standard style (rustfmt)
+- Comprehensive error types with context
+- Document all public functions
+- Use type aliases for clarity
+
+### Ruby Side
+- Follow Ruby Style Guide
+- Use YARD documentation format
+- Provide type signatures where helpful
+- Include usage examples in docs
+
+## Testing Philosophy
+
+1. **Comprehensive Coverage**
+   - Unit tests for all public methods
+   - Integration tests with real datasets
+   - Performance benchmarks
+   - Memory leak tests
+
+2. **Test Data**
+   - Use standard ML datasets (Iris, MNIST samples)
+   - Generate synthetic data for edge cases
+   - Test with various Ruby object types
+
+3. **Platform Testing**
+   - Test on multiple Ruby versions
+   - Test on different operating systems
+   - Verify precompiled gem distribution
+
+## Documentation Standards
+
+1. **README**
+   - Clear installation instructions
+   - Quick start example that works
+   - Link to full documentation
+   - Performance comparisons
+
+2. **API Documentation**
+   - Every public method documented
+   - Parameter types and ranges specified
+   - Return values clearly described
+   - Usage examples for complex methods
+
+3. **Tutorials**
+   - Jupyter notebook examples
+   - Common use case walkthroughs
+   - Integration examples with other gems
+
+## Common Patterns
+
+### Configuration Objects
+```ruby
+# Prefer configuration objects over many parameters
+config = Annembed::Config.new(
+  method: :umap,
+  n_neighbors: 15,
+  min_dist: 0.1
+)
+embedder = Annembed::Embedder.new(config)
+```
+
+### Progress Callbacks
+```ruby
+# Support progress monitoring
+embedder.on_progress do |iteration, total|
+  puts "Progress: #{iteration}/#{total}"
+end
+```
+
+### Flexible Input/Output
+```ruby
+# Accept multiple input formats
+embedder.fit_transform(data)  # Array, NArray, or CSV path
+
+# Support different output formats
+result = embedder.transform(data, output: :array)    # Ruby Array
+result = embedder.transform(data, output: :narray)   # Numo::NArray
+```
+
+## Development Workflow
+
+1. **Branch Strategy**
+   - `main` - stable release
+   - `develop` - integration branch
+   - `feature/*` - new features
+   - `fix/*` - bug fixes
+
+2. **Release Process**
+   - Version bump in version.rb
+   - Update CHANGELOG.md
+   - Run full test suite
+   - Build precompiled gems
+   - Tag release
+   - Push to RubyGems
+
+3. **Continuous Integration**
+   - Run tests on each push
+   - Build gems for multiple platforms
+   - Check documentation building
+   - Performance regression tests
+
+## Future Considerations
+
+1. **GPU Support**
+   - Monitor annembed for GPU features
+   - Plan bindings if GPU support is added
+   - Consider alternative GPU libraries
+
+2. **Web Integration**
+   - Consider Rails integration
+   - WebAssembly compilation?
+   - REST API wrapper?
+
+3. **Visualization**
+   - Built-in plotting helpers?
+   - Export to common formats
+   - Interactive visualizations?
+
+## Getting Help
+
+When implementing new features:
+1. Check existing patterns in lancelot and red-candle
+2. Consult annembed documentation
+3. Ask in ruby-nlp discussions
+4. Profile before optimizing
+
+Remember: The goal is to make advanced embedding algorithms accessible and performant for Ruby developers while maintaining the simplicity and elegance that makes Ruby special.

data/Cargo.toml

@@ -0,0 +1,8 @@
+[workspace]
+members = ["ext/clusterkit"]
+resolver = "2"
+
+[profile.release]
+opt-level = 3
+lto = true
+codegen-units = 1

data/Gemfile

@@ -0,0 +1,17 @@
+source "https://rubygems.org"
+
+# Specify your gem's dependencies in annembed-ruby.gemspec
+gemspec
+
+# Test-only dependencies
+group :test do
+  # Optional: For comparing with Python implementations
+  gem "pycall", "~> 1.4", require: false
+end
+
+# Development dependencies for generating test fixtures
+group :development do
+  # For generating real embeddings to use as test fixtures
+  # This avoids the hanging issues with random test data
+  gem "red-candle", "~> 1.0", require: false
+end

data/IMPLEMENTATION_NOTES.md

@@ -0,0 +1,143 @@
+# Implementation Notes for annembed-ruby
+
+## Architecture Overview
+
+The gem follows a three-layer architecture:
+
+1. **Ruby Layer** (`lib/annembed/`): User-facing API with Ruby idioms
+2. **Magnus Bridge** (`ext/annembed_ruby/src/`): Type conversions and bindings
+3. **annembed Core**: The underlying Rust embedding library
+
+## Key Implementation Challenges
+
+### 1. Data Type Conversions
+
+The main challenge is efficiently converting between Ruby and Rust types:
+
+```
+Ruby Array/Numo::NArray ↔ ndarray::Array2<f32> ↔ annembed matrices
+```
+
+Consider using views instead of copies where possible.
+
+### 2. Memory Management
+
+- Magnus handles Ruby GC integration
+- Need to be careful with large matrices
+- Consider streaming for datasets > available RAM
+
+### 3. Progress Reporting
+
+annembed operations can be long-running. Options:
+1. Polling from Ruby side (requires thread)
+2. Callback from Rust (needs GVL management)
+3. Async/await pattern (complex but clean)
+
+### 4. Error Handling
+
+Map annembed errors to Ruby exceptions:
+- `annembed::Error` → `Annembed::Error`
+- Panic → `RuntimeError` (avoid panics!)
+- Invalid input → `ArgumentError`
+
+## annembed API Mapping
+
+### Core Types
+- `EmbedderT` - Main embedding trait
+- `HnswParams` - HNSW graph parameters
+- `EmbedderParams` - Algorithm-specific params
+- `GraphProjection` - For graph-based methods
+
+### Key Functions
+```rust
+// Main embedding function
+pub fn get_embedder(
+    data: &Array2<f32>,
+    params: EmbedderParams,
+    hnsw_params: HnswParams
+) -> Result<Box<dyn EmbedderT>>
+
+// The embedder trait
+pub trait EmbedderT {
+    fn embed(&self) -> Result<Array2<f32>>;
+    fn get_hnsw(&self) -> &Hnsw<f32, DistL2>;
+}
+```
+
+## Performance Considerations
+
+1. **Parallelization**: annembed uses rayon, respect Ruby's thread settings
+2. **BLAS Backend**: Allow users to choose (OpenBLAS vs MKL)
+3. **Large Datasets**: Implement chunking for > 1M points
+4. **GPU Future**: Design API to allow GPU backend later
+
+## Testing Strategy
+
+### Unit Tests
+- Type conversions
+- Configuration parsing
+- Error handling
+
+### Integration Tests
+- Small datasets (Iris)
+- Medium datasets (MNIST subset)
+- Large datasets (if CI allows)
+
+### Benchmarks
+- vs Python UMAP
+- vs pure Ruby implementations
+- Memory usage profiling
+
+## Platform Support
+
+### Priorities
+1. Linux x86_64 (most servers)
+2. macOS arm64 (M1/M2 developers)
+3. macOS x86_64 (Intel Macs)
+4. Windows x86_64 (if feasible)
+
+### Build Considerations
+- Use rake-compiler-dock for cross-compilation
+- Static link BLAS when possible
+- Provide clear instructions for source builds
+
+## Future Enhancements
+
+1. **Incremental Learning**: Add new points to existing embedding
+2. **Custom Metrics**: Allow user-defined distance functions
+3. **Supervised Embedding**: Use labels to guide embedding
+4. **GPU Support**: If annembed adds it
+5. **Visualization**: Built-in plotting helpers?
+
+## Debugging Tips
+
+1. Use `RUST_BACKTRACE=1` for better errors
+2. Add logging with `env_logger` in Rust
+3. Use `rb_sys` debug mode for development
+4. Memory debugging with valgrind/ASAN
+
+## Code Organization
+
+Keep related functionality together:
+- `embedder.rs`: All embedding algorithms
+- `utils.rs`: Dimension estimation, hubness
+- `svd.rs`: Matrix decomposition
+- `conversions.rs`: Type conversion helpers
+
+## Release Checklist
+
+1. [ ] Update version in `version.rb`
+2. [ ] Update CHANGELOG.md
+3. [ ] Run full test suite on all platforms
+4. [ ] Build precompiled gems
+5. [ ] Test gem installation from .gem file
+6. [ ] Tag release in git
+7. [ ] Push to RubyGems.org
+8. [ ] Update documentation
+
+## References
+
+- annembed docs: https://docs.rs/annembed/
+- Magnus guide: https://github.com/matsadler/magnus
+- UMAP paper: https://arxiv.org/abs/1802.03426
+- t-SNE paper: https://lvdmaaten.github.io/tsne/

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2024 Your Name
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/PYTHON_COMPARISON.md

@@ -0,0 +1,183 @@
+# ClusterKit vs Python UMAP/HDBSCAN: A Comparison
+
+## Overview
+
+ClusterKit and Python's UMAP/HDBSCAN implementations share the same algorithmic foundations but take different approaches to error handling and data validation. This document outlines these differences to help users understand what to expect from each implementation.
+
+## Philosophical Approaches
+
+### Python UMAP/HDBSCAN
+Python's implementations prioritize **algorithmic robustness**, attempting to produce results even in edge cases. This approach:
+- Maximizes compatibility with diverse datasets
+- Minimizes interruptions to analysis workflows  
+- Trusts users to interpret results appropriately
+- Follows the scikit-learn pattern of "fit on anything"
+
+### Ruby ClusterKit
+ClusterKit prioritizes **guided analysis**, providing feedback when data characteristics may lead to suboptimal results. This approach:
+- Helps users understand their data's suitability for the algorithm
+- Provides actionable suggestions when issues arise
+- Encourages best practices in dimensionality reduction
+- Aims to prevent misinterpretation of results
+
+## Behavioral Differences
+
+### Small Datasets
+
+#### Scenario: 3 data points with n_neighbors=15
+
+**Python UMAP:**
+- Automatically adjusts n_neighbors silently
+- Returns a 2D embedding of the 3 points
+- The resulting triangle's shape depends on random initialization
+
+**Ruby ClusterKit:**
+- Explicitly adjusts n_neighbors with optional warning
+- Currently experiences performance issues on very small datasets (being addressed)
+- Provides context about why adjustment was needed
+
+### Data Quality Issues
+
+#### Scenario: Random data without inherent structure
+
+**Python UMAP:**
+- Processes the data without warnings
+- Returns an embedding that may show apparent "clusters"
+- These patterns are typically artifacts of the algorithm rather than real structure
+
+**Ruby ClusterKit:**
+- May raise `IsolatedPointError` with explanation
+- Suggests that the data lacks structure suitable for manifold learning
+- Recommends alternatives like PCA for unstructured data
+
+#### Scenario: Extreme outliers (points 1000x farther than main data)
+
+**Python UMAP:**
+- Embeds outliers far from main cluster
+- Main cluster may be compressed to accommodate outlier scale
+- Visualization scale dominated by outliers
+
+**Ruby ClusterKit:**
+- May raise `IsolatedPointError` 
+- Explains impact of outliers on manifold learning
+- Suggests preprocessing steps like outlier removal or normalization
+
+### Invalid Data
+
+#### Scenario: NaN or Infinite values
+
+**Both implementations** reject invalid numerical data, but with different messaging:
+
+**Python:** `"Input contains NaN"` or `"Input contains infinity"`
+
+**Ruby:** `"Element at position [5, 2] is NaN or Infinite"`
+
+The Ruby version provides specific location information to aid debugging.
+
+### Edge Cases
+
+#### Scenario: Single data point
+
+**Python UMAP:**
+- Returns `[[0, 0]]` or similar default position
+- No error or warning about meaningless result
+
+**Ruby ClusterKit:**
+- Raises error explaining that manifold learning requires multiple points
+- Suggests minimum data requirements
+
+#### Scenario: Empty dataset
+
+**Both implementations** appropriately reject empty input with clear error messages.
+
+## Parameter Handling
+
+### Auto-adjustment
+
+**Python UMAP:**
+- Silently adjusts parameters when necessary
+- May issue warnings through Python's warning system
+- Adjustments not always visible in normal workflow
+
+**Ruby ClusterKit:**
+- Adjusts parameters when needed
+- Optional verbose mode shows adjustments
+- Explains why adjustments were made
+
+### Validation
+
+**Python UMAP:**
+- Validates parameters against mathematical constraints
+- Generic `ValueError` for invalid parameters
+
+**Ruby ClusterKit:**
+- Validates parameters with context-aware messages
+- `InvalidParameterError` with specific guidance
+- Suggests valid parameter ranges based on data
+
+## Error Messages
+
+### Python Style
+Focuses on technical accuracy:
+```
+ValueError: n_neighbors must be less than or equal to the number of samples
+```
+
+### Ruby Style
+Focuses on user guidance:
+```
+The n_neighbors parameter (15) is too large for your dataset size (10).
+
+UMAP needs n_neighbors to be less than the number of samples.
+Suggested value: 5
+
+Example: UMAP.new(n_neighbors: 5)
+```
+
+## Performance Characteristics
+
+### Python
+- Mature optimization over many years
+- Handles edge cases without hanging
+- Extensive numerical stability improvements
+
+### Ruby
+- Newer implementation via Rust bindings
+- Excellent performance on standard datasets
+- Some edge cases still being optimized
+
+## When Each Approach Shines
+
+### Python UMAP/HDBSCAN is ideal when:
+- Working with well-understood data pipelines
+- Requiring maximum compatibility
+- Integrating with existing Python ML workflows
+- Batch processing diverse datasets
+
+### Ruby ClusterKit is ideal when:
+- Learning dimensionality reduction techniques
+- Working with new or unfamiliar datasets
+- Needing clear feedback about data issues
+- Prioritizing interpretable results
+
+## Convergence Behavior
+
+### Python
+- Continues optimization even with poor convergence
+- Returns best result found within iteration limit
+- May produce suboptimal embeddings silently
+
+### Ruby
+- Raises `ConvergenceError` with explanation
+- Suggests parameter adjustments to improve convergence
+- Helps users understand why convergence failed
+
+## Summary
+
+Both implementations are valuable tools for dimensionality reduction and clustering. Python's approach offers battle-tested robustness and broad compatibility, making it excellent for production pipelines and experienced practitioners. ClusterKit's approach provides more guidance and education, making it particularly valuable for exploratory analysis and users who want to understand their results deeply.
+
+The choice between them often depends on your specific needs:
+- Choose Python when you need maximum compatibility and robustness
+- Choose ClusterKit when you value clear feedback and guided analysis
+
+Both approaches reflect thoughtful design decisions optimized for their respective user communities and use cases.