clusterkit 0.3.0-x86_64-linux

data/Cargo.toml

@@ -0,0 +1,8 @@
+[workspace]
+members = ["ext/clusterkit"]
+resolver = "2"
+
+[profile.release]
+opt-level = 3
+lto = true
+codegen-units = 1

data/Gemfile

@@ -0,0 +1,17 @@
+source "https://rubygems.org"
+
+# Specify your gem's dependencies in annembed-ruby.gemspec
+gemspec
+
+# Test-only dependencies
+group :test do
+  # Optional: For comparing with Python implementations
+  gem "pycall", "~> 1.4", require: false
+end
+
+# Development dependencies for generating test fixtures
+group :development do
+  # For generating real embeddings to use as test fixtures
+  # This avoids the hanging issues with random test data
+  gem "red-candle", "~> 1.0", require: false
+end

data/IMPLEMENTATION_NOTES.md

@@ -0,0 +1,143 @@
+# Implementation Notes for annembed-ruby
+
+## Architecture Overview
+
+The gem follows a three-layer architecture:
+
+1. **Ruby Layer** (`lib/annembed/`): User-facing API with Ruby idioms
+2. **Magnus Bridge** (`ext/annembed_ruby/src/`): Type conversions and bindings
+3. **annembed Core**: The underlying Rust embedding library
+
+## Key Implementation Challenges
+
+### 1. Data Type Conversions
+
+The main challenge is efficiently converting between Ruby and Rust types:
+
+```
+Ruby Array/Numo::NArray ↔ ndarray::Array2<f32> ↔ annembed matrices
+```
+
+Consider using views instead of copies where possible.
+
+### 2. Memory Management
+
+- Magnus handles Ruby GC integration
+- Need to be careful with large matrices
+- Consider streaming for datasets > available RAM
+
+### 3. Progress Reporting
+
+annembed operations can be long-running. Options:
+1. Polling from Ruby side (requires thread)
+2. Callback from Rust (needs GVL management)
+3. Async/await pattern (complex but clean)
+
+### 4. Error Handling
+
+Map annembed errors to Ruby exceptions:
+- `annembed::Error` → `Annembed::Error`
+- Panic → `RuntimeError` (avoid panics!)
+- Invalid input → `ArgumentError`
+
+## annembed API Mapping
+
+### Core Types
+- `EmbedderT` - Main embedding trait
+- `HnswParams` - HNSW graph parameters
+- `EmbedderParams` - Algorithm-specific params
+- `GraphProjection` - For graph-based methods
+
+### Key Functions
+```rust
+// Main embedding function
+pub fn get_embedder(
+    data: &Array2<f32>,
+    params: EmbedderParams,
+    hnsw_params: HnswParams
+) -> Result<Box<dyn EmbedderT>>
+
+// The embedder trait
+pub trait EmbedderT {
+    fn embed(&self) -> Result<Array2<f32>>;
+    fn get_hnsw(&self) -> &Hnsw<f32, DistL2>;
+}
+```
+
+## Performance Considerations
+
+1. **Parallelization**: annembed uses rayon, respect Ruby's thread settings
+2. **BLAS Backend**: Allow users to choose (OpenBLAS vs MKL)
+3. **Large Datasets**: Implement chunking for > 1M points
+4. **GPU Future**: Design API to allow GPU backend later
+
+## Testing Strategy
+
+### Unit Tests
+- Type conversions
+- Configuration parsing
+- Error handling
+
+### Integration Tests
+- Small datasets (Iris)
+- Medium datasets (MNIST subset)
+- Large datasets (if CI allows)
+
+### Benchmarks
+- vs Python UMAP
+- vs pure Ruby implementations
+- Memory usage profiling
+
+## Platform Support
+
+### Priorities
+1. Linux x86_64 (most servers)
+2. macOS arm64 (M1/M2 developers)
+3. macOS x86_64 (Intel Macs)
+4. Windows x86_64 (if feasible)
+
+### Build Considerations
+- Use rake-compiler-dock for cross-compilation
+- Static link BLAS when possible
+- Provide clear instructions for source builds
+
+## Future Enhancements
+
+1. **Incremental Learning**: Add new points to existing embedding
+2. **Custom Metrics**: Allow user-defined distance functions
+3. **Supervised Embedding**: Use labels to guide embedding
+4. **GPU Support**: If annembed adds it
+5. **Visualization**: Built-in plotting helpers?
+
+## Debugging Tips
+
+1. Use `RUST_BACKTRACE=1` for better errors
+2. Add logging with `env_logger` in Rust
+3. Use `rb_sys` debug mode for development
+4. Memory debugging with valgrind/ASAN
+
+## Code Organization
+
+Keep related functionality together:
+- `embedder.rs`: All embedding algorithms
+- `utils.rs`: Dimension estimation, hubness
+- `svd.rs`: Matrix decomposition
+- `conversions.rs`: Type conversion helpers
+
+## Release Checklist
+
+1. [ ] Update version in `version.rb`
+2. [ ] Update CHANGELOG.md
+3. [ ] Run full test suite on all platforms
+4. [ ] Build precompiled gems
+5. [ ] Test gem installation from .gem file
+6. [ ] Tag release in git
+7. [ ] Push to RubyGems.org
+8. [ ] Update documentation
+
+## References
+
+- annembed docs: https://docs.rs/annembed/
+- Magnus guide: https://github.com/matsadler/magnus
+- UMAP paper: https://arxiv.org/abs/1802.03426
+- t-SNE paper: https://lvdmaaten.github.io/tsne/

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2024 Your Name
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/PYTHON_COMPARISON.md

@@ -0,0 +1,183 @@
+# ClusterKit vs Python UMAP/HDBSCAN: A Comparison
+
+## Overview
+
+ClusterKit and Python's UMAP/HDBSCAN implementations share the same algorithmic foundations but take different approaches to error handling and data validation. This document outlines these differences to help users understand what to expect from each implementation.
+
+## Philosophical Approaches
+
+### Python UMAP/HDBSCAN
+Python's implementations prioritize **algorithmic robustness**, attempting to produce results even in edge cases. This approach:
+- Maximizes compatibility with diverse datasets
+- Minimizes interruptions to analysis workflows  
+- Trusts users to interpret results appropriately
+- Follows the scikit-learn pattern of "fit on anything"
+
+### Ruby ClusterKit
+ClusterKit prioritizes **guided analysis**, providing feedback when data characteristics may lead to suboptimal results. This approach:
+- Helps users understand their data's suitability for the algorithm
+- Provides actionable suggestions when issues arise
+- Encourages best practices in dimensionality reduction
+- Aims to prevent misinterpretation of results
+
+## Behavioral Differences
+
+### Small Datasets
+
+#### Scenario: 3 data points with n_neighbors=15
+
+**Python UMAP:**
+- Automatically adjusts n_neighbors silently
+- Returns a 2D embedding of the 3 points
+- The resulting triangle's shape depends on random initialization
+
+**Ruby ClusterKit:**
+- Explicitly adjusts n_neighbors with optional warning
+- Currently experiences performance issues on very small datasets (being addressed)
+- Provides context about why adjustment was needed
+
+### Data Quality Issues
+
+#### Scenario: Random data without inherent structure
+
+**Python UMAP:**
+- Processes the data without warnings
+- Returns an embedding that may show apparent "clusters"
+- These patterns are typically artifacts of the algorithm rather than real structure
+
+**Ruby ClusterKit:**
+- May raise `IsolatedPointError` with explanation
+- Suggests that the data lacks structure suitable for manifold learning
+- Recommends alternatives like PCA for unstructured data
+
+#### Scenario: Extreme outliers (points 1000x farther than main data)
+
+**Python UMAP:**
+- Embeds outliers far from main cluster
+- Main cluster may be compressed to accommodate outlier scale
+- Visualization scale dominated by outliers
+
+**Ruby ClusterKit:**
+- May raise `IsolatedPointError` 
+- Explains impact of outliers on manifold learning
+- Suggests preprocessing steps like outlier removal or normalization
+
+### Invalid Data
+
+#### Scenario: NaN or Infinite values
+
+**Both implementations** reject invalid numerical data, but with different messaging:
+
+**Python:** `"Input contains NaN"` or `"Input contains infinity"`
+
+**Ruby:** `"Element at position [5, 2] is NaN or Infinite"`
+
+The Ruby version provides specific location information to aid debugging.
+
+### Edge Cases
+
+#### Scenario: Single data point
+
+**Python UMAP:**
+- Returns `[[0, 0]]` or similar default position
+- No error or warning about meaningless result
+
+**Ruby ClusterKit:**
+- Raises error explaining that manifold learning requires multiple points
+- Suggests minimum data requirements
+
+#### Scenario: Empty dataset
+
+**Both implementations** appropriately reject empty input with clear error messages.
+
+## Parameter Handling
+
+### Auto-adjustment
+
+**Python UMAP:**
+- Silently adjusts parameters when necessary
+- May issue warnings through Python's warning system
+- Adjustments not always visible in normal workflow
+
+**Ruby ClusterKit:**
+- Adjusts parameters when needed
+- Optional verbose mode shows adjustments
+- Explains why adjustments were made
+
+### Validation
+
+**Python UMAP:**
+- Validates parameters against mathematical constraints
+- Generic `ValueError` for invalid parameters
+
+**Ruby ClusterKit:**
+- Validates parameters with context-aware messages
+- `InvalidParameterError` with specific guidance
+- Suggests valid parameter ranges based on data
+
+## Error Messages
+
+### Python Style
+Focuses on technical accuracy:
+```
+ValueError: n_neighbors must be less than or equal to the number of samples
+```
+
+### Ruby Style
+Focuses on user guidance:
+```
+The n_neighbors parameter (15) is too large for your dataset size (10).
+
+UMAP needs n_neighbors to be less than the number of samples.
+Suggested value: 5
+
+Example: UMAP.new(n_neighbors: 5)
+```
+
+## Performance Characteristics
+
+### Python
+- Mature optimization over many years
+- Handles edge cases without hanging
+- Extensive numerical stability improvements
+
+### Ruby
+- Newer implementation via Rust bindings
+- Excellent performance on standard datasets
+- Some edge cases still being optimized
+
+## When Each Approach Shines
+
+### Python UMAP/HDBSCAN is ideal when:
+- Working with well-understood data pipelines
+- Requiring maximum compatibility
+- Integrating with existing Python ML workflows
+- Batch processing diverse datasets
+
+### Ruby ClusterKit is ideal when:
+- Learning dimensionality reduction techniques
+- Working with new or unfamiliar datasets
+- Needing clear feedback about data issues
+- Prioritizing interpretable results
+
+## Convergence Behavior
+
+### Python
+- Continues optimization even with poor convergence
+- Returns best result found within iteration limit
+- May produce suboptimal embeddings silently
+
+### Ruby
+- Raises `ConvergenceError` with explanation
+- Suggests parameter adjustments to improve convergence
+- Helps users understand why convergence failed
+
+## Summary
+
+Both implementations are valuable tools for dimensionality reduction and clustering. Python's approach offers battle-tested robustness and broad compatibility, making it excellent for production pipelines and experienced practitioners. ClusterKit's approach provides more guidance and education, making it particularly valuable for exploratory analysis and users who want to understand their results deeply.
+
+The choice between them often depends on your specific needs:
+- Choose Python when you need maximum compatibility and robustness
+- Choose ClusterKit when you value clear feedback and guided analysis
+
+Both approaches reflect thoughtful design decisions optimized for their respective user communities and use cases.