classifier 2.0.0 → 2.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fea14969bc8a61283823b0b0f5bae013af968caf4676c383155e3b8682b948de
-  data.tar.gz: 4d626c85d084ff75eba2ff305673734a6f25b668e773b1b5a3a0630a6b68df96
+  metadata.gz: 2ce325479bd32938cd7a7c694152983579266667e5def8d36725c38594eb822b
+  data.tar.gz: a35439c4ddfba77091297fee46930c279c54bfeb314b8b87fba8dd0ccbe6b292
 SHA512:
-  metadata.gz: ef53c06db3326b1b6ebc14255b4ba198286c06e291cba3afc67bba360ca766a173f89269405d216751806ca72f885a87ac80ec24a031053f8e6f2987e8e2267e
-  data.tar.gz: 8f120a9b78e802e6fd3e7172fd311b476745e27d5b3d301dc8d140296a451875e5aa33a901514bfdd1bc96c656ad1a43cbb3935a05223cd38548a71ba6a3a1c1
+  metadata.gz: e82b06f6449d3c5b089f840fe794b0e7f59bc1a00a379d4739010ad872ce7a90a4cbc1eee930b822b2761add461f2378dde8e015115c45b59c67e8aee0ad01b5
+  data.tar.gz: fa9271f248da8de9012ef923911b5f3d18cb9ced08ff9f85372e13622b8ab43a3f75365d21e9de3b3fa2849f3a54c3261da67c00c38a263b3ceee7ac0afd2598

data/CLAUDE.md

@@ -11,21 +11,28 @@ Ruby gem providing text classification via two algorithms:
 ## Common Commands
 
 ```bash
-# Run all tests
-rake test
+# Compile native C extension
+bundle exec rake compile
+
+# Run all tests (compiles first)
+bundle exec rake test
 
 # Run a single test file
 ruby -Ilib test/bayes/bayesian_test.rb
 ruby -Ilib test/lsi/lsi_test.rb
 
-# Run tests with native Ruby vector (without GSL)
-NATIVE_VECTOR=true rake test
+# Run tests with pure Ruby (no native extension)
+NATIVE_VECTOR=true bundle exec rake test
+
+# Run benchmarks
+bundle exec rake benchmark
+bundle exec rake benchmark:compare
 
 # Interactive console
-rake console
+bundle exec rake console
 
 # Generate documentation
-rake doc
+bundle exec rake doc
 ```
 
 ## Architecture
@@ -39,7 +46,7 @@ rake doc
 
 **LSI Classifier** (`lib/classifier/lsi.rb`)
 - Uses Singular Value Decomposition (SVD) for semantic analysis
-- Optional GSL gem for 10x faster matrix operations; falls back to pure Ruby SVD
+- Native C extension for 5-50x faster matrix operations; falls back to pure Ruby
 - Key operations: `add_item`, `classify`, `find_related`, `search`
 - `auto_rebuild` option controls automatic index rebuilding after changes
 
@@ -49,15 +56,18 @@ rake doc
 - Uses `fast-stemmer` gem for Porter stemming
 
 **Vector Extensions** (`lib/classifier/extensions/vector.rb`)
-- Pure Ruby SVD implementation (`Matrix#SV_decomp`)
+- Pure Ruby SVD implementation (`Matrix#SV_decomp`) - used as fallback
 - Vector normalization and magnitude calculations
 
-### GSL Integration
+### Native C Extension (`ext/classifier/`)
+
+LSI uses a native C extension for fast linear algebra operations:
+- `Classifier::Linalg::Vector` - Vector operations (alloc, normalize, dot product)
+- `Classifier::Linalg::Matrix` - Matrix operations (alloc, transpose, multiply)
+- Jacobi SVD implementation for singular value decomposition
 
-LSI checks for the `gsl` gem at load time. When available:
-- Uses `GSL::Matrix` and `GSL::Vector` for faster operations
-- Serialization handled via `vector_serialize.rb`
-- Test without GSL: `NATIVE_VECTOR=true rake test`
+Check current backend: `Classifier::LSI.backend` returns `:native` or `:ruby`
+Force pure Ruby: `NATIVE_VECTOR=true bundle exec rake test`
 
 ### Content Nodes (`lib/classifier/lsi/content_node.rb`)
 

data/README.md

@@ -4,256 +4,138 @@
 [![CI](https://github.com/cardmagic/classifier/actions/workflows/ruby.yml/badge.svg)](https://github.com/cardmagic/classifier/actions/workflows/ruby.yml)
 [![License: LGPL](https://img.shields.io/badge/License-LGPL_2.1-blue.svg)](https://opensource.org/licenses/LGPL-2.1)
 
-A Ruby library for text classification using Bayesian and Latent Semantic Indexing (LSI) algorithms.
+Text classification in Ruby. Five algorithms, native performance, streaming support.
 
-## Table of Contents
+**[Documentation](https://rubyclassifier.com/docs)** · **[Tutorials](https://rubyclassifier.com/docs/tutorials)** · **[API Reference](https://rubyclassifier.com/docs/api)**
 
-- [Installation](#installation)
-- [Bayesian Classifier](#bayesian-classifier)
-- [LSI (Latent Semantic Indexing)](#lsi-latent-semantic-indexing)
-- [Performance](#performance)
-- [Development](#development)
-- [Contributing](#contributing)
-- [License](#license)
+## Why This Library?
 
-## Installation
+| | This Gem | Other Forks |
+|:--|:--|:--|
+| **Algorithms** | ✅ 5 classifiers | ❌ 2 only |
+| **Incremental LSI** | ✅ Brand's algorithm (no rebuild) | ❌ Full SVD rebuild on every add |
+| **LSI Performance** | ✅ Native C extension (5-50x faster) | ❌ Pure Ruby or requires GSL |
+| **Streaming** | ✅ Train on multi-GB datasets | ❌ Must load all data in memory |
+| **Persistence** | ✅ Pluggable (file, Redis, S3) | ❌ Marshal only |
 
-Add to your Gemfile:
+## Installation
 
 ```ruby
 gem 'classifier'
 ```
 
-Then run:
+## Quick Start
 
-```bash
-bundle install
-```
-
-Or install directly:
+### Bayesian
 
-```bash
-gem install classifier
+```ruby
+classifier = Classifier::Bayes.new(:spam, :ham)
+classifier.train(spam: "Buy cheap viagra now!", ham: "Meeting at 3pm tomorrow")
+classifier.classify "You've won a prize!"  # => "Spam"
 ```
+[Bayesian Guide →](https://rubyclassifier.com/docs/guides/bayes/basics)
 
-### Optional: GSL for Faster LSI
-
-For significantly faster LSI operations, install the [GNU Scientific Library](https://www.gnu.org/software/gsl/).
-
-<details>
-<summary><strong>Ruby 3+</strong></summary>
-
-The released `gsl` gem doesn't support Ruby 3+. Install from source:
+### Logistic Regression
 
-```bash
-# Install GSL library
-brew install gsl        # macOS
-apt-get install libgsl-dev  # Ubuntu/Debian
-
-# Build and install the gem
-git clone https://github.com/cardmagic/rb-gsl.git
-cd rb-gsl
-git checkout fix/ruby-3.4-compatibility
-gem build gsl.gemspec
-gem install gsl-*.gem
+```ruby
+classifier = Classifier::LogisticRegression.new(:positive, :negative)
+classifier.train(positive: "Great product!", negative: "Terrible experience")
+classifier.classify "Loved it!"  # => "Positive"
 ```
-</details>
-
-<details>
-<summary><strong>Ruby 2.x</strong></summary>
+[Logistic Regression Guide →](https://rubyclassifier.com/docs/guides/logisticregression/basics)
 
-```bash
-# macOS
-brew install gsl
-gem install gsl
+### LSI (Latent Semantic Indexing)
 
-# Ubuntu/Debian
-apt-get install libgsl-dev
-gem install gsl
+```ruby
+lsi = Classifier::LSI.new
+lsi.add(pets: "Dogs are loyal", tech: "Ruby is elegant")
+lsi.classify "My puppy is playful"  # => "pets"
 ```
-</details>
+[LSI Guide →](https://rubyclassifier.com/docs/guides/lsi/basics)
 
-When GSL is installed, Classifier automatically uses it. To suppress the GSL notice:
+### k-Nearest Neighbors
 
-```bash
-SUPPRESS_GSL_WARNING=true ruby your_script.rb
+```ruby
+knn = Classifier::KNN.new(k: 3)
+knn.train(spam: "Free money!", ham: "Quarterly report attached")  # or knn.add()
+knn.classify "Claim your prize"  # => "spam"
 ```
+[k-Nearest Neighbors Guide →](https://rubyclassifier.com/docs/guides/knn/basics)
 
-### Compatibility
-
-| Ruby Version | Status |
-|--------------|--------|
-| 4.0          | Supported |
-| 3.4          | Supported |
-| 3.3          | Supported |
-| 3.2          | Supported |
-| 3.1          | EOL (unsupported) |
-
-## Bayesian Classifier
-
-Fast, accurate classification with modest memory requirements. Ideal for spam filtering, sentiment analysis, and content categorization.
-
-### Quick Start
+### TF-IDF
 
 ```ruby
-require 'classifier'
-
-classifier = Classifier::Bayes.new('Spam', 'Ham')
-
-# Train the classifier
-classifier.train_spam "Buy cheap viagra now! Limited offer!"
-classifier.train_spam "You've won a million dollars! Claim now!"
-classifier.train_ham "Meeting scheduled for tomorrow at 10am"
-classifier.train_ham "Please review the attached document"
-
-# Classify new text
-classifier.classify "Congratulations! You've won a prize!"
-# => "Spam"
+tfidf = Classifier::TFIDF.new
+tfidf.fit(["Dogs are pets", "Cats are independent"])
+tfidf.transform("Dogs are loyal")  # => {:dog => 0.707, :loyal => 0.707}
 ```
+[TF-IDF Guide →](https://rubyclassifier.com/docs/guides/tfidf/basics)
 
-### Persistence with Madeleine
+## Key Features
 
-```ruby
-require 'classifier'
-require 'madeleine'
+### Incremental LSI
 
-m = SnapshotMadeleine.new("classifier_data") {
-  Classifier::Bayes.new('Interesting', 'Uninteresting')
-}
+Add documents without rebuilding the entire index—400x faster for streaming data:
 
-m.system.train_interesting "fascinating article about science"
-m.system.train_uninteresting "boring repetitive content"
-m.take_snapshot
+```ruby
+lsi = Classifier::LSI.new(incremental: true)
+lsi.add(tech: ["Ruby is elegant", "Python is popular"])
+lsi.build_index
 
-# Later, restore and use:
-m.system.classify "new scientific discovery"
-# => "Interesting"
+# These use Brand's algorithm—no full rebuild
+lsi.add(tech: "Go is fast")
+lsi.add(tech: "Rust is safe")
 ```
 
-### Learn More
+[Learn more →](https://rubyclassifier.com/docs/guides/lsi/incremental)
 
-- [Bayesian Filtering Explained](http://www.process.com/precisemail/bayesian_filtering.htm)
-- [Wikipedia: Bayesian Filtering](http://en.wikipedia.org/wiki/Bayesian_filtering)
-- [Paul Graham: A Plan for Spam](http://www.paulgraham.com/spam.html)
-
-## LSI (Latent Semantic Indexing)
-
-Semantic analysis using Singular Value Decomposition (SVD). More flexible than Bayesian classifiers, providing search, clustering, and classification based on meaning rather than just keywords.
-
-### Quick Start
+### Persistence
 
 ```ruby
-require 'classifier'
-
-lsi = Classifier::LSI.new
-
-# Add documents with categories
-lsi.add_item "Dogs are loyal pets that love to play fetch", :pets
-lsi.add_item "Cats are independent and love to nap", :pets
-lsi.add_item "Ruby is a dynamic programming language", :programming
-lsi.add_item "Python is great for data science", :programming
-
-# Classify new text
-lsi.classify "My puppy loves to run around"
-# => :pets
+classifier.storage = Classifier::Storage::File.new(path: "model.json")
+classifier.save
 
-# Get classification with confidence score
-lsi.classify_with_confidence "Learning to code in Ruby"
-# => [:programming, 0.89]
+loaded = Classifier::Bayes.load(storage: classifier.storage)
 ```
 
-### Search and Discovery
+[Learn more →](https://rubyclassifier.com/docs/guides/persistence)
 
-```ruby
-# Find similar documents
-lsi.find_related "Dogs are great companions", 2
-# => ["Dogs are loyal pets that love to play fetch", "Cats are independent..."]
+### Streaming Training
 
-# Search by keyword
-lsi.search "programming", 3
-# => ["Ruby is a dynamic programming language", "Python is great for..."]
+```ruby
+classifier.train_from_stream(:spam, File.open("spam_corpus.txt"))
 ```
 
-### Learn More
-
-- [Wikipedia: Latent Semantic Analysis](http://en.wikipedia.org/wiki/Latent_semantic_analysis)
-- [C2 Wiki: Latent Semantic Indexing](http://www.c2.com/cgi/wiki?LatentSemanticIndexing)
+[Learn more →](https://rubyclassifier.com/docs/guides/streaming)
 
 ## Performance
 
-### GSL vs Native Ruby
-
-GSL provides dramatic speedups for LSI operations, especially `build_index` (SVD computation):
-
-| Documents | build_index | Overall |
-|-----------|-------------|---------|
-| 5         | 4x faster   | 2.5x    |
-| 10        | 24x faster  | 5.5x    |
-| 15        | 116x faster | 17x     |
-
-<details>
-<summary>Detailed benchmark (15 documents)</summary>
-
-```
-Operation              Native          GSL      Speedup
-----------------------------------------------------------
-build_index            0.1412       0.0012       116.2x
-classify               0.0142       0.0049         2.9x
-search                 0.0102       0.0026         3.9x
-find_related           0.0069       0.0016         4.2x
-----------------------------------------------------------
-TOTAL                  0.1725       0.0104        16.6x
-```
-</details>
+Native C extension provides 5-50x speedup for LSI operations:
 
-### Running Benchmarks
+| Documents | Speedup |
+|-----------|---------|
+| 10 | 25x |
+| 20 | 50x |
 
 ```bash
-rake benchmark              # Run with current configuration
-rake benchmark:compare      # Compare GSL vs native Ruby
+rake benchmark:compare  # Run your own comparison
 ```
 
 ## Development
 
-### Setup
-
 ```bash
-git clone https://github.com/cardmagic/classifier.git
-cd classifier
 bundle install
+rake compile  # Build native extension
+rake test     # Run tests
 ```
 
-### Running Tests
-
-```bash
-rake test                        # Run all tests
-ruby -Ilib test/bayes/bayesian_test.rb  # Run specific test file
-
-# Test without GSL (pure Ruby)
-NATIVE_VECTOR=true rake test
-```
-
-### Console
-
-```bash
-rake console
-```
-
-## Contributing
-
-1. Fork the repository
-2. Create your feature branch (`git checkout -b feature/amazing-feature`)
-3. Commit your changes (`git commit -am 'Add amazing feature'`)
-4. Push to the branch (`git push origin feature/amazing-feature`)
-5. Open a Pull Request
-
 ## Authors
 
-- **Lucas Carlson** - *Original author* - lucas@rufy.com
-- **David Fayram II** - *LSI implementation* - dfayram@gmail.com
+- **Lucas Carlson** - lucas@rufy.com
+- **David Fayram II** - dfayram@gmail.com
 - **Cameron McBride** - cameron.mcbride@gmail.com
 - **Ivan Acosta-Rubio** - ivan@softwarecriollo.com
 
 ## License
 
-This library is released under the [GNU Lesser General Public License (LGPL) 2.1](LICENSE).
+[LGPL 2.1](LICENSE)

data/ext/classifier/classifier_ext.c

@@ -0,0 +1,26 @@
+/*
+ * classifier_ext.c
+ * Main entry point for the Classifier native linear algebra extension
+ *
+ * This extension provides zero-dependency Vector, Matrix, and SVD
+ * implementations for the Classifier gem's LSI functionality.
+ */
+
+#include "linalg.h"
+
+VALUE mClassifierLinalg;
+VALUE cClassifierVector;
+VALUE cClassifierMatrix;
+
+void Init_classifier_ext(void)
+{
+    /* Define Classifier::Linalg module */
+    VALUE mClassifier = rb_define_module("Classifier");
+    mClassifierLinalg = rb_define_module_under(mClassifier, "Linalg");
+
+    /* Initialize Vector and Matrix classes */
+    Init_vector();
+    Init_matrix();
+    Init_svd();
+    Init_incremental_svd();
+}

data/ext/classifier/extconf.rb

@@ -0,0 +1,15 @@
+require 'mkmf'
+
+# rubocop:disable Style/GlobalVars
+if ENV['COVERAGE']
+  # Coverage flags: disable optimization for accurate line coverage
+  $CFLAGS << ' -O0 -g --coverage -Wall'
+  $LDFLAGS << ' --coverage'
+else
+  # Optimization flags for performance
+  $CFLAGS << ' -O3 -ffast-math -Wall'
+end
+# rubocop:enable Style/GlobalVars
+
+# Create the Makefile
+create_makefile('classifier/classifier_ext')