classifier 2.1.0 → 2.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e82726ac5c6e619e701be4591c8ed29e8b9f9a88236ad9d8d602fe3f748dcf43
-  data.tar.gz: 3771f2d2fce4992ed0cd5b1ad3849cc3e1c4fa310a144c1056e5f9d0c49579ef
+  metadata.gz: 2ce325479bd32938cd7a7c694152983579266667e5def8d36725c38594eb822b
+  data.tar.gz: a35439c4ddfba77091297fee46930c279c54bfeb314b8b87fba8dd0ccbe6b292
 SHA512:
-  metadata.gz: 913111c43ffd83a1a461023c6d1f4fef0f1efb7f4e591089db7dbf3d37a1cc98cb3b5de16888019ab1caf8db83ad828dbaf82d2cab237b991216bdb27f529d15
-  data.tar.gz: b7142000fa687a58481051921663665edad1ac3b183947e79cf63bb15a79a0ca7bc66cd76104fccea713162cf05e876bd609d5f4866a13ea2c184f8caa089547
+  metadata.gz: e82b06f6449d3c5b089f840fe794b0e7f59bc1a00a379d4739010ad872ce7a90a4cbc1eee930b822b2761add461f2378dde8e015115c45b59c67e8aee0ad01b5
+  data.tar.gz: fa9271f248da8de9012ef923911b5f3d18cb9ced08ff9f85372e13622b8ab43a3f75365d21e9de3b3fa2849f3a54c3261da67c00c38a263b3ceee7ac0afd2598

data/README.md

@@ -4,271 +4,138 @@
 [![CI](https://github.com/cardmagic/classifier/actions/workflows/ruby.yml/badge.svg)](https://github.com/cardmagic/classifier/actions/workflows/ruby.yml)
 [![License: LGPL](https://img.shields.io/badge/License-LGPL_2.1-blue.svg)](https://opensource.org/licenses/LGPL-2.1)
 
-A Ruby library for text classification using Bayesian and Latent Semantic Indexing (LSI) algorithms.
+Text classification in Ruby. Five algorithms, native performance, streaming support.
 
-**[Documentation](https://rubyclassifier.com/docs)** · **[Tutorials](https://rubyclassifier.com/docs/tutorials)** · **[Guides](https://rubyclassifier.com/docs/guides)**
+**[Documentation](https://rubyclassifier.com/docs)** · **[Tutorials](https://rubyclassifier.com/docs/tutorials)** · **[API Reference](https://rubyclassifier.com/docs/api)**
 
-## Table of Contents
+## Why This Library?
 
-- [Installation](#installation)
-- [Bayesian Classifier](#bayesian-classifier)
-- [LSI (Latent Semantic Indexing)](#lsi-latent-semantic-indexing)
-- [Persistence](#persistence)
-- [Performance](#performance)
-- [Development](#development)
-- [Contributing](#contributing)
-- [License](#license)
+| | This Gem | Other Forks |
+|:--|:--|:--|
+| **Algorithms** | ✅ 5 classifiers | ❌ 2 only |
+| **Incremental LSI** | ✅ Brand's algorithm (no rebuild) | ❌ Full SVD rebuild on every add |
+| **LSI Performance** | ✅ Native C extension (5-50x faster) | ❌ Pure Ruby or requires GSL |
+| **Streaming** | ✅ Train on multi-GB datasets | ❌ Must load all data in memory |
+| **Persistence** | ✅ Pluggable (file, Redis, S3) | ❌ Marshal only |
 
 ## Installation
 
-Add to your Gemfile:
-
 ```ruby
 gem 'classifier'
 ```
 
-Then run:
-
-```bash
-bundle install
-```
-
-Or install directly:
+## Quick Start
 
-```bash
-gem install classifier
-```
-
-### Native C Extension
-
-The gem includes a native C extension for fast LSI operations. It compiles automatically during gem installation. No external dependencies are required.
-
-To verify the native extension is active:
+### Bayesian
 
 ```ruby
-require 'classifier'
-puts Classifier::LSI.backend  # => :native
+classifier = Classifier::Bayes.new(:spam, :ham)
+classifier.train(spam: "Buy cheap viagra now!", ham: "Meeting at 3pm tomorrow")
+classifier.classify "You've won a prize!"  # => "Spam"
 ```
+[Bayesian Guide →](https://rubyclassifier.com/docs/guides/bayes/basics)
 
-To force pure Ruby mode (for debugging):
+### Logistic Regression
 
-```bash
-NATIVE_VECTOR=true ruby your_script.rb
+```ruby
+classifier = Classifier::LogisticRegression.new(:positive, :negative)
+classifier.train(positive: "Great product!", negative: "Terrible experience")
+classifier.classify "Loved it!"  # => "Positive"
 ```
+[Logistic Regression Guide →](https://rubyclassifier.com/docs/guides/logisticregression/basics)
 
-To suppress the warning when native extension isn't available:
+### LSI (Latent Semantic Indexing)
 
-```bash
-SUPPRESS_LSI_WARNING=true ruby your_script.rb
+```ruby
+lsi = Classifier::LSI.new
+lsi.add(pets: "Dogs are loyal", tech: "Ruby is elegant")
+lsi.classify "My puppy is playful"  # => "pets"
 ```
+[LSI Guide →](https://rubyclassifier.com/docs/guides/lsi/basics)
 
-### Compatibility
-
-| Ruby Version | Status |
-|--------------|--------|
-| 4.0          | Supported |
-| 3.4          | Supported |
-| 3.3          | Supported |
-| 3.2          | Supported |
-| 3.1          | EOL (unsupported) |
-
-## Bayesian Classifier
-
-Fast, accurate classification with modest memory requirements. Ideal for spam filtering, sentiment analysis, and content categorization.
-
-### Quick Start
+### k-Nearest Neighbors
 
 ```ruby
-require 'classifier'
-
-classifier = Classifier::Bayes.new('Spam', 'Ham')
-
-# Train the classifier
-classifier.train_spam "Buy cheap viagra now! Limited offer!"
-classifier.train_spam "You've won a million dollars! Claim now!"
-classifier.train_ham "Meeting scheduled for tomorrow at 10am"
-classifier.train_ham "Please review the attached document"
-
-# Classify new text
-classifier.classify "Congratulations! You've won a prize!"
-# => "Spam"
+knn = Classifier::KNN.new(k: 3)
+knn.train(spam: "Free money!", ham: "Quarterly report attached")  # or knn.add()
+knn.classify "Claim your prize"  # => "spam"
 ```
+[k-Nearest Neighbors Guide →](https://rubyclassifier.com/docs/guides/knn/basics)
 
-### Learn More
-
-- [Bayes Basics Guide](https://rubyclassifier.com/docs/guides/bayes/basics) - In-depth documentation
-- [Build a Spam Filter Tutorial](https://rubyclassifier.com/docs/tutorials/spam-filter) - Step-by-step guide
-- [Paul Graham: A Plan for Spam](http://www.paulgraham.com/spam.html)
-
-## LSI (Latent Semantic Indexing)
-
-Semantic analysis using Singular Value Decomposition (SVD). More flexible than Bayesian classifiers, providing search, clustering, and classification based on meaning rather than just keywords.
-
-### Quick Start
+### TF-IDF
 
 ```ruby
-require 'classifier'
-
-lsi = Classifier::LSI.new
-
-# Add documents with categories
-lsi.add_item "Dogs are loyal pets that love to play fetch", :pets
-lsi.add_item "Cats are independent and love to nap", :pets
-lsi.add_item "Ruby is a dynamic programming language", :programming
-lsi.add_item "Python is great for data science", :programming
+tfidf = Classifier::TFIDF.new
+tfidf.fit(["Dogs are pets", "Cats are independent"])
+tfidf.transform("Dogs are loyal")  # => {:dog => 0.707, :loyal => 0.707}
+```
+[TF-IDF Guide →](https://rubyclassifier.com/docs/guides/tfidf/basics)
 
-# Classify new text
-lsi.classify "My puppy loves to run around"
-# => :pets
+## Key Features
 
-# Get classification with confidence score
-lsi.classify_with_confidence "Learning to code in Ruby"
-# => [:programming, 0.89]
-```
+### Incremental LSI
 
-### Search and Discovery
+Add documents without rebuilding the entire index—400x faster for streaming data:
 
 ```ruby
-# Find similar documents
-lsi.find_related "Dogs are great companions", 2
-# => ["Dogs are loyal pets that love to play fetch", "Cats are independent..."]
+lsi = Classifier::LSI.new(incremental: true)
+lsi.add(tech: ["Ruby is elegant", "Python is popular"])
+lsi.build_index
 
-# Search by keyword
-lsi.search "programming", 3
-# => ["Ruby is a dynamic programming language", "Python is great for..."]
+# These use Brand's algorithm—no full rebuild
+lsi.add(tech: "Go is fast")
+lsi.add(tech: "Rust is safe")
 ```
 
-### Learn More
-
-- [LSI Basics Guide](https://rubyclassifier.com/docs/guides/lsi/basics) - In-depth documentation
-- [Wikipedia: Latent Semantic Analysis](http://en.wikipedia.org/wiki/Latent_semantic_analysis)
+[Learn more →](https://rubyclassifier.com/docs/guides/lsi/incremental)
 
-## Persistence
-
-Save and load trained classifiers with pluggable storage backends. Works with both Bayes and LSI classifiers.
-
-### File Storage
+### Persistence
 
 ```ruby
-require 'classifier'
-
-classifier = Classifier::Bayes.new('Spam', 'Ham')
-classifier.train_spam "Buy now! Limited offer!"
-classifier.train_ham "Meeting tomorrow at 3pm"
-
-# Configure storage and save
-classifier.storage = Classifier::Storage::File.new(path: "spam_filter.json")
+classifier.storage = Classifier::Storage::File.new(path: "model.json")
 classifier.save
 
-# Load later
 loaded = Classifier::Bayes.load(storage: classifier.storage)
-loaded.classify "Claim your prize now!"
-# => "Spam"
 ```
 
-### Custom Storage Backends
+[Learn more →](https://rubyclassifier.com/docs/guides/persistence)
 
-Create backends for Redis, PostgreSQL, S3, or any storage system:
+### Streaming Training
 
 ```ruby
-class RedisStorage < Classifier::Storage::Base
-  def initialize(redis:, key:)
-    super()
-    @redis, @key = redis, key
-  end
-
-  def write(data) = @redis.set(@key, data)
-  def read = @redis.get(@key)
-  def delete = @redis.del(@key)
-  def exists? = @redis.exists?(@key)
-end
-
-# Use it
-classifier.storage = RedisStorage.new(redis: Redis.new, key: "classifier:spam")
-classifier.save
+classifier.train_from_stream(:spam, File.open("spam_corpus.txt"))
 ```
 
-### Learn More
-
-- [Persistence Guide](https://rubyclassifier.com/docs/guides/persistence/basics) - Full documentation with examples
+[Learn more →](https://rubyclassifier.com/docs/guides/streaming)
 
 ## Performance
 
-### Native C Extension vs Pure Ruby
-
-The native C extension provides dramatic speedups for LSI operations, especially `build_index` (SVD computation):
-
-| Documents | build_index | Overall |
-|-----------|-------------|---------|
-| 5         | 7x faster   | 2.6x    |
-| 10        | 25x faster  | 4.6x    |
-| 15        | 112x faster | 14.5x   |
-| 20        | 385x faster | 48.7x   |
-
-<details>
-<summary>Detailed benchmark (20 documents)</summary>
-
-```
-Operation            Pure Ruby     Native C      Speedup
-----------------------------------------------------------
-build_index            0.5540       0.0014       384.5x
-classify               0.0190       0.0060         3.2x
-search                 0.0145       0.0037         3.9x
-find_related           0.0098       0.0011         8.6x
-----------------------------------------------------------
-TOTAL                  0.5973       0.0123        48.7x
-```
-</details>
+Native C extension provides 5-50x speedup for LSI operations:
 
-### Running Benchmarks
+| Documents | Speedup |
+|-----------|---------|
+| 10 | 25x |
+| 20 | 50x |
 
 ```bash
-rake benchmark              # Run with current configuration
-rake benchmark:compare      # Compare native C vs pure Ruby
+rake benchmark:compare  # Run your own comparison
 ```
 
 ## Development
 
-### Setup
-
 ```bash
-git clone https://github.com/cardmagic/classifier.git
-cd classifier
 bundle install
-rake compile  # Compile native C extension
-```
-
-### Running Tests
-
-```bash
-rake test                        # Run all tests (compiles first)
-ruby -Ilib test/bayes/bayesian_test.rb  # Run specific test file
-
-# Test with pure Ruby (no native extension)
-NATIVE_VECTOR=true rake test
+rake compile  # Build native extension
+rake test     # Run tests
 ```
 
-### Console
-
-```bash
-rake console
-```
-
-## Contributing
-
-1. Fork the repository
-2. Create your feature branch (`git checkout -b feature/amazing-feature`)
-3. Commit your changes (`git commit -am 'Add amazing feature'`)
-4. Push to the branch (`git push origin feature/amazing-feature`)
-5. Open a Pull Request
-
 ## Authors
 
-- **Lucas Carlson** - *Original author* - lucas@rufy.com
-- **David Fayram II** - *LSI implementation* - dfayram@gmail.com
+- **Lucas Carlson** - lucas@rufy.com
+- **David Fayram II** - dfayram@gmail.com
 - **Cameron McBride** - cameron.mcbride@gmail.com
 - **Ivan Acosta-Rubio** - ivan@softwarecriollo.com
 
 ## License
 
-This library is released under the [GNU Lesser General Public License (LGPL) 2.1](LICENSE).
+[LGPL 2.1](LICENSE)

data/ext/classifier/classifier_ext.c

@@ -22,4 +22,5 @@ void Init_classifier_ext(void)
     Init_vector();
     Init_matrix();
     Init_svd();
+    Init_incremental_svd();
 }