classifier 2.6.0 → 2.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 527f52e23121cbda5ce138a74c200f473813ea306aa04f50bd74f2d1230ff45b
-  data.tar.gz: ca52b0412149e3e6bf3d885d12ef5485ab1a70d6dcc4cb659121733bfcced2da
+  metadata.gz: 28ad3396d95e315e11525b87393aaf3d081cf537f63dee0ba89fdd1a8acab4ee
+  data.tar.gz: 03b0b12d04d9ed05878a1b5a659383b72eb4eba12b6eaecf6b02b0356ee524b9
 SHA512:
-  metadata.gz: 9ddcb1c5c85229beef3b8b048f65fae551ae97ac032e51f07937470070f48cb2d56adca0bd30df1f981a93e44004e572a258c9ecf7dbfed26f8005ca52247714
-  data.tar.gz: 79ffe1a234a988c5d026740405c350f2222a15eaf6a562e4b1da39cf6bdf1aaea43dbfe85bfc55db9830684a10eea3c9f4df0eae4921374fe2d6ac0d4c7b7606
+  metadata.gz: 3675894eaf015c2a2b1f49fcc62d4ad55d487955f5978628d48563f90d5513b20f0ed3e2d7f6b2fc9498c99971c71482c67a1342b3eb39f027e20ca0f2fa6ee5
+  data.tar.gz: ce38803fe759fce8f02013a9d5a6bc613af95ccf4c132ff8c203f961dffabcc0b1eca7db642fad4e3e488fdac5093dd62c8fdafde1831f2d9aa97d15f07fcb62

data/CHANGELOG.md

@@ -0,0 +1,165 @@
+# Changelog
+
+## 2.7.0 - 2026-08-15
+
+- Add a `keywords` executable for TF-IDF keyword extraction. `keywords fit`
+  builds a vocabulary from files or standard input, `keywords extract` reads a
+  file, `keywords info` prints model statistics, and a bare text argument
+  transforms that text. The command prints `term:score` pairs and maps stems
+  back to whole words, so `keywords "Ruby is elegant"` prints
+  `elegant:0.61 ruby:0.61`. Options set the model path, the top-N terms, quiet
+  mode, `--min-df`, `--max-df`, and `--ngram`. Usage errors exit 2 and other
+  errors exit 1. The gem now installs two executables, `classifier` and
+  `keywords`.
+- Treat each line as a separate document during `keywords fit`. The document
+  count controls the inverse document frequency, so a file with many lines
+  contributes many documents.
+- Add `Classifier::Streaming::MultiIO`. It reads several IO objects or file
+  paths as one sequential stream. It opens and closes each path one at a time,
+  so a corpus larger than the file descriptor limit still fits.
+- Add `String#stem_to_word_hash`. It maps each stemmed root to the most
+  frequent original word in the text.
+- Add the `min_df` and `max_df` readers to `Classifier::TFIDF`.
+- Accept a `MultiIO` in `TFIDF#fit_from_stream` and `Streaming::LineReader`.
+- Fix `Marshal` support in `Classifier::Bayes`. The dump left out
+  `min_word_length`, so the restored classifier raised
+  `ArgumentError: comparison of Integer with nil failed` on its first
+  `classify`. A dump written by an older version still loads, and takes the
+  configured default.
+- Fix `LSI#highest_relative_content`, which returned an Enumerator rather than
+  the documented array of documents.
+- Fix `LSI#highest_ranked_stems`, which repeated one stem when the document
+  vector held equal weights. It looked up each weight by value, so tied weights
+  all resolved to the same index.
+- Add a `docs/` reference for the command line tools, each classifier,
+  persistence, streaming, and configuration.
+
+## 2.6.0 - 2026-06-25
+
+- Add a `--search` flag to the command line tool to filter local models.
+- Add a model detail view to the command line tool.
+- Read model information in advance when the tool lists local models.
+
+## 2.5.0 - 2026-06-01
+
+- Accept keyword arguments in `train_from_stream`.
+
+## 2.4.0 - 2026-05-19
+
+- Make `min_word_length` configurable, so a caller can keep or drop short
+  words during tokenization.
+- Add a Claude Code plugin with a skill and slash commands.
+
+## 2.3.2 - 2026-01-01
+
+- Force UTF-8 encoding on the HTTP response body, so a remote model loads
+  under any locale.
+
+## 2.3.1 - 2026-01-01
+
+- Force UTF-8 encoding when the locale is not UTF-8. Model data and user input
+  no longer raise an encoding error.
+
+## 2.3.0 - 2025-12-31
+
+- Add a `classifier` executable with a model registry. The tool trains,
+  classifies, and manages saved models from the shell.
+- Fix the examples and the broken links in the README.
+- Add Dependabot for automated dependency updates.
+
+## 2.2.0 - 2025-12-29
+
+- **Breaking:** set `required_ruby_version` to `>= 3.1`. Older Ruby versions
+  are no longer supported.
+- Add a k-Nearest Neighbors classifier.
+- Add a Logistic Regression classifier.
+- Add a TF-IDF vectorizer.
+- Add streaming training and incremental SVD, so a corpus larger than memory
+  can train a model.
+- Add a hash-style API to add items to LSI.
+- Add keyword arguments to `Bayes#train` and `Bayes#untrain`.
+- Accept an array of categories in the classifier constructor.
+- Fix the sentence and paragraph splits in `Summary`.
+- Add property-based tests for the probabilistic invariants.
+
+## 2.1.0 - 2025-12-28
+
+- Replace the optional GSL dependency with a bundled C extension for LSI. The
+  extension has no external dependency and falls back to pure Ruby.
+- Add pluggable persistence backends through a storage API.
+- Add `save` and `load` methods for classifier persistence.
+- Add thread safety to the Bayes and LSI classifiers.
+- Expose the LSI tuning parameters, with validation and an introspection API.
+- Cache the expensive computations in the Bayes classifier.
+
+## 2.0.0 - 2025-12-27
+
+- **Breaking:** replace the fixed 0.1 constant in the Bayes classifier with
+  add-one (Laplace) smoothing, where
+  `P(word|category) = (count + 1) / (total + vocabulary_size)`. The smoothing
+  now scales with the vocabulary size and applies to seen and unseen words
+  alike. Classification scores change as a result.
+- Fix an LSI dimension mismatch in the pure Ruby SVD.
+- Fix the numerical stability of the SVD implementation.
+- Replace the separate RBS files with inline annotations.
+- Add a GitHub Actions workflow that publishes the gem on a version tag.
+- Add an LSI benchmark that compares GSL against pure Ruby.
+- Add RuboCop and SimpleCov.
+
+## 1.4.4 - 2024-07-31
+
+- Improve the scaling of the LSI content node.
+
+## 1.4.3 - 2024-07-31
+
+- Require `set` and use the explicit `::Set` namespace.
+- Refactor `prepare_category_name`.
+
+## 1.4.2 - 2024-07-31
+
+- Fix the word count when `remove_category` runs.
+- Add the `mutex_m` dependency and update the `fast-stemmer` version.
+
+## 1.4.1 - 2024-07-31
+
+- Add `remove_category` to the Bayes classifier.
+
+## 1.4.0 - 2024-07-31
+
+- Add `classify_with_confidence` to the LSI classifier.
+- Require `mathn` only for Ruby 2.5 and later, and add `cmath` for Ruby 2.7
+  and later.
+- Silence the warnings about an uninitialized `$GSL`, and correct the rb-gsl
+  URL hint.
+- Package the test files in the gem.
+- Add a Gemfile.lock.
+
+## 1.3.5 - 2018-04-17
+
+- Use Minitest for the test suite.
+- Add the `mathn` dependency, which Ruby 2.5.0 removed.
+- Make the gem installable through Bundler and a git remote.
+- Fix the gemspec and the unit tests.
+
+## 1.3.4 - 2013-12-31
+
+- Use a prior in the Bayes classifier.
+- Change the skip word list from an array to a set.
+- Reduce the number of regular expression matches during tokenization.
+- Stem only the words that the tokenizer keeps.
+- Default the word hash values to 0.
+- Add a gemspec, a README, and Travis CI.
+
+## 1.3.3 - 2010-07-06
+
+- Use fast-stemmer for the Porter stemmer.
+- Check `$GSL` before a call to `Matrix.diag`, so the code uses `GSL::Matrix`.
+
+## 1.3.2 - 2010-07-06
+
+- Fix the reported issue #1.
+
+## 1.3.1 and earlier
+
+Versions 1.0 through 1.3.1 reached RubyGems on 2009-07-25. The git history
+before 2010 is too sparse to attribute each change to one of these versions.

data/README.md

@@ -6,7 +6,7 @@
 
 Text classification in Ruby. Five algorithms, native performance, streaming support.
 
-**[Documentation](https://rubyclassifier.com/docs)** · **[Tutorials](https://rubyclassifier.com/docs/tutorials)** · **[API Reference](https://rubydoc.info/gems/classifier)**
+**[Reference](docs/)** · **[Documentation](https://rubyclassifier.com/docs)** · **[Tutorials](https://rubyclassifier.com/docs/tutorials)** · **[API Reference](https://rubydoc.info/gems/classifier)**
 
 ## Why This Library?
 
@@ -32,7 +32,7 @@ brew install cardmagic/tap/classifier
 
 ## Command Line
 
-Classify text instantly with pre-trained models—no coding required:
+Classify text instantly with pre-trained models. No code required:
 
 ```bash
 # Detect spam
@@ -63,7 +63,61 @@ classifier "Great product, highly recommend"
 # => positive
 ```
 
-[CLI Guide →](https://rubyclassifier.com/docs/guides/cli/basics)
+The `keywords` command scores term importance with TF-IDF. It has no
+pre-trained models, so build a vocabulary first. Every later command reads
+that model:
+
+```bash
+# Fit from multiple files. Each line becomes a separate document.
+keywords fit corpus/*.txt
+# => Saved to "/path/to/keywords.json"
+
+# Fit from stdin
+cat documents.txt | keywords fit
+
+# Tune the vocabulary filters during the fit
+keywords fit --min-df 2 --max-df 0.85 --ngram 1,2 corpus/*.txt
+```
+
+Then score any text against that vocabulary:
+
+```bash
+# Score a raw string
+keywords "Ruby is a programming language"
+# => language:0.58 programming:0.58 ruby:0.58
+
+# Score a file
+keywords extract article.txt
+# => machine:0.58 network:0.47 neural:0.47 learning:0.47
+
+# Pipeline with stdin and web data
+curl -s https://example.com/article | keywords extract
+
+# Get the top 5 terms only
+keywords -n 5 "long document with many terms..."
+
+# Use a different model file
+keywords -m custom_model.json "Ruby is a programming language"
+```
+
+Inspect the model:
+
+```bash
+keywords info
+# => Documents: 1,234
+# => Vocabulary: 5,678
+# => Min DF: 1
+# => Max DF: 1.0
+```
+
+The output maps stems back to whole words, so a model built from `programming`
+prints `programming`, not `program`. An n-gram label joins its parts with a
+space, as in `machine learning:0.35`.
+
+Run `keywords --help` for the full option list. A usage error exits 2 and any
+other error exits 1, so scripts can tell the two apart.
+
+[keywords reference →](docs/keywords.md) · [CLI Guide →](https://rubyclassifier.com/docs/guides/cli/basics)
 
 ### Claude Code Plugin
 
@@ -100,6 +154,7 @@ classifier.classify("Cheap pills!")  # => "Spam"
 classifier = Classifier::LogisticRegression.new(:positive, :negative)
 classifier.train(positive: "love amazing great wonderful")
 classifier.train(negative: "hate terrible awful bad")
+classifier.fit                     # required before the first classify
 classifier.classify("I love it!")  # => "Positive"
 ```
 [Logistic Regression Guide →](https://rubyclassifier.com/docs/guides/logisticregression/basics)
@@ -128,7 +183,7 @@ knn.classify("programming code")  # => "tech"
 ```ruby
 tfidf = Classifier::TFIDF.new
 tfidf.fit(["Ruby is great", "Python is great", "Ruby on Rails"])
-tfidf.transform("Ruby programming")  # => {:rubi => 1.0}
+tfidf.transform("Ruby programming")  # => {rubi: 1.0}
 ```
 [TF-IDF Guide →](https://rubyclassifier.com/docs/guides/tfidf/basics)
 
@@ -136,19 +191,29 @@ tfidf.transform("Ruby programming")  # => {:rubi => 1.0}
 
 ### Incremental LSI
 
-Add documents without rebuilding the entire index—400x faster for streaming data:
+Add documents without a rebuild of the whole index. Turn `auto_rebuild` off, add
+the starting corpus, then build once:
 
 ```ruby
-lsi = Classifier::LSI.new(incremental: true)
-lsi.add(tech: ["Ruby is elegant", "Python is popular"])
+lsi = Classifier::LSI.new(incremental: true, auto_rebuild: false)
+lsi.add(tech: [
+  "Ruby is an elegant programming language for web development",
+  "Python is a popular programming language for data science",
+  "JavaScript runs in browsers and powers modern web applications",
+  "Java is a compiled language used for enterprise backend systems",
+  "Rust provides memory safety without a garbage collector runtime"
+])
 lsi.build_index
 
-# These use Brand's algorithm—no full rebuild
-lsi.add(tech: "Go is fast")
-lsi.add(tech: "Rust is safe")
+# This uses Brand's algorithm. No full rebuild.
+lsi.add(tech: "Go is a fast compiled language for backend systems")
+lsi.incremental_enabled?  # => true
 ```
 
-[Learn more →](https://rubyclassifier.com/docs/guides/lsi/basics)
+Incremental mode needs the starting corpus in place before the first build, and
+it falls back to a full rebuild when one document grows the vocabulary too far.
+
+[Incremental LSI →](docs/lsi.md#incremental-mode) · [Learn more →](https://rubyclassifier.com/docs/guides/lsi/basics)
 
 ### Persistence
 

data/docs/README.md

@@ -0,0 +1,49 @@
+# Classifier reference
+
+Feature reference for the `classifier` gem. Every example here runs against the
+version in this repository.
+
+The [README](../README.md) gives the short tour. These pages give the detail.
+
+## Command line
+
+| Page | Contents |
+|:--|:--|
+| [classifier](cli.md) | Train, classify, and manage models from the shell |
+| [keywords](keywords.md) | TF-IDF keyword extraction and term scores |
+
+## Classifiers
+
+| Page | Use it for |
+|:--|:--|
+| [Bayes](bayes.md) | Fast probabilistic classification. The default choice |
+| [Logistic Regression](logistic-regression.md) | Linear classification with calibrated probabilities |
+| [LSI](lsi.md) | Semantic similarity, search, related documents, and summaries |
+| [k-Nearest Neighbors](knn.md) | Classification with the nearest examples and their votes |
+
+## Vectorization
+
+| Page | Contents |
+|:--|:--|
+| [TF-IDF](tfidf.md) | Term weights, n-grams, document frequency filters |
+
+## Shared behavior
+
+| Page | Contents |
+|:--|:--|
+| [Persistence](persistence.md) | Save, load, storage backends, and custom backends |
+| [Streaming](streaming.md) | Training on data larger than memory |
+| [Configuration](configuration.md) | Global settings and the native extension |
+
+## Which classifier
+
+Start with Bayes. It trains in one pass, needs no fit step, and handles most
+text classification tasks.
+
+- Choose **Logistic Regression** when you need a probability per category, and
+  you accept a `fit` step after training.
+- Choose **LSI** when you need similarity, search, or related documents, and not
+  only a label.
+- Choose **k-Nearest Neighbors** when you want to see which examples drove the
+  answer.
+- Choose **TF-IDF** when you want term weights rather than a category.

data/docs/bayes.md

@@ -0,0 +1,108 @@
+# Bayes
+
+`Classifier::Bayes` is a Naive Bayesian classifier. It trains in one pass, needs
+no fit step, and suits most text classification tasks.
+
+It uses log probabilities for numerical stability, and add-one (Laplace)
+smoothing, where `P(word|category) = (count + 1) / (total + vocabulary_size)`.
+
+## Train and classify
+
+```ruby
+require "classifier"
+
+classifier = Classifier::Bayes.new(:spam, :ham)
+classifier.train(spam: "Buy viagra cheap pills now")
+classifier.train(spam: "You won million dollars prize")
+classifier.train(ham: ["Meeting tomorrow at 3pm", "Quarterly report attached"])
+
+classifier.classify("Cheap pills!")
+# => "Spam"
+```
+
+A category name comes back capitalized. Pass an array to train several documents
+against one category in a single call.
+
+## Scores per category
+
+```ruby
+classifier.classifications("Cheap pills!")
+# => {"Spam" => -8.579980179515003, "Ham" => -9.680344001221918}
+```
+
+These are log probabilities, so they are negative, and the highest value wins.
+
+## Dynamic training methods
+
+A `train_<category>` method exists for every category:
+
+```ruby
+classifier = Classifier::Bayes.new(:spam, :ham)
+classifier.train_spam("cheap pills")
+classifier.train_ham("meeting tomorrow")
+classifier.classify("pills")
+# => "Spam"
+```
+
+`untrain_<category>` removes a document the same way.
+
+## Manage categories
+
+```ruby
+classifier.categories
+# => ["Spam", "Ham"]
+
+classifier.add_category(:other)
+classifier.categories
+# => ["Spam", "Ham", "Other"]
+
+classifier.remove_category(:other)
+classifier.categories
+# => ["Spam", "Ham"]
+```
+
+`remove_category` also removes that category's word counts. `append_category` is
+an alias of `add_category`.
+
+## Untrain
+
+```ruby
+classifier.untrain(spam: "Buy viagra cheap pills now")
+```
+
+Untrain the same text you trained. A document you never trained corrupts the
+counts.
+
+## Short words
+
+The tokenizer drops words shorter than `min_word_length`, which defaults to 3.
+Raise or lower it per classifier:
+
+```ruby
+classifier = Classifier::Bayes.new(:spam, :ham, min_word_length: 2)
+```
+
+See [Configuration](configuration.md) to change the default for every
+classifier.
+
+## Constructor
+
+```ruby
+Classifier::Bayes.new(*categories, min_word_length: 3)
+```
+
+An array of categories also works:
+
+```ruby
+Classifier::Bayes.new([:spam, :ham])
+```
+
+## Save and load
+
+```ruby
+classifier.save_to_file("model.json")
+loaded = Classifier::Bayes.load_from_file("model.json")
+```
+
+See [Persistence](persistence.md) for storage backends, and
+[Streaming](streaming.md) for corpora larger than memory.

data/docs/cli.md

@@ -0,0 +1,144 @@
+# classifier
+
+`classifier` trains and runs text classifiers from the shell. For term scores
+rather than categories, use [`keywords`](keywords.md).
+
+## Pre-trained models
+
+The `-r` flag pulls a model from the registry, so classification works with no
+training:
+
+```console
+$ classifier -r sms-spam-filter "You won a free iPhone"
+spam
+
+$ classifier -r imdb-sentiment "This movie was absolutely amazing"
+positive
+
+$ classifier models
+```
+
+## Train your own
+
+```console
+$ classifier train positive reviews/good/*.txt
+$ classifier train negative reviews/bad/*.txt
+
+$ classifier "Great product, highly recommend"
+positive
+```
+
+Training reads standard input when you name no file:
+
+```console
+$ echo "amazing fantastic" | classifier train positive
+```
+
+The default model file is `./classifier.json`. Use `-f` for any other path. Each
+`train` call updates that file in place.
+
+## Probabilities
+
+```console
+$ classifier -p "Great product, highly recommend"
+positive:0.89 negative:0.11
+```
+
+## Model information
+
+```console
+$ classifier info
+{
+  "file": "./classifier.json",
+  "type": "bayes",
+  "categories": [
+    "Positive",
+    "Negative"
+  ],
+  "category_stats": {
+    "Positive": {
+      "unique_words": 7,
+      "total_words": 7
+    },
+    "Negative": {
+      "unique_words": 7,
+      "total_words": 7
+    }
+  }
+}
+```
+
+## Other classifiers
+
+`-m` selects the algorithm. The default is `bayes`.
+
+```console
+$ classifier -m knn -k 3 train tech docs/tech/*.txt
+$ classifier -m lr train positive reviews/good/*.txt
+$ classifier -m lsi train dogs corpus/dogs/*.txt
+```
+
+Logistic regression needs a fit step after training:
+
+```console
+$ classifier -m lr train positive reviews/good/*.txt
+$ classifier -m lr train negative reviews/bad/*.txt
+$ classifier -m lr fit
+Model fitted successfully
+$ classifier -m lr "I love it"
+positive
+```
+
+## Search and related documents
+
+These two commands need an LSI model.
+
+```console
+$ classifier -m lsi search "machine learning"
+$ classifier -m lsi related article.txt
+```
+
+`-n` sets how many results come back. The default is 10.
+
+## Commands
+
+| Command | Action |
+|:--|:--|
+| `train <category> [files...]` | Train a category from files or standard input |
+| `info` | Print model information |
+| `fit` | Fit the model. Logistic regression only |
+| `search <query>` | Semantic search. LSI only |
+| `related <item>` | Find related documents. LSI only |
+| `models [registry]` | List the models in a registry |
+| `pull <model>` | Download a model from the registry |
+| `push <file>` | Contribute a model to the registry |
+| `<text>` | Classify the text. The default action |
+
+## Options
+
+| Option | Meaning |
+|:--|:--|
+| `-f`, `--file FILE` | Model file. Default `./classifier.json` |
+| `-m`, `--model TYPE` | Algorithm: `bayes`, `lsi`, `knn`, or `lr`. Default `bayes` |
+| `-r`, `--remote MODEL` | Use a remote model, by name or `@user/repo:name` |
+| `--search TEXT` | Search remote models by name and description, and local models by name |
+| `-o`, `--output FILE` | Output path for `pull` |
+| `-p` | Print probabilities |
+| `-n`, `--count N` | Result count for `search` and `related`. Default 10 |
+| `-k`, `--neighbors N` | Neighbor count for kNN. Default 5 |
+| `--weighted` | Use distance-weighted voting for kNN |
+| `--learning-rate N` | Learning rate for logistic regression. Default 0.1 |
+| `--regularization N` | L2 regularization for logistic regression. Default 0.01 |
+| `--max-iterations N` | Maximum iterations for logistic regression. Default 100 |
+| `-q` | Quiet mode |
+| `--local` | List locally cached models, with the `models` command |
+| `-v`, `--version` | Print the gem version |
+| `-h`, `--help` | Print the full usage |
+
+## Install without Ruby
+
+Homebrew installs the command line tools on their own:
+
+```bash
+brew install cardmagic/tap/classifier
+```