classifier 2.5.0 → 2.7.0

data/docs/persistence.md

@@ -0,0 +1,124 @@
+# Persistence
+
+Every classifier saves and loads the same way. `Classifier::Bayes`,
+`Classifier::LogisticRegression`, `Classifier::LSI`, `Classifier::KNN`, and
+`Classifier::TFIDF` all share this API.
+
+## Files
+
+```ruby
+require "classifier"
+
+classifier = Classifier::Bayes.new(:spam, :ham)
+classifier.train(spam: "cheap pills", ham: "meeting tomorrow")
+
+classifier.save_to_file("model.json")
+
+loaded = Classifier::Bayes.load_from_file("model.json")
+loaded.classify("pills")
+# => "Spam"
+```
+
+The format is JSON, so a saved model is readable and portable between Ruby
+versions.
+
+## Storage backends
+
+A backend separates the model from where it lives. Assign one, then call `save`
+and `load` with no path:
+
+```ruby
+classifier.storage = Classifier::Storage::File.new(path: "model.json")
+classifier.save
+
+loaded = Classifier::Bayes.load(storage: classifier.storage)
+```
+
+The gem ships two backends:
+
+| Backend | Use it for |
+|:--|:--|
+| `Classifier::Storage::File` | A model on disk |
+| `Classifier::Storage::Memory` | Tests, and a model that lives for one process |
+
+```ruby
+storage = Classifier::Storage::Memory.new
+
+classifier = Classifier::Bayes.new(:a, :b)
+classifier.train(a: "alpha", b: "beta")
+classifier.storage = storage
+classifier.save
+
+Classifier::Bayes.load(storage: storage).categories
+# => ["A", "B"]
+```
+
+## Write your own backend
+
+Subclass `Classifier::Storage::Base` and implement four methods:
+
+```ruby
+Classifier::Storage::Base.instance_methods(false).sort
+# => [:delete, :exists?, :read, :write]
+```
+
+| Method | Contract |
+|:--|:--|
+| `write(key, data)` | Store the serialized model |
+| `read(key)` | Return what `write` stored, or nil |
+| `exists?(key)` | Report whether a model is stored under the key |
+| `delete(key)` | Remove the stored model |
+
+A Redis backend looks like this:
+
+```ruby
+class RedisStorage < Classifier::Storage::Base
+  def initialize(redis:, namespace: "classifier")
+    @redis = redis
+    @namespace = namespace
+  end
+
+  def write(key, data) = @redis.set(namespaced(key), data)
+  def read(key) = @redis.get(namespaced(key))
+  def exists?(key) = @redis.exists?(namespaced(key))
+  def delete(key) = @redis.del(namespaced(key))
+
+  private
+
+  def namespaced(key) = "#{@namespace}:#{key}"
+end
+```
+
+The same shape covers S3, a SQL table, or any other store.
+
+## Track unsaved changes
+
+`dirty?` reports whether the model changed since the last save:
+
+```ruby
+classifier.dirty?
+```
+
+`reload` discards unsaved changes and reads the stored model again. `reload!`
+does the same and raises when no stored model exists.
+
+## Marshal
+
+Every classifier also supports `Marshal`:
+
+```ruby
+data = Marshal.dump(classifier)
+restored = Marshal.load(data)
+```
+
+Prefer JSON. Only load a marshalled model from a source you trust.
+
+## Checkpoints
+
+Streaming training writes checkpoints, so a long run resumes after a failure:
+
+```ruby
+Classifier::Bayes.load_checkpoint(storage: storage, checkpoint_id: "run-1")
+```
+
+See [Streaming](streaming.md).

data/docs/streaming.md

@@ -0,0 +1,104 @@
+# Streaming
+
+Streaming trains a classifier on data larger than memory. The reader pulls one
+batch of lines at a time, so peak memory stays flat whatever the corpus size.
+
+Each **line** is one document.
+
+## Train from a stream
+
+```ruby
+require "classifier"
+
+classifier = Classifier::Bayes.new(:spam, :ham)
+classifier.train_from_stream(:spam, File.open("spam_corpus.txt"))
+```
+
+`Classifier::LogisticRegression` and `Classifier::KNN` accept the same call.
+`Classifier::TFIDF` uses `fit_from_stream`, because it fits a vocabulary rather
+than a category.
+
+## Progress
+
+Pass a block to watch the run:
+
+```ruby
+classifier.train_from_stream(:spam, File.open("spam_corpus.txt")) do |progress|
+  puts "completed=#{progress.completed}"
+end
+```
+
+`Classifier::Streaming::Progress` reports `completed`, and `total` when the
+reader can estimate the line count from the file size.
+
+## Batch size
+
+The reader groups lines into batches. The default is
+`Classifier::Streaming::DEFAULT_BATCH_SIZE`.
+
+```ruby
+classifier.train_from_stream(:spam, File.open("corpus.txt"), batch_size: 500)
+```
+
+A larger batch does less bookkeeping and uses more memory.
+
+## Train from an array
+
+`train_batch` takes documents already in memory and uses the same batching:
+
+```ruby
+classifier.train_batch(:spam, ["cheap pills", "you won a prize"])
+```
+
+`Classifier::LSI` and `Classifier::KNN` name the same method `add_batch`, which
+matches their `add` API:
+
+```ruby
+lsi.add_batch(tech: ["Ruby is elegant", "Python is popular"])
+```
+
+## Read several files as one stream
+
+`Classifier::Streaming::MultiIO` presents many sources as one sequential
+stream. It accepts file paths, IO objects, or both:
+
+```ruby
+multi = Classifier::Streaming::MultiIO.new(["a.txt", "b.txt"])
+multi.each_line { |line| puts line }
+```
+
+```ruby
+Classifier::Streaming::MultiIO.new(Dir["corpus/*.txt"]).each_line.to_a
+```
+
+Given a path, `MultiIO` opens the file, reads it, and closes it before it moves
+to the next one. Only one file is ever open, so a corpus larger than the file
+descriptor limit still works. Given an IO object, it reads that object and
+leaves the closing to you.
+
+`each_line` returns an Enumerator when you pass no block.
+
+Combine it with a vectorizer to fit a whole corpus:
+
+```ruby
+tfidf = Classifier::TFIDF.new
+tfidf.fit_from_stream(
+  Classifier::Streaming::MultiIO.new(Dir["corpus/*.txt"])
+)
+tfidf.num_documents
+```
+
+## Checkpoints
+
+A long run writes checkpoints through the assigned storage backend, so a
+failure does not cost the whole pass:
+
+```ruby
+classifier.storage = Classifier::Storage::File.new(path: "model.json")
+resumed = Classifier::Bayes.load_checkpoint(
+  storage: classifier.storage,
+  checkpoint_id: "run-1"
+)
+```
+
+See [Persistence](persistence.md) for the backends.

data/docs/tfidf.md

@@ -0,0 +1,137 @@
+# TF-IDF
+
+`Classifier::TFIDF` turns text into term weights. It answers "which terms matter
+in this document", not "which category is this". For the same thing from a
+shell, use [`keywords`](keywords.md).
+
+TF-IDF raises the weight of a term that is frequent in one document, and lowers
+it for a term that is common across every document.
+
+## Fit and transform
+
+```ruby
+require "classifier"
+
+tfidf = Classifier::TFIDF.new
+tfidf.fit(["Ruby is great", "Python is great", "Ruby on Rails"])
+
+tfidf.transform("Ruby programming")
+# => {rubi: 1.0}
+```
+
+The keys are Porter stems. `fit_transform` does both steps at once:
+
+```ruby
+tfidf.fit_transform(["Ruby is great", "Python is great"])
+```
+
+`transform` raises `Classifier::NotFittedError` before a fit.
+
+## The vocabulary
+
+```ruby
+tfidf.feature_names   # every term in the vocabulary
+tfidf.vocabulary      # term => column index
+tfidf.idf             # term => inverse document frequency
+tfidf.num_documents   # documents seen during the fit
+tfidf.fitted?         # => true
+```
+
+## Document frequency filters
+
+`min_df` and `max_df` drop terms that are too rare or too common.
+
+```ruby
+tfidf = Classifier::TFIDF.new(min_df: 2, max_df: 0.85)
+```
+
+| Parameter | Type | Meaning |
+|:--|:--|:--|
+| `min_df` | Integer | Keep a term only when at least this many documents hold it |
+| `min_df` | Float | The same, as a ratio of the document count |
+| `max_df` | Float | Drop a term that appears in more than this ratio of documents |
+| `max_df` | Integer | The same, as an absolute document count |
+
+An Integer means a count and a Float means a ratio. A Float must fall between
+0.0 and 1.0, and an Integer must not be negative.
+
+Both values are readable after construction, and a saved model keeps them:
+
+```ruby
+tfidf.min_df   # => 2
+tfidf.max_df   # => 0.85
+```
+
+## N-grams
+
+`ngram_range` sets the shortest and longest phrase to index. It defaults to
+`[1, 1]`, which indexes single words only.
+
+```ruby
+tfidf = Classifier::TFIDF.new(ngram_range: [1, 2])
+tfidf.fit(["machine learning rocks", "machine learning is fun"])
+
+tfidf.feature_names.sort.first(6)
+# => [:fun, :learn, :learn_fun, :learn_rock, :machin, :machin_learn]
+```
+
+An n-gram key joins its stems with an underscore. Both bounds must be 1 or
+more, and the first must not exceed the second.
+
+## Sublinear term frequency
+
+`sublinear_tf: true` replaces the raw count with `1 + log(count)`, which damps
+the effect of a term repeated many times in one document:
+
+```ruby
+tfidf = Classifier::TFIDF.new(sublinear_tf: true)
+```
+
+## Constructor
+
+```ruby
+Classifier::TFIDF.new(
+  min_df: 1,
+  max_df: 1.0,
+  ngram_range: [1, 1],
+  sublinear_tf: false,
+  min_word_length: 3
+)
+```
+
+## Fit from a stream
+
+`fit_from_stream` reads line by line, so a corpus larger than memory still
+fits. Each line is one document.
+
+```ruby
+tfidf = Classifier::TFIDF.new
+tfidf.fit_from_stream(File.open("corpus.txt"))
+```
+
+Pass a [`MultiIO`](streaming.md) to read several files as one stream:
+
+```ruby
+tfidf.fit_from_stream(
+  Classifier::Streaming::MultiIO.new(Dir["corpus/*.txt"])
+)
+```
+
+## Map stems back to words
+
+`transform` returns stems. `String#stem_to_word_hash` maps each stem back to the
+most frequent original word, which is how `keywords` prints whole words:
+
+```ruby
+"Ruby programming is elegant and programming rocks".stem_to_word_hash
+# => {rubi: "ruby", program: "programming", eleg: "elegant", rock: "rocks"}
+```
+
+## Save and load
+
+```ruby
+tfidf.save_to_file("vectorizer.json")
+loaded = Classifier::TFIDF.load_from_file("vectorizer.json")
+```
+
+See [Persistence](persistence.md).

data/exe/classifier

@@ -1,7 +1,6 @@
 #!/usr/bin/env ruby
 # frozen_string_literal: true
 
-# Force UTF-8 encoding for proper handling of model data and user input
 Encoding.default_external = Encoding::UTF_8
 Encoding.default_internal = Encoding::UTF_8
 

data/exe/keywords

@@ -0,0 +1,15 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Force UTF-8 encoding for proper handling of model data and user input
+Encoding.default_external = Encoding::UTF_8
+Encoding.default_internal = Encoding::UTF_8
+
+require 'classifier/keywords/cli'
+
+result = Classifier::Keywords::CLI.new(ARGV).run
+
+warn result[:error] unless result[:error].empty?
+puts result[:output] unless result[:output].empty?
+
+exit result[:exit_code]

data/lib/classifier/bayes.rb

@@ -279,14 +279,17 @@ module Classifier
     # Custom marshal serialization to exclude mutex state
     # @rbs () -> Array[untyped]
     def marshal_dump
-      [@categories, @total_words, @category_counts, @category_word_count, @dirty]
+      [@categories, @total_words, @category_counts, @category_word_count, @dirty,
+       @min_word_length]
     end
 
     # Custom marshal deserialization to recreate mutex
     # @rbs (Array[untyped]) -> void
     def marshal_load(data)
       mu_initialize
-      @categories, @total_words, @category_counts, @category_word_count, @dirty = data
+      @categories, @total_words, @category_counts, @category_word_count, @dirty,
+        @min_word_length = data
+      @min_word_length ||= Classifier.config.min_word_length
       @cached_training_count = nil
       @cached_vocab_size = nil
       @storage = nil

data/lib/classifier/cli.rb

@@ -95,6 +95,13 @@ module Classifier
           @options[:type] = type
         end
 
+        opts.on(
+          '--search TEXT',
+          'Search remote models by name/description, and local models by name only. Use quotes for multiword search'
+        ) do |text|
+          @options[:search] = text
+        end
+
         opts.on('-r', '--remote MODEL', 'Use remote model: name or @user/repo:name') do |model|
           @options[:remote] = model
         end
@@ -370,18 +377,49 @@ module Classifier
     end
 
     def list_remote_models
-      registry_arg = @args.shift
-      registry = parse_registry(registry_arg) || DEFAULT_REGISTRY
+      registry_or_model_arg = @args.shift
+      registry, model = detect_registry_and_model(registry_or_model_arg)
+      registry = parse_registry(registry) || DEFAULT_REGISTRY
       index = fetch_registry_index(registry)
 
       return if @exit_code != 0
 
-      if index['models'].empty?
+      models = index['models']
+
+      if model
+        info = models[model]
+        if info.nil?
+          @output << "No model #{model.inspect} found in registry"
+          return
+        end
+
+        type = info['type'] || 'unknown'
+        size = info['size'] || 'unknown'
+        desc = info['description'] || ''
+        categories = (info['categories'] || []).map(&:downcase).join(', ')
+        version = info['version'] || ''
+        author = info['author'] || ''
+        @output << format(
+          "Name: %<name>s\nDescription: %<desc>s\nType: %<type>s\n" \
+          "Categories: %<categories>s\nVersion: %<version>s\nAuthor: %<author>s\nSize: %<size>s",
+          name: model, desc: desc, type: type, categories: categories,
+          version: version, author: author, size: size
+        )
+        return
+      end
+
+      if @options[:search]
+        models = models.filter do |name, info|
+          [name, info['description']].any?(/#{Regexp.escape(@options[:search])}/i)
+        end
+      end
+
+      if models.empty?
         @output << 'No models found in registry'
         return
       end
 
-      index['models'].each do |name, info|
+      models.each do |name, info|
         type = info['type'] || 'unknown'
         size = info['size'] || 'unknown'
         desc = info['description'] || ''
@@ -390,6 +428,8 @@ module Classifier
     end
 
     def list_local_models
+      model_arg = @args.shift
+
       models_dir = File.join(CACHE_DIR, 'models')
 
       unless Dir.exist?(models_dir)
@@ -413,14 +453,43 @@ module Classifier
 
       models = default_models + custom_models #: Array[{name: String, registry: String?, path: String}]
 
+      models.each do |model|
+        model[:info] = load_model_info(model[:path])
+      end
+
+      if model_arg
+        model = models.find { |model| model[:name] == model_arg }
+        if model.nil?
+          @output << "No local model #{model_arg.inspect} found"
+          return
+        end
+        display_name = model[:registry] ? "@#{model[:registry]}:#{model[:name]}" : model[:name]
+        type = model.dig(:info, 'type') || 'unknown'
+        version = model.dig(:info, 'version')
+        categories = (model.dig(:info, 'categories') || {}).keys.map(&:downcase).join(', ')
+        size = File.size(model[:path])
+        @output << format(
+          "Name: %<name>s\nType: %<type>s\n" \
+          "Categories: %<categories>s\nVersion: %<version>s\nSize: %<size>s",
+          name: display_name, type: type, categories: categories,
+          version: version, size: human_size(size)
+        )
+        return
+      end
+
+      if @options[:search]
+        models = models.filter do |model|
+          [model[:name], model.dig(:info, 'description')].any?(/#{Regexp.escape(@options[:search])}/i)
+        end
+      end
+
       if models.empty?
         @output << 'No local models found'
         return
       end
 
       models.each do |model|
-        info = load_model_info(model[:path])
-        type = info['type'] || 'unknown'
+        type = model.dig(:info, 'type') || 'unknown'
         display_name = model[:registry] ? "@#{model[:registry]}:#{model[:name]}" : model[:name]
         size = File.size(model[:path])
         @output << format('%-30<name>s (%<type>s, %<size>s)', name: display_name, type: type, size: human_size(size))
@@ -800,6 +869,15 @@ module Classifier
       @output << 'Run "classifier --help" for full usage.'
     end
 
+    # @rbs (String?) -> [String?, String?]
+    def detect_registry_and_model(arg)
+      return nil, nil if arg.nil?
+      return *arg.split(':', 2) if arg.include?(':')
+      return nil, arg unless arg.start_with?('@')
+
+      [arg, nil]
+    end
+
     # Parse @user/repo format to extract registry
     # @rbs (String?) -> String?
     def parse_registry(arg)

data/lib/classifier/extensions/word_hash.rb

@@ -33,6 +33,12 @@ class String
     word_hash_for_words(gsub(/[^\w\s]/, '').split, min_word_length)
   end
 
+  # Builds a mapping between stemmed roots and their most frequent original words.
+  # @rbs (?Integer) -> Hash[Symbol, String]
+  def stem_to_word_hash(min_word_length = 3)
+    mapping_stem_to_word_for_words(gsub(/[^\w\s]/, '').split, min_word_length)
+  end
+
   private
 
   # @rbs (Array[String], Integer) -> Hash[Symbol, Integer]
@@ -54,6 +60,20 @@ class String
     d
   end
 
+  # @rbs (Array[String], Integer) -> Hash[Symbol, Integer]
+  def mapping_stem_to_word_for_words(words, min_word_length)
+    h = {}
+    words.map { _1.tap(&:downcase!) }.tally.each do |word, count|
+      next unless !CORPUS_SKIP_WORDS.include?(word) && word.length >= min_word_length
+
+      stem = word.stem.intern
+      h[stem] ||= [word, count]
+      h[stem] = [word, count] if h.dig(stem, 1) < count
+    end
+    h.each_key { |k| h[k] = h[k].first }
+    h
+  end
+
   CORPUS_SKIP_WORDS = ::Set.new(%w[
                                   a
                                   again