classifier 2.6.0 → 2.7.0

data/docs/configuration.md

@@ -0,0 +1,100 @@
+# Configuration
+
+## Global settings
+
+`Classifier.configure` sets the defaults for every classifier:
+
+```ruby
+require "classifier"
+
+Classifier.configure do |config|
+  config.min_word_length = 2
+end
+
+Classifier.config.min_word_length
+# => 2
+```
+
+| Setting | Default | Effect |
+|:--|:--|:--|
+| `min_word_length` | 3 | The tokenizer drops any word shorter than this |
+
+Set the configuration once at startup. The lazy setup is not thread-safe, so do
+not first touch it from several threads at once.
+
+Every classifier also takes `min_word_length` on its own, which overrides the
+global value:
+
+```ruby
+Classifier::Bayes.new(:spam, :ham, min_word_length: 2)
+```
+
+## Tokenization
+
+The tokenizer downcases the text, strips punctuation, drops the stop words in
+`CORPUS_SKIP_WORDS`, drops words shorter than `min_word_length`, and reduces
+each remaining word to its Porter stem.
+
+```ruby
+"Ruby programming is elegant".word_hash
+# => {rubi: 1, program: 1, eleg: 1}
+```
+
+`clean_word_hash` skips the punctuation strip when the text is already clean.
+`stem_to_word_hash` maps each stem back to the most frequent original word:
+
+```ruby
+"Ruby programming is elegant and programming rocks".stem_to_word_hash
+# => {rubi: "ruby", program: "programming", eleg: "elegant", rock: "rocks"}
+```
+
+## Native extension
+
+LSI uses a C extension for its linear algebra. It has no external dependency
+and builds during `gem install`. Pure Ruby runs when the extension is absent,
+with the same results and less speed.
+
+```ruby
+Classifier::LSI.backend
+# => :native
+```
+
+The value is `:native` or `:ruby`.
+
+Force pure Ruby with an environment variable, which is useful to compare the
+two:
+
+```bash
+NATIVE_VECTOR=true bundle exec rake test
+```
+
+Build the extension from a checkout:
+
+```bash
+bundle exec rake compile
+```
+
+Silence the startup notice about the missing extension:
+
+```bash
+SUPPRESS_LSI_WARNING=true
+```
+
+## Errors
+
+Every error inherits from `Classifier::Error`.
+
+| Error | Raised when |
+|:--|:--|
+| `Classifier::NotFittedError` | A model is used before its fit. Logistic regression and TF-IDF |
+| `Classifier::UnsavedChangesError` | `reload!` would discard unsaved changes |
+| `Classifier::StorageError` | A storage backend operation fails |
+
+```ruby
+begin
+  classifier.classify("text")
+rescue Classifier::NotFittedError
+  classifier.fit
+  retry
+end
+```

data/docs/keywords.md

@@ -0,0 +1,171 @@
+# keywords
+
+`keywords` scores the terms of a text with TF-IDF. It prints `term:score` pairs
+in descending order of score.
+
+The gem installs this command next to `classifier`.
+
+## A model comes first
+
+`keywords` ships no pre-trained models. Build a vocabulary before you score any
+text:
+
+```console
+$ keywords fit corpus/*.txt
+Saved to "/path/to/keywords.json"
+```
+
+A command that needs a model and finds none exits 2:
+
+```console
+$ keywords "Ruby is elegant"
+Error: No model found; run 'keywords fit' first or pass correct model using the '-m' option.
+```
+
+The default model path is `./keywords.json`. Use `-m` for any other path.
+
+## Commands
+
+| Command | Action |
+|:--|:--|
+| `keywords fit <files...>` | Build a vocabulary from files or standard input |
+| `keywords extract <file>` | Score the contents of one file |
+| `keywords info` | Print the model statistics |
+| `keywords <text>` | Score the text given as arguments |
+
+With no arguments and no piped input, `keywords` prints a short guide.
+
+## fit
+
+Each **line** becomes a separate document. The document count drives the inverse
+document frequency, so a file of 200 lines contributes 200 documents.
+
+```console
+$ keywords fit corpus/*.txt
+Saved to "/path/to/keywords.json"
+
+$ cat documents.txt | keywords fit
+Saved to "/path/to/keywords.json"
+
+$ keywords fit --min-df 2 --max-df 0.85 --ngram 1,2 corpus/*.txt
+Saved to "/path/to/keywords.json"
+```
+
+`fit` skips a directory and keeps the files beside it, so a shell glob works
+even when the directory holds a subdirectory:
+
+```console
+$ ls corpus
+a.txt   b.txt   archive/
+$ keywords fit corpus/*
+Saved to "/path/to/keywords.json"
+```
+
+A path that matches nothing stops the run, so a typo never produces a smaller
+model in silence:
+
+```console
+$ keywords fit corpus/a.txt corpus/NOPE.txt
+Error: No files matched "corpus/NOPE.txt"
+```
+
+An argument set with no readable file at all reports `No files to fit`. Empty
+input reports `No documents found to save the model`. Neither writes a model.
+
+`fit` reads one file at a time, so a corpus larger than the file descriptor
+limit still works.
+
+## extract
+
+```console
+$ keywords extract article.txt
+machine:0.58 network:0.47 neural:0.47 learning:0.47
+
+$ curl -s https://example.com/article | keywords extract
+```
+
+`extract` requires a real file. A path that does not exist, or a directory,
+exits 2. To score literal text, use the bare form instead.
+
+## info
+
+```console
+$ keywords info
+Documents: 1,234
+Vocabulary: 5,678
+Min DF: 1
+Max DF: 1.0
+```
+
+## Options
+
+| Option | Meaning |
+|:--|:--|
+| `-m`, `--model FILE` | Model file. Default `./keywords.json` |
+| `-n`, `--top N` | Print the top N terms only. N must be positive |
+| `-q` | Quiet. Suppress the `Saved to` line from `fit` |
+| `--min-df N` | Minimum document frequency, as a count. Default 1 |
+| `--max-df N` | Maximum document frequency, as a ratio from 0.0 to 1.0. Default 1.0 |
+| `--ngram MIN,MAX` | N-gram range. Default `1,1` |
+| `-v`, `--version` | Print the gem version |
+| `-h`, `--help` | Print the full usage |
+
+`--min-df` and `--max-df` apply during `fit`. The model stores them, and `info`
+reports them back.
+
+`-q` suppresses progress text but never the term scores. A scripted `fit` stays
+silent, and a scripted score still produces its data.
+
+## Output format
+
+Terms print as `term:score`, separated by spaces, sorted by descending score.
+
+The command maps stems back to whole words. A model built from `programming`
+prints `programming`, not the `program` stem:
+
+```console
+$ keywords "Ruby is a programming language"
+language:0.58 programming:0.58 ruby:0.58
+```
+
+An n-gram label joins its parts with a space:
+
+```console
+$ keywords fit --ngram 1,2 -m ng.json corpus/*.txt
+$ keywords -m ng.json "machine learning neural networks"
+machine learning:0.46 machine:0.46 neural networks:0.38 networks:0.38 neural:0.38 learning:0.38
+```
+
+Each score depends on the corpus you fitted, so your numbers will differ.
+
+That space sits inside a label in an otherwise space-separated stream. Parse
+n-gram output on the `:` separator, not on whitespace.
+
+## Exit codes
+
+| Code | Meaning |
+|:--|:--|
+| 0 | Success |
+| 1 | An unexpected error |
+| 2 | A usage error, such as a bad option, a missing model, or a path that matches nothing |
+
+Scripts can rely on 2 for every input mistake:
+
+```bash
+keywords fit corpus/*.txt || echo "fit failed with $?"
+```
+
+## Equivalent Ruby
+
+The command wraps [`Classifier::TFIDF`](tfidf.md). This code does what
+`keywords fit` does:
+
+```ruby
+require "classifier"
+
+tfidf = Classifier::TFIDF.new(min_df: 2, max_df: 0.85, ngram_range: [1, 2])
+tfidf.fit_from_stream(
+  Classifier::Streaming::MultiIO.new(Dir["corpus/*.txt"])
+)
+tfidf.save_to_file("keywords.json")
+```

data/docs/knn.md

@@ -0,0 +1,92 @@
+# k-Nearest Neighbors
+
+`Classifier::KNN` classifies text by the categories of its nearest examples. It
+shows which examples drove the answer, which makes it useful when you must
+explain a result.
+
+It builds on [LSI](lsi.md) for the similarity measure.
+
+## Add examples and classify
+
+```ruby
+require "classifier"
+
+knn = Classifier::KNN.new(k: 3)
+%w[laptop coding software developer programming].each { |w| knn.add(tech: w) }
+%w[football basketball soccer goal team].each { |w| knn.add(sports: w) }
+
+knn.classify("programming code")
+# => "tech"
+```
+
+`train` is an alias of `add`, so the call reads the same as Bayes:
+
+```ruby
+knn.train(tech: "compiler", sports: "referee")
+```
+
+## See the neighbors
+
+```ruby
+knn.classify_with_neighbors("programming code")
+```
+
+The result holds the winning category, the neighbors that voted, the vote
+tally, and a confidence value:
+
+```ruby
+{
+  category: "tech",
+  neighbors: [
+    { item: "programming", category: "tech", similarity: 0.9999999999999993 },
+    { item: "coding",      category: "tech", similarity: 0.9999999999999992 },
+    { item: "team",        category: "sports", similarity: 2.6e-16 }
+  ],
+  votes: { "tech" => 2.0, "sports" => 1.0 },
+  confidence: 0.6666666666666666
+}
+```
+
+`confidence` is the winning share of the votes.
+
+## Choose k
+
+`k` sets how many neighbors vote. It defaults to 5.
+
+```ruby
+knn = Classifier::KNN.new(k: 3)
+knn.k          # => 3
+knn.k = 5
+```
+
+A small `k` follows the data closely and reacts to noise. A large `k` smooths
+the boundary. Keep `k` at or below the number of examples you added.
+
+## Weighted voting
+
+By default every neighbor casts an equal vote. Weighted voting scales each vote
+by similarity, so a close neighbor counts for more:
+
+```ruby
+knn = Classifier::KNN.new(k: 5, weighted: true)
+```
+
+Set it later with `knn.weighted = true`.
+
+## Inspect the model
+
+```ruby
+knn.categories                   # => ["tech", "sports"]
+knn.items                        # every added example
+knn.categories_for("programming")
+knn.remove_item("programming")
+```
+
+## Save and load
+
+```ruby
+knn.save_to_file("model.json")
+loaded = Classifier::KNN.load_from_file("model.json")
+```
+
+See [Persistence](persistence.md).

data/docs/logistic-regression.md

@@ -0,0 +1,107 @@
+# Logistic Regression
+
+`Classifier::LogisticRegression` is a linear classifier. It gives calibrated
+probabilities that sum to 1.0, which Bayes does not.
+
+## Train, fit, then classify
+
+Unlike Bayes, this classifier needs a `fit` call after training. `classify`
+raises `Classifier::NotFittedError` before that call.
+
+```ruby
+require "classifier"
+
+classifier = Classifier::LogisticRegression.new(:positive, :negative)
+classifier.train(positive: "love amazing great wonderful")
+classifier.train(negative: "hate terrible awful bad")
+classifier.fit
+
+classifier.classify("I love it!")
+# => "Positive"
+```
+
+Train more documents at any time. Call `fit` again before the next classify.
+
+```ruby
+classifier.fitted?
+# => true
+```
+
+## Probabilities
+
+```ruby
+classifier.probabilities("I love it!")
+# => {"Positive" => 0.7398506195705559, "Negative" => 0.26014938042944413}
+```
+
+The values sum to 1.0. Use `classifications` for the raw scores before the
+sigmoid:
+
+```ruby
+classifier.classifications("I love it!")
+# => {"Positive" => 0.5225961471158276, "Negative" => -0.5225961471158275}
+```
+
+## Inspect the weights
+
+`weights` shows which terms drive a category:
+
+```ruby
+classifier.weights("positive")
+# => {hate: -0.5225, terribl: -0.5225, aw: -0.5225, bad: -0.5225,
+#     love: 0.5225, amaz: 0.5225, great: 0.5225, wonder: 0.5225}
+```
+
+The keys are Porter stems. The order runs by **absolute** value, so the terms
+that matter most come first whichever way they point. A positive weight argues
+for the category and a negative weight argues against it.
+
+`limit` caps the count:
+
+```ruby
+classifier.weights("positive", limit: 3)
+```
+
+Terms of equal absolute weight tie, and a tie has no defined order. A toy
+corpus like the one above gives every term the same magnitude, so `limit` there
+returns an arbitrary three. Real training data separates the weights.
+
+## Tuning
+
+```ruby
+Classifier::LogisticRegression.new(
+  :positive, :negative,
+  learning_rate: 0.1,
+  regularization: 0.01,
+  max_iterations: 100
+)
+```
+
+| Parameter | Default | Effect |
+|:--|:--|:--|
+| `learning_rate` | 0.1 | Step size per iteration. Raise it to train faster, lower it for stability |
+| `regularization` | 0.01 | L2 penalty. Raise it to reduce overfit |
+| `max_iterations` | 100 | Gradient descent iterations during `fit` |
+
+## Categories
+
+```ruby
+classifier.categories
+# => ["Positive", "Negative"]
+
+classifier.add_category(:neutral)
+```
+
+Call `fit` again after you add a category and train it.
+
+## Save and load
+
+```ruby
+classifier.save_to_file("model.json")
+loaded = Classifier::LogisticRegression.load_from_file("model.json")
+```
+
+A saved model keeps its fitted weights, so a loaded model classifies with no
+further `fit`.
+
+See [Persistence](persistence.md) and [Streaming](streaming.md).

data/docs/lsi.md

@@ -0,0 +1,219 @@
+# LSI
+
+`Classifier::LSI` implements Latent Semantic Indexing. It finds documents that
+share meaning, not only shared words, so it answers similarity, search, and
+related-document questions that a word-count classifier cannot.
+
+It uses Singular Value Decomposition. A [native C extension](configuration.md)
+makes that 5 to 50 times faster, and pure Ruby runs when the extension is
+absent.
+
+## Classify
+
+```ruby
+require "classifier"
+
+lsi = Classifier::LSI.new
+lsi.add(dog: "dog puppy canine bark fetch", cat: "cat kitten feline meow purr")
+
+lsi.classify("My puppy barks")
+# => "dog"
+```
+
+## Confidence
+
+```ruby
+lsi.classify_with_confidence("My puppy barks")
+# => ["dog", 1.0]
+```
+
+The second value runs from 0.0 to 1.0.
+
+## Search
+
+```ruby
+lsi.search("puppy", 2)
+# => ["dog puppy canine bark fetch", "cat kitten feline meow purr"]
+```
+
+The second argument caps the result count. Results come back in descending
+order of similarity.
+
+## Related documents
+
+```ruby
+lsi.find_related("dog puppy canine bark fetch", 1)
+```
+
+## Add documents
+
+`add` takes categories as keywords:
+
+```ruby
+lsi.add(dog: "dog puppy canine bark fetch")
+lsi.add(tech: ["Ruby is elegant", "Python is popular"])
+```
+
+`add_item` takes the item first, then its categories, and accepts a block that
+converts the item to text:
+
+```ruby
+lsi.add_item("dog puppy canine", :dog)
+lsi.add_item(article, :tech) { |a| a.body }
+```
+
+## The index
+
+LSI builds an index before it answers a query. By default it rebuilds whenever
+it needs to. Turn that off to add many documents and rebuild once:
+
+```ruby
+lsi = Classifier::LSI.new(auto_rebuild: false)
+lsi.add(dog: "dog puppy canine bark fetch")
+lsi.add(cat: "cat kitten feline meow purr")
+lsi.build_index
+```
+
+```ruby
+lsi.needs_rebuild?
+# => false
+```
+
+## Incremental mode
+
+Incremental mode adds documents through Brand's algorithm, with no full
+rebuild.
+
+Turn `auto_rebuild` off. Incremental mode needs the whole starting corpus in
+place before the first index build:
+
+```ruby
+lsi = Classifier::LSI.new(incremental: true, auto_rebuild: false, max_rank: 100)
+lsi.add(tech: [
+  "Ruby is an elegant programming language for web development",
+  "Python is a popular programming language for data science",
+  "JavaScript runs in browsers and powers modern web applications",
+  "Java is a compiled language used for enterprise backend systems",
+  "Rust provides memory safety without a garbage collector runtime"
+])
+lsi.build_index
+
+lsi.incremental_enabled?
+# => true
+
+lsi.add(tech: "Go is a fast compiled language for backend systems")
+lsi.incremental_enabled?
+# => true
+```
+
+`build_index` stores the U matrix that later updates need. It stores that
+matrix only while incremental mode is on.
+
+**Leave `auto_rebuild` at its default and incremental mode never starts.** Each
+`add` rebuilds at once, so the index builds from the first two documents, and
+the next `add` measures its vocabulary growth against that tiny start. The
+growth trips the threshold below, incremental mode switches off, and a later
+`build_index` cannot turn it back on.
+
+### The fallback
+
+An added document that grows the vocabulary by more than 20 percent of its
+size at the first build is too large a shift for an incremental update. LSI
+then turns incremental mode off and rebuilds in full. The results stay correct.
+The speed advantage stops.
+
+The fallback is permanent. Call `enable_incremental_mode!` to resume:
+
+```ruby
+lsi.enable_incremental_mode!(max_rank: 100)
+lsi.build_index(force: true)
+```
+
+`current_rank` reports the count of positive singular values.
+`disable_incremental_mode!` turns the mode off by hand.
+
+A corpus of a few documents grows its vocabulary quickly, so incremental mode
+suits a large starting corpus and small later additions.
+
+## Inspect the model
+
+```ruby
+lsi.items          # every indexed document
+lsi.categories_for("dog puppy canine bark fetch")
+lsi.remove_item("dog puppy canine bark fetch")
+```
+
+`singular_values` returns the raw values after `build_index`, and
+`singular_value_spectrum` returns the variance each dimension explains.
+
+`highest_ranked_stems` names the stems that carry a document:
+
+```ruby
+lsi.highest_ranked_stems("dog puppy canine bark fetch loyal", 3)
+# => [:dog, :puppi, :canin]
+```
+
+The document must already be indexed, or the call raises.
+
+`highest_relative_content` returns the documents nearest the center of the
+whole set, which describes what a corpus is mostly about:
+
+```ruby
+lsi.highest_relative_content(2)
+```
+
+It returns an empty array while the index still needs a rebuild.
+
+## Add without categories
+
+`<<` indexes a document with no category, for search and similarity only:
+
+```ruby
+lsi << "bird sparrow robin fly nest feather"
+```
+
+## Add in batches
+
+`add_batch` turns `auto_rebuild` off for the run, adds everything, then builds
+once. It reports progress like the streaming API:
+
+```ruby
+lsi.add_batch(
+  tech: ["Ruby is elegant", "Python is popular"],
+  sports: ["soccer goal", "basketball hoop"]
+) { |progress| puts progress.completed }
+```
+
+See [Streaming](streaming.md).
+
+## Summaries
+
+The gem adds `summary` to `String`:
+
+```ruby
+text = "The dog barks loudly. The cat sleeps quietly. " \
+       "Birds sing sweetly in the morning light."
+
+text.summary(1)
+# => "The cat sleeps quietly."
+```
+
+The argument sets how many sentences come back.
+
+## Constructor options
+
+| Option | Default | Meaning |
+|:--|:--|:--|
+| `auto_rebuild` | `true` | Rebuild the index automatically after a change |
+| `incremental` | `false` | Use Brand's algorithm to add documents |
+| `max_rank` | 100 | Rank cap in incremental mode |
+| `min_word_length` | 3 | Drop words shorter than this |
+
+## Save and load
+
+```ruby
+lsi.save_to_file("model.json")
+loaded = Classifier::LSI.load_from_file("model.json")
+```
+
+See [Persistence](persistence.md).