minifts 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 4da0fc15a39db6e325bc33fae8d81c0c03f1d916d5a6b200a309fdccfdb2ddd3
+  data.tar.gz: ec3f3dbce15dd997865a0c520dd13d5df80ec5e71792ad93cff1d32f5ebf3cdd
+SHA512:
+  metadata.gz: 658d33e390731ffa0ad1783d7f86006155b20d22df882ca3d069808ff33b9aaa6e794f909158baf4a6d20cc311a3d88b1810ba4b65067838e862993e0cab34a1
+  data.tar.gz: f980ef1aa9fac904faa7a5bff362bb65edb24ee4ff9e9c7877e8352ad46b789c4c9d76bef8c4f75585c20f8120ae1fe939e5ea9d044d14427415d8b7cc55ffec

data/CHANGELOG.md

@@ -0,0 +1,25 @@
+## [1.0.0] - 2026-07-18
+
+- Initial release: a pure-Ruby port of the JavaScript
+  [MiniSearch](https://github.com/lucaong/minisearch) full-text search engine.
+- In-memory inverted index backed by a radix tree (`MiniFTS::SearchableMap`)
+  with exact, prefix, and fuzzy (Levenshtein) lookup.
+- BM25+ relevance scoring with field boosting, term boosting, document boosting,
+  and per-field weights.
+- Query combinators (`OR`, `AND`, `AND_NOT`), combination-tree queries, result
+  filtering, and wildcard queries.
+- Auto-suggestions (`auto_suggest`).
+- Incremental `add` / `remove` / `discard` / `replace`, with lazy index cleanup
+  and synchronous (auto-)vacuuming.
+- Byte-identical JSON serialization with the JavaScript library (`to_json` /
+  `load_json`): a Ruby-serialized index loads and searches identically in
+  JavaScript, and vice versa, for any corpus without astral-plane (above
+  U+FFFF) characters. Verified bidirectionally against the real
+  `minisearch@7.2.0` across 32 scenarios (`rake compat`).
+- Runs on every Ruby since 2.4; no runtime dependencies.
+- Configurable `logger` for index-corruption warnings; setting `logger: nil`
+  silences them without raising.
+- Verified bit-for-bit against the reference JavaScript implementation via
+  golden and randomized differential tests, plus a full parity suite mirroring
+  the reference's own (non-async) test cases, including the movie/song ranking
+  sets.

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,10 @@
+# Code of Conduct
+
+"minifts" follows [The Ruby Community Conduct Guideline](https://www.ruby-lang.org/en/conduct) in all "collaborative space", which is defined as community communications channels (such as mailing lists, submitted patches, commit comments, etc.):
+
+* Participants will be tolerant of opposing views.
+* Participants must ensure that their language and actions are free of personal attacks and disparaging personal remarks.
+* When interpreting the words and actions of others, participants should always assume good intentions.
+* Behaviour which can be reasonably considered harassment will not be tolerated.
+
+If you have any concerns about behaviour within this project, please contact us at ["rodrigo.serradura@gmail.com"](mailto:"rodrigo.serradura@gmail.com").

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Rodrigo Serradura
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,302 @@
+<h1 align="center">MiniFTS</h1>
+
+<p align="center">
+  A tiny, dependency-free full-text search engine for Ruby, held entirely in memory —<br>
+  a faithful, bit-for-bit port of the JavaScript <a href="https://github.com/lucaong/minisearch">MiniSearch</a> library.
+</p>
+
+<p align="center">
+  <a href="https://rubygems.org/gems/minifts"><img src="https://img.shields.io/gem/v/minifts" alt="Gem version"></a>
+  <a href="https://rubygems.org/gems/minifts"><img src="https://img.shields.io/gem/dt/minifts" alt="Downloads"></a>
+  <a href="https://github.com/serradura/minifts/actions/workflows/main.yml"><img src="https://github.com/serradura/minifts/actions/workflows/main.yml/badge.svg" alt="CI"></a>
+  <a href="https://github.com/serradura/minifts"><img src="https://img.shields.io/badge/ruby-%3E%3D%202.4-black" alt="Ruby >= 2.4"></a>
+  <a href="LICENSE.txt"><img src="https://img.shields.io/badge/license-MIT-blue" alt="License: MIT"></a>
+</p>
+
+It has the same BM25+ scoring, prefix and fuzzy matching, boosting, query
+combinators, auto-suggestions, and JSON index format as the original — the
+public API mirrors it, translated to idiomatic Ruby.
+
+- **Pure Ruby, zero runtime dependencies.** No native extensions, no database,
+  no build step. It runs on the Ruby your OS already ships — the floor is
+  **Ruby 2.4**.
+- **Bit-for-bit compatible with MiniSearch.** Scores, ranking, and the
+  serialized index are identical to the JavaScript library (verified against it
+  across thousands of generated cases — see [Fidelity](#fidelity)). A JSON index
+  written by one can be loaded by the other.
+- **Good for the "just add search" case.** When you want relevance-ranked search
+  over a modest corpus (a few docs to hundreds of thousands) without reaching for
+  SQLite FTS5, Elasticsearch, or a service.
+
+## Installation
+
+```bash
+gem install minifts
+```
+
+Or in a `Gemfile`:
+
+```ruby
+gem "minifts"
+```
+
+## Quick start
+
+```ruby
+require "minifts"
+
+documents = [
+  { "id" => 1, "title" => "Moby Dick",   "text" => "Call me Ishmael. Some years ago...", "category" => "fiction" },
+  { "id" => 2, "title" => "Zen and the Art of Motorcycle Maintenance", "text" => "I can see by my watch...", "category" => "fiction" },
+  { "id" => 3, "title" => "Neuromancer", "text" => "The sky above the port was...", "category" => "sci-fi" },
+]
+
+ms = MiniFTS.new(fields: %w[title text], store_fields: %w[title category])
+ms.add_all(documents)
+
+ms.search("zen motorcycle")
+# => [
+#   { id: 2, score: 2.77, terms: ["zen", "motorcycle"], query_terms: ["zen", "motorcycle"],
+#     match: { "zen" => ["title"], "motorcycle" => ["title"] },
+#     "title" => "Zen and the Art of Motorcycle Maintenance", "category" => "fiction" },
+# ]
+```
+
+Each result is a Hash: `:id`, `:score`, `:terms` (the matched *document* terms),
+`:query_terms` (the matched query terms), `:match` (term → fields it matched in),
+plus any `store_fields` under their **string** keys.
+
+## Search options
+
+Pass options as the second argument to `search`:
+
+```ruby
+ms.search("moto", prefix: true)                          # prefix search
+ms.search("ishmael", fuzzy: 0.2)                         # fuzzy (edit distance = 0.2 * term length)
+ms.search("zen art", combine_with: "AND")               # require all terms
+ms.search("zen", boost: { "title" => 2 })               # weight matches in :title higher
+ms.search("art", fields: ["title"])                     # restrict to certain fields
+ms.search("art", filter: ->(r) { r["category"] == "fiction" })
+ms.search("art", boost_document: ->(id, term, stored) { stored["featured"] ? 2 : 1 })
+ms.search("zen art", boost_term: ->(term, i, terms) { term == "zen" ? 2 : 1 })  # weight some query terms higher
+ms.search("moto", prefix: true, weights: { prefix: 0.1 })   # down-weight non-exact (prefix/fuzzy) matches
+ms.search("zen", bm25: { k: 1.2, b: 0.7, d: 0.5 })          # tune BM25+ scoring (defaults shown)
+ms.search(MiniFTS::WILDCARD)                          # match every document
+```
+
+Query strings can also be combination trees:
+
+```ruby
+ms.search(
+  combine_with: "AND",
+  queries: ["zen", { combine_with: "OR", queries: %w[motorcycle archery] }]
+)
+```
+
+Supported search options: `:fields`, `:filter`, `:boost`, `:boost_term`,
+`:weights`, `:boost_document`, `:prefix`, `:fuzzy`, `:max_fuzzy`,
+`:combine_with`, `:tokenize`, `:process_term`, `:bm25`.
+
+`:weights` scales non-exact matches — defaults `{ fuzzy: 0.45, prefix: 0.375 }`,
+so a fuzzy or prefix hit counts for less than an exact one. `:boost_term` takes a
+callable returning a per-*query-term* multiplier (default `1`). `:bm25` tunes the
+BM25+ parameters `k`, `b`, and `d` (defaults `{ k: 1.2, b: 0.7, d: 0.5 }`).
+Defaults can be set once via the constructor's `:search_options`.
+
+## Auto-suggestions
+
+```ruby
+ms.auto_suggest("neuro")
+# => [{ suggestion: "neuromancer", terms: ["neuromancer"], score: 0.46 }]
+
+ms.auto_suggest("zen ar")
+# => [{ suggestion: "zen archery art", terms: [...], score: 1.73 },
+#     { suggestion: "zen art", terms: [...], score: 1.21 }]
+```
+
+## Adding, removing, and updating documents
+
+```ruby
+ms.add(document)          # add one
+ms.add_all(documents)     # add many
+ms.remove(document)       # remove (needs the full, unchanged document)
+ms.discard(id)            # remove by ID (lazy cleanup; see below)
+ms.discard_all([id, ...])
+ms.replace(updated)       # discard + add, same ID
+ms.remove_all             # clear everything
+ms.vacuum                 # reclaim space from discarded documents
+```
+
+`discard` is the convenient counterpart to `remove`: it only needs the ID and
+takes effect immediately, cleaning up the inverted index lazily (during later
+searches, and via `vacuum`). Auto-vacuuming runs on your behalf once enough
+documents have been discarded.
+
+Introspection: `document_count`, `term_count`, `has?(id)`,
+`get_stored_fields(id)`, `dirt_count`, `dirt_factor`.
+
+## Configuration
+
+```ruby
+MiniFTS.new(
+  fields: %w[title text],       # REQUIRED: field names (strings) to index
+  id_field: "id",               # unique-ID field (default "id")
+  store_fields: %w[title],      # fields to keep and return in results
+  extract_field: ->(doc, field) { doc[field] },   # how to read a field
+  stringify_field: ->(value, field) { value.to_s },
+  tokenize: ->(text, field = nil) { text.split(/\s+/) },
+  process_term: ->(term, field = nil) { term.downcase },  # normalize/stem; return nil to drop
+  search_options: { prefix: true },      # default options for every search
+  auto_suggest_options: { fuzzy: 0.2 },  # default options for every auto_suggest
+  logger: ->(level, msg, code) { warn("#{level}: #{msg}") },  # warning sink; default logs to stderr
+  auto_vacuum: true
+)
+```
+
+Callables are anything responding to `call` (lambdas, procs, method objects).
+`MiniFTS.get_default(:tokenize)` returns the built-in default for any
+constructor option (`:tokenize`, `:process_term`, `:extract_field`, ...), handy
+when you want to wrap the default rather than replace it.
+
+### Documents with symbol keys
+
+By default documents are Hashes with **string** keys matching the field names.
+For symbol-keyed documents, supply an extractor:
+
+```ruby
+MiniFTS.new(fields: ["title"], extract_field: ->(doc, field) { doc[field.to_sym] })
+```
+
+### Stemming, stop words, synonyms
+
+Do it in `process_term` (return `nil`/`false` to drop a term, or an array to
+expand one):
+
+```ruby
+STOP = %w[the a an of and].to_set
+MiniFTS.new(
+  fields: ["text"],
+  process_term: ->(t, _f = nil) { d = t.downcase; STOP.include?(d) ? nil : d }
+)
+```
+
+## Serialization and JavaScript interop
+
+The index serializes to JSON in exactly MiniSearch's format:
+
+```ruby
+json = ms.to_json
+File.write("index.json", json)
+
+# Later — pass the same options used to build it:
+ms = MiniFTS.load_json(File.read("index.json"), fields: %w[title text], store_fields: %w[title])
+```
+
+If you already hold the index as a Ruby Hash (string keys) rather than a JSON
+string, use `ms.as_plain_object` to get one and `MiniFTS.load(hash, **opts)`
+to load it back — `to_json`/`load_json` are exactly those two with JSON in
+between.
+
+Because the format is identical, an index built in Ruby can be loaded by
+JavaScript MiniSearch in the browser (and vice-versa) — index server-side, search
+client-side.
+
+## The radix tree
+
+The inverted index is backed by `MiniFTS::SearchableMap`, a radix tree with
+`Map`-like semantics plus `at_prefix` and `fuzzy_get`. It is exported for
+standalone use:
+
+```ruby
+map = MiniFTS::SearchableMap.new
+map.set("motorcycle", 1).set("motor", 2)
+map.at_prefix("moto").keys       # => ["motor", "motorcycle"]
+map.fuzzy_get("moter", 1)        # => { "motor" => [2, 1] }
+```
+
+## Performance
+
+Everything is in-memory and the heavy lifting is a radix-tree lookup plus BM25
+scoring, so it is fast for its class and scales with the number of *matching*
+documents rather than the total. Indicative figures from
+`benchmarks/search_bench.rb` (5,000 synthetic documents, pure Ruby):
+
+| operation      | throughput       |
+| -------------- | ---------------- |
+| index build    | ~3,000 docs/sec  |
+| exact search   | ~400 queries/sec |
+| prefix search  | ~330 queries/sec |
+| fuzzy search   | ~180 queries/sec |
+
+On the same corpus, exact search is ~6× faster than a naive "scan every
+document" approach — a gap that widens as the corpus grows.
+
+### Optimization
+
+The implementation was profiled and tuned against a real-world corpus. Relative
+to an early baseline, on the same workload with output verified byte-identical
+(2,627 result rows, scores matched to 17 significant figures):
+
+| metric          | change vs. baseline        |
+| --------------- | -------------------------- |
+| indexing memory | **4.3× lighter** (−76.6%)  |
+| search memory   | **1.8× lighter** (−44.7%)  |
+| indexing speed  | +37% (1.37×)               |
+| search speed    | +15% (1.15×)               |
+
+Memory (allocated bytes) is deterministic; the throughput figures are wall-clock
+medians and directional. The memory wins dominate because the hot path was
+allocation-bound: indexing 500 documents churned ~11.5M objects and now churns
+~2.4M — far lower GC pressure and peak memory, which is what matters most for an
+in-memory index at scale.
+
+## Fidelity
+
+This port is validated against the original JavaScript MiniSearch, not just by
+hand-written expectations:
+
+- **Golden cases** (`test/golden.json`, `test/lifecycle.json`): every documented
+  feature and lifecycle operation, with the exact output the JS library produces.
+- **Randomized differential testing** (`test/fuzz.json`): 250 documents and 700
+  randomly generated queries with random options, run through the real JS library
+  and replayed in Ruby — ~150,000 assertions, scores matched to full double
+  precision.
+- **Byte-identical serialization**: a Ruby-serialized index is byte-for-byte
+  identical to the JavaScript library's — index in one, search in the other. The
+  lone exception is a corpus containing astral-plane characters (above U+FFFF,
+  e.g. most emoji), whose radix-tree terms sort by UTF-8 code point here vs
+  UTF-16 code unit in JS; such indexes still load and search identically, only
+  their serialized byte order differs. Interchange is verified in both directions
+  across 32 scenarios by `rake compat` (see `fidelity/`).
+
+The whole suite runs on **Ruby 2.4** (verified in a `ruby:2.4` container).
+
+### Differences from the JavaScript library
+
+- Options and result metadata use **snake_case symbol keys** (`:id_field`,
+  `:store_fields`, `:query_terms`, ...). Stored fields keep their string keys.
+- Field names are **strings**; documents are read with `doc[field]` by default.
+- Vacuuming is **synchronous** (Ruby has no main thread to protect), so the
+  async variants (`addAllAsync`, `loadJSONAsync`, batched `vacuum`) are omitted;
+  `vacuum` cleans up immediately and `is_vacuuming` does not apply.
+- The wildcard is `MiniFTS::WILDCARD` (or `MiniFTS.wildcard`).
+
+## Development
+
+```bash
+bin/setup          # install dependencies
+rake test          # run the test suite
+rake rubocop       # lint (Ruby 2.4 target)
+rake               # both
+ruby -Ilib benchmarks/search_bench.rb [num_docs] [num_queries]
+```
+
+## Credits
+
+A Ruby port of [MiniSearch](https://github.com/lucaong/minisearch) by Luca
+Ongaro. All the search design is theirs; this project translates it to idiomatic
+Ruby with no dependencies.
+
+## License
+
+Available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+
+# rake/testtask (not minitest/test_task) so `rake test` runs on every supported
+# Ruby — minitest's own task class needs minitest 5.16+, which needs Ruby 2.6.
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test" << "lib"
+  t.test_files = FileList["test/**/test_*.rb"].exclude("test/test_helper.rb")
+  t.warning = false
+end
+
+# Ruby <-> JavaScript index interchange suite. Kept out of `test`/`default`
+# because it needs Node and the real `minisearch` npm package; the pure-Ruby
+# suite (including the Ruby 2.4 floor) must not depend on either.
+desc "Run the Ruby <-> JavaScript index interchange suite (needs Node in fidelity/)"
+task :compat do
+  Dir.chdir("fidelity") do
+    sh "npm install --silent" unless Dir.exist?("node_modules")
+    sh "ruby bin/build_ruby.rb"      # stage 1: Ruby produces the index fixtures
+    sh "node bin/check_js.mjs"       # stage 2: JS loads them + builds native
+    sh "ruby -Itest test/test_interchange.rb" # stage 3: Ruby loads the JS indexes
+  end
+end
+
+# RuboCop only installs on newer Rubies (see Gemfile); the default task degrades
+# to test-only where it is absent.
+begin
+  require "rubocop/rake_task"
+  RuboCop::RakeTask.new
+  task default: %i[test rubocop]
+rescue LoadError
+  task default: %i[test]
+end