spellkit 0.3.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a16cdfa3277c6161351b4b9f2ba35720dd78487f7e7b49f6c3cebc6a6a4d2d4b
-  data.tar.gz: 71ec217bee6087915e2fa2fec3229ba4fa1ee8b354c437acf68ef5abcef19b0f
+  metadata.gz: 529c36dc34cc7b87f8942db94f4df9c33831d7e9adac6d78c6205b96a5c08d20
+  data.tar.gz: 25bb9acef445ba184adeccb94da81252c78ccff57f654e18b37b26c85053a0bf
 SHA512:
-  metadata.gz: 47903715b908a7d5e55837dc94edacf50f91399a74b2ec12174c22015657864624436cc59b5d9ec93de349d1dd17da45edda3f879a8622c320190568335d4f85
-  data.tar.gz: 0ef569d807b305ff9c3f1598e0c705ae72559aefd3efd644c90dc8f749db389d05861a698b39dd9ead5fe95652b5350e7519c25e7b17fc622422d91900cb43f9
+  metadata.gz: f5c337f5032e5e700c4fbd96f490a99996f7f7dff394187424e231f6ec96b5c69c66d9d9a98f52a39589acadb8dbf3cea24cd988554ba39bd8b44c5c56bc0ead
+  data.tar.gz: 65f2840f521fd8f870e14b082b66e7aba65c1b0b6520637018f9de0205d706f31985d40443ae23d24791839bd53b731558603342b03489571b07add6080e5e30

data/README.md

@@ -240,9 +240,91 @@ MyBrand
 SpecialTerm
 ```
 
+## Dictionary Packs
+
+SpellKit still bundles no dictionaries. A **pack** is a separate gem that carries the data
+and registers itself when it loads, so you choose packs in your Gemfile:
+
+```ruby
+gem "spellkit"
+gem "spellkit-general-medical"   # drug, condition and gene vocabulary + general English
+```
+
+```ruby
+SpellKit.enable_dictionary(:general_medical, lazy: true)
+SpellKit.correct("acetaminphen")   # => "acetaminophen"
+```
+
+A pack ships the tuning that was measured against its own data, so you don't have to
+rediscover `edit_distance` and `frequency_threshold` yourself. Override anything:
+
+```ruby
+SpellKit.enable_dictionary(:general_medical, edit_distance: 1)
+```
+
+SpellKit knows the name of no pack — it provides only the mechanism, so a pack can ship
+new terms or new tuning without SpellKit releasing anything.
+
+### If you load this from a Rails initializer, read this
+
+**An initializer runs in every process that boots the app** — web, `rails db:migrate`,
+`rake`, `console`, sidecars — not just the one that searches. Loading a pack eagerly makes
+all of them pay, and a ~200k-term pack at `edit_distance: 2` measures **~2.1 GB resident
+and ~5.5s**. That has already OOM-killed a memory-constrained migrate init container in
+production.
+
+Pass `lazy: true`. Registration then costs nothing and the index is built on first real
+use, which a migrate or rake process never reaches:
+
+| | time | RSS |
+|---|---|---|
+| `enable_dictionary(..., lazy: true)` | 0.000s | +0 MB |
+| first `correct()` after that | ~5.5s | +2,076 MB |
+| every later call | ~0.0001s | +0 MB |
+
+**Lazy moves the cost, it doesn't remove it.** For a web server you usually want the
+server to pay it rather than the first user, so warm it at worker boot:
+
+```ruby
+# config/puma.rb
+on_worker_boot { SpellKit.load_dictionary! }
+```
+
+`stats` and `healthcheck` deliberately **do not** trigger the load — they report
+`{"loaded" => false, "deferred" => true}`. A liveness probe must not be able to build the
+index in the very process `lazy` protects.
+
+```ruby
+SpellKit.dictionary_loaded?   # false until something forces the load
+SpellKit.load_dictionary!     # force it now; idempotent, no-op when eager
+```
+
+Memory scales steeply with `edit_distance`: the same pack is **484 MB at 1** versus
+**2.1 GB at 2**, because SymSpell's deletion index grows sharply with distance. If you are
+memory-constrained, that knob matters more than lazy loading does.
+
+### Writing a pack gem
+
+A pack gem ships its data and one registration call:
+
+```ruby
+# lib/spellkit-my-domain.rb
+require "spellkit"
+
+SpellKit::Packs.register(:my_domain,
+  dictionary: File.expand_path("../data/dictionary.tsv", __dir__),
+  protected_path: File.expand_path("../data/protected.txt", __dir__),
+  defaults: {edit_distance: 1, frequency_threshold: 10.0},
+  summary: "What this pack covers")
+```
+
+Bundler requires it automatically, so `SpellKit.enable_dictionary(:my_domain)` then works.
+Registration verifies the files exist, so a packaging mistake fails while the stack still
+points at your gem.
+
 ## Dictionary Sources
 
-SpellKit doesn't bundle dictionaries, but works with several sources:
+SpellKit works with several raw dictionary sources directly:
 
 ### Use the Default Dictionary (Recommended)
 ```ruby

data/lib/spellkit/lazy_checker.rb

@@ -0,0 +1,91 @@
+# frozen_string_literal: true
+
+module SpellKit
+  # A checker that has not built its index yet.
+  #
+  # WHY. A Rails initializer runs in EVERY process that boots the app - web, db:migrate,
+  # rake, console, sidecars - but usually only the web server ever spell-checks anything.
+  # Loading eagerly makes all of them pay, and the cost is not small: a ~200k-term pack at
+  # edit_distance 2 measures ~2.1 GB resident and several seconds to index. That is not
+  # hypothetical - it OOM-killed a memory-constrained migrate init container in production.
+  #
+  # WHAT DOES NOT TRIGGER A LOAD. `stats` and `healthcheck` deliberately do not, and that
+  # matters more than it first appears: a liveness probe hitting a health endpoint would
+  # otherwise build the whole index in exactly the process this class exists to protect,
+  # silently reintroducing the bug. They report the deferred state instead.
+  #
+  # THREAD SAFETY. A threaded server can take two concurrent first requests; without the
+  # mutex both would build a multi-gigabyte index. The build happens at most once.
+  class LazyChecker
+    # Lookups need a real index. Introspection must not build one.
+    FORCES_LOAD = %i[correct correct? suggestions correct_tokens].freeze
+
+    attr_reader :pack_name
+
+    def initialize(pack_name, options)
+      @pack_name = pack_name
+      @options = options
+      @mutex = Mutex.new
+      @checker = nil
+    end
+
+    def loaded?
+      !@checker.nil?
+    end
+
+    # Build the index now. Idempotent and thread-safe.
+    #
+    # Lazy loading MOVES the cost rather than removing it, so for a web process you
+    # usually want the server to pay it instead of the first user:
+    #
+    #   # config/puma.rb
+    #   on_worker_boot { SpellKit.load_dictionary! }
+    def load_now!
+      return @checker if @checker
+
+      @mutex.synchronize do
+        # Re-check inside the lock; another thread may have loaded while we waited.
+        @checker ||= Checker.new.load!(**resolved_options)
+      end
+    end
+
+    FORCES_LOAD.each do |name|
+      define_method(name) { |*args| load_now!.public_send(name, *args) }
+    end
+
+    def stats
+      loaded? ? @checker.stats : deferred_report
+    end
+
+    def healthcheck
+      loaded? ? @checker.healthcheck : deferred_report
+    end
+
+    def method_missing(name, *args, &block)
+      if Checker.method_defined?(name)
+        load_now!.public_send(name, *args, &block)
+      else
+        super
+      end
+    end
+
+    def respond_to_missing?(name, include_private = false)
+      Checker.method_defined?(name) || super
+    end
+
+    private
+
+    # Resolved here rather than through a module-level helper so that helper can stay
+    # private: 1.0.0 freezes the public surface, and an internal seam that only exists to
+    # let one collaborator reach a private method should not be part of it.
+    def resolved_options
+      return @options if @pack_name.nil?
+
+      Packs.fetch(@pack_name).load_options(**@options)
+    end
+
+    def deferred_report
+      {"loaded" => false, "deferred" => true, "pack" => @pack_name&.to_s}
+    end
+  end
+end

data/lib/spellkit/packs.rb

@@ -0,0 +1,107 @@
+# frozen_string_literal: true
+
+module SpellKit
+  # A dictionary pack that some other gem has registered.
+  #
+  # SpellKit ships NO packs and knows the name of none. It provides the mechanism; a
+  # separate data gem (spellkit-general-medical, say) requires itself, calls
+  # SpellKit::Packs.register, and supplies both the files and the tuning that was measured
+  # against them. That keeps the "SpellKit doesn't bundle dictionaries" promise in the
+  # README literally true while still letting `SpellKit.enable_dictionary(:some_pack)`
+  # work, and it means a pack can ship a new version - new terms, new tuning - without
+  # SpellKit releasing anything at all.
+  Pack = Struct.new(:name, :dictionary, :protected_path, :defaults, :summary, keyword_init: true) do
+    # The keyword arguments to hand to load!, with caller overrides applied last.
+    def load_options(**overrides)
+      options = {dictionary: dictionary.to_s}
+      options[:protected_path] = protected_path.to_s if protected_path
+      defaults.merge(options).merge(overrides)
+    end
+  end
+
+  # The registry of packs some installed gem has announced.
+  module Packs
+    class << self
+      def registry
+        @registry ||= {}
+      end
+
+      # Called by a data gem when it loads.
+      #
+      #   SpellKit::Packs.register(:general_medical,
+      #     dictionary: "#{__dir__}/../data/dictionary.tsv",
+      #     protected_path: "#{__dir__}/../data/protected.txt",
+      #     defaults: {edit_distance: 2, frequency_threshold: 1.0})
+      #
+      # `defaults` is how a pack ships the tuning that was measured against ITS data,
+      # rather than leaving every consumer to rediscover it.
+      def register(name, dictionary:, protected_path: nil, defaults: {}, summary: nil)
+        key = normalize(name)
+
+        # Checked here, at require time, rather than on first use: a data gem whose files
+        # did not survive packaging is broken, and saying so while the stack still points
+        # at that gem beats an inexplicable failure during someone's first search. This is
+        # a stat, so it costs nothing and does not defeat lazy loading, which exists to
+        # defer the index BUILD.
+        unless File.exist?(dictionary.to_s)
+          raise FileNotFoundError,
+            "Pack #{key.inspect} registered a dictionary that does not exist: #{dictionary}"
+        end
+
+        if protected_path && !File.exist?(protected_path.to_s)
+          raise FileNotFoundError,
+            "Pack #{key.inspect} registered a protected-terms file that does not exist: #{protected_path}"
+        end
+
+        registry[key] = Pack.new(
+          name: key, dictionary: dictionary, protected_path: protected_path,
+          defaults: symbolize(defaults), summary: summary
+        )
+      end
+
+      def registered?(name)
+        registry.key?(normalize(name))
+      end
+
+      # [:general_medical, ...]
+      def names
+        registry.keys.map(&:to_sym)
+      end
+
+      def fetch(name)
+        registry.fetch(normalize(name)) do
+          raise UnknownPackError, unknown_message(name)
+        end
+      end
+
+      # Test hook.
+      def reset!
+        @registry = {}
+      end
+
+      # :general_medical, "general_medical" and "general-medical" all address one pack.
+      def normalize(name)
+        name.to_s.strip.downcase.tr("-", "_")
+      end
+
+      private
+
+      def unknown_message(name)
+        if registry.empty?
+          "No dictionary packs are registered. A pack lives in its own gem - add one to " \
+          "your Gemfile (e.g. gem \"spellkit-general-medical\") and it registers itself " \
+          "when it loads. To use your own files instead, pass them directly: " \
+          "SpellKit.enable_dictionary(dictionary: \"path/to.tsv\")."
+        else
+          "Unknown dictionary pack #{name.inspect}. Registered: #{names.inspect}. A pack " \
+          "registers itself when its gem loads, so a missing one usually means the gem is " \
+          "not in your Gemfile."
+        end
+      end
+
+      def symbolize(hash)
+        (hash || {}).each_with_object({}) { |(key, value), acc| acc[key.to_sym] = value }
+      end
+    end
+  end
+end

data/lib/spellkit/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SpellKit
-  VERSION = "0.3.0"
+  VERSION = "1.0.0"
 end

data/lib/spellkit.rb

@@ -17,6 +17,9 @@ rescue LoadError
   require "spellkit/spellkit"
 end
 
+require_relative "spellkit/packs"
+require_relative "spellkit/lazy_checker"
+
 module SpellKit
   class Error < StandardError; end
   class NotLoadedError < Error; end
@@ -24,6 +27,10 @@ module SpellKit
   class InvalidArgumentError < Error; end
   class DownloadError < Error; end
 
+  # Raised when a pack name is not registered - almost always a data gem missing from
+  # the Gemfile, since a pack registers itself when its gem loads.
+  class UnknownPackError < Error; end
+
   # Default dictionary: SymSpell English 80k frequency dictionary
   DEFAULT_DICTIONARY_URL = "https://raw.githubusercontent.com/wolfgarbe/SymSpell/master/SymSpell.FrequencyDictionary/en-80k.txt"
 
@@ -98,6 +105,83 @@ module SpellKit
     def healthcheck
       default.healthcheck
     end
+
+    # ----------------------------------------------------------------------------------
+    # Dictionary packs
+    #
+    # SpellKit still bundles no dictionaries. A pack lives in its own gem, registers
+    # itself on load (see SpellKit::Packs), and brings the tuning that was measured
+    # against its own data. These methods are the mechanism only - SpellKit knows the
+    # name of no pack.
+    # ----------------------------------------------------------------------------------
+
+    # Configure the DEFAULT checker from a registered pack, so plain SpellKit.correct
+    # becomes domain-aware.
+    #
+    #   SpellKit.enable_dictionary(:general_medical)
+    #   SpellKit.enable_dictionary(:general_medical, lazy: true)     # defer the index build
+    #   SpellKit.enable_dictionary(:general_medical, edit_distance: 1)
+    #   SpellKit.enable_dictionary(dictionary: "my.tsv")             # your own files
+    #
+    # `lazy: true` is strongly recommended from a Rails initializer - see LazyChecker for
+    # why. Eager stays the default so this is non-breaking.
+    def enable_dictionary(pack = nil, lazy: false, **options)
+      # Resolve eagerly even when lazy, so an unregistered pack (usually a data gem
+      # missing from the Gemfile) fails at boot rather than on a user's first search.
+      @dictionary_pack = pack.nil? ? nil : Packs.fetch(pack)
+
+      self.default = if lazy
+        LazyChecker.new(pack, options)
+      else
+        load!(**pack_load_options(pack, **options))
+      end
+    end
+
+    # An INDEPENDENT checker, for running more than one pack at once. Does not touch the
+    # default checker.
+    def dictionary_checker(pack = nil, lazy: false, **options)
+      Packs.fetch(pack) unless pack.nil?
+      return LazyChecker.new(pack, options) if lazy
+
+      Checker.new.load!(**pack_load_options(pack, **options))
+    end
+
+    # True once the default checker holds a real index.
+    def dictionary_loaded?
+      checker = @default
+      return false if checker.nil?
+      return checker.loaded? if checker.is_a?(LazyChecker)
+
+      true
+    end
+
+    # Force a deferred load now; no-op when already loaded or configured eagerly.
+    def load_dictionary!
+      checker = @default
+      checker.load_now! if checker.is_a?(LazyChecker)
+      checker
+    end
+
+    # The pack backing the default checker, or nil when configured from raw files.
+    attr_reader :dictionary_pack
+
+    private
+
+    # Internal: resolve a pack name (or a bare options hash) into load! keyword arguments.
+    # Private on purpose - see LazyChecker#resolved_options.
+    def pack_load_options(pack = nil, **overrides)
+      if pack.nil?
+        unless overrides.key?(:dictionary)
+          raise InvalidArgumentError,
+            "Pass a registered pack name or a dictionary: of your own. " \
+            "Registered packs: #{Packs.names.inspect}"
+        end
+
+        return overrides
+      end
+
+      Packs.fetch(pack).load_options(**overrides)
+    end
   end
 end
 
@@ -113,10 +197,9 @@ class SpellKit::Checker
   alias_method :_rust_healthcheck, :healthcheck
 
   def load!(dictionary: nil, protected_path: nil, protected_patterns: [],
-            edit_distance: 1, frequency_threshold: 10.0,
-            skip_urls: false, skip_emails: false, skip_hostnames: false,
-            skip_code_patterns: false, skip_numbers: false, **_options)
-
+    edit_distance: 1, frequency_threshold: 10.0,
+    skip_urls: false, skip_emails: false, skip_hostnames: false,
+    skip_code_patterns: false, skip_numbers: false, **_options)
     # Validate dictionary parameter
     raise SpellKit::InvalidArgumentError, "dictionary parameter is required" if dictionary.nil?
 
@@ -320,7 +403,7 @@ class SpellKit::Checker
     raise SpellKit::InvalidArgumentError, "Invalid URL: #{url} (#{e.message})"
   rescue Timeout::Error => e
     raise SpellKit::DownloadError, "Download timed out: #{url} (#{e.message})"
-  rescue StandardError => e
+  rescue => e
     raise SpellKit::DownloadError, "Failed to download dictionary: #{e.message}"
   end
 
@@ -356,13 +439,13 @@ class SpellKit::Checker
         raise SpellKit::DownloadError, "HTTP #{response.code}: #{response.message} (#{url})"
       end
     end
-  rescue Net::OpenTimeout => e
+  rescue Net::OpenTimeout
     raise Timeout::Error, "Connection timeout after #{open_timeout}s: #{url}"
-  rescue Net::ReadTimeout => e
+  rescue Net::ReadTimeout
     raise Timeout::Error, "Read timeout after #{read_timeout}s: #{url}"
   rescue SocketError => e
     raise SpellKit::DownloadError, "Network error: #{e.message} (#{url})"
   rescue OpenSSL::SSL::SSLError => e
     raise SpellKit::DownloadError, "SSL verification failed: #{e.message} (#{url})"
   end
-end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: spellkit
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 1.0.0
 platform: ruby
 authors:
 - Chris Petersen
@@ -179,6 +179,8 @@ files:
 - ext/spellkit/src/lib.rs
 - ext/spellkit/src/symspell.rs
 - lib/spellkit.rb
+- lib/spellkit/lazy_checker.rb
+- lib/spellkit/packs.rb
 - lib/spellkit/version.rb
 homepage: https://github.com/scientist-labs/spellkit
 licenses: