did_you_mean 1.2.1 → 1.5.0

data/documentation/human_typo_api.md

@@ -0,0 +1,20 @@
+# HumanTypo API
+## Description
+Simulate an error prone human typist.  Assumes typographical errors are Poisson distributed and
+each error is either a deletion, insertion, substitution, or transposition
+## Initialization
+```
+def initialize(input, lambda: 0.05)
+end
+```
+where
+### input: A string with the word to be corrupted.
+## lambda: Error rate of the poisson process
+The default of 0.05 corresponds to one error every 20 characters, and is thought to approximate the average, competent typist
+## Methods
+```
+def call
+end
+```
+Returns a word with typographical errors.
+

data/documentation/tree_spell_algorithm.md

@@ -0,0 +1,82 @@
+# TreeSpellChecker Algorithm
+## Overview
+The algorithm is designed to work on a dictionary that has a rooted tree structure.
+
+The algorithm treats the problem as a hidden state system, which tries to identify the true state of the input. Due to typographical errors, the state of the input is a hidden version of the true system state.  Each word in the dictionary is mapped to a multi-dimensional state, with the first dimension being being the root, the second dimension being the next branch, and so on.  Each dimension is discrete with a finite number of elements.  The first dimension corresponds to the root, so only has one element.
+
+The algorithm assumes the input state has the correct structure, and so generates the state of the input.  It starts with the root of the input word and maps it to the root element. It then looks at the value of second dimension of the input word and chooses closest elements of the possible second dimension elements.  It then continues to the third and higher dimensions. It terminates when it has worked out possible elements corresponding to the highest dimension of the input word.  At this point it has the possible elements at for each dimension of the input word. It then generates all possible legitimate states from these elements.  Finally it then compares the possible leaves at the end of these legitimate states with the leaf of the input word. From this process it produces suggested states.
+
+## Accuracy
+The accuracy of the algorithm was tested using the HumanTypo class. It simulates a human typist by assuming that errors are Poisson distributed at a rate of one typo per 20 characters. Typos can be either a deletion, an insertion, substitution or a transposition.
+
+I ran 10,000 repititions on both the `test` directory of the ```did_you_mean``` gem and on the ```spec``` directory of the ```rspec-core``` gem.  
+
+The results were as follows:
+```
+                               Minitest Summary                                
+--------------------------------------------------------------------------------
+ Method  |   First Time (%)    Mean Suggestions       Failures (%)              
+--------------------------------------------------------------------------------
+ Tree                98.0                1.1                 2.0                 
+ Standard            98.1                2.2                 1.6                 
+ Augmented           100.0                1.1                 0.0  
+```
+and
+```
+                               Rspec Summary                                
+--------------------------------------------------------------------------------
+ Method  |   First Time (%)    Mean Suggestions       Failures (%)              
+--------------------------------------------------------------------------------
+ Tree                94.7                1.0                 5.3                 
+ Standard            98.2                4.2                 1.1                 
+ Augmented           99.7                1.2                 0.2   
+```
+As well, I checked the results on the ```test``` directory with ```HumanTypo``` generating errors at three times the rate:
+```
+                Minitest Summary  (lambda = 0.15)                             
+--------------------------------------------------------------------------------
+ Method  |   First Time (%)    Mean Suggestions       Failures (%)              
+--------------------------------------------------------------------------------
+ Tree               88.9                1.0                 11.0                 
+ Standard           95.0                1.4                 4.3                 
+ Combined           99.0                1.0                 0.8                 
+```
+In all cases, the tree speller, when augmented by the standard spell checker performed with higher accuracy, and giving far fewer suggestions.
+
+## Execution Speed
+
+I tested the execution time on the ```test ``` directory:
+```
+Testing execution time of Standard
+Average time (ms): 5.2
+
+Testing execution time of Tree
+Average time (ms): 1.1
+
+Testing execution time of Augmented Tree
+Average time (ms): 1.2
+```
+and on the ```spec``` directory
+```
+
+Testing execution time of Standard
+Average time (ms): 40.6
+
+Testing execution time of Tree
+Average time (ms): 2.7
+
+Testing execution time of Augmented Tree
+Average time (ms): 4.5
+```
+
+I was surprised by how much faster the tree checker was compared to the standard checker. I think the reason is that the predominant computational load will scale with O(log n) where n is the total number of words in the dictionary. My reasoning is that the algorithm very quickly prunes out states as it moves through the dimensions.
+## Augmentation option
+Given the major difference in speed between the standard and tree checker, and the likelihood that the disparity will grow rapidly with the size of the dictionary, then I suspect for some applications, it will not be not practicable to augment the tree checker by using the standard checker when the tree checker fails to find a suggestion.  Accordingly, I have added an option, ```:augment?```. The default is nil, but if true, then the standard checker is used if there are no suggestions.
+## Generation of Performance data
+This is done using ```test/tree_spell/explore_test.rb```. This is not a proper test file in that there are no assertions in it. As well, it takes over ten minutes to run, accordingly, I have disabled it by setting the constant TREE_SPELL_EXPLORE to false at the top of the file.  To run the file, set TREE_SPELL_EXPLORE to true.  It is also possible to run quick assessments by using a smaller value of n_repeat in the various tests.
+## Future Work
+I have identified two categories of remaining errors. The first class is when one of the elements is corrupted to being very small. Then the standard checker does not suggest the correct element, e.g. if an element is ```core``` and it is reduced to ```co```, the standard speller will not make a suggestion.  The second class of error is when the structure of the word is broken because one of the separators has been removed.  I think it might be possible to remove the first type of error and dramatically reduce the second type of error in a future version. This would be done as follows:
+- At each level, choose the dictionary element with the smallest distance.
+- This will work well unless the structure is broken, in which case it could return a wildly wrong suggestion.
+- To guard against this, the suggestion could be checked against the input word using the standard checker. If the standard checker rejects the suggestion, then it is assumed the structure is broken.
+- A large proportion of the time, a broken structure will be just due to one separator missing, and the order of the elements will not be affected. Accordingly, the structure can be fixed by comparing the input elements with a concatenation of two levels of the dictionary elements.  It would be possible to use the same idea to fix more than one separator missing, but this could quickly become computationally expensive.

data/documentation/tree_spell_checker_api.md

@@ -0,0 +1,24 @@
+# TreeSpellChecker API
+## Description
+
+## Initialization
+```
+def initialize(dictionary:, separator: '/', augment: nil)
+end
+```
+where
+####dictionary: The dictionary is a list of possible words
+    * that are used to correct a misspelling
+    * The dictionary must be tree structured with a single character separator
+    * e.g 'spec/models/goals_spec_rb'.
+####separator: A single charactor.  Cannot be cannot be alphabetical, '@' or '.'.
+####augment: When set to true, the checker will used the standard ```SpellChecker``` to find possible suggestions.
+## Methods
+```
+def correct(input)
+end
+```
+where
+####input: Is the input word to be corrected.
+
+

data/lib/did_you_mean.rb

@@ -1,69 +1,112 @@
-require "did_you_mean/version"
-require "did_you_mean/core_ext/name_error"
+require_relative "did_you_mean/version"
+require_relative "did_you_mean/core_ext/name_error"
 
-require "did_you_mean/spell_checker"
-require 'did_you_mean/spell_checkers/name_error_checkers'
-require 'did_you_mean/spell_checkers/method_name_checker'
-require 'did_you_mean/spell_checkers/key_error_checker'
-require 'did_you_mean/spell_checkers/null_checker'
-
-require "did_you_mean/formatters/plain_formatter"
+require_relative "did_you_mean/spell_checker"
+require_relative 'did_you_mean/spell_checkers/name_error_checkers'
+require_relative 'did_you_mean/spell_checkers/method_name_checker'
+require_relative 'did_you_mean/spell_checkers/key_error_checker'
+require_relative 'did_you_mean/spell_checkers/null_checker'
+require_relative 'did_you_mean/spell_checkers/require_path_checker'
+require_relative 'did_you_mean/formatters/plain_formatter'
+require_relative 'did_you_mean/tree_spell_checker'
 
+# The +DidYouMean+ gem adds functionality to suggest possible method/class
+# names upon errors such as +NameError+ and +NoMethodError+. In Ruby 2.3 or
+# later, it is automatically activated during startup.
+#
+# @example
+#
+#   methosd
+#   # => NameError: undefined local variable or method `methosd' for main:Object
+#   #   Did you mean?  methods
+#   #                  method
+#
+#   OBject
+#   # => NameError: uninitialized constant OBject
+#   #    Did you mean?  Object
+#
+#   @full_name = "Yuki Nishijima"
+#   first_name, last_name = full_name.split(" ")
+#   # => NameError: undefined local variable or method `full_name' for main:Object
+#   #    Did you mean?  @full_name
+#
+#   @@full_name = "Yuki Nishijima"
+#   @@full_anme
+#   # => NameError: uninitialized class variable @@full_anme in Object
+#   #    Did you mean?  @@full_name
+#
+#   full_name = "Yuki Nishijima"
+#   full_name.starts_with?("Y")
+#   # => NoMethodError: undefined method `starts_with?' for "Yuki Nishijima":String
+#   #    Did you mean?  start_with?
+#
+#   hash = {foo: 1, bar: 2, baz: 3}
+#   hash.fetch(:fooo)
+#   # => KeyError: key not found: :fooo
+#   #    Did you mean?  :foo
+#
+#
+# == Disabling +did_you_mean+
+#
+# Occasionally, you may want to disable the +did_you_mean+ gem for e.g.
+# debugging issues in the error object itself. You can disable it entirely by
+# specifying +--disable-did_you_mean+ option to the +ruby+ command:
+#
+#   $ ruby --disable-did_you_mean -e "1.zeor?"
+#   -e:1:in `<main>': undefined method `zeor?' for 1:Integer (NameError)
+#
+# When you do not have direct access to the +ruby+ command (e.g.
+# +rails console+, +irb+), you could applyoptions using the +RUBYOPT+
+# environment variable:
+#
+#   $ RUBYOPT='--disable-did_you_mean' irb
+#   irb:0> 1.zeor?
+#   # => NoMethodError (undefined method `zeor?' for 1:Integer)
+#
+#
+# == Getting the original error message
+#
+# Sometimes, you do not want to disable the gem entirely, but need to get the
+# original error message without suggestions (e.g. testing). In this case, you
+# could use the +#original_message+ method on the error object:
+#
+#   no_method_error = begin
+#                       1.zeor?
+#                     rescue NoMethodError => error
+#                       error
+#                     end
+#
+#   no_method_error.message
+#   # => NoMethodError (undefined method `zeor?' for 1:Integer)
+#   #    Did you mean?  zero?
+#
+#   no_method_error.original_message
+#   # => NoMethodError (undefined method `zeor?' for 1:Integer)
+#
 module DidYouMean
-  class DeprecatedIgnoredCallers < Array
-    %i(
-      +
-      <<
-      []=
-      insert
-      unshift
-      push
-    ).each do |method_name|
-      eval <<-RUBY, nil, __FILE__, __LINE__ + 1
-        def #{method_name}(*)
-          warn "IGNORED_CALLERS has been deprecated and has no effect."
+  # Map of error types and spell checker objects.
+  SPELL_CHECKERS = Hash.new(NullChecker)
 
-          super
-        end
-      RUBY
-    end
+  # Adds +DidYouMean+ functionality to an error using a given spell checker
+  def self.correct_error(error_class, spell_checker)
+    SPELL_CHECKERS[error_class.name] = spell_checker
+    error_class.prepend(Correctable) unless error_class < Correctable
   end
 
-  IGNORED_CALLERS = DeprecatedIgnoredCallers.new
-
-  SPELL_CHECKERS = Hash.new(NullChecker)
-  SPELL_CHECKERS.merge!({
-    "NameError"     => NameErrorCheckers,
-    "NoMethodError" => MethodNameChecker,
-    "KeyError"      => KeyErrorChecker
-  })
-
-  NameError.prepend DidYouMean::Correctable
-  KeyError.prepend DidYouMean::Correctable
+  correct_error NameError, NameErrorCheckers
+  correct_error KeyError, KeyErrorChecker
+  correct_error NoMethodError, MethodNameChecker
+  correct_error LoadError, RequirePathChecker if RUBY_VERSION >= '2.8.0'
 
+  # Returns the currently set formatter. By default, it is set to +DidYouMean::Formatter+.
   def self.formatter
     @@formatter
   end
 
+  # Updates the primary formatter used to format the suggestions.
   def self.formatter=(formatter)
     @@formatter = formatter
   end
 
   self.formatter = PlainFormatter.new
-
-  # Deprecated formatter
-  class Formatter #:nodoc:
-    def initialize(corrections = [])
-      @corrections = corrections
-    end
-
-    def to_s
-      return "" if @corrections.empty?
-
-      output = "\nDid you mean?  ".dup
-      output << @corrections.join("\n               ")
-    end
-  end
-
-  deprecate_constant :Formatter
 end

data/lib/did_you_mean/core_ext/name_error.rb

@@ -6,22 +6,20 @@ module DidYouMean
 
     def to_s
       msg = super.dup
+      suggestion = DidYouMean.formatter.message_for(corrections)
 
-      if !cause.respond_to?(:corrections) || cause.corrections.empty?
-        msg << DidYouMean.formatter.message_for(corrections)
-      end
-
+      msg << suggestion if !msg.end_with?(suggestion)
       msg
     rescue
       super
     end
 
     def corrections
-      spell_checker.corrections
+      @corrections ||= spell_checker.corrections
     end
 
     def spell_checker
-      @spell_checker ||= SPELL_CHECKERS[self.class.to_s].new(self)
+      SPELL_CHECKERS[self.class.to_s].new(self)
     end
   end
 end

data/lib/did_you_mean/experimental.rb

@@ -1,2 +1,2 @@
-require 'did_you_mean/experimental/initializer_name_correction'
-require 'did_you_mean/experimental/ivar_name_correction'
+warn "Experimental features in the did_you_mean gem has been removed " \
+     "and `require \"did_you_mean/experimental\"' has no effect."

data/lib/did_you_mean/formatters/plain_formatter.rb

@@ -1,7 +1,31 @@
 # frozen-string-literal: true
 
 module DidYouMean
+  # The +DidYouMean::PlainFormatter+ is the basic, default formatter for the
+  # gem. The formatter responds to the +message_for+ method and it returns a
+  # human readable string.
   class PlainFormatter
+
+    # Returns a human readable string that contains +corrections+. This
+    # formatter is designed to be less verbose to not take too much screen
+    # space while being helpful enough to the user.
+    #
+    # @example
+    #
+    #   formatter = DidYouMean::PlainFormatter.new
+    #
+    #   # displays suggestions in two lines with the leading empty line
+    #   puts formatter.message_for(["methods", "method"])
+    #
+    #   Did you mean?  methods
+    #                   method
+    #   # => nil
+    #
+    #   # displays an empty line
+    #   puts formatter.message_for([])
+    #
+    #   # => nil
+    #
     def message_for(corrections)
       corrections.empty? ? "" : "\nDid you mean?  #{corrections.join("\n               ")}"
     end

data/lib/did_you_mean/formatters/verbose_formatter.rb

@@ -1,7 +1,43 @@
 # frozen-string-literal: true
 
 module DidYouMean
+  # The +DidYouMean::VerboseFormatter+ uses extra empty lines to make the
+  # suggestion stand out more in the error message.
+  #
+  # In order to activate the verbose formatter,
+  #
+  # @example
+  #
+  #   OBject
+  #   # => NameError: uninitialized constant OBject
+  #   #    Did you mean?  Object
+  #
+  #   require 'did_you_mean/verbose'
+  #
+  #   OBject
+  #   # => NameError: uninitialized constant OBject
+  #   #
+  #   #        Did you mean? Object
+  #   #
+  #
   class VerboseFormatter
+
+    # Returns a human readable string that contains +corrections+. This
+    # formatter is designed to be less verbose to not take too much screen
+    # space while being helpful enough to the user.
+    #
+    # @example
+    #
+    #   formatter = DidYouMean::PlainFormatter.new
+    #
+    #   puts formatter.message_for(["methods", "method"])
+    #
+    #
+    #       Did you mean? methods
+    #                     method
+    #
+    #   # => nil
+    #
     def message_for(corrections)
       return "" if corrections.empty?
 

data/lib/did_you_mean/levenshtein.rb

@@ -41,7 +41,7 @@ module DidYouMean
 
     # detects the minimum value out of three arguments. This method is
     # faster than `[a, b, c].min` and puts less GC pressure.
-    # See https://github.com/yuki24/did_you_mean/pull/1 for a performance
+    # See https://github.com/ruby/did_you_mean/pull/1 for a performance
     # benchmark.
     def min3(a, b, c)
       if a < b && a < c

data/lib/did_you_mean/spell_checker.rb

@@ -1,11 +1,11 @@
 # frozen-string-literal: true
 
-require "did_you_mean/levenshtein"
-require "did_you_mean/jaro_winkler"
+require_relative "levenshtein"
+require_relative "jaro_winkler"
 
 module DidYouMean
   class SpellChecker
-    def initialize(dictionary: )
+    def initialize(dictionary:)
       @dictionary = dictionary
     end
 
@@ -13,14 +13,14 @@ module DidYouMean
       input     = normalize(input)
       threshold = input.length > 3 ? 0.834 : 0.77
 
-      words = @dictionary.select {|word| JaroWinkler.distance(normalize(word), input) >= threshold }
-      words.reject! {|word| input == word.to_s }
-      words.sort_by! {|word| JaroWinkler.distance(word.to_s, input) }
+      words = @dictionary.select { |word| JaroWinkler.distance(normalize(word), input) >= threshold }
+      words.reject! { |word| input == word.to_s }
+      words.sort_by! { |word| JaroWinkler.distance(word.to_s, input) }
       words.reverse!
 
       # Correct mistypes
       threshold   = (input.length * 0.25).ceil
-      corrections = words.select {|c| Levenshtein.distance(normalize(c), input) <= threshold }
+      corrections = words.select { |c| Levenshtein.distance(normalize(c), input) <= threshold }
 
       # Correct misspells
       if corrections.empty?
@@ -38,13 +38,7 @@ module DidYouMean
     private
 
     def normalize(str_or_symbol) #:nodoc:
-      str = if str_or_symbol.is_a?(String)
-              str_or_symbol.dup
-            else
-              str_or_symbol.to_s
-            end
-
-      str.downcase!
+      str = str_or_symbol.to_s.downcase
       str.tr!("@", "")
       str
     end