spellr 0.4.1 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e259544899eb28d6a4b19e521eed16b12bb316488241d3a79af01c22873515f1
-  data.tar.gz: f519dcb4f8a434e1cc619f389c0719263d894ca44ac3996b07acef89afcef1c4
+  metadata.gz: b12a7a52fc68f4960a9156f4d500366225de366b3ca20213231a22176c898030
+  data.tar.gz: 816aafdee27ec4c997e96b3726f37f1e08dc22aaee9f29680aee9f86b8481693
 SHA512:
-  metadata.gz: eef1d7df649bf7f6b1a5b2c3e7cac423627b4aea17221e8ca5464c380d5de9ee7fba45183afc5373aeaa974272a7cd7127809935cdd3addae94d30c0d9f858d4
-  data.tar.gz: c314a2230af151f7098cc344d78b90f9b3c9696870b1056cc50ac3b5547cdbd367e62ebb98e2bdbd101f130881b80b4f050cd71ffc9db3c62fd16e074ac5c1e3
+  metadata.gz: 52e520722759e56c001ee0271408a7bb87e47cfb3ed1cba15ae1c208356a26cae1be01f05e72c316a6141e0e7e71aebe3878862de975cf12eedd77e64c2c1c3f
+  data.tar.gz: 3f11b304e48cc73823dd66b23daec1622ab1ee0e94545929a15d0baf80a4fc616a3dfa8f2ec9dda28e9792701d277fb0dc8e3cbbf673531dc8aca3fe9947ffd0

data/CHANGELOG.md

@@ -1,3 +1,8 @@
+# v0.5.0
+- Removed the fetch command it was just unnecessarily slow and awkward. instead use `locale: [US,AU]`
+- Added usage documentation
+- Fixed an issue where file-specific wordlists couldn't be added to
+
 # v0.4.1
 - fix the private method 'touch' issue when generating wordlists
 - fix the js/javascript defaults being named differently (now consistently is named javascript)

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    spellr (0.4.1)
+    spellr (0.5.0)
       fast_ignore (~> 0.4.0)
 
 GEM

data/README.md

@@ -4,29 +4,209 @@
 
 Spell check your source code for fun and occasionally finding bugs
 
-This is inspired by https://github.com/myint/scspell but i wanted a ruby gem
+This is inspired by https://github.com/myint/scspell, and uses wordlists from [SCOWL](http://wordlist.aspell.net) - see the license [here](https://github.com/robotdana/spellr/blob/master/wordlists/english.LICENSE.txt).
+
+## What makes a spell checker a source code spell checker?
+
+1. It tokenizes CamelCase and snake_case and kebab-case and checks these as independent words including CAMELCase with acronyms.
+2. It skips urls
+3. It skips things that heuristically look like base64 or hex strings rather than words. This uses a bayesian classifier and is not magic. Find the balance of false-positive to false-negative that works for you with the `key_heuristic_weight` [configuration](#configuration) option.
+4. It comes with some wordlists for built in commands in some common programming languages, and recognizes hashbangs.
+5. Configure whether you want US, AU, CA, or GB english (or all of them).
+6. It checks directories recursively, obeying .gitignore
+7. It's easy to add terms to wordlists
+8. It's easy to integrate with CI pipelines
+7. It's very configurable
+
+## A brief aside on "correct" spelling.
+
+There's no correct way to spell anything. You can't trust dictionaries, they only react to the way everyone else uses words. Any agreement about certain spellings is a collective hallucination, and is a terrible proxy for attention or intelligence or education or value. Those who get to declare what "correct" spelling is, or even what counts as a real word, tend to be those groups that have more social power and it's (sometimes unconsciously) used as a way to maintain that power.
+
+However, in a programming context spelling things _consistently_ is useful, where method definitions must match method calls, and comments about these are clearer when also matching. It also makes grepping easier, not that you'd find the word 'grepping' in most dictionaries.
 
 ## Installation
 
-Add this line to your application's Gemfile:
+### With Bundler
+
+Add this line to your application's `Gemfile`:
 
 ```ruby
 gem 'spellr', require: false
 ```
 
-And then execute:
+Then execute:
+
+```bash
+$ bundle install
+```
+
+### With Rubygems
 
-    $ bundle
+```bash
+$ gem install spellr
+```
 
-Or install it yourself as:
+### With Docker
 
-    $ gem install spellr
+execute this command instead of `spellr`. This is otherwise identical to using the gem version
+
+```bash
+$ docker run -it -v $PWD:/app robotdana/spellr
+```
 
 ## Usage
 
-To start an interactive spell checking session
+The main way to interact with `spellr` is through the executable.
+
+```bash
+$ spellr # will run the spell checker
+$ spellr --interactive # will run the spell checker, interactively
+$ spellr --wordlist # will output all words that fail the spell checker in spellr wordlist format
+$ spellr --quiet # will suppress all output
+```
+
+To check a single file or subset of files, just add paths or globs:
+```bash
+$ spellr --interactive path/to/my/file.txt and/another/file.sh
+$ spellr --wordlist '*.rb' '*_test.js'
+```
+
+There are some support commands available:
+
+```bash
+$ spellr --dry-run # list files that will be checked
+$ spellr --version # for the current version
+$ spellr --help # for the list of flags available
+```
+
+### First run
+
+Feel free to just `spellr --interactive` and go, but I prefer this process when first adding spellr to a large project.
+
+```bash
+$ spellr --dry-run
+```
+
+Look at the list of files, are there some that shouldn't be checked (generated files etc)? .gitignored files and some binary file extensions are already skipped by default.
+
+Add any additional files to ignore to a `.spellr.yml` file in your project root directory.
+```yml
+excludes:
+  - ignore
+  - /generated
+  - "!files"
+  - in/*
+  - .gitignore
+  - "*.format"
+```
+
+Then output the existing words that fail the default dictionaries.
+```bash
+$ spellr --wordlist > .spellr-wordlists/english.txt
+```
+
+Open `.spellr-wordlists/english.txt` and remove those lines that look like typos or mistakes, leaving the file in ascii order.
+
+Now it's time to run the interactive spell checker
+
+```bash
+$ spellr --interactive
+```
+
+### Interactive spell checking
+
+To start an interactive spell checking session:
+```bash
+$ spellr --interactive
+```
+
+You'll be shown each word that's not found in a dictionary, it's location (path:line:column), along with a prompt.
+```
+file.rb:1:0 notaword
+[r,R,s,S,a,e?]
+```
+
+Type `?` for this list of what each letter command does
+```
+[r] Replace notaword
+[R] Replace all future instances of notaword
+[s] Skip notaword
+[S] Skip all future instances of notaword
+[a] Add notaword to a word list
+[e] Edit the whole line
+[?] Show this help
+```
+
+If you type `r`, `R` or `e` you'll be shown a prompt with the original word and it prefilled ready for correcting:
+```
+file.txt:1:0 notaword
+>> notaword
+=> not_a_word
+```
+To submit your choice and continue with the spell checking click enter. Your replacement word will be immediately spellchecked. To instead go back press Ctrl-C once (pressing it twice will exit the spell checking).
+
+If you instead type `s` or `S` it will skip this word and continue with the spell checking.
+
+---
+
+If you instead type `a` you'll be shown a list of possible wordlists to add to. This list is based on the file path, and is configurable in `.spellr.yml`.
+```
+Add notaword to wordlist:
+[e] english
+[r] ruby
+```
+Type `e` to add this word to the english wordlist and continue on through the spell checking. To instead go back to the prompt press Ctrl-C once (pressing it twice will exit the spell checking).
+
+### Disabling the tokenizer
+
+If the tokenizer finds a word you don't want to add to the wordlist (perhaps it's an intentional example of a typo, or a non-word string not excluded by the heuristic) then place on the lines before and after
+```ruby
+# spellr:disable
+"Test typo of the: teh"
+# spellr:enable
+```
+
+## Configuration
+
+Spellr's configuration is a `.spellr.yml` file in your project root. This is combined with the gem defaults defined [here](https://github.com/robotdana/spellr/blob/master/lib/.spellr.yml).
+There are top-level keys and per-language keys.
+```yml
+word_minimum_length: 3 # any words shorter than this will be ignored
+key_minimum_length: 6 # any strings shorter than this won't be considered non-word strings
+key_heuristic_weight: 5 # higher values mean strings are more likely to be considered words or non-words by the classifier.
+excludes:
+  - ignore
+  - "!files"
+  - in/*
+  - .gitignore
+  - "*.format"
+includes:
+  - limit to
+  - "files*"
+  - in/*
+  - .gitignore-esque
+  - "*.format"
 ```
-spellr --interactive
+The includes format is documented [here](https://github.com/robotdana/fast_ignore#using-an-includes-list).
+
+Also within this file are language definitions:
+```yml
+languages:
+  english: # this must match exactly the name of the file in .spellr-wordlists/
+    locale: # US, AU, CA, or GB
+      - US
+      - AU
+  ruby:
+    includes:
+      - patterns*
+      - "*_here.rb"
+      - limit-which-files
+      - the/wordlist/**/*
+      - /applies_to/
+    key: r # this is the letter used to choose this wordlist when using `spellr --interactive`.
+    hashbangs:
+      - ruby # if the hashbang contains ruby, this file will match,
+             # even if it doesn't otherwise match the includes pattern.
 ```
 
 ## Development

data/lib/.spellr.yml

@@ -28,8 +28,7 @@ excludes: # this list is parsed with the .gitignore format
 
 languages:
   english:
-    generate: "fetch english"
-    # TODO: don't generate the ruby file until you actually need one
+    locale: US # options US, CA, AU, GBs (GB with -ise endings), GBz (GB with -ize endings)
   ruby:
     includes: # Filtered using gitignore format
       - '*.rb'

data/lib/spellr/cli.rb

@@ -7,8 +7,7 @@ require 'open3'
 require_relative '../spellr'
 
 module Spellr
-  class CLI # rubocop:disable Metrics/ClassLength
-    attr_writer :fetch_output_dir
+  class CLI
     attr_reader :argv
 
     def initialize(argv)
@@ -67,108 +66,13 @@ module Spellr
       exit
     end
 
-    def get_wordlist_option(command)
-      get_wordlist_dir.join(command)
-    end
-
-    def fetch_output_dir
-      @fetch_output_dir ||= Pathname.pwd.join('.spellr_wordlists/generated').expand_path
-    end
-
-    def fetch_words_for_wordlist(wordlist)
-      wordlist_command(wordlist, *argv)
-    end
-
-    def wordlist_command(wordlist, *args)
-      require 'shellwords'
-      command = fetch_wordlist_dir.join(wordlist).to_s
-      fetch_output_dir.mkpath
-
-      command_with_args = args.unshift(command).shelljoin
-
-      out, err, status = Open3.capture3(command_with_args)
-      puts err unless err.empty?
-      return out if status.exitstatus == 0
-
-      exit
-    end
-
-    def replace_wordlist(words, wordlist)
-      require_relative '../../lib/spellr/wordlist'
-
-      Spellr::Wordlist.new(fetch_output_dir.join("#{wordlist}.txt")).clean(StringIO.new(words))
-    end
-
-    def extract_and_write_license(words, wordlist)
-      words, license = words.split('---', 2).reverse
-
-      fetch_output_dir.join("#{wordlist}.LICENSE.txt").write(license) if license
-
-      words
-    end
-
-    def fetch
-      wordlist = argv.shift
-      puts "Fetching #{wordlist} wordlist"
-      words = fetch_words_for_wordlist(wordlist)
-      puts "Preparing #{wordlist} wordlist"
-      words = extract_and_write_license(words, wordlist)
-      puts "cleaning #{wordlist} wordlist"
-      replace_wordlist(words, wordlist)
-    end
-
-    def output_option(dir)
-      self.fetch_output_dir = Pathname.pwd.join(dir).expand_path
-    end
-
-    def wordlists
-      fetch_wordlist_dir.children.map { |p| p.basename.to_s }
-    end
-
-    def fetch_wordlist_dir
-      @fetch_wordlist_dir ||= Pathname.new(__dir__).parent.parent.join('bin', 'fetch_wordlist').expand_path
-    end
-
     def parse_command
-      case argv.first
-      when 'fetch'
-        parse_fetch_options
-        fetch
-      else
-        parse_options
-        check
-      end
-    end
-
-    def fetch_options
-      @fetch_options ||= begin
-        opts = OptionParser.new
-        opts.banner = "Usage: spellr fetch [options] WORDLIST [wordlist options]\nAvailable wordlists: #{wordlists}"
-
-        opts.separator('')
-        opts.on('-o', '--output=OUTPUT', 'Outputs the fetched wordlist to OUTPUT/WORDLIST.txt', &method(:output_option))
-        opts.on('-h', '--help', 'Shows help for fetch', &method(:fetch_options_help))
-
-        opts
-      end
-    end
-
-    def fetch_options_help(*_)
-      puts fetch_options.help
-
-      wordlist = argv.first
-      if wordlist
-        puts
-        wordlist_command('english', '--help')
-      end
-
-      exit
+      parse_options
+      check
     end
 
     def options_help(_)
       puts options.help
-      puts
-      puts fetch_options.help
 
       exit
     end
@@ -177,14 +81,6 @@ module Spellr
       options.parse!(argv)
     end
 
-    def parse_fetch_options
-      argv.shift
-      fetch_options.order!(argv) do |non_arg|
-        argv.unshift(non_arg)
-        break
-      end
-    end
-
     def options # rubocop:disable Metrics/MethodLength, Metrics/AbcSize
       @options ||= begin
         opts = OptionParser.new

data/lib/spellr/interactive.rb

@@ -50,7 +50,7 @@ module Spellr
     end
 
     def prompt(token)
-      print bold('[a,s,S,r,R,e,?]')
+      print bold('[r,R,s,S,a,e,?]')
 
       handle_response(token)
     rescue Interrupt
@@ -120,10 +120,10 @@ module Spellr
     # TODO: handle more than 16 options
     def handle_add(token) # rubocop:disable Metrics/AbcSize, Metrics/MethodLength
       puts "Add #{red(token)} to wordlist:"
-      languages = Spellr.config.languages_for(token.location.file)
+      languages = Spellr.config.languages_for(token.location.file.path)
 
-      languages.each do |wordlist|
-        puts "[#{wordlist.key}] #{wordlist.name}"
+      languages.each do |language|
+        puts "[#{language.key}] #{language.name}"
       end
       choice = STDIN.getch
       clear_current_line

data/lib/spellr/language.rb

@@ -13,7 +13,8 @@ module Spellr
       only: [],
       includes: [],
       description: '',
-      hashbangs: [])
+      hashbangs: [],
+      locale: [])
       unless only.empty?
         warn <<~WARNING
           \e[33mSpellr: `only:` language yaml key with a list of fnmatch rules is deprecated.
@@ -21,12 +22,20 @@ module Spellr
           see github.com/robotdana/fast_ignore#using-an-includes-list for details\e[0m
         WARNING
       end
+
+      if generate
+        warn <<~WARNING
+          \e[33mSpellr: `generate:` and generation is now deprecated. Choose the language
+          using the key `locale:` (any of US,AU,CA,GB,GBz,GBs as a string or array).\e[0m
+        WARNING
+      end
+
       @name = name
       @key = key
       @description = description
-      @generate = generate
       @includes = only + includes
       @hashbangs = hashbangs
+      @locales = Array(locale)
     end
 
     def matches?(file)
@@ -43,24 +52,9 @@ module Spellr
     end
 
     def wordlists
-      generate_wordlist unless generated_project_wordlist.exist?
       default_wordlists.select(&:exist?)
     end
 
-    def generate_wordlist
-      return [] unless generate
-
-      require_relative 'cli'
-      require 'shellwords'
-      warn "Generating wordlist for #{name}"
-
-      generated_project_wordlist.touch
-
-      Spellr::CLI.new(generate.shellsplit)
-
-      default_wordlists
-    end
-
     def gem_wordlist
       @gem_wordlist ||= Spellr::Wordlist.new(
         Pathname.new(__dir__).parent.parent.join('wordlists', "#{name}.txt")
@@ -74,17 +68,17 @@ module Spellr
       )
     end
 
-    def generated_project_wordlist
-      @generated_project_wordlist ||= Spellr::Wordlist.new(
-        Pathname.pwd.join('.spellr_wordlists', 'generated', "#{name}.txt")
-      )
+    def locale_wordlists
+      @locale_wordlists ||= @locales.map do |locale|
+        Spellr::Wordlist.new(
+          Pathname.new(__dir__).parent.parent.join('wordlists', name.to_s, "#{locale}.txt")
+        )
+      end
     end
 
     private
 
-    attr_reader :generate
-
-    def load_wordlists(name, paths, _generate)
+    def load_wordlists(name, paths)
       wordlists = paths + default_wordlist_paths(name)
 
       wordlists.map(&Spellr::Wordlist.method(:new))
@@ -93,8 +87,8 @@ module Spellr
     def default_wordlists
       [
         gem_wordlist,
-        generated_project_wordlist,
-        project_wordlist
+        project_wordlist,
+        *locale_wordlists
       ]
     end
   end