syntax_suggest 0.0.1

data/README.md

@@ -0,0 +1,229 @@
+# SyntaxSuggest
+
+An error in your code forces you to stop. SyntaxSuggest helps you find those errors to get you back on your way faster.
+
+```
+Unmatched `end', missing keyword (`do', `def`, `if`, etc.) ?
+
+  1  class Dog
+❯ 2    defbark
+❯ 4    end
+  5  end
+```
+
+## Installation in your codebase
+
+To automatically annotate errors when they happen, add this to your Gemfile:
+
+```ruby
+gem 'syntax_suggest'
+```
+
+And then execute:
+
+    $ bundle install
+
+If your application is not calling `Bundler.require` then you must manually add a require:
+
+```ruby
+require "syntax_suggest"
+```
+
+If you're using rspec add this to your `.rspec` file:
+
+```
+--require syntax_suggest
+```
+
+> This is needed because people can execute a single test file via `bundle exec rspec path/to/file_spec.rb` and if that file has a syntax error, it won't load `spec_helper.rb` to trigger any requires.
+
+## Install the CLI
+
+To get the CLI and manually search for syntax errors (but not automatically annotate them), you can manually install the gem:
+
+    $ gem install syntax_suggest
+
+This gives you the CLI command `$ syntax_suggest` for more info run `$ syntax_suggest --help`.
+
+## Editor integration
+
+An extension is available for VSCode:
+
+- Extension: https://marketplace.visualstudio.com/items?itemName=Zombocom.dead-end-vscode
+- GitHub: https://github.com/zombocom/dead_end-vscode
+
+## What syntax errors does it handle?
+
+Syntax suggest will fire against all syntax errors and can isolate any syntax error. In addition, syntax_suggest attempts to produce human readable descriptions of what needs to be done to resolve the issue. For example:
+
+- Missing `end`:
+
+<!--
+```ruby
+class Dog
+  def bark
+    puts "bark"
+end
+```
+-->
+
+```
+Unmatched keyword, missing `end' ?
+
+❯ 1  class Dog
+❯ 2    def bark
+❯ 4  end
+```
+
+- Missing keyword
+<!--
+```ruby
+class Dog
+  def speak
+    @sounds.each |sound|
+      puts sound
+    end
+  end
+end
+```
+-->
+
+```
+Unmatched `end', missing keyword (`do', `def`, `if`, etc.) ?
+
+  1  class Dog
+  2    def speak
+❯ 3      @sounds.each |sound|
+❯ 5      end
+  6    end
+  7  end
+```
+
+- Missing pair characters (like `{}`, `[]`, `()` , or `|<var>|`)
+<!--
+
+```ruby
+class Dog
+  def speak(sound
+    puts sound
+  end
+end
+```
+-->
+
+```
+Unmatched `(', missing `)' ?
+
+  1  class Dog
+❯ 2    def speak(sound
+❯ 4    end
+  5  end
+```
+
+- Any ambiguous or unknown errors will be annotated by the original ripper error output:
+
+<!--
+class Dog
+  def meals_last_month
+    puts 3 *
+  end
+end
+-->
+
+```
+syntax error, unexpected end-of-input
+
+  1  class Dog
+  2    def meals_last_month
+❯ 3      puts 3 *
+  4    end
+  5  end
+```
+
+## How is it better than `ruby -wc`?
+
+Ruby allows you to syntax check a file with warnings using `ruby -wc`. This emits a parser error instead of a human focused error. Ruby's parse errors attempt to narrow down the location and can tell you if there is a glaring indentation error involving `end`.
+
+The `syntax_suggest` algorithm doesn't just guess at the location of syntax errors, it re-parses the document to prove that it captured them.
+
+This library focuses on the human side of syntax errors. It cares less about why the document could not be parsed (computer problem) and more on what the programmer needs (human problem) to fix the problem.
+
+## Sounds cool, but why isn't this baked into Ruby directly?
+
+We are now talking about it https://bugs.ruby-lang.org/issues/18159#change-93682.
+
+## Artificial Inteligence?
+
+This library uses a goal-seeking algorithm for syntax error detection similar to that of a path-finding search. For more information [read the blog post about how it works under the hood](https://schneems.com/2020/12/01/squash-unexpectedend-errors-with-syntaxsearch/).
+
+## How does it detect syntax error locations?
+
+We know that source code that does not contain a syntax error can be parsed. We also know that code with a syntax error contains both valid code and invalid code. If you remove the invalid code, then we can programatically determine that the code we removed contained a syntax error. We can do this detection by generating small code blocks and searching for which blocks need to be removed to generate valid source code.
+
+Since there can be multiple syntax errors in a document it's not good enough to check individual code blocks, we've got to check multiple at the same time. We will keep creating and adding new blocks to our search until we detect that our "frontier" (which contains all of our blocks) contains the syntax error. After this, we can stop our search and instead focus on filtering to find the smallest subset of blocks that contain the syntax error.
+
+Here's an example:
+
+![](assets/syntax_search.gif)
+
+## Use internals
+
+To use the `syntax_suggest` gem without monkeypatching you can  `require 'syntax_suggest/api'`. This will allow you to load `syntax_suggest` and use its internals without mutating `require`.
+
+Stable internal interface(s):
+
+- `SyntaxSuggest.handle_error(e)`
+
+Any other entrypoints are subject to change without warning. If you want to use an internal interface from `syntax_suggest` not on this list, open an issue to explain your use case.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+### How to debug changes to output display
+
+You can see changes to output against a variety of invalid code by running specs and using the `DEBUG_DISPLAY=1` environment variable. For example:
+
+```
+$ DEBUG_DISPLAY=1 bundle exec rspec spec/ --format=failures
+```
+
+### Run profiler
+
+You can output profiler data to the `tmp` directory by running:
+
+```
+$ DEBUG_PERF=1 bundle exec rspec spec/integration/syntax_suggest_spec.rb
+```
+
+Some outputs are in text format, some are html, the raw marshaled data is available in `raw.rb.marshal`. See https://ruby-prof.github.io/#reports for more info. One interesting one, is the "kcachegrind" interface. To view this on mac:
+
+```
+$ brew install qcachegrind
+```
+
+Open:
+
+```
+$ qcachegrind tmp/last/profile.callgrind.out.<numbers>
+```
+
+## Environment variables
+
+- `SYNTAX_SUGGEST_DEBUG` - Enables debug output to STDOUT/STDERR and/or disk at `./tmp`. The contents of debugging output are not stable and may change. If you would like stability, please open an issue to explain your use case.
+- `SYNTAX_SUGGEST_TIMEOUT` - Changes the default timeout value to the number set (in seconds).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/zombocom/syntax_suggest. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/zombocom/syntax_suggest/blob/main/CODE_OF_CONDUCT.md).
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Code of Conduct
+
+Everyone interacting in the SyntaxSuggest project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/zombocom/syntax_suggest/blob/main/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "syntax_suggest"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/dead_end.gemspec

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+begin
+  require_relative "lib/syntax_suggest/version"
+rescue LoadError # Fallback to load version file in ruby core repository
+  require_relative "version"
+end
+
+Gem::Specification.new do |spec|
+  spec.name = "syntax_suggest"
+  spec.version = SyntaxSuggest::VERSION
+  spec.authors = ["schneems"]
+  spec.email = ["richard.schneeman+foo@gmail.com"]
+
+  spec.summary = "Find syntax errors in your source in a snap"
+  spec.description = 'When you get an "unexpected end" in your syntax this gem helps you find it'
+  spec.homepage = "https://github.com/zombocom/syntax_suggest.git"
+  spec.license = "MIT"
+  spec.required_ruby_version = Gem::Requirement.new(">= 2.5.0")
+
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = "https://github.com/zombocom/syntax_suggest.git"
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  spec.files = Dir.chdir(File.expand_path("..", __FILE__)) do
+    `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features|assets)/}) }
+  end
+  spec.bindir = "exe"
+  spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+end

data/exe/syntax_suggest

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+
+require_relative "../lib/syntax_suggest/api"
+
+SyntaxSuggest::Cli.new(
+  argv: ARGV
+).call

data/lib/syntax_suggest/api.rb

@@ -0,0 +1,199 @@
+# frozen_string_literal: true
+
+require_relative "version"
+
+require "tmpdir"
+require "stringio"
+require "pathname"
+require "ripper"
+require "timeout"
+
+module SyntaxSuggest
+  # Used to indicate a default value that cannot
+  # be confused with another input.
+  DEFAULT_VALUE = Object.new.freeze
+
+  class Error < StandardError; end
+  TIMEOUT_DEFAULT = ENV.fetch("SYNTAX_SUGGEST_TIMEOUT", 1).to_i
+
+  # SyntaxSuggest.handle_error [Public]
+  #
+  # Takes a `SyntaxError` exception, uses the
+  # error message to locate the file. Then the file
+  # will be analyzed to find the location of the syntax
+  # error and emit that location to stderr.
+  #
+  # Example:
+  #
+  #   begin
+  #     require 'bad_file'
+  #   rescue => e
+  #     SyntaxSuggest.handle_error(e)
+  #   end
+  #
+  # By default it will re-raise the exception unless
+  # `re_raise: false`. The message output location
+  # can be configured using the `io: $stderr` input.
+  #
+  # If a valid filename cannot be determined, the original
+  # exception will be re-raised (even with
+  # `re_raise: false`).
+  def self.handle_error(e, re_raise: true, io: $stderr)
+    unless e.is_a?(SyntaxError)
+      io.puts("SyntaxSuggest: Must pass a SyntaxError, got: #{e.class}")
+      raise e
+    end
+
+    file = PathnameFromMessage.new(e.message, io: io).call.name
+    raise e unless file
+
+    io.sync = true
+
+    call(
+      io: io,
+      source: file.read,
+      filename: file
+    )
+
+    raise e if re_raise
+  end
+
+  # SyntaxSuggest.call [Private]
+  #
+  # Main private interface
+  def self.call(source:, filename: DEFAULT_VALUE, terminal: DEFAULT_VALUE, record_dir: DEFAULT_VALUE, timeout: TIMEOUT_DEFAULT, io: $stderr)
+    search = nil
+    filename = nil if filename == DEFAULT_VALUE
+    Timeout.timeout(timeout) do
+      record_dir ||= ENV["DEBUG"] ? "tmp" : nil
+      search = CodeSearch.new(source, record_dir: record_dir).call
+    end
+
+    blocks = search.invalid_blocks
+    DisplayInvalidBlocks.new(
+      io: io,
+      blocks: blocks,
+      filename: filename,
+      terminal: terminal,
+      code_lines: search.code_lines
+    ).call
+  rescue Timeout::Error => e
+    io.puts "Search timed out SYNTAX_SUGGEST_TIMEOUT=#{timeout}, run with DEBUG=1 for more info"
+    io.puts e.backtrace.first(3).join($/)
+  end
+
+  # SyntaxSuggest.record_dir [Private]
+  #
+  # Used to generate a unique directory to record
+  # search steps for debugging
+  def self.record_dir(dir)
+    time = Time.now.strftime("%Y-%m-%d-%H-%M-%s-%N")
+    dir = Pathname(dir)
+    dir.join(time).tap { |path|
+      path.mkpath
+      FileUtils.ln_sf(time, dir.join("last"))
+    }
+  end
+
+  # SyntaxSuggest.valid_without? [Private]
+  #
+  # This will tell you if the `code_lines` would be valid
+  # if you removed the `without_lines`. In short it's a
+  # way to detect if we've found the lines with syntax errors
+  # in our document yet.
+  #
+  #   code_lines = [
+  #     CodeLine.new(line: "def foo\n",   index: 0)
+  #     CodeLine.new(line: "  def bar\n", index: 1)
+  #     CodeLine.new(line: "end\n",       index: 2)
+  #   ]
+  #
+  #   SyntaxSuggest.valid_without?(
+  #     without_lines: code_lines[1],
+  #     code_lines: code_lines
+  #   )                                    # => true
+  #
+  #   SyntaxSuggest.valid?(code_lines) # => false
+  def self.valid_without?(without_lines:, code_lines:)
+    lines = code_lines - Array(without_lines).flatten
+
+    if lines.empty?
+      true
+    else
+      valid?(lines)
+    end
+  end
+
+  # SyntaxSuggest.invalid? [Private]
+  #
+  # Opposite of `SyntaxSuggest.valid?`
+  def self.invalid?(source)
+    source = source.join if source.is_a?(Array)
+    source = source.to_s
+
+    Ripper.new(source).tap(&:parse).error?
+  end
+
+  # SyntaxSuggest.valid? [Private]
+  #
+  # Returns truthy if a given input source is valid syntax
+  #
+  #   SyntaxSuggest.valid?(<<~EOM) # => true
+  #     def foo
+  #     end
+  #   EOM
+  #
+  #   SyntaxSuggest.valid?(<<~EOM) # => false
+  #     def foo
+  #       def bar # Syntax error here
+  #     end
+  #   EOM
+  #
+  # You can also pass in an array of lines and they'll be
+  # joined before evaluating
+  #
+  #   SyntaxSuggest.valid?(
+  #     [
+  #       "def foo\n",
+  #       "end\n"
+  #     ]
+  #   ) # => true
+  #
+  #   SyntaxSuggest.valid?(
+  #     [
+  #       "def foo\n",
+  #       "  def bar\n", # Syntax error here
+  #       "end\n"
+  #     ]
+  #   ) # => false
+  #
+  # As an FYI the CodeLine class instances respond to `to_s`
+  # so passing a CodeLine in as an object or as an array
+  # will convert it to it's code representation.
+  def self.valid?(source)
+    !invalid?(source)
+  end
+end
+
+# Integration
+require_relative "cli"
+
+# Core logic
+require_relative "code_search"
+require_relative "code_frontier"
+require_relative "explain_syntax"
+require_relative "clean_document"
+
+# Helpers
+require_relative "lex_all"
+require_relative "code_line"
+require_relative "code_block"
+require_relative "block_expand"
+require_relative "ripper_errors"
+require_relative "priority_queue"
+require_relative "unvisited_lines"
+require_relative "around_block_scan"
+require_relative "priority_engulf_queue"
+require_relative "pathname_from_message"
+require_relative "display_invalid_blocks"
+require_relative "parse_blocks_from_indent_line"