seeing_is_believing 1.0.1 → 2.0.0.beta1

data/Readme.md

@@ -141,16 +141,37 @@ Known Issues
 ============
 
 * `BEGIN/END` breaks things and I probably won't take the time to fix it, becuase it's nontrivial and its really meant for command-line scripts, but there is currently a cuke for it
-* Heredocs aren't recorded. It might actually be possible if the ExpressionList were to get smarter
 
 Todo
 ====
 
-* Make a Lines class which is a collection of lines, responsible for managing the trailing newlines in Binary::PrintResultsNextToLines and SeeingIsBelieving/ExpressionList
 * Add examples of invocations to the help screen
 * Add xmpfilter option to sublime text
 * Update TextMate examples to use same keys as sublime, add xmpfilter option on cmd+opt+N
-* Move as much of the SyntaxAnalyzer as possible over to Parser and ditch Ripper altogether
+* Remove the SyntaxAnalyzer altogether!
+* How about if begin/rescue/end was able to record the result on the rescue section
+* Check how begin/rescue/end with multiple rescue blocks works
+* What about recording the result of a line inside of a string interpolation, e.g. "a#{\n1\n}b" could record line 2 is 1 and line 3 is "a\n1\nb"
+* Be able to clean an invalid file (used to be able to do this, but parser can't identify comments in an invalid file the way that I'm currently using it, cuke is still there, marked as @not-implemented)
+* Add a flag to allow you to just get the results so that it can be easily used without a Ruby runtime
+
+Up Next
+=======
+
+rename ProgramRewriter -> WrapExpressions
+find that fucking regex bug
+add the inspected result --debug output
+Fix high-level shit:
+  remove all previous output except if -x flag is set, then leave `# =>`
+  `gsub(/\s*$/, '')`
+  run it through sib
+  if -x flag is set
+    for each comment
+      if it is `# =>`, update it
+  else
+    printable_list = get a list of each line that is an actual end and has no comments with sib (can ignore heredocs b/c they will have no value)
+                     this list is like line_number => [col_number, character_number] (e.g. where to insert on that line)
+    for each line, add it at that character location, adjusting from the col_number to pad it correctly
 
 
 License

data/features/errors.feature

@@ -29,11 +29,11 @@ Feature: Running the binary unsuccessfully
     """
     def first_defined
       second_defined
-    end                # => nil
+    end
 
     def second_defined
       require_relative 'raises_exception'  # ~> RuntimeError: ZOMG\n!!!!
-    end                                    # => nil
+    end
 
     first_defined
 

data/features/examples.feature

@@ -64,14 +64,14 @@ Feature: Running the binary successfully
     And stdout is:
     """
     # iteration
-    5.times do |i|
+    5.times do |i|  # => 5
       i * 2         # => 0, 2, 4, 6, 8
     end             # => 5
 
     # method and invocations
     def meth(n)
       n          # => "12", "34"
-    end          # => nil
+    end
 
     meth "12"  # => "12"
     meth "34"  # => "34"
@@ -89,25 +89,25 @@ Feature: Running the binary successfully
      b/x  # => /a\n b/x
 
     # don't record heredocs b/c they're just too fucking different
-    <<HERE
+    <<HERE    # => "is a doc\n"
     is a doc
     HERE
 
     # method invocation that occurs entirely on the next line
-    [*1..10]
-      .select(&:even?)
+    [*1..10]              # => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
+      .select(&:even?)    # => [2, 4, 6, 8, 10]
       .map { |n| n * 2 }  # => [4, 8, 12, 16, 20]
 
     # mutliple levels of nesting
     class User
       def initialize(name)
         @name = name        # => "Josh", "Rick"
-      end                   # => nil
+      end
 
       def name
         @name   # => "Josh", "Rick"
-      end       # => nil
-    end         # => nil
+      end
+    end
 
     User.new("Josh").name  # => "Josh"
     User.new("Rick").name  # => "Rick"
@@ -187,7 +187,7 @@ Feature: Running the binary successfully
     __LINE__            # => 2
     $stdout.puts "omg"  # => nil
     $stderr.puts "hi"   # => nil
-    DATA.read           # => "1\n2"
+    DATA.read           # => "1\n2\n"
     __LINE__            # => 6
 
     # >> omg

data/features/flags.feature

@@ -459,6 +459,8 @@ Feature: Using flags
     2+2    # => 10
     "a
      b" # =>
+    /a
+     b/ # =>
     1
     "omg"
     # =>
@@ -474,6 +476,8 @@ Feature: Using flags
     2+2    # => 4
     "a
      b" # => "a\n b"
+    /a
+     b/ # => /a\n b/
     1
     "omg"
     # => "omg"
@@ -481,6 +485,7 @@ Feature: Using flags
     # => "omg2"
     """
 
+  # TODO: do we want to print the source without its comments?
   Scenario: --debug
     Given the file "simple_program.rb":
     """
@@ -492,16 +497,13 @@ Feature: Using flags
     Then stderr is empty
     And the exit status is 0
     # source without comments
-    And stdout includes "SOURCE WITHOUT COMMENTS:"
-    And stdout includes:
-    """
-    # encoding: utf-8
-    1
-    2
-    """
-    # expression evaluation
-    And stdout includes "EXPRESSION EVALUATION:"
-    And stdout includes "GENERATED"
+    # And stdout includes "SOURCE WITHOUT COMMENTS:"
+    # And stdout includes:
+    # """
+    # # encoding: utf-8
+    # 1
+    # 2
+    # """
     # translated program
     And stdout includes "TRANSLATED PROGRAM:"
     And stdout includes "$seeing_is_believing_current_result"

data/features/regression.feature

@@ -62,10 +62,10 @@ Feature:
     Then stdout is:
     """
     def m
-      if true
-        return 1
+      if true     # => true
+        return 1  # => 1
       end
-    end           # => nil
+    end
     m             # => 1
     """
 
@@ -163,3 +163,17 @@ Feature:
     # encoding: utf-8
     'ç'  # => "ç"
     """
+
+  @not-implemented
+  Scenario: Some strings look like comments
+    Given the file "strings_that_look_like_comments.rb":
+    """
+    "1
+     #{2}"
+    """
+    When I run "seeing_is_believing strings_that_look_like_comments.rb"
+    Then stdout is:
+    """
+    "1
+     #{2}"  # => "1\n 2"
+    """

data/lib/seeing_is_believing.rb

@@ -6,9 +6,10 @@ require 'seeing_is_believing/queue'
 require 'seeing_is_believing/result'
 require 'seeing_is_believing/version'
 require 'seeing_is_believing/debugger'
-require 'seeing_is_believing/expression_list'
 require 'seeing_is_believing/remove_inline_comments'
 require 'seeing_is_believing/evaluate_by_moving_files'
+require 'seeing_is_believing/program_rewriter'
+require 'seeing_is_believing/syntax_analyzer' # can we get rid of this?
 
 # might not work on windows b/c of assumptions about line ends
 class SeeingIsBelieving
@@ -20,109 +21,50 @@ class SeeingIsBelieving
   end
 
   def initialize(program, options={})
-    program_string   = RemoveInlineComments::NonLeading.call program
-    @stream          = to_stream program_string
+    @program         = program
     @matrix_filename = options[:matrix_filename]
     @filename        = options[:filename]
     @stdin           = to_stream options.fetch(:stdin, '')
     @require         = options.fetch :require, []
     @load_path       = options.fetch :load_path, []
     @encoding        = options.fetch :encoding, nil
-    @line_number     = 1
     @timeout         = options[:timeout]
     @debugger        = options.fetch :debugger, Debugger.new(enabled: false)
-
-    debugger.context("SOURCE WITHOUT COMMENTS") { program_string }
   end
 
-  # I'd like to refactor this, but I was unsatisfied with the three different things I tried.
-  # In the end, I prefer keeping all manipulation of the line number here in the main function
-  # And I like that the higher-level construct of how the program gets built can be found here.
   def call
     @memoized_result ||= begin
-      leading_comments = ''
-
-
-      # extract leading comments (e.g. encoding) so they don't get wrapped in begin/rescue/end
-      while SyntaxAnalyzer.line_is_comment?(next_line_queue.peek)
-        leading_comments << next_line_queue.dequeue << "\n"
-        @line_number += 1
-      end
-
-      # extract leading =begin/=end so they don't get wrapped in begin/rescue/end
-      while SyntaxAnalyzer.begins_multiline_comment?(next_line_queue.peek)
-        lines = next_line_queue.dequeue << "\n"
-        @line_number += 1
-        until SyntaxAnalyzer.begin_and_end_comments_are_complete? lines
-          lines << next_line_queue.dequeue << "\n"
-          @line_number += 1
-        end
-        leading_comments << lines
-      end
-
-      # extract program body
-      body = ''
-      until next_line_queue.empty? || data_segment?
-        expression, expression_size = expression_list.call
-        body << expression
-        track_line_number @line_number
-        @line_number += expression_size
-      end
-
-      # extract data segment
-      data_segment = ''
-      data_segment = "\n#{the_rest_of_the_stream}" if data_segment?
-
-      # build the program
-      program = leading_comments << record_exceptions_in(body) << data_segment
-      debugger.context("TRANSLATED PROGRAM") { program }
-
-      # return the result
-      result_for program, max_line_number
+      # must use newline after code, or comments will comment out rescue section
+      # FIXME: IS THIS STILL TRUE?
+      wrapped = ProgramReWriter.call "#@program\n",
+                                     before_all:  "begin;",
+                                     after_all:   "\n"\
+                                                  "rescue Exception;"\
+                                                    "line_number = $!.backtrace.grep(/\#{__FILE__}/).first[/:\\d+/][1..-1].to_i;"\
+                                                    "$seeing_is_believing_current_result.record_exception line_number, $!;"\
+                                                    "$seeing_is_believing_current_result.exitstatus = 1;"\
+                                                    "$seeing_is_believing_current_result.exitstatus = $!.status if $!.kind_of? SystemExit;"\
+                                                  "end",
+                                     before_each: -> line_number { "($seeing_is_believing_current_result.record_result(#{line_number}, (" },
+                                     after_each:  -> line_number { ")))" }
+      debugger.context("TRANSLATED PROGRAM") { wrapped }
+      result = result_for wrapped
+      debugger.context("RESULT") { result.inspect }
+      result
     end
   end
 
-  private
-
-  attr_reader :stream, :matrix_filename, :debugger
 
-  def expression_list
-    @expression_list ||= ExpressionList.new debugger:       debugger,
-                                            get_next_line:  lambda { next_line_queue.dequeue },
-                                            peek_next_line: lambda { next_line_queue.peek },
-                                            on_complete:    lambda { |line, children, completions, offset|
-                                              expression = [line, *children, *completions].map(&:chomp).join("\n")
+  private
 
-                                              if do_not_record? expression
-                                                expression + "\n"
-                                              else
-                                                record_yahself(expression, @line_number+offset) + "\n"
-                                              end
-                                            }
-  end
+  attr_reader :matrix_filename, :debugger
 
   def to_stream(string_or_stream)
     return string_or_stream if string_or_stream.respond_to? :gets
     StringIO.new string_or_stream
   end
 
-  def record_yahself(line, line_number)
-    "($seeing_is_believing_current_result.record_result(#{line_number}, (#{line})))"
-  end
-
-  def record_exceptions_in(code)
-    # must use newline after code, or comments will comment out rescue section
-    "begin;"\
-      "#{code}\n"\
-    "rescue Exception;"\
-      "line_number = $!.backtrace.grep(/\#{__FILE__}/).first[/:\\d+/][1..-1].to_i;"\
-      "$seeing_is_believing_current_result.record_exception line_number, $!;"\
-      "$seeing_is_believing_current_result.exitstatus = 1;"\
-      "$seeing_is_believing_current_result.exitstatus = $!.status if $!.kind_of? SystemExit;"\
-    "end"
-  end
-
-  def result_for(program, max_line_number)
+  def result_for(program)
     Dir.mktmpdir "seeing_is_believing_temp_dir" do |dir|
       filename = @filename || File.join(dir, 'program.rb')
       EvaluateByMovingFiles.new(program,
@@ -134,31 +76,6 @@ class SeeingIsBelieving
                                 encoding:        @encoding,
                                 timeout:         @timeout)
                            .call
-                           .track_line_number(max_line_number)
     end
   end
-
-  def eof?
-    next_line_queue.peek.nil?
-  end
-
-  def data_segment?
-    SyntaxAnalyzer.begins_data_segment?(next_line_queue.peek)
-  end
-
-  def next_line_queue
-    @next_line_queue ||= Queue.new { (line = stream.gets) && line.chomp }
-  end
-
-  # eh? -.^ (can't we pull the rest of this out of the queue instead of breaking encapsulation?)
-  def the_rest_of_the_stream
-    next_line_queue.dequeue << "\n" << stream.read
-  end
-
-  def do_not_record?(code)
-    code =~ BLANK_REGEX                           ||
-      SyntaxAnalyzer.ends_in_comment?(code)       ||
-      SyntaxAnalyzer.void_value_expression?(code) ||
-      SyntaxAnalyzer.here_doc?(code)
-  end
 end

data/lib/seeing_is_believing/binary/add_annotations.rb

@@ -89,9 +89,9 @@ class SeeingIsBelieving
       def add_line(line, line_number)
         should_record = should_record? line, line_number
         if should_record && xmpfilter_style && line.strip =~ /^# =>/
-          new_body << xmpfilter_update(line, file_result[line_number - 1])
+          new_body << xmpfilter_update(line, file_result[line_number - 1]) # There is probably a bug in this since it doesn't go through the LineFormatter it can probably be to long
         elsif should_record && xmpfilter_style
-          new_body << xmpfilter_update(line, file_result[line_number])
+          new_body << xmpfilter_update(line, file_result[line_number]) # There is probably a bug in this since it doesn't go through the LineFormatter it can probably be to long
         elsif should_record
           new_body << format_line(line.chomp, line_number, file_result[line_number])
         else
@@ -108,7 +108,7 @@ class SeeingIsBelieving
 
       # again, this is too naive, should actually parse the comments and update them
       def xmpfilter_update(line, line_results)
-        line.gsub /# =>.*?$/, "# => #{line_results.join ', '}"
+        line.gsub /# =>.*?$/, "# => #{line_results.join(', ').gsub("\n", '\n')}"
       end
 
       def add_stdout

data/lib/seeing_is_believing/program_rewriter.rb

@@ -0,0 +1,280 @@
+require 'parser/current'
+
+# hack rewriter to apply insertions in stable order
+# until https://github.com/whitequark/parser/pull/102 gets merged in
+module Parser
+  module Source
+    class Rewriter
+      def process
+        adjustment   = 0
+        source       = @source_buffer.source.dup
+        sorted_queue = @queue.sort_by.with_index do |action, index|
+          [action.range.begin_pos, index]
+        end
+        sorted_queue.each do |action|
+          begin_pos = action.range.begin_pos + adjustment
+          end_pos   = begin_pos + action.range.length
+
+          source[begin_pos...end_pos] = action.replacement
+
+          adjustment += (action.replacement.length - action.range.length)
+        end
+
+        source
+      end
+    end
+  end
+end
+
+
+# comprehensive list of syntaxes that can come up
+# https://github.com/whitequark/parser/blob/master/doc/AST_FORMAT.md
+class SeeingIsBelieving
+  class ProgramReWriter
+    def self.call(program, wrappings)
+      new(program, wrappings).call
+    end
+
+    def initialize(program, wrappings)
+      self.program     = program
+      self.before_all  = wrappings.fetch :before_all,  ''.freeze
+      self.after_all   = wrappings.fetch :after_all,   ''.freeze
+      self.before_each = wrappings.fetch :before_each, -> * { '' }
+      self.after_each  = wrappings.fetch :after_each,  -> * { '' }
+      self.buffer      = Parser::Source::Buffer.new('program-without-annotations')
+      buffer.source    = program
+      self.root        = Parser::CurrentRuby.new.parse buffer
+      self.rewriter    = Parser::Source::Rewriter.new buffer
+      self.wrappings   = {}
+    rescue Parser::SyntaxError => e
+      raise ::SyntaxError, e.message
+    end
+
+    def call
+      @called ||= begin
+        find_wrappings
+
+        if root # file may be empty
+          rewriter.insert_before root.location.expression, before_all
+
+          wrappings.each do |line_num, (range, last_col)|
+            rewriter.insert_before range, before_each.call(line_num)
+            rewriter.insert_after  range, after_each.call(line_num)
+          end
+
+          rewriter.insert_after root.location.expression, after_all
+        end
+
+        rewriter.process
+      end
+    end
+
+    private
+
+    attr_accessor :program, :before_all, :after_all, :before_each, :after_each, :buffer, :root, :rewriter, :wrappings
+
+    def add_to_wrappings(range_or_ast)
+      range = range_or_ast
+      range = range_or_ast.location.expression if range.kind_of? ::AST::Node
+      line, col = buffer.decompose_position range.end_pos
+      _, prev_col = wrappings[line]
+      wrappings[line] = (!wrappings[line] || prev_col < col ? [range, col] : wrappings[line] )
+    end
+
+    def add_children(ast, omit_first = false)
+      (omit_first ? ast.children.drop(1) : ast.children)
+        .each { |child| find_wrappings child }
+    end
+
+    def find_wrappings(ast=root)
+      return wrappings unless ast.kind_of? ::AST::Node
+
+      case ast.type
+      when :args, :redo, :retry, :alias, :undef, :splat, :match_current_line
+        # no op
+      when :rescue, :ensure, :def, :return, :break, :next
+        add_children ast
+      when :if
+        if ast.location.kind_of? Parser::Source::Map::Ternary
+          add_to_wrappings ast unless ast.children.any? { |child| void_value? child }
+          add_children ast
+        else
+          keyword = ast.location.keyword.source
+          if (keyword == 'if' || keyword == 'unless') && ast.children.none? { |child| void_value? child }
+            add_to_wrappings ast
+          end
+          add_children ast
+        end
+      when :when, :pair, :defs, :class, :module, :sclass
+        find_wrappings ast.children.last
+      when :resbody
+        exception_type, variable_name, body = ast.children
+        find_wrappings body
+      when :block
+        add_to_wrappings ast
+
+        # a {} comes in as
+        #   (block
+        #     (send nil :a)
+        #     (args) nil)
+        #
+        # a.b {} comes in as
+        #   (block
+        #     (send
+        #       (send nil :a) :b)
+        #     (args) nil)
+        #
+        # we don't want to wrap the send itself, otherwise could come in as <a>{}
+        # but we do want ot wrap its first child so that we can get <<a>\n.b{}>
+        #
+        # I can't think of anything other than a :send that could be the first child
+        # but I'll check for it anyway.
+        the_send = ast.children[0]
+        find_wrappings the_send.children.first if the_send.type == :send
+        add_children ast, true
+      when :masgn
+        # we must look at RHS because [1,<<A] and 1,<<A are both allowed
+        #
+        # in the first case, we must take the end_pos of the array, or we'll insert the after_each in the wrong location
+        #
+        # in the second, there is an implicit Array wrapped around it, with the wrong end_pos,
+        # so we must take the end_pos of the last arg
+        array = ast.children.last
+        if array.location.expression.source.start_with? '['
+          add_to_wrappings ast
+          find_wrappings array
+        else
+          begin_pos = ast.location.expression.begin_pos
+          end_pos   = heredoc_hack(ast.children.last.children.last).location.expression.end_pos
+          range     = Parser::Source::Range.new buffer, begin_pos, end_pos
+          add_to_wrappings range
+          add_children ast.children.last
+        end
+      when :lvasgn
+        # because the RHS can be a heredoc, and parser currently handles heredocs locations incorrectly
+        # we must hack around this
+
+        # can have one or two children:
+        #   in a=1 (has children :a, and 1)
+        #   in a,b=1,2 it comes out like:
+        #     (masgn
+        #       (mlhs
+        #         (lvasgn :a) <-- one child
+        #
+        #         (lvasgn :b))
+        #       (array
+        #         (int 1)
+        #         (int 2)))
+        if ast.children.size == 2
+          begin_pos = ast.location.expression.begin_pos
+          end_pos   = heredoc_hack(ast.children.last).location.expression.end_pos
+          range     = Parser::Source::Range.new buffer, begin_pos, end_pos
+          add_to_wrappings range
+          add_children ast
+        end
+      when :send
+        # because the target and the last child can be heredocs
+        # and the method may or may not have parens,
+        # it can inadvertently inherit the incorrect location of the heredocs
+        # so we check for this case, that way we can construct the correct range instead
+        range = ast.location.expression
+
+        # first two children: target, message, so we want the last child only if it is an argument
+        target, message, *, last_arg = ast.children
+
+        # last arg is a heredoc, use the closing paren, or the end of the first line of the heredoc
+        if heredoc? last_arg
+          end_pos = heredoc_hack(last_arg).location.expression.end_pos
+          if buffer.source[ast.location.selector.end_pos] == '('
+            end_pos += 1 until buffer.source[end_pos] == ')'
+            end_pos += 1
+          end
+
+        # the last arg is not a heredoc, the range of the expression can be trusted
+        elsif last_arg
+          end_pos = ast.location.expression.end_pos
+
+        # there is no last arg, but there are parens, find the closing paren
+        # we can't trust the expression range because the *target* could be a heredoc
+        # FIXME: This blows up on 2.0 with ->{}.() because it has no selector, so in this case
+        #        we would want to use the expression, but I'm ignoring that for now, because
+        #        we would have to check the target to see whether to use the selector or the expression
+        elsif buffer.source[ast.location.selector.end_pos] == '('
+          closing_paren_index = ast.location.selector.end_pos + 1
+          closing_paren_index += 1 until buffer.source[closing_paren_index] == ')'
+          end_pos = closing_paren_index + 1
+
+        # use the selector because we can't trust expression since target can be a heredoc
+        elsif heredoc? target
+          end_pos = ast.location.selector.end_pos
+
+        # use the expression because it could be something like !1, in which case the selector would return the rhs of the !
+        else
+          end_pos = ast.location.expression.end_pos
+        end
+
+        begin_pos = ast.location.expression.begin_pos
+        range     = Parser::Source::Range.new(buffer, begin_pos, end_pos)
+        add_to_wrappings range
+        add_children ast
+      when :begin
+        last_child = ast.children.last
+        if heredoc? last_child
+          range = Parser::Source::Range.new buffer,
+                                            ast.location.expression.begin_pos,
+                                            heredoc_hack(last_child).location.expression.end_pos
+          add_to_wrappings range unless void_value? ast.children.last
+        end
+
+        add_children ast
+      when :str, :dstr, :xstr, :regexp
+        add_to_wrappings heredoc_hack ast
+      else
+        add_to_wrappings ast
+        add_children ast
+      end
+    rescue
+      # TODO: delete this rescue block once things get stabler
+      puts ast.type
+      puts $!
+      require "pry"
+      binding.pry
+    end
+
+    def heredoc_hack(ast)
+      return ast unless heredoc? ast
+      Parser::AST::Node.new :str,
+                            [],
+                            location: Parser::Source::Map.new(ast.location.begin)
+    end
+
+    # this is the scardest fucking method I think I've ever written.
+    # *anything* can go wrong!
+    def heredoc?(ast)
+      # some strings are fucking weird.
+      # e.g. the "1" in `%w[1]` returns nil for ast.location.begin
+      # and `__FILE__` is a string whose location is a Parser::Source::Map instead of a Parser::Source::Map::Collection, so it has no #begin
+      ast.kind_of?(Parser::AST::Node)           &&
+        (ast.type == :dstr || ast.type == :str) &&
+        (location  = ast.location)              &&
+        (location.respond_to?(:begin))          &&
+        (the_begin = location.begin)            &&
+        (the_begin.source =~ /^\<\<-?/)
+    end
+
+    def void_value?(ast)
+      case ast && ast.type
+      when :begin, :kwbegin, :resbody
+        void_value?(ast.children[-1])
+      when :rescue, :ensure
+        ast.children.any? { |child| void_value? child }
+      when :if
+        void_value?(ast.children[1]) || void_value?(ast.children[2])
+      when :return, :next, :redo, :retry, :break
+        true
+      else
+        false
+      end
+    end
+  end
+end