oedipus_lex 2.3.2 → 2.5.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 2d1d4969de9074724f4c4491ba4ed4e11c49625d
-  data.tar.gz: db2fff4a5947473754ff79c4b0ea7bce982496d6
+SHA256:
+  metadata.gz: 9822a393543024bafcbb4c490751008d5ae468d3348d3397df454b622e5f8fb1
+  data.tar.gz: 5910e4bf0720dc2724bf01dde8629f02616958a1d8eab65005bdfd2d5da35690
 SHA512:
-  metadata.gz: 03883e37fc06e56850ded37019c6b500e54d30ee0d455f529d4f5b9fbfe4f31baf053305bd503fa7bbf69ac4bacb11fe0c832128fe1d024d6e7d4c2d7d374f8f
-  data.tar.gz: ff54b5cb64e1ca96a20c3a6f094ed925dfb845dba5a6e66de5e73925e24a6c5655fa4cb78d283e46f9bb2649ae2b9361377d891b657ee99c2d2dbc27fec977d1
+  metadata.gz: b1c26b081d86155d0bade889a37a80e98acd35dc5dfc8bec0009fe38f18064e7d7a0c61e11a6ba037dd024fedda76597bab9787b21fed1a87d80ceaf666fb22f
+  data.tar.gz: e9ad55f6f9d155af3a64af6edd81ffa41c33c70f9e55148a9a002b3dfa3ac7441c819c9bfa6c885903425549926f2a2f3e823791557facb49ded71f16a2bf1df

data/History.rdoc

@@ -1,3 +1,50 @@
+=== 2.5.2 / 2020-06-14
+
+* 1 minor enhancement:
+
+  * Speedup of column position computation. It went from roughly 10s to 2s for a big file! (vdbijl)
+
+=== 2.5.1 / 2019-06-03
+
+* 1 minor enhancement:
+
+  * Added full rdoc an re-bootstrapped.
+
+* 1 bug fix:
+
+  * Fixed a deprecation warning in ruby 2.6+.
+
+=== 2.5.0 / 2016-11-30
+
+* 5 minor enhancements:
+
+  * Added #location to generated template, provides file:line:column per options.
+  * Added LexerError and made ScanError subclass it.
+  * Added column option.
+  * Errors try to provide location now.
+  * Re-bootstrapped.
+
+* 2 bug fixes:
+
+  * Fixed some whitespace generation when using :column.
+  * Fixed wiring on column. (steakknife)
+
+=== 2.4.1 / 2016-01-21
+
+* 1 minor enhancement:
+
+  * Use `skip` and `match?` instead of `scan` and `check`. Better on GC. (presidentbeef)
+
+=== 2.4.0 / 2014-08-29
+
+* 1 minor enhancement:
+
+  * Added column option & accessor.
+
+* 1 bug fix:
+
+  * lineno shouldn't be visible at all if the option isn't on.
+
 === 2.3.2 / 2014-08-06
 
 * 1 bug fix:

data/README.rdoc

@@ -19,7 +19,7 @@ At the very least, you need to add slashes to all your regexps.
 
 Oedipus, like rexical, is based primarily on generating code much like
 you would a hand-written lexer. It is _not_ a table or hash driven
-lexer. It use StrScanner within a multi-level case statement. As such,
+lexer. It uses StrScanner within a multi-level case statement. As such,
 Oedipus matches on the _first_ match, not the longest (like lex and
 its ilk).
 
@@ -49,6 +49,7 @@ resource for CS learning. Books... books are good. I like books.
                   | /debug/i
                   | /do_parse/i
                   | /lineno/i
+                  | /column/i
 
   inner_section   = /inner/ NL (misc_line)*
 
@@ -129,6 +130,8 @@ Specify `lineno` to generate automatic line number handling at the
 beginning of `next_token`. This was the default in 1.0.0 and you must
 now activate it.
 
+Specify `column` to generate automatic column number handling.
+
 ==== Inner
 
 The inner section is just code, like header or footer, but inner gets

data/Rakefile

@@ -3,8 +3,6 @@
 require "rubygems"
 require "hoe"
 
-Hoe.plugin :debugging
-Hoe.plugin :git
 Hoe.plugin :isolate
 Hoe.plugin :seattlerb
 
@@ -16,6 +14,10 @@ Hoe.spec "oedipus_lex" do
   self.history_file     = "History.rdoc"
 end
 
+Hoe.bad_plugins.each do |bad|
+  warn "BAD: Hoe.plugin :#{bad}"
+end
+
 task :bootstrap do
   ruby "-Ilib lib/oedipus_lex.rb lib/oedipus_lex.rex > lib/oedipus_lex.rex.rb.new"
   system "diff -uw lib/oedipus_lex.rex.rb lib/oedipus_lex.rex.rb.new"
@@ -78,4 +80,9 @@ task :debug do
   puts rex.generate
 end
 
+task :wtf => :isolate do
+  puts `~/.rbenv/versions/2.2.0/bin/ruby -S gem env`
+  puts `~/.rbenv/versions/2.2.0/bin/ruby -S gem list`
+end
+
 # vim: syntax=ruby

data/lib/oedipus_lex.rb

@@ -3,71 +3,160 @@ require 'strscan'
 require "erb"
 require "oedipus_lex.rex"
 
+##
+# Oedipus Lex is a lexer generator in the same family as Rexical and
+# Rex. Oedipus Lex is my independent lexer fork of Rexical. Rexical
+# was in turn a fork of Rex. We've been unable to contact the author
+# of rex in order to take it over, fix it up, extend it, and relicense
+# it to MIT. So, Oedipus was written clean-room in order to bypass
+# licensing constraints (and because bootstrapping is fun).
+#
+# Oedipus brings a lot of extras to the table and at this point is
+# only historically related to rexical. The syntax has changed enough
+# that any rexical lexer will have to be tweaked to work inside of
+# oedipus. At the very least, you need to add slashes to all your
+# regexps.
+#
+# Oedipus, like rexical, is based primarily on generating code much
+# like you would a hand-written lexer. It is _not_ a table or hash
+# driven lexer. It uses StrScanner within a multi-level case
+# statement. As such, Oedipus matches on the _first_ match, not the
+# longest (like lex and its ilk).
+#
+# This documentation is not meant to bypass any prerequisite knowledge
+# on lexing or parsing. If you'd like to study the subject in further
+# detail, please try [TIN321] or the [LLVM Tutorial] or some other
+# good resource for CS learning. Books... books are good. I like
+# books.
+
 class OedipusLex
-  VERSION = "2.3.2"
+  VERSION = "2.5.2" # :nodoc:
+
+  ##
+  # The class name to generate.
 
   attr_accessor :class_name
+
+  ##
+  # An array of header lines to have before the lexer class.
+
   attr_accessor :header
+
+  ##
+  # An array of lines to have after the lexer class.
+
   attr_accessor :ends
+
+  ##
+  # An array of lines to have inside (but at the bottom of) the lexer
+  # class.
+
   attr_accessor :inners
+
+  ##
+  # An array of name/regexp pairs to generate constants inside the
+  # lexer class.
+
   attr_accessor :macros
+
+  ##
+  # A hash of options for the code generator. See README.rdoc for
+  # supported options.
+
   attr_accessor :option
+
+  ##
+  # The rules for the lexer.
+
   attr_accessor :rules
+
+  ##
+  # An array of lines of code to generate into the top of the lexer
+  # (next_token) loop.
+
   attr_accessor :starts
+
+  ##
+  # An array of all the groups within the lexer rules.
+
   attr_accessor :group
 
-  DEFAULTS = {
+  DEFAULTS = { # :nodoc:
     :debug    => false,
     :do_parse => false,
     :lineno   => false,
+    :column   => false,
     :stub     => false,
   }
 
+  ##
+  # A Rule represents the main component of Oedipus Lex. These are the
+  # things that "get stuff done" at the lexical level. They consist of:
+  #
+  # + an optional required start state symbol or predicate method name
+  # + a regexp to match on
+  # + an optional action method or block
+
   class Rule < Struct.new :start_state, :regexp, :action
+    ##
+    # What group this rule is in, if any.
+
     attr_accessor :group
-    alias :group? :group
+
+    alias :group? :group # :nodoc:
+
+    ##
+    # A simple constructor
 
     def self.[] start, regexp, action
       new start, regexp.inspect, action
     end
 
-    def initialize start_state, regexp, action
+    def initialize start_state, regexp, action # :nodoc:
       super
       self.group = nil
     end
 
     undef_method :to_a
 
+    ##
+    # Generate equivalent ruby code for the rule.
+
     def to_ruby state, predicates, exclusive
       return unless group? or
         start_state == state or
         (state.nil? and predicates.include? start_state)
 
-      cond =
-        if exclusive or not start_state then
-          "when text = ss.scan(#{regexp}) then"
-        elsif start_state =~ /^:/ then
-          "when (state == #{start_state}) && (text = ss.scan(#{regexp})) then"
-        else
-          "when #{start_state} && (text = ss.scan(#{regexp})) then"
-        end
+      uses_text = false
 
       body =
         case action
         when nil, false then
           "  # do nothing"
         when /^\{/ then
+          uses_text = action =~ /\btext\b/
           "  action #{action}"
         when /^:/, "nil" then
           "  [:state, #{action}]"
-        else
+        else # plain method name
+          uses_text = true
           "  #{action} text"
         end
 
-      [cond, body]
+      check = uses_text ? "text = ss.scan(#{regexp})" : "ss.skip(#{regexp})"
+
+      cond = if exclusive or not start_state then
+               check
+             elsif /^:/.match?(start_state) then
+               "(state == #{start_state}) && (#{check})"
+             else # predicate method
+               "#{start_state} && (#{check})"
+             end
+
+      ["when #{cond} then", body]
     end
 
-    def pretty_print pp
+    def pretty_print pp # :nodoc:
       pp.text "Rule"
       pp.group 2, "[", "]" do
         pp.pp start_state
@@ -79,27 +168,39 @@ class OedipusLex
     end
   end
 
+  ##
+  # A group allows you to group up multiple rules under a single
+  # regular prefix expression, allowing optimized code to be generated
+  # that skips over all actions if the prefix isn't matched.
+
   class Group < Struct.new :regex, :rules
     alias :start_state :regex
 
+    ##
+    # A convenience method to create a new group with a +start+ and
+    # given +subrules+.
+
     def self.[] start, *subrules
       r = new start.inspect
       r.rules.concat subrules
       r
     end
 
-    def initialize start
+    def initialize start # :nodoc:
       super(start, [])
     end
 
+    ##
+    # Add a rule to this group.
+
     def << rule
       rules << rule
       nil
     end
 
-    def to_ruby state, predicates, exclusive
+    def to_ruby state, predicates, exclusive # :nodoc:
       [
-       "when ss.check(#{regex}) then",
+       "when ss.match?(#{regex}) then",
        "  case",
        rules.map { |subrule|
          s = subrule.to_ruby(state, predicates, exclusive)
@@ -109,7 +210,7 @@ class OedipusLex
       ]
     end
 
-    def pretty_print pp
+    def pretty_print pp # :nodoc:
       pp.text "Group"
       pp.group 2, "[", "]" do
         pp.seplist([regex] + rules, lambda { pp.comma_breakable }, :each) { |v|
@@ -119,6 +220,10 @@ class OedipusLex
     end
   end
 
+  ##
+  # A convenience method to create a new lexer with a +name+ and given
+  # +rules+.
+
   def self.[](name, *rules)
     r = new
     r.class_name = name
@@ -126,7 +231,7 @@ class OedipusLex
     r
   end
 
-  def initialize opts = {}
+  def initialize opts = {} # :nodoc:
     self.option     = DEFAULTS.merge opts
     self.class_name = nil
 
@@ -139,7 +244,7 @@ class OedipusLex
     self.group   = nil
   end
 
-  def == o
+  def == o # :nodoc:
     (o.class      == self.class      and
      o.class_name == self.class_name and
      o.header     == self.header     and
@@ -150,7 +255,7 @@ class OedipusLex
      o.starts     == self.starts)
   end
 
-  def pretty_print pp
+  def pretty_print pp # :nodoc:
     commas = lambda { pp.comma_breakable }
 
     pp.text "Lexer"
@@ -159,67 +264,109 @@ class OedipusLex
     end
   end
 
+  ##
+  # Process a +class+ lexeme.
+
   def lex_class prefix, name
     header.concat prefix.split(/\n/)
     self.class_name = name
   end
 
+  ##
+  # Process a +comment+ lexeme.
+
   def lex_comment line
     # do nothing
   end
 
+  ##
+  # Process an +end+ lexeme.
+
   def lex_end line
     ends << line
   end
 
+  ##
+  # Process an +inner+ lexeme.
+
   def lex_inner line
     inners << line
   end
 
+  ##
+  # Process a +start+ lexeme.
+
   def lex_start line
     starts << line.strip
   end
 
+  ##
+  # Process a +macro+ lexeme.
+
   def lex_macro name, value
     macros << [name, value]
   end
 
+  ##
+  # Process an +option+ lexeme.
+
   def lex_option option
     self.option[option.to_sym] = true
   end
 
+  ##
+  # Process a +X+ lexeme.
+
   def lex_rule start_state, regexp, action = nil
     rules << Rule.new(start_state, regexp, action)
   end
 
+  ##
+  # Process a +group head+ lexeme.
+
   def lex_grouphead re
     end_group if group
     self.state = :group
     self.group = Group.new re
   end
 
+  ##
+  # Process a +group+ lexeme.
+
   def lex_group start_state, regexp, action = nil
     rule = Rule.new(start_state, regexp, action)
     rule.group = group
     self.group << rule
   end
 
+  ##
+  # End a group.
+
   def end_group
     rules << group
     self.group = nil
     self.state = :rule
   end
 
+  ##
+  # Process the end of a +group+ lexeme.
+
   def lex_groupend start_state, regexp, action = nil
     end_group
     lex_rule start_state, regexp, action
   end
 
-  def lex_state new_state
+  ##
+  # Process a +state+ lexeme.
+
+  def lex_state _new_state
     end_group if group
     # do nothing -- lexer switches state for us
   end
 
+  ##
+  # Generate the lexer.
+
   def generate
     filter = lambda { |r| Rule === r && r.start_state || nil }
     _mystates = rules.map(&filter).flatten.compact.uniq
@@ -232,12 +379,20 @@ class OedipusLex
       all_states = [[nil, *inclusives],          # nil+incls # eg [[nil, :a],
                     *exclusives.map { |s| [s] }] # [excls]   #     [:A], [:B]]
 
-    encoding = header.shift if header.first =~ /encoding:/
+    encoding = header.shift if /encoding:/.match?(header.first)
     encoding ||= "# encoding: UTF-8"
 
-    ERB.new(TEMPLATE, nil, "%").result binding
+    erb = if RUBY_VERSION >= "2.6.0" then
+            ERB.new(TEMPLATE, trim_mode:"%")
+          else
+            ERB.new(TEMPLATE, nil, "%")
+          end
+
+    erb.result binding
   end
 
+  # :stopdoc:
+
   TEMPLATE = <<-'REX'.gsub(/^ {6}/, '')
       <%= encoding %>
       #--
@@ -254,36 +409,89 @@ class OedipusLex
 %   end
 
 % end
+
+      ##
+      # The generated lexer <%= class_name %>
+
       class <%= class_name %>
         require 'strscan'
 
 % unless macros.empty? then
+        # :stopdoc:
 %   max = macros.map { |(k,_)| k.size }.max
 %   macros.each do |(k,v)|
         <%= "%-#{max}s = %s" % [k, v] %>
 %   end
-
+        # :startdoc:
 % end
-        class ScanError < StandardError ; end
+        # :stopdoc:
+        class LexerError < StandardError ; end
+        class ScanError < LexerError ; end
+        # :startdoc:
+
+% if option[:lineno] then
+        ##
+        # The current line number.
 
         attr_accessor :lineno
+% end
+        ##
+        # The file name / path
+
         attr_accessor :filename
+
+        ##
+        # The StringScanner for this lexer.
+
         attr_accessor :ss
+
+        ##
+        # The current lexical state.
+
         attr_accessor :state
 
         alias :match :ss
 
+        ##
+        # The match groups for the current scan.
+
         def matches
           m = (1..9).map { |i| ss[i] }
           m.pop until m[-1] or m.empty?
           m
         end
 
+        ##
+        # Yields on the current action.
+
         def action
           yield
         end
 
+% if option[:column] then
+        ##
+        # The previous position. Only available if the :column option is on.
+
+        attr_accessor :old_pos
+
+        ##
+        # The position of the start of the current line. Only available if the
+        # :column option is on.
+
+        attr_accessor :start_of_current_line_pos
+
+        ##
+        # The current column, starting at 0. Only available if the
+        # :column option is on.
+        def column
+          old_pos - start_of_current_line_pos
+        end
+
+% end
 % if option[:do_parse] then
+        ##
+        # Parse the file by getting all tokens and calling lex_+type+ on them.
+
         def do_parse
           while token = next_token do
             type, *vals = token
@@ -293,18 +501,33 @@ class OedipusLex
         end
 
 % end
+
+        ##
+        # The current scanner class. Must be overridden in subclasses.
+
         def scanner_class
           StringScanner
         end unless instance_methods(false).map(&:to_s).include?("scanner_class")
 
+        ##
+        # Parse the given string.
+
         def parse str
           self.ss     = scanner_class.new str
+% if option[:lineno] then
           self.lineno = 1
+% end
+% if option[:column] then
+          self.start_of_current_line_pos = 0
+% end
           self.state  ||= nil
 
           do_parse
         end
 
+        ##
+        # Read in and parse the file at +path+.
+
         def parse_file path
           self.filename = path
           open path do |f|
@@ -312,6 +535,26 @@ class OedipusLex
           end
         end
 
+        ##
+        # The current location in the parse.
+
+        def location
+          [
+            (filename || "<input>"),
+% if option[:lineno] then
+            lineno,
+% elsif option[:column] then
+            "?",
+% end
+% if option[:column] then
+            column,
+% end
+          ].compact.join(":")
+        end
+
+        ##
+        # Lex the next token.
+
         def next_token
 % starts.each do |s|
           <%= s %>
@@ -321,7 +564,16 @@ class OedipusLex
 
           until ss.eos? or token do
 % if option[:lineno] then
-            self.lineno += 1 if ss.peek(1) == "\n"
+            if ss.peek(1) == "\n"
+              self.lineno += 1
+% if option[:column] then
+              # line starts 1 position after the newline
+              self.start_of_current_line_pos = ss.pos + 1
+% end
+            end
+% end
+% if option[:column] then
+            self.old_pos = ss.pos
 % end
             token =
               case state
@@ -336,17 +588,17 @@ class OedipusLex
 %   end # the_states.each
                 else
                   text = ss.string[ss.pos .. -1]
-                  raise ScanError, "can not match (#{state.inspect}): '#{text}'"
+                  raise ScanError, "can not match (#{state.inspect}) at #{location}: '#{text}'"
                 end
 % end # all_states
               else
-                raise ScanError, "undefined state: '#{state}'"
+                raise ScanError, "undefined state at #{location}: '#{state}'"
               end # token = case state
 
             next unless token # allow functions to trigger redo w/ nil
           end # while
 
-          raise "bad lexical result: #{token.inspect}" unless
+          raise LexerError, "bad lexical result at #{location}: #{token.inspect}" unless
             token.nil? || (Array === token && token.size >= 2)
 
           # auto-switch state
@@ -382,13 +634,16 @@ class OedipusLex
           begin
             rex.parse_file path
           rescue
-            $stderr.printf "%s:%d:%s\n", rex.filename, rex.lineno, $!.message
+            lineno = rex.respond_to?(:lineno) ? rex.lineno : -1
+            $stderr.printf "%s:%d:%s\n", rex.filename, lineno, $!.message
             exit 1
           end
         end
       end
 % end
   REX
+
+  # :startdoc:
 end
 
 if $0 == __FILE__ then

data/lib/oedipus_lex.rex

@@ -4,6 +4,7 @@ option
 
   do_parse
   lineno
+  column
 
 macro
   ST    /(?:(:\S+|\w+\??))/
@@ -31,6 +32,7 @@ rule
   :option       /debug/i                { [:option, text] }
   :option       /do_parse/i             { [:option, text] }
   :option       /lineno/i               { [:option, text] }
+  :option       /column/i               { [:option, text] }
 
   :inner        /.*/                    { [:inner, text] }
 

data/lib/oedipus_lex.rex.rb

@@ -1,36 +1,85 @@
 # encoding: UTF-8
 #--
 # This file is automatically generated. Do not modify it.
-# Generated by: oedipus_lex version 2.3.0.
+# Generated by: oedipus_lex version 2.5.2.
 # Source: lib/oedipus_lex.rex
 #++
 
+
+##
+# The generated lexer OedipusLex
+
 class OedipusLex
   require 'strscan'
 
+  # :stopdoc:
   ST  = /(?:(:\S+|\w+\??))/
   RE  = /(\/(?:\\.|[^\/])+\/[ion]?)/
   ACT = /(\{.*|:?\w+)/
+  # :startdoc:
+  # :stopdoc:
+  class LexerError < StandardError ; end
+  class ScanError < LexerError ; end
+  # :startdoc:
 
-  class ScanError < StandardError ; end
+  ##
+  # The current line number.
 
   attr_accessor :lineno
+  ##
+  # The file name / path
+
   attr_accessor :filename
+
+  ##
+  # The StringScanner for this lexer.
+
   attr_accessor :ss
+
+  ##
+  # The current lexical state.
+
   attr_accessor :state
 
   alias :match :ss
 
+  ##
+  # The match groups for the current scan.
+
   def matches
     m = (1..9).map { |i| ss[i] }
     m.pop until m[-1] or m.empty?
     m
   end
 
+  ##
+  # Yields on the current action.
+
   def action
     yield
   end
 
+  ##
+  # The previous position. Only available if the :column option is on.
+
+  attr_accessor :old_pos
+
+  ##
+  # The position of the start of the current line. Only available if the
+  # :column option is on.
+
+  attr_accessor :start_of_current_line_pos
+
+  ##
+  # The current column, starting at 0. Only available if the
+  # :column option is on.
+  def column
+    old_pos - start_of_current_line_pos
+  end
+
+  ##
+  # Parse the file by getting all tokens and calling lex_+type+ on them.
+
   def do_parse
     while token = next_token do
       type, *vals = token
@@ -39,18 +88,29 @@ class OedipusLex
     end
   end
 
+
+  ##
+  # The current scanner class. Must be overridden in subclasses.
+
   def scanner_class
     StringScanner
   end unless instance_methods(false).map(&:to_s).include?("scanner_class")
 
+  ##
+  # Parse the given string.
+
   def parse str
     self.ss     = scanner_class.new str
     self.lineno = 1
+    self.start_of_current_line_pos = 0
     self.state  ||= nil
 
     do_parse
   end
 
+  ##
+  # Read in and parse the file at +path+.
+
   def parse_file path
     self.filename = path
     open path do |f|
@@ -58,35 +118,54 @@ class OedipusLex
     end
   end
 
+  ##
+  # The current location in the parse.
+
+  def location
+    [
+      (filename || "<input>"),
+      lineno,
+      column,
+    ].compact.join(":")
+  end
+
+  ##
+  # Lex the next token.
+
   def next_token
-    self.lineno += 1 if ss.peek(1) == "\n"
 
     token = nil
 
     until ss.eos? or token do
+      if ss.peek(1) == "\n"
+        self.lineno += 1
+        # line starts 1 position after the newline
+        self.start_of_current_line_pos = ss.pos + 1
+      end
+      self.old_pos = ss.pos
       token =
         case state
         when nil, :option, :inner, :start, :macro, :rule, :group then
           case
-          when text = ss.scan(/options?.*/) then
+          when ss.skip(/options?.*/) then
             [:state, :option]
-          when text = ss.scan(/inner.*/) then
+          when ss.skip(/inner.*/) then
             [:state, :inner]
-          when text = ss.scan(/macros?.*/) then
+          when ss.skip(/macros?.*/) then
             [:state, :macro]
-          when text = ss.scan(/rules?.*/) then
+          when ss.skip(/rules?.*/) then
             [:state, :rule]
-          when text = ss.scan(/start.*/) then
+          when ss.skip(/start.*/) then
             [:state, :start]
-          when text = ss.scan(/end/) then
+          when ss.skip(/end/) then
             [:state, :END]
-          when text = ss.scan(/\A((?:.|\n)*)class ([\w:]+.*)/) then
+          when ss.skip(/\A((?:.|\n)*)class ([\w:]+.*)/) then
             action { [:class, *matches] }
-          when text = ss.scan(/\n+/) then
+          when ss.skip(/\n+/) then
             # do nothing
           when text = ss.scan(/\s*(\#.*)/) then
             action { [:comment, text] }
-          when (state == :option) && (text = ss.scan(/\s+/)) then
+          when (state == :option) && (ss.skip(/\s+/)) then
             # do nothing
           when (state == :option) && (text = ss.scan(/stub/i)) then
             action { [:option, text] }
@@ -96,44 +175,46 @@ class OedipusLex
             action { [:option, text] }
           when (state == :option) && (text = ss.scan(/lineno/i)) then
             action { [:option, text] }
+          when (state == :option) && (text = ss.scan(/column/i)) then
+            action { [:option, text] }
           when (state == :inner) && (text = ss.scan(/.*/)) then
             action { [:inner, text] }
           when (state == :start) && (text = ss.scan(/.*/)) then
             action { [:start, text] }
-          when (state == :macro) && (text = ss.scan(/\s+(\w+)\s+#{RE}/o)) then
+          when (state == :macro) && (ss.skip(/\s+(\w+)\s+#{RE}/o)) then
             action { [:macro, *matches] }
-          when (state == :rule) && (text = ss.scan(/\s*#{ST}?[\ \t]*#{RE}[\ \t]*#{ACT}?/o)) then
+          when (state == :rule) && (ss.skip(/\s*#{ST}?[\ \t]*#{RE}[\ \t]*#{ACT}?/o)) then
             action { [:rule, *matches] }
-          when (state == :rule) && (text = ss.scan(/\s*:[\ \t]*#{RE}/o)) then
+          when (state == :rule) && (ss.skip(/\s*:[\ \t]*#{RE}/o)) then
             action { [:grouphead, *matches] }
-          when (state == :group) && (text = ss.scan(/\s*:[\ \t]*#{RE}/o)) then
+          when (state == :group) && (ss.skip(/\s*:[\ \t]*#{RE}/o)) then
             action { [:grouphead, *matches] }
-          when (state == :group) && (text = ss.scan(/\s*\|\s*#{ST}?[\ \t]*#{RE}[\ \t]*#{ACT}?/o)) then
+          when (state == :group) && (ss.skip(/\s*\|\s*#{ST}?[\ \t]*#{RE}[\ \t]*#{ACT}?/o)) then
             action { [:group, *matches] }
-          when (state == :group) && (text = ss.scan(/\s*#{ST}?[\ \t]*#{RE}[\ \t]*#{ACT}?/o)) then
+          when (state == :group) && (ss.skip(/\s*#{ST}?[\ \t]*#{RE}[\ \t]*#{ACT}?/o)) then
             action { [:groupend, *matches] }
           else
             text = ss.string[ss.pos .. -1]
-            raise ScanError, "can not match (#{state.inspect}): '#{text}'"
+            raise ScanError, "can not match (#{state.inspect}) at #{location}: '#{text}'"
           end
         when :END then
           case
-          when text = ss.scan(/\n+/) then
+          when ss.skip(/\n+/) then
             # do nothing
           when text = ss.scan(/.*/) then
             action { [:end, text] }
           else
             text = ss.string[ss.pos .. -1]
-            raise ScanError, "can not match (#{state.inspect}): '#{text}'"
+            raise ScanError, "can not match (#{state.inspect}) at #{location}: '#{text}'"
           end
         else
-          raise ScanError, "undefined state: '#{state}'"
+          raise ScanError, "undefined state at #{location}: '#{state}'"
         end # token = case state
 
       next unless token # allow functions to trigger redo w/ nil
     end # while
 
-    raise "bad lexical result: #{token.inspect}" unless
+    raise LexerError, "bad lexical result at #{location}: #{token.inspect}" unless
       token.nil? || (Array === token && token.size >= 2)
 
     # auto-switch state

data/test/test_oedipus_lex.rb

@@ -1,4 +1,3 @@
-gem "minitest"
 require "minitest/autorun"
 require "oedipus_lex"
 require "stringio"
@@ -83,6 +82,26 @@ class TestOedipusLex < Minitest::Test
     assert_equal expected_msg, e.message
   end
 
+  def assert_token_error grammar, input, expected_msg
+    _, mod = eval_lexer grammar
+
+    calc = mod::Calculator.new
+
+    def calc.do_parse
+      tokens = []
+      while token = next_token
+        tokens << token
+      end
+      tokens
+    end
+
+    e = assert_raises mod::Calculator::LexerError do
+      calc.parse input
+    end
+
+    assert_equal expected_msg, e.message
+  end
+
   def test_simple_scanner
     src = <<-'REX'
       class Calculator
@@ -137,7 +156,7 @@ class TestOedipusLex < Minitest::Test
       end
     REX
 
-    assert_generate_error src, "can not match (:rule): '"
+    assert_generate_error src, "can not match (:rule) at <input>:4:0: '"
   end
 
   def test_simple_scanner_macro
@@ -260,6 +279,30 @@ class TestOedipusLex < Minitest::Test
     assert_match "[:op, \"+\"]", out
   end
 
+  def test_column
+    src = <<-'REX'
+      class Calculator
+      rule
+              /\d+/       { [:number, text.to_i, lineno, column] }
+              /\s+/
+              /[+-]/      { [:op, text, lineno, column] }
+      end
+    REX
+
+    txt = "1 + 2\n+ 30"
+
+    exp = [[:number, 1,   1, 0],
+           [:op,     "+", 1, 2],
+           [:number, 2,   1, 4],
+           [:op,     "+", 2, 0],
+           [:number, 30,  2, 2]]
+
+    option[:column] = true
+    option[:lineno] = true
+
+    assert_lexer src, txt, exp
+  end
+
   def test_simple_scanner_debug_src
     src = <<-'REX'
       class Calculator
@@ -434,7 +477,7 @@ class TestOedipusLex < Minitest::Test
 
     ruby = generate_lexer src
 
-    assert_match "when ss.check(/\\d/) then", ruby
+    assert_match "when ss.match?(/\\d/) then", ruby
     assert_match "when text = ss.scan(/\\d+\\.\\d+/) then", ruby
     assert_match "when text = ss.scan(/\\d+/) then", ruby
     assert_match "end # group /\\d/", ruby
@@ -457,12 +500,12 @@ class TestOedipusLex < Minitest::Test
 
     ruby = generate_lexer src
 
-    assert_match "when ss.check(/\\d/) then", ruby
+    assert_match "when ss.match?(/\\d/) then", ruby
     assert_match "when text = ss.scan(/\\d+\\.\\d+/) then", ruby
     assert_match "when text = ss.scan(/\\d+/) then", ruby
     assert_match "end # group /\\d/", ruby
 
-    assert_match "when ss.check(/\\+/) then", ruby
+    assert_match "when ss.match?(/\\+/) then", ruby
     assert_match "when xx? && (text = ss.scan(/\\+whatever/)) then", ruby
     assert_match "when (state == :x) && (text = ss.scan(/\\+\\d+/)) then", ruby
     assert_match "end # group /\\d/", ruby
@@ -737,9 +780,50 @@ class TestOedipusLex < Minitest::Test
     assert_lexer src, txt, exp
 
     txt = "aa"
-    exp = [[:A, 'a'], [:B, 'b'], [:A, 'a'], [:B, 'b'], [:A, 'a']]
 
-    assert_lexer_error src, txt, "can not match (:B): 'a'"
+    assert_lexer_error src, txt, "can not match (:B) at <input>: 'a'"
+  end
+
+  def test_error_undefined_state
+    src = <<-'REX'
+      class Calculator
+      rule
+             /a/       { self.state = :C  ; [:A, text] }
+        :B   /b/       { self.state = nil ; [:B, text] }
+      end
+    REX
+
+    txt = "aa"
+
+    assert_lexer_error src, txt, "undefined state at <input>: 'C'"
+  end
+
+  def test_error_bad_token
+    src = <<-'REX'
+      class Calculator
+      rule
+             /a/       { self.state = :B  ; :A }
+        :B   /b/       { self.state = nil ; [:B, text] }
+      end
+    REX
+
+    txt = "aa"
+
+    assert_token_error src, txt, "bad lexical result at <input>: :A"
+  end
+
+  def test_error_bad_token_size
+    src = <<-'REX'
+      class Calculator
+      rule
+             /a/       { self.state = :B  ; [:A] }
+        :B   /b/       { self.state = nil ; [:B, text] }
+      end
+    REX
+
+    txt = "aa"
+
+    assert_token_error src, txt, "bad lexical result at <input>: [:A]"
   end
 
   def test_incrementing_lineno_on_nil_token
@@ -758,4 +842,36 @@ class TestOedipusLex < Minitest::Test
 
     assert_lexer src, txt, exp
   end
+
+  def assert_location exp, option = {}
+    self.option = option
+
+    src = "class Calculator\nrule\n  /\\d+/ { [:number, text.to_i] }\nend\n"
+
+    _, mod = eval_lexer src
+
+    calc = mod::Calculator.new
+    def calc.do_parse
+      [next_token]
+    end
+
+    calc.filename = option[:filename] if option[:filename]
+    calc.parse "42"
+
+    assert_equal exp, calc.location
+  end
+
+  def test_location
+    t = true
+
+    assert_location "<input>"
+    assert_location "<input>:1",   :lineno => t
+    assert_location "<input>:?:0", :column => t
+    assert_location "<input>:1:0", :lineno => t, :column => t
+
+    assert_location "blah",     :filename => "blah"
+    assert_location "blah:1",   :filename => "blah", :lineno => t
+    assert_location "blah:?:0", :filename => "blah", :column => t
+    assert_location "blah:1:0", :filename => "blah", :lineno => t, :column => t
+  end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: oedipus_lex
 version: !ruby/object:Gem::Version
-  version: 2.3.2
+  version: 2.5.2
 platform: ruby
 authors:
 - Ryan Davis
@@ -10,9 +10,9 @@ bindir: bin
 cert_chain:
 - |
   -----BEGIN CERTIFICATE-----
-  MIIDPjCCAiagAwIBAgIBATANBgkqhkiG9w0BAQUFADBFMRMwEQYDVQQDDApyeWFu
+  MIIDPjCCAiagAwIBAgIBBDANBgkqhkiG9w0BAQsFADBFMRMwEQYDVQQDDApyeWFu
   ZC1ydWJ5MRkwFwYKCZImiZPyLGQBGRYJemVuc3BpZGVyMRMwEQYKCZImiZPyLGQB
-  GRYDY29tMB4XDTEzMDkxNjIzMDQxMloXDTE0MDkxNjIzMDQxMlowRTETMBEGA1UE
+  GRYDY29tMB4XDTE5MTIxMzAwMDIwNFoXDTIwMTIxMjAwMDIwNFowRTETMBEGA1UE
   AwwKcnlhbmQtcnVieTEZMBcGCgmSJomT8ixkARkWCXplbnNwaWRlcjETMBEGCgmS
   JomT8ixkARkWA2NvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALda
   b9DCgK+627gPJkB6XfjZ1itoOQvpqH1EXScSaba9/S2VF22VYQbXU1xQXL/WzCkx
@@ -21,58 +21,50 @@ cert_chain:
   GiadM9GHRaDiaxuX0cIUBj19T01mVE2iymf9I6bEsiayK/n6QujtyCbTWsAS9Rqt
   qhtV7HJxNKuPj/JFH0D2cswvzznE/a5FOYO68g+YCuFi5L8wZuuM8zzdwjrWHqSV
   gBEfoTEGr7Zii72cx+sCAwEAAaM5MDcwCQYDVR0TBAIwADALBgNVHQ8EBAMCBLAw
-  HQYDVR0OBBYEFEfFe9md/r/tj/Wmwpy+MI8d9k/hMA0GCSqGSIb3DQEBBQUAA4IB
-  AQCFZ7JTzoy1gcG4d8A6dmOJy7ygtO5MFpRIz8HuKCF5566nOvpy7aHhDDzFmQuu
-  FX3zDU6ghx5cQIueDhf2SGOncyBmmJRRYawm3wI0o1MeN6LZJ/3cRaOTjSFy6+S6
-  zqDmHBp8fVA2TGJtO0BLNkbGVrBJjh0UPmSoGzWlRhEVnYC33TpDAbNA+u39UrQI
-  ynwhNN7YbnmSR7+JU2cUjBFv2iPBO+TGuWC+9L2zn3NHjuc6tnmSYipA9y8Hv+As
-  Y4evBVezr3SjXz08vPqRO5YRdO3zfeMT8gBjRqZjWJGMZ2lD4XNfrs7eky74CyZw
-  xx3n58i0lQkBE1EpKE0lFu/y
+  HQYDVR0OBBYEFEfFe9md/r/tj/Wmwpy+MI8d9k/hMA0GCSqGSIb3DQEBCwUAA4IB
+  AQCkkcHqAa6IKLYGl93rn78J3L+LnqyxaA059n4IGMHWN5bv9KBQnIjOrpLadtYZ
+  vhWkunWDKdfVapBEq5+T4HzqnsEXC3aCv6JEKJY6Zw7iSzl0M8hozuzRr+w46wvT
+  fV2yTN6QTVxqbMsJJyjosks4ZdQYov2zdvQpt1HsLi+Qmckmg8SPZsd+T8uiiBCf
+  b+1ORSM5eEfBQenPXy83LZcoQz8i6zVB4aAfTGGdhxjoMGUEmSZ6xpkOzmnGa9QK
+  m5x9IDiApM+vCELNwDXXGNFEnQBBK+wAe4Pek8o1V1TTOxL1kGPewVOitX1p3xoN
+  h7iEjga8iM1LbZUfiISZ+WrB
   -----END CERTIFICATE-----
-date: 2014-08-07 00:00:00.000000000 Z
+date: 2020-06-14 00:00:00.000000000 Z
 dependencies:
-- !ruby/object:Gem::Dependency
-  name: minitest
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ~>
-      - !ruby/object:Gem::Version
-        version: '5.4'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ~>
-      - !ruby/object:Gem::Version
-        version: '5.4'
 - !ruby/object:Gem::Dependency
   name: rdoc
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - ">="
       - !ruby/object:Gem::Version
         version: '4.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '7'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - ">="
       - !ruby/object:Gem::Version
         version: '4.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '7'
 - !ruby/object:Gem::Dependency
   name: hoe
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.12'
+        version: '3.22'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.12'
+        version: '3.22'
 description: |-
   Oedipus Lex is a lexer generator in the same family as Rexical and
   Rex. Oedipus Lex is my independent lexer fork of Rexical. Rexical was
@@ -88,7 +80,7 @@ description: |-
 
   Oedipus, like rexical, is based primarily on generating code much like
   you would a hand-written lexer. It is _not_ a table or hash driven
-  lexer. It use StrScanner within a multi-level case statement. As such,
+  lexer. It uses StrScanner within a multi-level case statement. As such,
   Oedipus matches on the _first_ match, not the longest (like lex and
   its ilk).
 
@@ -106,8 +98,7 @@ extra_rdoc_files:
 - README.rdoc
 - sample/error1.txt
 files:
-- .autotest
-- .gemtest
+- ".autotest"
 - History.rdoc
 - Manifest.txt
 - README.rdoc
@@ -137,28 +128,27 @@ files:
 homepage: http://github.com/seattlerb/oedipus_lex
 licenses:
 - MIT
-metadata: {}
+metadata:
+  homepage_uri: http://github.com/seattlerb/oedipus_lex
 post_install_message: 
 rdoc_options:
-- --main
+- "--main"
 - README.rdoc
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
-  - - '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.2.1
+rubygems_version: 3.0.3
 signing_key: 
 specification_version: 4
 summary: Oedipus Lex is a lexer generator in the same family as Rexical and Rex
-test_files:
-- test/test_oedipus_lex.rb
+test_files: []

metadata.gz.sig

@@ -1,3 +1 @@
-N�o�n	���QR��5?	�F5�de`®�n���
-	�4�s���-�
[ ����X���/{��sI���\��CU�4��7�Z0�st㋾-���CD�m푹�&��<_�+Vʈs��T>�m4���	�/�ŠG�TT�-Akch]�U
-�`zU�QSu��|��"����%\֙3���Zp^�-]�������qs�Z���T���`s�x�vG�[�m�6�f%�����j+��A��4��װ�\��[����
+jl�f��J*��_X%��d��s��	�g�Cao���G7�!�ߡm�������:J6���M�͡���I� Ϟ�p��@�5)�}G�0F^�cń�o�_����zx,������L/���s+sQ7�6��g�-���J��+I�0�3�#`o,�gw��ԁ�#���Yr��"p�a�w ���u�k|��tc$x�=�M��Ѥ�[e$����`�R����ke�i��&�'���ŗ�����)