rubylexer 0.6.2

data/README

@@ -0,0 +1,134 @@
+                              -=RubyLexer 0.6.2=-
+
+RubyLexer is a lexer library for Ruby, written in Ruby. My goal with Rubylexer
+was to create a lexer for Ruby that's complete and correct; all legal Ruby 
+code should be lexed correctly by RubyLexer as well. Just enough parsing 
+capability is included to give RubyLexer enough context to tokenize correctly
+in all cases. (This turned out to be more parsing than I had thought or 
+wanted to take on at first.)
+
+Other Ruby lexers exist, but most are inadequate. For instance, irb has it's 
+own little lexer, as does, (I believe) RDoc, so do all the ide's that can 
+colorize. I've seen several stand-alone libraries as well. All or almost all 
+suffer from the same problems: they skip the hard part of lexing. RubyLexer 
+handles the hard things like complicated strings, the ambiguous nature of 
+some punctuation characters and keywords in ruby, and distinguishing methods 
+and local variables.
+
+RubyLexer is not particularly clean code. As I progressed in writing this, 
+I've learned a little about how these things are supposed to be done; the 
+lexer is not supposed to have any state of it's own, instead it gets whatever 
+it needs to know from the parser. As a stand-alone lexer, Rubylexer maintains 
+quite a lot of state. Every instance variable in the RubyLexer class is some 
+sort of lexer state. Most of the complication and ugly code in RubyLexer is 
+in maintaining or using this state.
+
+For information about using RubyLexer in your program, please see howtouse.txt.
+
+For my notes on the testing of RubyLexer, see testing.txt.
+
+If you have any questions, comments, problems, new feature requests, or just
+want to figure out how to make it work for what you need to do, contact me: 
+       rubylexer _at_ inforadical.net
+
+RubyLexer is a RubyForge project. RubyForge is another good place to send your
+bug reports or whatever:  http://rubyforge.org/projects/rubylexer/
+
+(There aren't any bug filed against RubyLexer there yet, but don't be afraid 
+that your report will get lonely.)
+
+Status:
+RubyLexer can correctly lex all legal Ruby 1.8 code that I've been able to 
+find on my Debian system. It can also handle (most of) my catalog of nasty 
+test cases (in testdata/p.rb). At this point, new bugs are almost exclusively 
+found by my home-grown test code, rather than ruby code gathered 'from the 
+wild'. A largish sample of ruby recently tested for the first time (that is, 
+Rubyx) had _0_ lex errors. (And this is not the only example.) There are a 
+number of issues i know about and plan to fix, but it seems that Ruby coders 
+don't write code complex enough to trigger them very often. Although 
+incomplete, RubyLexer is nevertheless better than many existing ad-hoc 
+lexers. For instance, RubyLexer can correctly distinguish all cases of the 
+different uses the following operators, depending on context:
+  %   can be modulus operator or start of fancy string
+  /   can be division operator or start of regex
+  * & + -  can be unary or binary operator
+  []  can be for array literal or index method
+  <<  can be here document or left shift operator (or in class<<obj expr)
+  ::  can be unary or binary operator
+  :   can be start of symbol, substitute for then, or part of ternary op
+      (there are other uses too, but they're not supported yet.)
+  ?   can be start of character constant or ternary operator
+  `   can be method name or start of exec string
+
+todo:
+test w/ more code (rubygems, rpa, obfuscated ruby contest, rubicon, others?)
+these 5 should be my standard test suite: p.rb, (matz') test.rb, tk.rb, obfuscated ruby contest, rubicon
+test more ways: cvt source to dos or mac fmt before testing
+test more ways: run unit tests after passing thru rubylexer (0.7)
+test more ways: test require'd, load'd, or eval'd code as well (0.7)
+lex code a line (or chunk) at a time and save state for next line (irb wants this) (0.8)
+incremental lexing (ides want this (for performance))
+put everything in a namespace
+integrate w/ other tools...
+html colorized output?
+move more state onto @bracestack (ongoing)
+expand on test documentation above
+the new cases in p.rb now compile, but won't run
+use want_op_name more
+return result as a half-parsed tree (with parentheses and the like matched)
+emit advisory tokens when see beginword, then (or equivalent), or end... what else does florian want?
+strings are still slow
+rantfile
+emit advisory tokens when local var defined/goes out of scope (or hidden/unhidden?)
+fakefile should be a mixin
+token pruning in dumptokens...
+
+new ruby features not yet supported:
+procs without proc keyword, looks like hash to current lexer
+keyword arguments, in hash immediates or actual param lists (&formal param lists?)
+unicode (0.9)
+:wrap and friends... (i wish someone would make a list of all the uses of colon in ruby.)
+parens in block param list (works, but hacky)
+
+
+known issues: (and planned fix release)
+context not really preserved when entering or leaving string inclusions. this causes
+a number or problems. (0.8)
+string tokenization sometimes a little different from ruby around newlines
+  (htree/template.rb) (0.8)
+string contents might not be correctly translated in a few cases (0.8?)
+the implicit tokens might be emitted at the wrong times. (or not at the right times) (need more test code) (0.7)
+local variables should be temporarily hidden by class, module, and def (0.7)
+windows or mac newline in source are likely to cause problems in obscure cases (need test case)
+line numbers are sometimes off... probably to do with multi-line strings (=begin...=end causes this) (0.8)
+symbols which contain string interpolations are flattened into one token. eg :"foo#{bar}" (0.8)
+methnames and varnames might get mixed up in def header (in idents after the 'def' but before param list) (0.7)
+FileAndLineToken not emitted everywhere it should be (0.8)
+'\r' whitespace sometimes seen in dos-formatted output.. shouldn't be (eg pre.rb) (0.7)
+no way to get offset of __END__ (??) (0.7)
+put things in lib/
+
+
+fixed issues in 0.6.2:
+testcode/dumptokens.rb charhandler.rb doesn't work... but does after unix2dos (not reproducible)
+files should be opened in binmode to avoid all possible eol translation
+(x.+?x) doesn't work
+methname/varname mixups in some cases
+performance, in most important cases.
+error handling tokens should be emitted on error input... ErrorToken mixin module
+but old error handling interface should be preserved and made available.
+move readahead and friends into IOext. make optimized readahead et al for fakefile.
+dos newlines (and newlines generally) can't be fancy string delimiters
+do,if,until, etc, have no way to tell if an end is associated
+break readme into pieces
+
+
+fixed issues in 0.6.0:
+the implicit tokens might be emitted at the wrong times. (or not at the right times) (partly fixed) (0.6)
+: operator might be a synonym for 'then' (0.6)
+variables other than the last are not recognized in multiple assignment. (0.7)
+variables created by for and rescue are not recognized. (0.7)
+token following :: should not be BareSymbolToken if begins with A-Z (unless obviously a func, eg b/c followed by func param list)
+read code to be lexed from a string. (irb wants this) (0.7)
+fancy symbols don't work at all. (like this:  %s{abcdefg}) (0.7) [this is regressing now]
+Newline,EscNl,BareSymbolToken may get renamed

data/Rantfile

@@ -0,0 +1,37 @@
+  import %w(rubydoc rubypackage)
+
+
+    test_files=Dir["test{code,data}/*.rb"] 
+    lib_files = Dir["lib/**/*.rb"]  #need to actually put files here...
+    dist_files = lib_files + %w(Rantfile README COPYING) + test_files
+
+    desc "Run unit tests."
+    task :test do
+      sys.mkdir 'testresults'
+      test_files.each{|f| 
+            sys.ruby "testcode/rubylexervsruby.rb", f
+      }
+      lib_files.each{|f| 
+            sys.ruby "testcode/rubylexervsruby.rb", f
+      }
+      system 'which locate grep' && 
+        sys.ruby "testcode/rubylexervsruby.rb",  `locate /tk.rb|grep 'tk.rb$'`
+    end
+
+    desc "Generate html documentation."
+    gen RubyDoc do |t|
+        t.opts = %w(--title RubyLexer --main README README)
+    end
+
+    desc "Create packages."
+    gen RubyPackage, :rubylexer do |t|
+        t.version = "0.6.1"
+        t.summary = "A complete lexer of ruby in ruby."
+        t.files = dist_files
+        t.package_task :gem
+        #need more here
+    end
+
+    task :clean do
+        sys.rm_rf %w(doc pkg testresults)
+    end

data/assert.rb

@@ -0,0 +1,31 @@
+=begin copyright
+    rubylexer - a ruby lexer written in ruby
+    Copyright (C) 2004,2005  Caleb Clausen
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+=end
+require 'set'
+
+
+def assert(expr,msg="assertion failed")
+  $DEBUG and (expr or raise msg)
+end
+
+@@printed=Set.new
+def fixme(s)
+  @@printed.include?( s) and return
+  $DEBUG and STDERR.print "FIXME: #{s}\n"
+  @@printed.add s
+end

data/charhandler.rb

@@ -0,0 +1,84 @@
+=begin copyright
+    rubylexer - a ruby lexer written in ruby
+    Copyright (C) 2004,2005  Caleb Clausen
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+=end
+
+require "charset"
+#------------------------------------
+class CharHandler
+  #-----------------------------------
+  CHARSETSPECIALS=CharSet[?[ ,?] ,?\\ ,?-]
+  def initialize(receiver,default,hash) 
+    @default=default 
+    @receiver=receiver
+ #breakpoint
+    @table=Array.new(0)
+    @matcher='^[^'
+
+    hash.each_pair {|pattern,action|
+      case pattern
+      when Range
+        pattern.each { |c|
+          c.kind_of? String and c=c[0] #cvt to integer  #still needed?
+          self[c]=action
+        }
+      when String
+        pattern.each_byte {|b| self[b]=action }
+      when Fixnum
+        self[pattern]=action
+      else
+        raise "invalid pattern class #{pattern.class}"
+      end
+    }
+
+    @matcher += ']$'
+    @matcher=Regexp.new(@matcher)
+
+    freeze
+  end
+
+  #-----------------------------------
+  def []=(b,action)  #for use in initialize only
+    assert b >= ?\x00
+    assert b <= ?\xFF
+    assert !frozen?
+
+    @table[b]=action
+    @matcher<<?\\ if CHARSETSPECIALS===b
+    @matcher<<b
+  end
+  private :[]=
+
+  #-----------------------------------
+  def go(b,*args)
+    @receiver.send((@table[b] or @default), b.chr, *args)
+  end
+
+  #-----------------------------------
+  def eat_file(file,blocksize,*args)
+    begin
+      chars=file.read(blocksize)
+      md=@matcher.match(chars)
+      mychar=md[0][0]
+      #get file back in the right pos
+      file.pos+=md.offset(0)[0] - chars.length
+      @receiver.send(@default,md[0])
+    end until go(mychar,*args)
+  end
+end
+
+

data/charset.rb

@@ -0,0 +1,76 @@
+=begin copyright
+    rubylexer - a ruby lexer written in ruby
+    Copyright (C) 2004,2005  Caleb Clausen
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+=end
+
+class CharSet
+   def initialize(*charss)
+      clear
+      charss.each {|chars| add chars }
+   end
+
+   def CharSet.[](*charss)        CharSet.new(*charss)       end
+
+   def clear
+      @bitset=0
+   end
+
+   def add(chars)
+      case chars
+      when String
+         chars.each_byte {|c| @bitset |= (1<<c) }
+      when Fixnum then        @bitset |= (1<<chars)
+      else chars.each    {|c| @bitset |= (1<<c) }
+      end
+   end
+
+   def remove(chars)
+      case chars
+      when String
+         chars.each_byte {|c| @bitset &= ~(1<<c) }
+      when Fixnum then        @bitset &= ~(1<<chars)
+      else chars.each    {|c| @bitset &= ~(1<<c) }
+      end
+      #this math works right with bignums... (i'm pretty sure)
+   end
+
+   def ===(c) #c is String|Fixnum|nil
+      c.nil? and return false
+      c.kind_of? String and c=c[0]
+      return ( @bitset[c] != 0 )
+   end
+
+   #enumerate the chars in n AS INTEGERS
+   def each_byte(&block)
+      #should use ffs... not available in ruby
+      (0..255).each { |n|
+         @bitset[n] and block[n]
+      }
+   end
+
+   def each(&block)
+      each_byte{|n| block[n.chr] }
+   end
+
+   def chars #turn bitset back into a string
+      result=''
+      each {|c| result << c }
+      return result
+   end
+end
+
+

data/context.rb

@@ -0,0 +1,174 @@
+module NestedContexts
+  class NestedContext
+    attr :starter
+    attr :ender
+    attr :linenum
+    def initialize(starter,ender,linenum)
+      @starter,@ender,@linenum=starter,ender,linenum
+    end
+
+    alias dflt_initialize initialize
+
+    def matches?(tok)
+      @ender==tok
+    end
+    
+    def see stack,msg; end
+  end
+
+  class ListContext < NestedContext 
+  end
+  
+  class ListImmedContext < ListContext
+    def initialize(starter,linenum)
+      assert '{['[starter]
+      super(starter, starter.tr('{[','}]') ,linenum)
+    end
+  end
+
+  class ParenContext < NestedContext
+    def initialize(linenum)
+      super('(', ')' ,linenum)
+    end
+  end
+
+  class BlockContext  < NestedContext
+    def initialize(linenum)
+      super('{','}',linenum)
+    end
+  end
+
+  class BlockParamListContext  < ListContext
+    def initialize(linenum)
+      super('|','|',linenum)
+    end
+  end
+
+  class ParamListContext < ListContext
+    def initialize(linenum)
+      super('(', ')',linenum)
+    end
+  end
+  
+  class ImplicitContext < ListContext
+  end
+
+  class ParamListContextNoParen < ImplicitContext
+    def initialize(linenum)
+      dflt_initialize(nil,nil,linenum)
+    end
+  end
+
+  class KwParamListContext < ImplicitContext
+    def initialize(starter,linenum)
+      dflt_initialize(starter,nil,linenum)
+    end
+  end
+
+  class AssignmentRhsContext < ImplicitContext
+    def initialize(linenum)
+      dflt_initialize(nil,nil,linenum)
+    end
+  end
+
+  class WantsEndContext < NestedContext
+    def initialize(starter,linenum)
+      super(starter,'end',linenum)
+    end
+    
+    def see stack,msg
+      msg==:rescue ? stack.push_rescue_sm : super
+    end
+  end
+
+  class StringContext < NestedContext #not used yet
+    def initialize(starter,linenum)
+      super(starter, starter[-1,1].tr!('{[(','}])'),linenum)
+    end
+  end
+
+  class HereStringContext < StringContext #not used yet
+    def initialize(ender,linenum)
+      dflt_initialize("\n",ender,linenum)
+    end
+  end
+
+  class TopLevelContext < NestedContext
+    def initialize
+      dflt_initialize('','',1)
+    end
+  end
+
+
+  class RescueSMContext < NestedContext
+    #normal progression: rescue => arrow => then
+    EVENTS=[:rescue,:arrow,:then,:semi,:colon]
+    LEGAL_SUCCESSORS={nil=> [:rescue], :rescue => [:arrow,:then,:semi,:colon],:arrow => [:then,:semi,:colon],:then => [nil]}
+    #note on :semi and :colon events: in arrow state (and only then), 
+    #      (unescaped) newline, semicolon, and (unaccompanied) colon 
+    #      also trigger the :then event. otherwise, they are ignored.
+    attr :state
+    
+    def initialize linenum
+      dflt_initialize("rescue","then",linenum)
+      @state=nil
+      @state=:rescue 
+    end
+    
+    def see(stack,msg)
+      case msg
+      when :rescue: 
+        WantsEndContext===stack.last or 
+          BlockContext===stack.last or 
+          ParenContext===stack.last or 
+          raise 'syntax error: rescue not expected at this time'
+      when :arrow: #local var defined in this state
+      when :then,:semi,:colon:
+        msg=:then
+        RescueSMContext===stack.pop or raise 'syntax error: then not expected at this time'
+                  #pop self off owning context stack
+      else super
+      end
+      LEGAL_SUCCESSORS[@state].include? msg or raise "rescue syntax error: #{msg} unexpected in #@state"
+      @state=msg
+    end
+    
+  end
+
+  class ForSMContext < NestedContext
+    #normal progression: for => in 
+    EVENTS=[:for,:in]
+    LEGAL_SUCCESSORS={nil=> :for, :for => :in,:in => nil}
+    #note on :semi and :colon events: in :in state (and only then), 
+    #      (unescaped) newline, semicolon, and (unaccompanied) colon 
+    #      also trigger the :then event. otherwise, they are ignored.
+    attr :state
+    
+    def initialize linenum
+      dflt_initialize("for","in",linenum)
+      @state=:for
+    end
+    
+    def see(stack,msg)
+      case msg
+      when :for: WantsEndContext===stack.last or raise 'syntax error: for not expected at this time'
+                 #local var defined in this state
+      when :in:  ForSMContext===stack.pop or raise 'syntax error: in not expected at this time'
+                 stack.push ExpectDoOrNlContext.new("for",/(do|;|:|\n)/,@linenum) 
+                 #pop self off owning context stack and push ExpectDoOrNlContext
+      else super
+      end
+      LEGAL_SUCCESSORS[@state] == msg or raise 'for syntax error: #{msg} unexpected in #@state'
+      @state=msg
+    end    
+  end
+
+  class ExpectDoOrNlContext < NestedContext
+  end
+
+  class TernaryContext < NestedContext
+    def initialize(linenum)
+      dflt_initialize('?',':',linenum)
+    end
+  end
+end