rubylexer 0.6.2

data/howtouse.txt

@@ -0,0 +1,136 @@
+
+Using rubylexer:
+require "rubylexer.rb"
+ ,then
+lexer=RubyLexer.new(a_file_name, opened_File_or_String)
+until EoiToken===(tok=lexer.get1token)
+  ...do stuff w/ toks...
+end
+
+For a slightly expanded version of this example, see testcode/dumptokens.rb.
+
+tok will be a subclass of Token. there are many token classes (see token.rb)
+however, all tokens have some common methods:
+to_s   #return a string containing ruby code representing that token
+ident  #return internal form of token; use with caution
+offset #offset in file of start of token
+error  #returns a string if there was a lex error at this position, else nil
+
+here's a list of token subclasses and their meaning:
+(note: indentation indicates inheiritance)
+
+WToken       #(mostly useless?) abstract superclass for KeywordToken, 
+             #OperatorToken, VarNameToken, and HerePlaceholderToken
+             #but not (confusingly) MethNameToken (perhaps that'll change)
+ KeywordToken #a ruby keyword or non-overridable punctuation char(s)
+ OperatorToken #overrideable operators
+ VarNameToken #a name that represents a variable
+ HerePlaceholderToken #represents the header of a here string. subclass of WToken
+MethNameToken  #the name of a method: the uncoloned
+                 #symbols allowed in 'alias' and 'undef' statements and all names
+                 #which follow a 'def',
+                 #'::', or '.', as well as other call sites. operators used as
+                 #method names will appear as methnametokens.
+                 #confusingly, this is not a WToken.
+NumberToken  #a literal number, including character constants
+SymbolToken  #a symbol
+NewlineToken #represents an (unescaped) newline.
+StringToken  #represents a string. unlike all other tokens, strings might contain
+             #other tokens. if the string used interpolation, tokens inside #{ }
+             #are considered subtokens of the string. StringToken#elems returns
+             #an array whose elements are sections of uninterpolated string (in
+             #the even indeces) and arrays of subtokens (in the odd indeces).
+             #this notion of subtokens is an unfortunate one and will go away in
+             #a future release.
+ RenderExactlyStringToken  #a subclass of StringToken; used to represent regexes and other string-like thingys
+
+ErrorToken   #actually a module that may be mixed in to any token. indicates an error in the input at (or
+             #near) that position. You may continue getting tokens after an error token is encountered, 
+             #and I try to make this work as well as possible, but I can not guarantee correctness after
+             #an error.
+             #note: any token may be an ErrorToken, including IgnoreToken, EoiToken, a subtoken of a 
+             #StringToken, etc. Please take this into account in your error processing.
+
+IgnoreToken  #superclass for tokens without semantic meaning to a parser
+ WsToken      #whitespace
+ EscNlToken   #implicitly or explicitly escaped newline
+ EoiToken     #end of source file. always the last token
+ HereBodyToken        #the actual body of the here string. subclass of IgnoreToken
+  OutlinedHereBodyToken #hacky subclass of HereBodyToken... will disappear once strings are done right.
+
+ ZwToken      #informational IgnoreTokens. (parsers might need to look at some of these, actually.)
+  NoWsToken    #no whitespace was on either side of this token. kind of a hack
+               #to help TokenPrinter work correctly in certain cases.
+
+  ImplicitParamListStartToken #if you leave the parentheses out in a function
+  ImplicitParamListEndToken   #call, a pair of these will be generated instead
+
+  KwParamListStartToken       #the when,for,and rescue keywords take a comma-
+  KwParamListEndToken         #delimited list. these tokens enclose those lists.
+
+  AssignmentRhsListStartToken #encloses the right hand side of an assignment,
+  AssignmentRhsListEndToken   #including both single and multiple assignment.
+
+  FileAndLineToken   #generated at every newline, escaped or unescaped. the file
+                     #and line methods of this class return the file and line at
+                     #that point in the token stream. (not always working right now.)
+
+
+Subclasses of WToken provide an === method for comparing the token to a String or Regexp.
+
+The different types of string and how to distinguish: 
+For the most part you can tell what was what by looking at StringToken#char. 
+Single and double quotes can't be distinguished this way, and neither can you tell a fancy 
+string (starting with %) from the regular kind. If you want to have more... one option 
+with the current code is to just go and look in your input what was at StringToken#offset. 
+
+Eventually, (in version 0.8) string boundaries, bodies, and inclusions will all be separate 
+tokens laid out linearly in the token stream. This is the way matz handles things, and it's 
+much cleaner.  I'll make sure that the string start token at that time contains all the 
+info you could want. If you really can't wait for 0.8 and can't stand #offset, I can hack 
+in a method to StringToken that tell you exactly what char(s) opened the string.
+
+Certain keywords (if, unless, while, until, do) may or may not have an associated end keyword.
+For instance, these have ends associated:
+
+if somthing then
+  do_somthing
+end
+
+a.each do|x| 
+  x.something_about_it 
+end
+
+And these do not:
+
+do_something if something
+
+for x in a do 
+  x.something_about_it 
+end #paired to the for, not the do!
+
+A KeywordToken is generated by rubylexer in either case, but you can now use the has_end?
+method of KeywordToken to determine whether an end should be expected for a particular if or not. 
+
+api stability:
+Future changes to the user-visible api will happen in a backwards-compatible way, so that
+if the interface changes, there will be a (probably quite long) transition period during 
+which both the old and new interfaces are supported. The idea is to give users plenty of 
+time to adapt to changes. That promise goes for all the changes described below.
+
+In cases where the 2 are incompatible, (inspired by rubygems) I've come up with this:
+
+  RubyLexer.version(0.6).new(...args...)  #request the 0.6 api
+  
+This actually works currently; it enables the old api where errors cause an exception instead
+of generating ErrorTokens. The default will always be to use the new api.
+
+StringToken will go away; replaced by multiple token types, like in ruby.  StringToken
+subclasses will need reorganization at this point too... tokens in an interpolation
+will no longer be 'subtokens' but full-fledged tokens in their own right.
+i intend to make a namespace for all rubylexer classes at some point... shouldn't
+be a big deal; old clients can just include the namespace module.
+Token#ident may be taken away or change without notice.
+MethNameToken may become a WToken
+HereBodyToken should really be a string subclass...
+

data/io.each_til_charset.rb

@@ -0,0 +1,247 @@
+=begin copyright
+    rubylexer - a ruby lexer written in ruby
+    Copyright (C) 2004,2005  Caleb Clausen
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+=end
+
+
+module IOext
+   #read until a character in a user-supplied set is found.
+   #charrex must be a regexp that contains _only_ a single character class 
+   def til_charset(charrex,blocksize=16)
+      blocks=[]
+      m=nil
+      until eof?
+         block=read blocksize
+         #if near eof, less than a full block may have been read
+
+         if m=charrex .match(block)
+            self.pos-=m.post_match.length+1
+            #'self.' shouldn't be needed... but is
+
+            blocks.push m.pre_match if m.pre_match.length>0
+            break
+         end
+         blocks<<block
+      end
+      return blocks.to_s
+   end
+   
+   
+   
+
+
+   #-----------------------------------
+   #read and return next char if it matches ch
+   #else, leave input unread and return nil or false
+   def eat_next_if(ch)
+      oldpos=pos
+      c=read(1)
+
+      ch.kind_of? Integer and ch=ch.chr
+
+      return case c
+         when ch then c
+         when '' then self.pos=oldpos; nil
+         else    self.pos=oldpos; false
+      end
+   end
+
+   #-----------------------------------
+   def eat_while(pat)
+      pat.kind_of? Integer and pat=pat.chr
+
+      result=''
+      loop {
+         ch=read(1)
+         unless pat===ch
+            back1char unless ch.nil? #nil ch mean eof
+            return result
+         end
+         result << ch
+      }
+      return result
+   end
+
+   #-----------------------------------
+   #returns previous character in stream
+   #without changing stream position
+   #or '' if at beginning
+   def prevchar
+      pos==0 and return ''
+
+      back1char
+      return getc.chr
+   end
+
+   #-----------------------------------
+   #returns next character in stream
+   #without changing stream position
+   #or nil if at end
+   def nextchar
+      eof? and return nil
+
+      result=getc
+      back1char
+      return result
+   end
+
+   #-----------------------------------
+   #this should really be in class File...
+   def getchar
+      eof? and return ''
+      return getc.chr
+   end
+
+   #-----------------------------------
+   def back1char()    self.pos-=1   end
+
+   #-----------------------------------
+   def readahead(len)
+      oldpos=pos
+         result=read(len)
+      self.pos=oldpos
+
+      return result
+   end
+
+   #-----------------------------------
+   def readback(len)
+      oldpos=pos
+         self.pos-=len
+         result=read(len)
+      self.pos=oldpos
+
+      return result
+   end
+
+   #-----------------------------------
+   def readuntil(pat)
+      each(pat) { |match|
+          return match
+      }
+   end
+
+
+
+#-----------------------------------------------------------------------   
+        #a String with the duck-type of a File
+        #just enough is emulated to fool RubyLexer
+		class FakeFile < ::String  #thanks to murphy for this lovely.
+
+			def initialize(*)
+				super
+				@pos = 0
+			end
+			
+			attr_accessor :pos
+			
+			def read x
+				pos = @pos
+				@pos += x
+                @pos>size and @pos=size
+				self[pos ... @pos]
+			end
+
+			def getc
+                eof? and return nil
+				pos = @pos
+				@pos += 1
+				self[pos]
+			end
+
+			def eof?
+				@pos >= size
+			end
+
+			def each_byte
+				until eof?
+					yield getc
+				end
+			end
+			
+            def stat  #cheezy cheat to make #stat.size work
+              self
+            end
+            
+            def close; end
+            
+            def binmode; end
+            
+            include IOext
+            
+          
+   #-----------------------------------
+   #read and return next char if it matches ch
+   #else, leave input unread and return nil or false
+   def eat_next_if(ch)
+      c=self[@pos,1]
+
+      ch.kind_of? Integer and ch=ch.chr
+
+      case c
+         when ch then @pos+=1;c
+         when '' then nil
+         else  false
+      end
+   end
+
+   #-----------------------------------
+   #returns previous character in stream
+   #without changing stream position
+   #or '' if at beginning
+   def prevchar #returns Fixnum
+      pos==0 ? '' : self[@pos-1]
+   end
+
+   #-----------------------------------
+   #returns next character in stream
+   #without changing stream position
+   #or nil if at end
+   def nextchar #returns Fixnum
+      self[@pos]
+   end
+
+   #-----------------------------------
+   def getchar  #returns String
+      eof? and return ''
+				pos = @pos
+				@pos += 1
+				self[pos,1]
+   end
+
+   #-----------------------------------
+   def back1char()    @pos-=1   end
+
+   #-----------------------------------
+   def readahead(len)
+     self[@pos,len]
+   end
+
+   #-----------------------------------
+   def readback(len)
+      assert @pos-len>=0
+      self[@pos-len,len]
+   end
+  
+            
+		end
+
+end
+
+class IO
+  include IOext
+end

data/require.rb

@@ -0,0 +1,103 @@
+
+#wrapper versions of all commands that import code into a running program:
+#require, load, eval and friends. the wrapped versions pass the code to
+#import to rubylexervsruby, to test whether it gets lexed correctly. an 
+#exception is raised if an lex error happens, else the code should behave 
+#as normal, just much slower.
+class Kernel
+  
+  System_extension_extension=
+    case RUBY_PLATFORM
+      when /darwin/:   'o'
+      when /windows/i: 'dll'
+      else             'so'
+    end
+  System_ext_rex=/\.#{System_extension_extension}$/o
+  
+  def require_name_resolve(name)
+    add_ext=case name
+      when System_ext_rex,/\.rb$/:
+      else name=/(#{name})(#{System_ext_rex}|\.rb)?/
+    end
+    name=/#{File::SEPARATOR}#{name}$/
+    $:.find{|dir| 
+      dir.chomp File::SEPARATOR
+      Dir[/#{dir}#{name}/] 
+    }
+  end
+  
+  
+  #reallyy jonesing for :wrap here
+  alias stdlib__require require
+  def require feat
+    name=feat
+    name=require_name_resolve(name)    unless File.abs_path?(name)
+    return(false) unless name
+    return(true) if $".grep(feat)
+    $"<<feat
+    return stdlib__require(name) if name[System_ext_rex]
+    load name 
+  end
+
+  alias stdlib__load load
+  def load name,wrap=false
+    name=$:.find{|dir| Dir[dir,name]}    unless File.abs_path?(name)
+    if wrap then Module.new {
+           eval File.read(name), huh binding, name,1
+    } else eval File.read(name), huh binding, name,1
+    end
+    true
+  end
+  
+  @@evalpos=1  #eval saves a position for the next eval sometimes... when?
+  alias stdlib__eval eval
+  def eval code,binding=nil,name='(eval)',linenum=1
+    if binding 
+      rubylexervsruby(code, :name=>name, :linenum=>linenum, :locals=>eval("local_variables",binding))
+    
+      return stdlib__eval code,binding,filename,linenum
+    end
+    huh Binding.of_caller{|bg| eval code,bg,name,linenum}
+  end
+
+  huh#got to do module_eval, class_eval, instance_eval,  etc
+end
+
+class Object
+  alias stdlib__instance_eval instance_eval
+  def instance_eval(code,&block)
+    block and return stdlib__instance_eval &block
+    eval code, stdlib__instance_eval{binding}
+  end
+end
+
+class Module
+  alias stdlib__module_eval module_eval
+  alias module_eval instance_eval
+end
+
+class Class
+  alias stdlib__class_eval class_eval
+  alias class_eval instance_eval
+end
+
+
+class Binding
+  alias stdlib__eval eval
+  def eval code,name='(eval)',linenum=1
+      rubylexervsruby(code, :name=>name, :linenum=>linenum, :locals=>eval("local_variables",binding))
+    
+      huh #should set code to (effectively) output of tokentest
+      #how to do that within rubylexervsruby
+    
+      return stdlib__eval code,self,filename,linenum
+  end
+end
+
+=begin
+def Module
+  def new
+    o=Object.extend self    
+  end
+end
+=end