rubylexer 0.6.2 → 0.7.0

data/History.txt

@@ -0,0 +1,55 @@
+=== 0.7.0/2-15-2008
+* implicit tokens are now emitted at the right times (need more test code)
+* local variables are now temporarily hidden by class, module, and def
+* line numbers should always be correct now (=begin...=end caused this) (??)
+* fixed meth/var-name confusion in idents after 'def' but before params
+* FileAndLineToken now emitted on all new lines (0.8)
+* offset of __END__ now works(??) 
+* put files into lib/
+* improvements in parsing unary * and &  (??)
+* input can now come from a string
+* local vars (defs and uses) are recognized in string inclusions
+* explicitly decimal numbers, eg: 0d123456789
+* eof in unusual cases is better handled
+* __END__ is not treated as a keyword
+* '|' as goalpost is now better handled
+* a number of things have been renamed internally
+* no more implicit parens for setter method calls
+* '{' after return, break, and next is now the start of a hash.
+* ambiguous cases of '::','(',':',and '?' are now better handled.
+* more start/end hint tokens (for 'when', 'rescue', and 'for')
+* bugfixes in rhs hint tokens
+* parsing of def headers for singleton methods is improved
+* rescue as operator is now handled
+* block param list lookahead is simplified
+* unary ops (including * and &) can be easily distinguished in output
+* here document bodies better handled, esp after escaped newline
+* symbols like %s{symbol} now actually work
+* implicit parens around => better handled...
+* different cases of '{' can now be easily distinguished
+* ImplicitParamList Start and End are now Keyword, not Ignore tokens.
+
+=== 0.6.2
+* testcode/dumptokens.rb charhandler.rb doesn't work... but does after unix2dos (not reproducible)
+* files are opened in binmode to avoid all possible eol translation
+* (x.+?x) now works
+* methname/varname mixups fixed in some cases
+* performance improvements, in most important cases
+* error handling tokens should be emitted on error input... ErrorToken mixin module
+* but old error handling interface should be preserved and made available
+* moved readahead and friends into IOext
+* made optimized readahead et al for fakefile
+* dos newlines (and newlines generally) can be fancy string delimiters
+* do,if,until, etc, have a way to tell if an end is associated
+* broke readme into pieces
+
+=== 0.6.0
+* partly fixed the implicit tokens at the wrong times. (or not at the 
+* right times) (partly fixed) 
+* : operator might be a synonym for 'then' 
+* variables other than the last are now recognized in multiple assignment
+* variables created by for and rescue are now recognized
+* token following :: should not be BareSymbolToken if begins with A-Z (unless obviously a func)
+* read code to be lexed from a string. (irb wants this)
+* fancy symbols weren't supported at all. (like this:  %s{abcdefg})
+

data/Manifest.txt

@@ -0,0 +1,67 @@
+COPYING
+README.txt
+Manifest.txt
+Rakefile
+howtouse.txt
+History.txt
+testing.txt
+lib/rubylexer/rubycode.rb
+lib/rubylexer/context.rb
+lib/rubylexer/token.rb
+lib/rubylexer/0.6.rb
+lib/rubylexer/0.6.2.rb
+lib/rubylexer/0.7.0.rb
+lib/rubylexer/version.rb
+lib/rubylexer/rulexer.rb
+lib/rubylexer/tokenprinter.rb
+lib/rubylexer/charset.rb
+lib/rubylexer/symboltable.rb
+lib/rubylexer/charhandler.rb
+lib/assert.rb
+lib/rubylexer.rb
+test/data/gemlist.txt
+test/data/blockassigntest.rb
+test/data/for.rb
+test/data/chunky_bacon.rb
+test/data/and.rb
+test/data/pre.unix.rb
+test/data/untermed_string.rb.broken
+test/data/__end__2.rb
+test/data/w.rb
+test/data/if.rb
+test/data/pre.rb
+test/data/jarh.rb
+test/data/regtest.rb
+test/data/chunky_bacon4.rb
+test/data/__end__.rb
+test/data/strinc.rb
+test/data/lbrace.rb
+test/data/p.rb
+test/data/chunky.plain.rb
+test/data/noeolatend.rb
+test/data/g.rb
+test/data/23.rb
+test/data/lbrack.rb
+test/data/untitled1.rb
+test/data/rescue.rb
+test/data/tokentest.assert.rb.can
+test/data/pleac.rb.broken
+test/data/heart.rb
+test/data/s.rb
+test/data/wsdlDriver.rb
+test/data/p-op.rb
+test/data/1.rb.broken
+test/data/untermed_here.rb.broken
+test/data/newsyntax.rb
+test/data/chunky_bacon3.rb
+test/data/chunky_bacon2.rb
+test/data/format.rb
+test/code/locatetest.rb
+test/code/rubylexervsruby.rb
+test/code/dl_all_gems.rb
+test/code/unpack_all_gems.rb
+test/code/tokentest.rb
+test/code/dumptokens.rb
+test/code/torment
+test/code/locatetest
+test/code/deletewarns.rb

data/README.txt

@@ -0,0 +1,103 @@
+= RubyLexer
+
+*
+*
+*
+
+=== DESCRIPTION:
+
+RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is meant
+as a lexer for Ruby that's complete and correct; all legal Ruby 
+code should be lexed correctly by RubyLexer as well. Just enough parsing 
+capability is included to give RubyLexer enough context to tokenize correctly
+in all cases. (This turned out to be more parsing than I had thought or 
+wanted to take on at first.) RubyLexer handles the hard things like 
+complicated strings, the ambiguous nature of some punctuation characters and 
+keywords in ruby, and distinguishing methods and local variables.
+
+RubyLexer is not particularly clean code. As I progressed in writing this, 
+I've learned a little about how these things are supposed to be done; the 
+lexer is not supposed to have any state of it's own, instead it gets whatever 
+it needs to know from the parser. As a stand-alone lexer, Rubylexer maintains 
+quite a lot of state. Every instance variable in the RubyLexer class is some 
+sort of lexer state. Most of the complication and ugly code in RubyLexer is 
+in maintaining or using this state.
+
+For information about using RubyLexer in your program, please see howtouse.txt.
+
+For my notes on the testing of RubyLexer, see testing.txt.
+
+If you have any questions, comments, problems, new feature requests, or just
+want to figure out how to make it work for what you need to do, contact me: 
+       rubylexer _at_ inforadical _dot_ net
+
+RubyLexer is a RubyForge project. RubyForge is another good place to send your
+bug reports or whatever:  http://rubyforge.org/projects/rubylexer/
+
+(There aren't any bug filed against RubyLexer there yet, but don't be afraid 
+that your report will get lonely.)
+
+==SYNOPSIS:
+require "rubylexer.rb"
+ #then later
+lexer=RubyLexer.new(a_file_name, opened_File_or_String)
+until EoiToken===(token=lexer.get1token)
+  #...do stuff w/ token...
+end
+
+== Status
+RubyLexer can correctly lex all legal Ruby 1.8 code that I've been able to 
+find on my Debian system. It can also handle (most of) my catalog of nasty 
+test cases (in testdata/p.rb) (see below for known problems). At this point, 
+new bugs are almost exclusively found by my home-grown test code, rather 
+than ruby code gathered 'from the wild'. There are a number of issues I know 
+about and plan to fix, but it seems that Ruby coders don't write code complex 
+enough to trigger them very often. Although incomplete, RubyLexer can 
+correctly distinguish these ambiguous uses of the following operator and 
+keywords, depending on context:
+  %   can be modulus operator or start of fancy string
+  /   can be division operator or start of regex
+  * & + - :: can be unary or binary operator
+  []  can be for array literal or [] method (or []=)
+  <<  can be here document or left shift operator (or in class<<obj expr)
+  :   can be start of symbol, substitute for then, or part of ternary op
+      (there are other uses too, but they're not supported yet.)
+  ?   can be start of character constant or ternary operator
+  `   can be method name or start of exec string
+  any overrideable operator and most keywords can also be method names
+
+== todo
+test w/ more code (rubygems, rpa, obfuscated ruby contest, rubicon, others?)
+these 5 should be my standard test suite: p.rb, (matz') test.rb, tk.rb, obfuscated ruby contest, rubicon
+test more ways: cvt source to dos or mac fmt before testing
+test more ways: run unit tests after passing thru rubylexer (0.7)
+test more ways: test require'd, load'd, or eval'd code as well (0.7)
+lex code a line (or chunk) at a time and save state for next line (irb wants this) (0.8)
+incremental lexing (ides want this (for performance))
+put everything in a namespace
+integrate w/ other tools...
+html colorized output?
+move more state onto @parsestack (ongoing)
+the new cases in p.rb now compile, but won't run
+expand on test documentation
+use want_op_name more
+return result as a half-parsed tree (with parentheses and the like matched)
+emit advisory tokens when see beginword, then (or equivalent), or end... what else does florian want?
+strings are still slow
+emit advisory tokens when local var defined/goes out of scope (or hidden/unhidden?)
+token pruning in dumptokens...
+
+== known issues: (and planned fix release)
+context not really preserved when entering or leaving string inclusions. this causes
+a number or problems. local variables are ok now, but here document headers started
+in a string inclusion with the body outside will be a problem. (0.8)
+string tokenization sometimes a little different from ruby around newlines
+  (htree/template.rb) (0.8)
+string contents might not be correctly translated in a few cases (0.8?)
+symbols which contain string interpolations are flattened into one token. eg :"foo#{bar}" (0.8)
+'\r' whitespace sometimes seen in dos-formatted output.. shouldn't be (eg pre.rb) (0.7)
+windows or mac newline in source are likely to cause problems in obscure cases (need test case)
+unterminated =begin is not an error (0.8)
+ruby 1.9 completely unsupported (0.9)
+character sets other than ascii are not supported at all (1.0)
+

data/Rakefile

@@ -0,0 +1,24 @@
+# Copyright (C) 2008  Caleb Clausen
+# Distributed under the terms of Ruby's license.
+require 'rubygems'
+require 'hoe'
+require 'lib/rubylexer/version.rb'
+ 
+ 
+   readme=open("README.txt")
+   readme.readline("\n=== DESCRIPTION:")
+   readme.readline("\n\n")
+   desc=readme.readline("\n\n")
+ 
+   hoe=Hoe.new("rubylexer", RubyLexer::VERSION) do |_|
+     _.author = "Caleb Clausen"
+     _.email = "rubylexer-owner @at@ inforadical .dot. net"
+     _.url = "http://rubylexer.rubyforge.org/"
+     _.extra_deps = ["sequence"]
+     _.test_globs=["test/{code/*,data/*rb*,results/}"]
+     _.description=desc
+     _.summary=desc[/\A[^.]+\./]
+     _.spec_extras={:bindir=>''}
+   end
+
+

data/howtouse.txt

@@ -1,13 +1,13 @@
 
 Using rubylexer:
 require "rubylexer.rb"
- ,then
+  #then later
 lexer=RubyLexer.new(a_file_name, opened_File_or_String)
-until EoiToken===(tok=lexer.get1token)
-  ...do stuff w/ toks...
+until EoiToken===(token=lexer.get1token)
+  #...do stuff w/ token...
 end
 
-For a slightly expanded version of this example, see testcode/dumptokens.rb.
+For a slightly expanded version of this example, see test/code/dumptokens.rb.
 
 tok will be a subclass of Token. there are many token classes (see token.rb)
 however, all tokens have some common methods:
@@ -23,7 +23,8 @@ WToken       #(mostly useless?) abstract superclass for KeywordToken,
              #OperatorToken, VarNameToken, and HerePlaceholderToken
              #but not (confusingly) MethNameToken (perhaps that'll change)
  KeywordToken #a ruby keyword or non-overridable punctuation char(s)
- OperatorToken #overrideable operators
+ OperatorToken #overrideable operators. 
+               #use #unary? and #binary? to find out how many arguments it takes.
  VarNameToken #a name that represents a variable
  HerePlaceholderToken #represents the header of a here string. subclass of WToken
 MethNameToken  #the name of a method: the uncoloned
@@ -120,7 +121,8 @@ time to adapt to changes. That promise goes for all the changes described below.
 
 In cases where the 2 are incompatible, (inspired by rubygems) I've come up with this:
 
-  RubyLexer.version(0.6).new(...args...)  #request the 0.6 api
+  require 'rubylexer/0.6'
+  rl=RubyLexer.new(...args...)  #request the 0.6 api
   
 This actually works currently; it enables the old api where errors cause an exception instead
 of generating ErrorTokens. The default will always be to use the new api.
@@ -133,4 +135,5 @@ be a big deal; old clients can just include the namespace module.
 Token#ident may be taken away or change without notice.
 MethNameToken may become a WToken
 HereBodyToken should really be a string subclass...
+Newline,EscNl,BareSymbolToken may get renamed
 

data/{assert.rb → lib/assert.rb}

@@ -1,5 +1,4 @@
 =begin copyright
-    rubylexer - a ruby lexer written in ruby
     Copyright (C) 2004,2005  Caleb Clausen
 
     This library is free software; you can redistribute it and/or
@@ -16,16 +15,17 @@
     License along with this library; if not, write to the Free Software
     Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 =end
-require 'set'
 
+module Kernel
+  def assert(expr,msg="assertion failed")
+    defined? $Debug and $Debug and (expr or raise msg)
+  end
 
-def assert(expr,msg="assertion failed")
-  $DEBUG and (expr or raise msg)
-end
-
-@@printed=Set.new
-def fixme(s)
-  @@printed.include?( s) and return
-  $DEBUG and STDERR.print "FIXME: #{s}\n"
-  @@printed.add s
+  @@printed={}
+  def fixme(s)
+    unless @@printed[s] 
+      @@printed[s]=1
+      defined? $Debug and $Debug and $stderr.print "FIXME: #{s}\n"
+    end
+  end
 end

data/{rubylexer.rb → lib/rubylexer.rb}

@@ -19,15 +19,18 @@
 
 
 
-require "rulexer"
-require "symboltable"
-require "io.each_til_charset"
-require "context.rb"
-
+require 'rubylexer/rulexer' #must be 1st!!!
+require 'rubylexer/version'
+require 'rubylexer/token'
+require 'rubylexer/charhandler'
+require 'rubylexer/symboltable'
+#require "io.each_til_charset"
+require 'rubylexer/context'
+require 'rubylexer/tokenprinter'
 
 
 #-----------------------------------
-class RubyLexer < RuLexer
+class RubyLexer
   include NestedContexts
  
    RUBYSYMOPERATORREX=
@@ -39,7 +42,7 @@ class RubyLexer < RuLexer
       #or .. ... ?:
       #for that use:
    RUBYNONSYMOPERATORREX=
-      %r{^([%^~/\-+]=|(\|\|?|&&?)=?|(<<|>>|\*\*?)=|\.{1,3}|[?:,;]|=>?|![=~]?)$}
+      %r{^([%^~/\-+|&]=|(\|\||&&)=?|(<<|>>|\*\*?)=|\.{1,3}|[?:,;]|=>?|![=~]?)$}
    RUBYOPERATORREX=/#{RUBYSYMOPERATORREX}|#{RUBYNONSYMOPERATORREX}/o
    UNSYMOPS=/^[~!]$/ #always unary
    UBSYMOPS=/^([*&+-]|::)$/  #ops that could be unary or binary
@@ -50,16 +53,18 @@ class RubyLexer < RuLexer
    VARLIKE_KEYWORDS=/^(__FILE__|__LINE__|false|nil|self|true)$/
    INNERBOUNDINGWORDS="(else|elsif|ensure|in|then|rescue|when)"
    BINOPWORDS="(and|or)"
-   NEVERSTARTPARAMLISTWORDS=/^(#{OPORBEGINWORDS}|#{INNERBOUNDINGWORDS}|#{BINOPWORDS}|end)[^a-zA-Z0-9_!?=]?/o
-   NEVERSTARTPARAMLISTFIRST=CharSet[%[aoeitrwu]]  #char set that begins NEVERSTARTPARAMLIST
-   NEVERSTARTPARAMLISTMAXLEN=7            #max len of a NEVERSTARTPARAMLIST
+   NEVERSTARTPARAMLISTWORDS=/^(#{OPORBEGINWORDS}|#{INNERBOUNDINGWORDS}|#{BINOPWORDS}|end)([^a-zA-Z0-9_!?=]|\Z)/o
+   NEVERSTARTPARAMLISTFIRST=CharSet['aoeitrwu']  #chars that begin NEVERSTARTPARAMLIST
+   NEVERSTARTPARAMLISTMAXLEN=7     #max len of a NEVERSTARTPARAMLIST
    
    RUBYKEYWORDS=%r{
-     ^(alias|#{BINOPWORDS}|not|undef|__END__|end|
+     ^(alias|#{BINOPWORDS}|not|undef|end|
        #{VARLIKE_KEYWORDS}|#{FUNCLIKE_KEYWORDS}|
        #{INNERBOUNDINGWORDS}|#{BEGINWORDS}
      )$
    }xo
+      #__END__ should not be in this set... its handled in start_of_line_directives
+
    CHARMAPPINGS = {
          ?$ => :dollar_identifier,
          ?@ => :at_identifier,
@@ -67,8 +72,7 @@ class RubyLexer < RuLexer
          ?A..?Z => :identifier,
          ?_     => :identifier,
          ?0..?9 => :number,
-         ?" => :double_quote,
-         ?' => :single_quote,
+         %{"'} => :double_quote,
          ?` => :back_quote,
 
          WHSP => :whitespace, #includes \r
@@ -83,7 +87,8 @@ class RubyLexer < RuLexer
 
          #these ones could signal either an op or a term
          ?/ => :regex_or_div,
-         "|>" => :quadriop,
+         "|" => :conjunction_or_goalpost,
+         ">" => :quadriop,
          "*&" => :star_or_amp,        #could be unary
          "+-" => :plusminus, #could be unary
          ?< => :lessthan,
@@ -103,22 +108,27 @@ class RubyLexer < RuLexer
          ?# => :comment
    }
 
-   attr :incomplete_here_tokens
+   attr_reader :incomplete_here_tokens, :parsestack
 
 
    #-----------------------------------
    def initialize(filename,file,linenum=1)
       super(filename,file, linenum)
       @start_linenum=linenum
-      @bracestack=[TopLevelContext.new]
+      @parsestack=[TopLevelContext.new]
       @incomplete_here_tokens=[]
-      @localvars=SymbolTable.new
+      @localvars_stack=[SymbolTable.new]
       @defining_lvar=nil
+      @in_def_name=false
 
       @toptable=CharHandler.new(self, :illegal_char, CHARMAPPINGS)
 
       start_of_line_directives
    end
+   
+   def localvars;
+     @localvars_stack.last
+   end
 
    #-----------------------------------
    def get1token
@@ -129,25 +139,23 @@ class RubyLexer < RuLexer
 
       #check for bizarre token types
       case result
-      when IgnoreToken#,nil
-         return result
+      when StillIgnoreToken#,nil
+          result
       when Token#,String
+          @last_operative_token=result
+          assert !(IgnoreToken===@last_operative_token)
+          result
       else
-         raise "#{@filename}:#{linenum}:token is a #{result.class}, last is #{@last_operative_token}"
+          raise "#{@filename}:#{linenum}:token is a #{result.class}, last is #{@last_operative_token}"
       end
-
-      @last_operative_token=result
-
-      return result
    end
-   
 
 
    #-----------------------------------
    def balanced_braces?
 
-       #@bracestack.empty?
-       @bracestack.size==1 and TopLevelContext===@bracestack.first
+       #@parsestack.empty?
+       @parsestack.size==1 and TopLevelContext===@parsestack.first
    end
 
    #-----------------------------------
@@ -182,7 +190,7 @@ private
 
    #-----------------------------------
    def expect_do_or_end_or_nl!(st)
-     @bracestack.push ExpectDoOrNlContext.new(st,/(do|;|:|\n)/,@linenum)
+     @parsestack.push ExpectDoOrNlContext.new(st,/(do|;|:|\n)/,@linenum)
    end
 
   #-----------------------------------
@@ -199,31 +207,38 @@ private
   end
 
   #-----------------------------------
-  WSCHARSET=CharSet["#\\\n\s\t\v\r\f"]
-  def ignored_tokens(allow_eof=false)
+  WSCHARSET=/[#\\\n\s\t\v\r\f]/
+  def ignored_tokens(allow_eof=false,allow_eol=true)
     result=[]
-    result<<@moretokens.shift while IgnoreToken===@moretokens.first
+    result<<@moretokens.shift while StillIgnoreToken===@moretokens.first
     @moretokens.empty? or return result
-if true
     loop do
       unless @moretokens.empty?
-        IgnoreToken===@moretokens.first or NewlineToken===@moretokens.first or
-        break
+        case @moretokens.first
+        when StillIgnoreToken
+        when NewlineToken: allow_eol or break
+        else break
+        end 
       else
-        WSCHARSET===nextchar or break
+      
+        break unless ch=nextchar
+        ch=ch.chr
+        break unless WSCHARSET===ch
+        break if ch[/[\r\n]/] and !allow_eol
       end
+      
 
       tok=get1token
-      result<<tok
+      result << tok
       case tok
-        when NewlineToken : block_given? and yield tok
-        when EoiToken : allow_eof or lexerror tok,"end of file not expected here(2)"
-        when IgnoreToken
-        else raise "impossible"
+        when NewlineToken; assert allow_eol; block_given? and yield tok
+        when EoiToken; allow_eof or lexerror tok,"end of file not expected here(2)"
+        when StillIgnoreToken
+        else raise "impossible token: #{tok.inspect}"
       end
     end
 
-else
+=begin
       @whsphandler||=CharHandler.new(self, :==,
          "#" => :comment,
          "\n" => :newline,
@@ -235,18 +250,18 @@ else
          block_given? and NewlineToken===tok and yield tok
          result << tok
       end
-end
+=end
     return result
   end
 
    #-----------------------------------
    def safe_recurse
       old_moretokens=@moretokens
-      #old_bracestack=@bracestack.dup
+      #old_parsestack=@parsestack.dup
       @moretokens=[]
       result=   yield @moretokens
       #assert @incomplete_here_tokens.empty?
-      #assert @bracestack==old_bracestack
+      #assert @parsestack==old_parsestack
       @moretokens= old_moretokens.concat @moretokens
       return result
       #need to do something with @last_operative_token?
@@ -258,7 +273,7 @@ end
       result = ((
       #order matters here, but it shouldn't
       #(but til_charset must be last)
-         eat_next_if(/^[!@&+`'=~\/\\,.;<>*"$?:]$/) or
+         eat_next_if(/[!@&+`'=~\/\\,.;<>*"$?:]/) or
          (eat_next_if('-') and ("-"+getchar)) or
          (?0..?9)===nextchar ? til_charset(/[^\d]/) : nil
       ))
@@ -266,7 +281,7 @@ end
 
    #-----------------------------------
    def identifier(context=nil)
-      oldpos=@file.pos
+      oldpos= input_position
       str=identifier_as_string(context)
 
       #skip keyword processing if 'escaped' as it were, by def, . or ::
@@ -279,8 +294,8 @@ end
       @moretokens.unshift(*parse_keywords(str,oldpos) do
         #if not a keyword,
         case str
-          when FUNCLIKE_KEYWORDS: #do nothing
-          when VARLIKE_KEYWORDS,RUBYKEYWORDS: raise "shouldnt see keywords here, now"
+          when FUNCLIKE_KEYWORDS; #do nothing
+          when VARLIKE_KEYWORDS,RUBYKEYWORDS; raise "shouldnt see keywords here, now"
         end
         safe_recurse { |a| var_or_meth_name(str,@last_operative_token,oldpos) }
       end)
@@ -290,7 +305,7 @@ end
    #-----------------------------------
    def identifier_as_string(context)
       #must begin w/ letter or underscore
-      str=eat_next_if(/^[_a-z]$/i) or return nil
+      str=eat_next_if(/[_a-z]/i) or return nil
 
       #equals, question mark, and exclamation mark
       #might be allowed at the end in some contexts.
@@ -305,18 +320,20 @@ end
          when ?:    then [?=, ??, ?!]
          else            [nil,??, ?!]
       end
+      
+      @in_def_name and maybe_eq= ?=
 
       str<<til_charset(/[^a-z0-9_]/i)
 
       #look for ?, !, or =, if allowed
-      case b=@file.getc
+      case b=getc
       when nil #means we're at eof
          #handling nil here prevents b from ever matching
          #a nil value of maybe_qm, maybe_ex or maybe_eq
       when maybe_qm
          str << b
       when maybe_ex
-         nc=(nextchar unless @file.eof?)
+         nc=(nextchar unless eof?)
          #does ex appear to be part of a larger operator?
          if nc==?= #or nc==?~
            back1char
@@ -324,7 +341,7 @@ end
            str << b
          end
       when maybe_eq
-         nc=(nextchar unless @file.eof?)
+         nc=(nextchar unless eof?)
          #does eq appear to be part of a larger operator?
          if nc==?= or nc==?~ or nc==?>
            back1char
@@ -342,34 +359,37 @@ end
   #-----------------------------------
   #contexts in which comma may appear in ruby:
     #multiple lhs (terminated by assign op)
-    #multiple rhs (in implicit context) (tbd)
+    #multiple rhs (in implicit context) 
     #method actual param list (in ( or implicit context)
     #method formal param list (in ( or implicit context)
-    #block  formal param list (in | context) (tbd)
+    #block  formal param list (in | context) 
+    #nested multiple rhs 
+    #nested multiple lhs 
+    #nested block formal list 
+    #element reference/assignment (in [] or []= method actual parameter context)
     #hash immediate (in imm{ context)
     #array immediate (in imm[ context)
-    #element reference/assignment (in [] or []= method actual parameter context)
-    #list after for
+    #list between 'for' and 'in'
     #list after rescue
     #list after when
     #list after undef
 
-    #note: comma in parens not around a param list is illegal
+    #note: comma in parens not around a param list or lhs or rhs is illegal
 
    #-----------------------------------
    #a comma has been seen. are we in an
    #lvalue list or some other construct that uses commas?
    def comma_in_lvalue_list?
-     not ListContext===@bracestack.last
+     @parsestack.last.lhs= (not ListContext===@parsestack.last)
    end
    
    #-----------------------------------
    def in_lvar_define_state
      #@defining_lvar is a hack
-     @defining_lvar or case ctx=@bracestack.last
-       when ForSMContext: ctx.state==:for
-       when RescueSMContext: ctx.state==:arrow
-       when BlockParamListContext: true
+     @defining_lvar or case ctx=@parsestack.last
+       when ForSMContext; ctx.state==:for
+       when RescueSMContext; ctx.state==:arrow
+       #when BlockParamListLhsContext; true
      end 
    end
    
@@ -391,66 +411,102 @@ end
      #look for and ignore local variable names
 
      assert String===name
-
-     #fixme: keywords shouldn't be treated specially after :: and .
      
      #maybe_local really means 'maybe local or constant'
      maybe_local=case name
-       when /[^a-z_0-9]$/i: #do nothing
-       when /^[a-z_]/:  (@localvars===name or VARLIKE_KEYWORDS===name or in_lvar_define_state) and not lasttok===/^(\.|::)$/
-       when /^[A-Z]/: is_const=true;not lasttok==='.'  #this is the right algorithm for constants... 
+       when /[^a-z_0-9]$/i; #do nothing
+       when /^[a-z_]/;  (localvars===name or VARLIKE_KEYWORDS===name or in_lvar_define_state) and not lasttok===/^(\.|::)$/
+       when /^[A-Z]/; is_const=true;not lasttok==='.'  #this is the right algorithm for constants... 
      end 
           
      assert(@moretokens.empty?)
+     
+     oldlast=@last_operative_token
 
      tok=@last_operative_token=VarNameToken.new(name,pos)
 
-     oldpos=@file.pos
+     oldpos= input_position
      sawnl=false
      result=ws_toks=ignored_tokens(true) {|nl| sawnl=true }
-     sawnl || @file.eof?  and  return result.unshift(
-         *if  maybe_local : [tok] 
-         else [MethNameToken.new(name,pos),  #insert implicit parens right after tok
-               ImplicitParamListStartToken.new( oldpos),
-               ImplicitParamListEndToken.new( oldpos) ]
+     if sawnl || eof? 
+         if  maybe_local then 
+           if in_lvar_define_state
+             if /^[a-z_][a-zA-Z_0-9]*$/===name 
+               assert !(lasttok===/^(\.|::)$/)
+               localvars[name]=true
+             else
+               lexerror tok,"not a valid variable name: #{name}"
+             end
+             return result.unshift(tok)
+           end
+           return result.unshift(tok) #if is_const
+         else 
+           return result.unshift(
+             MethNameToken.new(name,pos),  #insert implicit parens right after tok
+             ImplicitParamListStartToken.new( oldpos),
+             ImplicitParamListEndToken.new( oldpos) 
+           )
          end
-     )
+     end
      
      #if next op is assignment (or comma in lvalue list)
      #then omit implicit parens
      assignment_coming=case nc=nextchar
-       when ?=:  not /^=[=~]$/===readahead(2)
-       when ?,: comma_in_lvalue_list?
-       when ?>,?<: /^([<>])\1=$/===readahead(3)
-       when ?*,?|,?&: /^([*|&])\1?=/===readahead(3)
-       when ?%,?/,?-,?+,?^: readahead(2)[1..1]=='='
+       when ?=;  not /^=[>=~]$/===readahead(2)
+       when ?,; comma_in_lvalue_list? 
+       when ?); last_context_not_implicit.lhs
+       when ?>,?<; /^(.)\1=$/===readahead(3)
+       when ?*,?&; /^(.)\1?=/===readahead(3)
+       when ?|; /^\|\|?=/===readahead(3) or
+                #is it a goalpost?
+                BlockParamListLhsContext===last_context_not_implicit &&
+                readahead(2)[1] != ?|
+       when ?%,?/,?-,?+,?^; readahead(2)[1]== ?=
      end 
-     if (assignment_coming or in_lvar_define_state)
+     if (assignment_coming && !(lasttok===/^(\.|::)$/) or in_lvar_define_state)
         tok=VarNameToken.new(name,pos)
         if /[^a-z_0-9]$/i===name 
           lexerror tok,"not a valid variable name: #{name}"
         elsif /^[a-z_]/===name and !(lasttok===/^(\.|::)$/)
-          @localvars[name]=true
+          localvars[name]=true
         end
         return result.unshift(tok)
      end
-
-     implicit_parens_to_emit=case nc
-       when ?!: readahead(2)=='!=' ? 2 : 1
+     
+     implicit_parens_to_emit=
+     if assignment_coming
+       @parsestack.push AssignmentContext.new(nil) if nc==?% or nc==?/
+       0
+     else
+     case nc
+       when nil: 2
+       when ?!; readahead(2)=='!=' ? 2 : 1
        when NEVERSTARTPARAMLISTFIRST
          (NEVERSTARTPARAMLISTWORDS===readahead(NEVERSTARTPARAMLISTMAXLEN)) ? 2 : 1
-       when ?",?',?`,?a..?z,?A..?Z,?0..?9,?_,?@,?$,?~: 1
-       when ?{: maybe_local=false; 2
-       when ?(: maybe_local=false; 0
-       when ?},?],?),?;,?^, ?|, ?>, ?,, ?., ?=: 2
-       when ?+, ?-, ?*, ?&, ?%, ?/, ?:, ??: (ws_toks.empty? || readahead(2)[/^.[#{WHSPLF}]/]) ? 2 : 3
-       when ?<: (ws_toks.empty? || readahead(3)[/^<<[^"'`a-zA-Z_0-9-]/]) ? 2 : 3 
-       when ?[: ws_toks.empty? ? 2 : 3
-       when ?\\, ?\s, ?\t, ?\n, ?\r, ?\v, ?#: raise 'failure'
-       else raise "unknown char after ident: #{nextchar.chr}"
+       when ?",?',?`,?a..?z,?A..?Z,?0..?9,?_,?@,?$,?~; 1
+       when ?{
+         maybe_local=false
+         x=2
+         x-=1 if /\A(return|break|next)\Z/===name and 
+                 !(KeywordToken===oldlast and oldlast===/\A(\.|::)\Z/)
+         x
+       when ?(; 
+         maybe_local=false; !(ws_toks.empty? or lasttok===/^(\.|::)$/)? 1 : 0
+       when ?},?],?),?;,?^, ?|, ?>, ?,, ?., ?=; 2
+       when ?+, ?-, ?*, ?&, ?%, ?/; (ws_toks.empty? || readahead(2)[/^.[#{WHSPLF}]/]) ? 2 : 3
+       when ?:,??; next2=readahead(2);
+                   WHSPLF[next2[1].chr] || next2=='::' ? 2 : 3
+#       when ?:,??; (readahead(2)[/^.[#{WHSPLF}]/]) ? 2 : 3
+       when ?<; (ws_toks.empty? || readahead(3)[/^<<["'`a-zA-Z_0-9-]/]) ? 3 : 2 
+       when ?[; ws_toks.empty? ? 2 : 3
+       when ?\\, ?\s, ?\t, ?\n, ?\r, ?\v, ?#; raise 'failure'
+       else raise "unknown char after ident: #{nc=nextchar ? nc.chr : "<<EOF>>"}"
+     end
+     end
+
+     if is_const and implicit_parens_to_emit==3 then
+       implicit_parens_to_emit=1
      end
-     
-     implicit_parens_to_emit==3 and is_const and implicit_parens_to_emit=1
      
      tok=if maybe_local and implicit_parens_to_emit>=2 
        implicit_parens_to_emit=0
@@ -459,15 +515,18 @@ end
        MethNameToken
      end.new(name,pos)
      
-     
      case implicit_parens_to_emit
-     when 2:
+     when 2;
        result.unshift ImplicitParamListStartToken.new(oldpos),
                  ImplicitParamListEndToken.new(oldpos)
-     when 1,3:
-       result.unshift ImplicitParamListStartToken.new(oldpos)
-       @bracestack.push ParamListContextNoParen.new(@linenum)
-     when 0: #do nothing
+     when 1,3;
+       arr,pass=*param_list_coming_with_2_or_more_params?
+       result.push( *arr )
+       unless pass
+         result.unshift ImplicitParamListStartToken.new(oldpos)
+         @parsestack.push ParamListContextNoParen.new(@linenum)
+       end
+     when 0; #do nothing
      else raise 'invalid value of implicit_parens_to_emit'
      end
      return result.unshift(tok)
@@ -476,22 +535,43 @@ end
      # '\n (unescaped) and or'
      # 'then else elsif rescue ensure (illegal in value context)'
 
-     # 'need to pop noparen from bracestack on these tokens: (in operator context)'
+     # 'need to pop noparen from parsestack on these tokens: (in operator context)'
      # 'not ok:'
      # 'not (but should it be?)'
    end
 
+   #-----------------------------------
+   def param_list_coming_with_2_or_more_params?
+     WHSPCHARS[prevchar] && (?(==nextchar) or return [[],false]
+       basesize=@parsestack.size
+       result=[get1token]
+       pass=loop{
+         tok=get1token
+         result<<tok
+         if @parsestack.size==basesize
+           break false
+         elsif ','==tok.to_s and @parsestack.size==basesize+1
+           break true 
+         elsif EoiToken===tok
+           lexerror tok, "unexpected eof in parameter list"
+         end
+       }
+       return [result,pass]
+   end
+
   #-----------------------------------
   CONTEXT2ENDTOK={AssignmentRhsContext=>AssignmentRhsListEndToken, 
                   ParamListContextNoParen=>ImplicitParamListEndToken,
-                  KwParamListContext=>KwParamListEndToken
+                  WhenParamListContext=>KwParamListEndToken,
+                  RescueSMContext=>KwParamListEndToken
                  }
   def abort_noparens!(str='')
     #assert @moretokens.empty?
     result=[]
-    while klass=CONTEXT2ENDTOK[@bracestack.last.class]
-      result << klass.new(@file.pos-str.length)
-      @bracestack.pop
+    while klass=CONTEXT2ENDTOK[@parsestack.last.class]
+      result << klass.new(input_position-str.length)
+      break if RescueSMContext===@parsestack.last
+      @parsestack.pop 
     end
     return result
   end
@@ -501,13 +581,13 @@ if false #no longer used
    def abort_1_noparen!(offs=0)
      assert @moretokens.empty?
      result=[]
-     while AssignmentRhsContext===@bracestack.last
-       @bracestack.pop
-       result << AssignmentRhsListEndToken.new(@file.pos-offs)
+     while AssignmentRhsContext===@parsestack.last
+       @parsestack.pop
+       result << AssignmentRhsListEndToken.new(input_position-offs)
      end
-     ParamListContextNoParen===@bracestack.last or lexerror huh,'{} with no matching callsite'
-       @bracestack.pop
-       result << ImplicitParamListEndToken.new(@file.pos-offs)
+     ParamListContextNoParen===@parsestack.last or lexerror huh,'{} with no matching callsite'
+       @parsestack.pop
+       result << ImplicitParamListEndToken.new(input_position-offs)
      return result
    end
 end
@@ -523,30 +603,31 @@ end
       case str
       when "end"
          result.unshift(*abort_noparens!(str))
-         @bracestack.last.see @bracestack,:semi #sorta hacky... should make an :end event instead?
+         @parsestack.last.see self,:semi #sorta hacky... should make an :end event instead?
 
 =begin not needed?
-         if ExpectDoOrNlContext===@bracestack.last
-            @bracestack.pop
-            assert @bracestack.last.starter[/^(while|until|for)$/]
+         if ExpectDoOrNlContext===@parsestack.last
+            @parsestack.pop
+            assert @parsestack.last.starter[/^(while|until|for)$/]
          end
 =end
 
-         WantsEndContext===@bracestack.last or lexerror result.last, 'unbalanced end'
-         ctx=@bracestack.pop
+         WantsEndContext===@parsestack.last or lexerror result.last, 'unbalanced end'
+         ctx=@parsestack.pop
          start,line=ctx.starter,ctx.linenum
          BEGINWORDS===start or lexerror result.last, "end does not match #{start or "nil"}"
-         /^(class|module|def|do)$/===start and @localvars.end_block
+         /^(do)$/===start and localvars.end_block
+         /^(class|module|def)$/===start and @localvars_stack.pop
 
       when "class","module"
          result.first.has_end!
-         @bracestack.push WantsEndContext.new(str,@linenum)
-         @localvars.start_block
-
+         @parsestack.push WantsEndContext.new(str,@linenum)
+         @localvars_stack.push SymbolTable.new 
+         
       when "if","unless" #could be infix form without end
          if after_nonid_op?{false} #prefix form
             result.first.has_end!
-            @bracestack.push WantsEndContext.new(str,@linenum)
+            @parsestack.push WantsEndContext.new(str,@linenum)
             
 
          else #infix form
@@ -554,11 +635,11 @@ end
          end
       when "begin","case"
          result.first.has_end!
-         @bracestack.push WantsEndContext.new(str,@linenum)
+         @parsestack.push WantsEndContext.new(str,@linenum)
       when "while","until" #could be infix form without end
          if after_nonid_op?{false} #prefix form
            result.first.has_end!
-           @bracestack.push WantsEndContext.new(str,@linenum)
+           @parsestack.push WantsEndContext.new(str,@linenum)
            expect_do_or_end_or_nl! str
 
          else #infix form
@@ -566,24 +647,26 @@ end
          end
       when "for"
          result.first.has_end!
-         @bracestack.push WantsEndContext.new(str,@linenum)
+         result.push KwParamListStartToken.new(offset+str.length)
+         # corresponding EndToken emitted leaving ForContext ("in" branch, below)
+         @parsestack.push WantsEndContext.new(str,@linenum)
          #expect_do_or_end_or_nl! str #handled by ForSMContext now
-         @bracestack.push ForSMContext.new(@linenum)
+         @parsestack.push ForSMContext.new(@linenum)
       when "do"
          result.unshift(*abort_noparens!(str))
-         if ExpectDoOrNlContext===@bracestack.last
-            @bracestack.pop
-            assert WantsEndContext===@bracestack.last
+         if ExpectDoOrNlContext===@parsestack.last
+            @parsestack.pop
+            assert WantsEndContext===@parsestack.last
          else
             result.last.has_end!
-            @bracestack.push WantsEndContext.new(str,@linenum)
-            @localvars.start_block
+            @parsestack.push WantsEndContext.new(str,@linenum)
+            localvars.start_block
             block_param_list_lookahead
          end
       when "def"
          result.first.has_end!
-         @bracestack.push WantsEndContext.new("def",@linenum)
-         @localvars.start_block
+         @parsestack.push WantsEndContext.new("def",@linenum)
+         @localvars_stack.push SymbolTable.new
          safe_recurse { |aa|
             @last_operative_token=KeywordToken.new "def" #hack
             result.concat ignored_tokens
@@ -591,7 +674,7 @@ end
             #read an expr like a.b.c or a::b::c
             #or (expr).b.c
             if nextchar==?( #look for optional parenthesised head
-              old_size=@bracestack.size
+              old_size=@parsestack.size
               parencount=0
               begin
                 tok=get1token
@@ -601,22 +684,58 @@ end
                 end
                 EoiToken===tok and lexerror tok, "eof in def header"
                 result<<tok
-              end until  parencount==0 #@bracestack.size==old_size
+              end until  parencount==0 #@parsestack.size==old_size
             else #no parentheses, all tail
               @last_operative_token=KeywordToken.new "." #hack hack
-              result << symbol(false,false)
-              #this isn't quite right.... if a.b.c.d is seen, a, b, and c
+              tokindex=result.size
+              result << tok=symbol(false,false)
+              name=tok.to_s
+              assert !in_lvar_define_state
+     
+              #maybe_local really means 'maybe local or constant'
+              maybe_local=case name
+                when /[^a-z_0-9]$/i; #do nothing
+                when /^[@$]/; true
+                when VARLIKE_KEYWORDS,FUNCLIKE_KEYWORDS; ty=KeywordToken
+                when /^[a-z_]/;  localvars===name 
+                when /^[A-Z]/; is_const=true  #this is the right algorithm for constants... 
+              end
+              if !ty and maybe_local 
+                result.push(  *ignored_tokens(false,false)  )
+                nc=nextchar
+                if nc==?: || nc==?.
+                  ty=VarNameToken
+                end
+              end  
+              unless ty
+                ty=MethNameToken
+                endofs=tok.offset+tok.to_s.length
+                result[tokindex+1...tokindex+1]=
+                  [ImplicitParamListStartToken.new(endofs),ImplicitParamListEndToken.new(endofs)]
+              end
+
+              assert result[tokindex].equal?(tok)
+              result[tokindex]=ty.new(tok.to_s,tok.offset)
+              
+
+              #if a.b.c.d is seen, a, b, and c
               #should be considered maybe varname instead of methnames.
               #the last (d in the example) is always considered a methname;
               #it's what's being defined.
+              #b and c should be considered varnames only if 
+              #they are capitalized and preceded by :: .
+              #a could even be a keyword (eg self or block_given?).
             end
             #read tail: .b.c.d etc
-            @last_operative_token=result.last
+            result.reverse_each{|res| break @last_operative_token=res unless StillIgnoreToken===res}
+            ###@last_operative_token=result.last #naive
+            assert !(IgnoreToken===@last_operative_token)
             state=:expect_op
+            @in_def_name=true
             loop do
 
                #look for start of parameter list
-               nc=(@moretokens.first or nextchar.chr)
+               nc=(@moretokens.empty? ? nextchar.chr : @moretokens.first.to_s[0,1])
                if state==:expect_op and /^[a-z_(&*]/i===nc
                   result.concat def_param_list
                   break
@@ -627,8 +746,8 @@ end
                case tok
                when EoiToken
                   lexerror tok,'unexpected eof in def header'
-               when IgnoreToken
-               when MethNameToken #,VarNameToken # /^[a-z_]/i.token_pat
+               when StillIgnoreToken
+               when MethNameToken ,VarNameToken # /^[a-z_]/i.token_pat
                   lexerror tok,'expected . or ::' unless state==:expect_name
                   state=:expect_op
                when /^(\.|::)$/.token_pat
@@ -642,6 +761,7 @@ end
                            "#{tok}:#{tok.class}")
                end
             end
+            @in_def_name=false
          }
       when "alias"
          safe_recurse { |a|
@@ -663,6 +783,7 @@ end
                tok or lexerror(result.first,"bad symbol in undef")
                result<< tok
                @last_operative_token=tok
+               assert !(IgnoreToken===@last_operative_token)
 
                sawnl=false
                result.concat ignored_tokens(true){|nl| sawnl=true}
@@ -674,26 +795,47 @@ end
          }
          
 #      when "defined?"
-#         huh
          #defined? might have a baresymbol following it
          #does it need to be handled specially?
+         #it would seem not.....
 
       when "when"
+         #abort_noparens! emits EndToken on leaving context
          result.unshift(*abort_noparens!(str))
-         @bracestack.push KwParamListContext.new(str,@linenum)
+         result.push KwParamListStartToken.new( offset+str.length)
+         @parsestack.push WhenParamListContext.new(str,@linenum)
 
       when "rescue"
-         result.unshift(*abort_noparens!(str))
-         @bracestack.push RescueSMContext.new(@linenum)
+         unless after_nonid_op? {false}
+           #rescue needs to be treated differently when in operator context... 
+           #i think no RescueSMContext should be pushed on the stack...
+           #plus, the rescue token should be marked as infix
+           result.first.set_infix!
+         else         
+           result.push KwParamListStartToken.new(offset+str.length)
+           #corresponding EndToken emitted by abort_noparens! on leaving rescue context
+           result.unshift(*abort_noparens!(str))
+           @parsestack.push RescueSMContext.new(@linenum)
+         end
 
-      when "then","in"
+      when "then"
+         result.unshift(*abort_noparens!(str))
+         @parsestack.last.see self,:then
+         
+      when "in"
+         result.unshift KwParamListEndToken.new( offset)
          result.unshift(*abort_noparens!(str))
-         @bracestack.last.see @bracestack,str.to_sym
+         @parsestack.last.see self,:in
          
-      when /^(#{BINOPWORDS}|#{INNERBOUNDINGWORDS})$/o
+      when /\A(#{BINOPWORDS}|#{INNERBOUNDINGWORDS})\Z/o
          result.unshift(*abort_noparens!(str))
 
-      when FUNCLIKE_KEYWORDS: result=yield
+      when /\A(return|break|next)\Z/
+         result=yield
+         result.first.has_no_block! unless KeywordToken===@last_operative_token and @last_operative_token===/\A(\.|::)\Z/
+
+      when FUNCLIKE_KEYWORDS
+         result=yield
 
       when RUBYKEYWORDS
         #do nothing
@@ -706,6 +848,36 @@ end
    end
 
 
+   #-----------------------------------
+   def parsestack_lastnonassign_is?(obj)
+     @parsestack.reverse_each{|ctx|
+       case ctx
+  #    when klass: return true
+       when AssignmentRhsContext
+       else return ctx.object_id==obj.object_id
+       end
+     }
+   end
+
+   #-----------------------------------
+   #what's inside goalposts (the block formal parameter list) 
+   #is considered the left hand side of an assignment.
+   #inside goalposts, a local variable is declared if
+   #it has one of the following tokens on both sides:
+   #   ,  (if directly inside goalposts or nested lhs) 
+   #   |  (as a goalpost)
+   #   * or & (unary only)
+   #   ( or ) (if they form a nested left hand side)
+   #parens form a nested lhs if they're not part of an actual
+   #parameter list and have a comma directly in them somewhere
+   #a nested lhs _must_ have a comma in it somewhere. this is
+   #not legal:
+   #  (foo)=[1]
+   #whereas this is:
+   #  (foo,)=[1]
+
+
+   
    #-----------------------------------
    def block_param_list_lookahead
       safe_recurse{ |la|
@@ -713,27 +885,45 @@ end
          a=ignored_tokens
 
          if eat_next_if(?|)
-           a<<KeywordToken.new("|",@file.pos-1)
+           a<<KeywordToken.new("|", input_position-1)
+if true
+           @parsestack.push mycontext=BlockParamListLhsContext.new(@linenum)
+           nextchar==?| and a.push NoWsToken.new(input_position)
+else
            if eat_next_if(?|)
-             a.concat [NoWsToken.new(@file.pos-1),
-                       KeywordToken.new('|',@file.pos-1)]
+             a.concat [NoWsToken.new(input_position-1),
+                       KeywordToken.new('|', input_position-1)]
            else
              assert !@defining_lvar
              @defining_lvar=true
              assert((@last_operative_token===';' or NewlineToken===@last_operative_token))
-             @bracestack.push BlockParamListContext.new(@linenum)
-             #block param initializers are not supported here, because ruby doesn't allow them!
-             begin
+             @parsestack.push mycontext=BlockParamListLhsContext.new(@linenum)
+             #block param initializers ARE supported here, even tho ruby doesn't allow them!
+             tok=nil
+             loop do
                tok=get1token
-               EoiToken===tok and lexerror tok,"eof in block parameter list"
+               case tok
+               when EoiToken; lexerror tok,"eof in block parameter list"
+               when AssignmentRhsListStartToken; @defining_lvar=false
+               when AssignmentRhsListEndToken; parsestack_lastnonassign_is?(mycontext) and @defining_lvar=true
+               end
+               
+               tok==='|' and parsestack_lastnonassign_is?(mycontext) and break
                a<<tok
-             end until tok==='|'
-             assert@defining_lvar
+             end 
+             assert@defining_lvar || AssignmentRhsContext===@parsestack.last
              @defining_lvar=false
-             BlockParamListContext===@bracestack.last or raise 'expected BlockParamListContext atop @bracestack'
-             @bracestack.pop
+             while AssignmentRhsContext===@parsestack.last 
+               a.push( *abort_noparens!('|') )
+             end
+             
+             @parsestack.last.object_id==mycontext.object_id or raise 'expected my BlockParamListLhsContext atop @parsestack'
+             @parsestack.pop
+             
+             a<<KeywordToken.new('|',tok.offset)
              @moretokens.empty? or
                fixme %#moretokens might be set from get1token call above...might be bad#
+end
            end
          end
 
@@ -755,8 +945,9 @@ end
    #then match the following tokens until
    #the matching endbrace is found
    def def_param_list
+      @in_def_name=false
       result=[]
-      normal_comma_level=old_bracestack_size=@bracestack.size
+      normal_comma_level=old_parsestack_size=@parsestack.size
       safe_recurse { |a|
          assert(@moretokens.empty?)
          assert((not IgnoreToken===@moretokens[0]))
@@ -770,9 +961,9 @@ end
             assert(tok==='(')
 
 
-            #bracestack was changed by get1token above...
+            #parsestack was changed by get1token above...
             normal_comma_level+=1
-            assert(normal_comma_level==@bracestack.size)
+            assert(normal_comma_level==@parsestack.size)
             endingblock=proc{|tok| tok===')' }
          else
             endingblock=proc{|tok| tok===';' or NewlineToken===tok}
@@ -785,36 +976,48 @@ end
          #read local parameter names
          loop do
             expect_name=(@last_operative_token===',' and
-                         normal_comma_level==@bracestack.size)
+                         normal_comma_level==@parsestack.size)
             expect_name and @defining_lvar||=true
             result << tok=get1token
             lexerror tok, "unexpected eof in def header" if EoiToken===tok
 
             #break if at end of param list
                endingblock===tok and 
-                 old_bracestack_size>=@bracestack.size and break
+                 old_parsestack_size>=@parsestack.size and break
 
             #next token is a local var name
             #(or the one after that if unary ops present)
             #result.concat ignored_tokens
-            expect_name and case tok
-               when IgnoreToken#, /^[A-Z]/ #do nothing
-               when VarNameToken
+            if expect_name 
+              case tok
+                when IgnoreToken #, /^[A-Z]/ #do nothing
+                when /^,$/.token_pat #hack
+               
+               
+                when VarNameToken
                   assert@defining_lvar
                   @defining_lvar=false
                   assert((not @last_operative_token===','))
-               when /^[&*]$/.token_pat #unary form...
+                when /^[&*]$/.token_pat #unary form...
                   #a NoWsToken is also expected... read it now
                   result.concat maybe_no_ws_token #not needed?
                   @last_operative_token=KeywordToken.new ','
-               else lexerror tok,"unfamiliar var name '#{tok}'"
+                else 
+                  lexerror tok,"unfamiliar var name '#{tok}'"
+              end
+            elsif /^,$/.token_pat===tok and
+                  normal_comma_level+1==@parsestack.size and
+                  AssignmentRhsContext===@parsestack.last
+              #seeing comma here should end implicit rhs started within the param list
+              result[-1,0]=AssignmentRhsListEndToken.new(tok.offset)
+              @parsestack.pop
             end
          end
          
          @defining_lvar=false
                    
 
-         assert(@bracestack.size <= old_bracestack_size)
+         assert(@parsestack.size <= old_parsestack_size)
          assert(endingblock[tok])
 
          #hack: force next token to look like start of a
@@ -846,19 +1049,19 @@ end
    end
 
    #-----------------------------------
-   #handle * in ruby code. is unary or binary operator?
+   #handle * & in ruby code. is unary or binary operator?
    def star_or_amp(ch)
       assert('*&'[ch])
-      if unary_op_expected? ch
+      want_unary=unary_op_expected? ch
+      result=(quadriop ch)
+      if want_unary
          #readahead(2)[1..1][/[\s\v#\\]/] or #not needed?
-         result=operator_or_methname_token getchar
+         assert OperatorToken===result
+         result.unary=true         #result should distinguish unary+binary *&
          WHSPLF[nextchar.chr] or
-           @moretokens << NoWsToken.new(@file.pos)
-         return result
-      else
-         return(quadriop ch)
+           @moretokens << NoWsToken.new(input_position)
       end
-         #result should distinguish unary+binary *&
+      result
    end
 
    #-----------------------------------
@@ -868,7 +1071,7 @@ end
          getchar
          NumberToken.new getchar_maybe_escape
       else
-         @bracestack.push TernaryContext.new(@linenum)
+         @parsestack.push TernaryContext.new(@linenum)
          KeywordToken.new getchar   #operator
       end
    end
@@ -888,18 +1091,19 @@ end
    end
 
    #-----------------------------------
-   #return true if tok corresponds to a variable or constant, false if its for a method, nil for something else
-   #we assume tok is a valid token with a correctly formed name.
+   #return true if last tok corresponds to a variable or constant, 
+   #false if its for a method, nil for something else
+   #we assume it is a valid token with a correctly formed name.
    #...should really be called was_var_name
    def is_var_name?
      (tok=@last_operative_token)
 
      s=tok.to_s
      case s
-     when /[^a-z_0-9]$/i: false
-     when /^[a-z_]/: @localvars===s or VARLIKE_KEYWORDS===s
-     when /^[A-Z]/: VarNameToken===tok
-     when /^[@$<]/: true
+     when /[^a-z_0-9]$/i; false
+     when /^[a-z_]/; localvars===s or VARLIKE_KEYWORDS===s
+     when /^[A-Z]/; VarNameToken===tok
+     when /^[@$<]/; true
      else raise "not var or method name: #{s}"
      end   
    end
@@ -907,21 +1111,26 @@ end
    #-----------------------------------
    def colon_quote_expected?(ch) #yukko hack
      assert ':?'[ch]
-     readahead(2)[/^(\?[^#{WHSPLF}]|:[$@a-zA-Z_'"`\[*~+\-\/%<=>&|^])$/o]   or return false
+     readahead(2)[/^(\?[^#{WHSPLF}]|:[^\s\r\n\t\f\v :])$/o]   or return false
 
      after_nonid_op? {
        #possible func-call as operator
 
-       !is_var_name?
+       not is_var_name? and
+         if ch==':'
+           not TernaryContext===@parsestack.last
+         else
+           !readahead(3)[/^\?[a-z0-9_]{2}/i]
+         end
      }
    end
 
    #-----------------------------------
    def symbol_or_op(ch)
-      startpos=@file.pos
+      startpos= input_position
       qe= colon_quote_expected?(ch)
       lastchar=prevchar
-      eat_next_if(ch) or raise "needed: "+ch
+      eat_next_if(ch[0]) or raise "needed: "+ch
 
       #handle quoted symbols like  :"foobar",  :"[]"
       qe and return symbol(':')
@@ -932,13 +1141,13 @@ end
         @moretokens.push(*abort_noparens!(':'))
         
         #end ternary context, if any
-        @bracestack.last.see @bracestack,:colon
+        @parsestack.last.see self,:colon
         
-        TernaryContext===@bracestack.last and @bracestack.pop #should be in the context's see handler
+        TernaryContext===@parsestack.last and @parsestack.pop #should be in the context's see handler
         
-        if ExpectDoOrNlContext===@bracestack.last #should be in the context's see handler
-          @bracestack.pop
-          assert @bracestack.last.starter[/^(while|until|for)$/]
+        if ExpectDoOrNlContext===@parsestack.last #should be in the context's see handler
+          @parsestack.pop
+          assert @parsestack.last.starter[/^(while|until|for)$/]
         end
         
         @moretokens.push KeywordToken.new(':',startpos)
@@ -965,17 +1174,17 @@ end
    #-----------------------------------
    def symbol(notbare,couldbecallsite=!notbare)
      assert !couldbecallsite
-     start=@file.pos
+     start= input_position
      notbare and start-=1
      klass=(notbare ? SymbolToken : MethNameToken)
      
      #look for operators
      opmatches=readahead(3)[RUBYSYMOPERATORREX]
-     result= opmatches ? @file.read(opmatches.size) :
+     result= opmatches ? read(opmatches.size) :
        case nc=nextchar
          when ?" then assert notbare;double_quote('"')
          when ?' then assert notbare;double_quote("'")
-         when ?` then @file.read(1)
+         when ?` then read(1)
          when ?@ then at_identifier.to_s
          when ?$ then dollar_identifier.to_s
          when ?_,?a..?z then identifier_as_string(?:)
@@ -991,19 +1200,24 @@ end
      return lexerror(klass.new(result,start),error)
    end
 
+   def merge_assignment_op_in_setter_callsites?
+     false
+   end
    #-----------------------------------
    def callsite_symbol(tok_to_errify)
-     start=@file.pos
+     start= input_position
      
      #look for operators
      opmatches=readahead(3)[RUBYSYMOPERATORREX]
-     return [opmatches ? @file.read(opmatches.size) :
+     return [opmatches ? read(opmatches.size) :
        case nc=nextchar
-         when ?` then @file.read(1)
-         when ?_,?a..?z,?A..?Z then identifier_as_string(?:)
+         when ?` then read(1)
+         when ?_,?a..?z,?A..?Z then 
+           context=merge_assignment_op_in_setter_callsites? ? ?: : nc
+           identifier_as_string(context)
          else 
            @last_operative_token=KeywordToken.new(';')
-           lexerror(tok_to_errify,"unexpected char starting symbol: #{nc.chr}")
+           lexerror(tok_to_errify,"unexpected char starting callsite symbol: #{nc.chr}, tok=#{tok_to_errify.inspect}")
            nil
        end, start
       ]
@@ -1011,10 +1225,10 @@ end
 
    #-----------------------------------
    def here_header
-      @file.read(2)=='<<' or raise "parser insanity"
+      read(2)=='<<' or raise "parser insanity"
 
       dash=eat_next_if(?-)
-      quote=eat_next_if( /^['"`]$/)
+      quote=eat_next_if( /['"`]/)
       if quote
         ender=til_charset(/[#{quote}]/)
         (quote==getchar) or 
@@ -1042,8 +1256,8 @@ end
            #handle case of here header in a string inclusion, but
            #here body outside it.
            cnt=0
-           1.upto @bracestack.size do |i|
-             case @bracestack[-i]
+           1.upto @parsestack.size do |i|
+             case @parsestack[-i]
                when AssignmentRhsContext,ParamListContextNoParen,TopLevelContext
                else cnt+=1
              end
@@ -1054,11 +1268,11 @@ end
            end
            
            tok=get1token
-           assert(a.object_id==@moretokens.object_id)
+           assert(a.equal?( @moretokens))
            toks<<tok
            EoiToken===tok and lexerror tok, "here body expected before eof"
          end while res.unsafe_to_use
-         assert(a.object_id==@moretokens.object_id)
+         assert(a.equal?( @moretokens))
          a[0,0]= toks   #same as a=toks+a, but keeps a's id
       }
 
@@ -1076,9 +1290,9 @@ end
            if quote_expected?(ch) #and @last_operative_token!='class' #not needed?
               here_header
            else
-              operator_or_methname_token @file.read(2)
+              operator_or_methname_token read(2)
            end
-        when "<=>" then operator_or_methname_token @file.read(3)
+        when "<=>" then operator_or_methname_token read(3)
         else quadriop(ch)
       end
    end
@@ -1087,115 +1301,152 @@ end
    def escnewline(ch)
       assert ch == '\\'
       
-      pos=@file.pos
+      pos= input_position
       result=getchar
       if nl=readnl
         result+=nl
       else
         error='illegal escape sequence'
       end
-      lexerror EscNlToken.new(@filename,@linenum,result,pos), error
-   end
+      
+      @moretokens.unshift FileAndLineToken.new(@filename,ln=@linenum,input_position)
+      optional_here_bodies
 
+      lexerror EscNlToken.new(@filename,ln-1,result,pos), error
+   end
+  
    #-----------------------------------
-   def newline(ch)
-      assert("\r\n"[nextchar.chr])
+   def optional_here_bodies
 
       #handle here bodies queued up by previous line
       #(we should be more compatible with dos/mac style newlines...)
-      if tofill=@incomplete_here_tokens.shift
-         tofill.string.offset=@file.pos
+      while tofill=@incomplete_here_tokens.shift
+         tofill.string.offset= input_position
          loop {
-            assert("\r\n"[nextchar.chr])
-
-            #retr evrything til next nl
-            line=all_quote(/^[\r\n]$/, tofill.quote, /^[\r\n]$/, :regex_esc_seq)
-            #(you didn't know all_quote could take a regex, did you?)
+            assert("\r\n"[prevchar])
 
-            #get rid of fals that otherwise appear to be in the middle of
-            #a string (and are emitted out of order)
-            fal=@moretokens.pop
-            assert FileAndLineToken===fal || fal.nil?
-
-            back1char
-            assert("\r\n"[nextchar.chr])
-
-            #matches terminating reg expr?
-            break if line.elems.size==1 and
-                     line.elems[0][tofill.termex]
-
-            tofill.append_token line
-            tofill.append readnl
-            back1char
+            #here body terminator?
+            oldpos= input_position
+            if tofill.dash
+              til_charset(/[^#{WHSP}]/o)
+            end
+            break if eof?
+            break if read(tofill.ender.size)==tofill.ender and readnl
+            input_position_set oldpos
+            
+            if tofill.quote=="'" 
+              line=til_charset(/[\r\n]/)+readnl
+              line.gsub! "\\\\", "\\"
+              tofill.append line
+              assert(line[-1..-1][/[\r\n]/])
+            else
+
+              back1char  #-1 to make newline char the next to read
+              @linenum-=1
+  
+              #retr evrything til next nl
+              line=all_quote(INET_NL_REX, tofill.quote, INET_NL_REX)
+              #(you didn't know all_quote could take a regex, did you?)
+  
+              #get rid of fals that otherwise appear to be in the middle of
+              #a string (and are emitted out of order)
+              fal=@moretokens.pop
+              assert FileAndLineToken===fal || fal.nil?
+  
+              back1char
+              @linenum-=1
+              assert("\r\n"[nextchar.chr])
+              tofill.append_token line
+              tofill.append readnl
+            end
          }
 
-         assert("\r\n"[nextchar.chr])
+         assert(eof?  || "\r\n"[prevchar])
          tofill.unsafe_to_use=false
+         tofill.line=@linenum-1
 
-         return tofill.bodyclass.new(tofill)
+         @moretokens.push \
+           tofill.bodyclass.new(tofill), 
+           FileAndLineToken.new(@filename,@linenum,input_position)
       end
+ 
+   end
+
+   #-----------------------------------
+   def newline(ch)
+      assert("\r\n"[nextchar.chr])
+
+      
 
       #ordinary newline handling (possibly implicitly escaped)
       assert("\r\n"[nextchar.chr])
+                   assert !@parsestack.empty?
       assert @moretokens.empty?
       result=if NewlineToken===@last_operative_token or #hack
                 @last_operative_token===/^(;|begin|do|#{INNERBOUNDINGWORDS})$/ or #hack
                 !after_nonid_op?{false}
              then   #hack-o-rama: probly cases left out above
                    a= abort_noparens!
-                   ExpectDoOrNlContext===@bracestack.last and @bracestack.pop
-                   @bracestack.last.see @bracestack,:semi
+                   ExpectDoOrNlContext===@parsestack.last and @parsestack.pop
+                   assert !@parsestack.empty?
+                   @parsestack.last.see self,:semi
 
                    a << super(ch)
                    @moretokens.replace a+@moretokens
                    @moretokens.shift
              else
-                   offset=@file.pos
-                   #@moretokens <<
-                      EscNlToken.new(@filename,@linenum,readnl,offset)
+                   offset= input_position
+                   nl=readnl
+                   @moretokens << FileAndLineToken.new(@filename,@linenum,input_position)
+                   EscNlToken.new(@filename,@linenum-1,nl,offset)
                    #WsToken.new ' ' #why?  #should be "\\\n" ?
              end
 
+      optional_here_bodies
+
       start_of_line_directives
 
       return result
    end
 
    #-----------------------------------
-   EQBEGIN=%r/^=begin[^a-zA-Z_0-9]$/
+   EQBEGIN=%r/^=begin[ \t\v\r\n\f]$/
    EQBEGINLENGTH=7
    EQEND='=end'
-   ENDMARKER=/^__END__[\r\n]$/
+   EQENDLENGTH=4
+   ENDMARKER=/^__END__[\r\n]?\Z/
    ENDMARKERLENGTH=8
    def start_of_line_directives
       #handle =begin...=end (at start of a line)
       while EQBEGIN===readahead(EQBEGINLENGTH)
-         startpos=@file.pos
-         more=@file.read(EQBEGINLENGTH-1)   #get =begin
-
-         #keep reading til /\n=end.*\n/
-         @file.each(EQEND) {|cblock|
-            more << cblock
-            #must be at start of line
-            break if /^[\r\n]#{EQEND}/o===readback(EQEND.length+1)
-         }
+         startpos= input_position
+         more= read(EQBEGINLENGTH-1)   #get =begin
+
+         begin
+           eof? and raise "eof before =end"
+           more<<til_charset(/[\r\n]/)
+           more<<readnl
+         end until readahead(EQENDLENGTH)==EQEND
+
          #read rest of line after =end
-         more << @file.til_charset(/[\r\n]/)  
+         more << til_charset(/[\r\n]/)  
          assert((?\r===nextchar or ?\n===nextchar))
          assert !(/[\r\n]/===more[-1,1])
+         more<< readnl
 
-         newls= more.scan(/\r\n?|\n\r?/)
-         @linenum+= newls.size
+#         newls= more.scan(/\r\n?|\n\r?/)
+#         @linenum+= newls.size
 
          #inject the fresh comment into future token results
-         @moretokens.push IgnoreToken.new(more,startpos)
+         @moretokens.push IgnoreToken.new(more,startpos),
+                          FileAndLineToken.new(@filename,@linenum,input_position)
       end
 
       #handle __END__
       if ENDMARKER===readahead(ENDMARKERLENGTH)
-         assert !(ImplicitContext===@bracestack.last)
-         @moretokens.unshift endoffile_detected(@file.read(6))
-         @file.pos=@file.stat.size
+         assert !(ImplicitContext===@parsestack.last)
+         @moretokens.unshift endoffile_detected(read(7))
+#         input_position_set @file.size
       end
    end
 
@@ -1221,11 +1472,15 @@ end
    #used to resolve the ambiguity of
    # <<, %, ? in ruby
    #returns whether current token is to be the start of a literal
-   #/ is not handled right here if whitespace immediately follows the /
    def quote_expected?(ch) #yukko hack
+     if AssignmentContext===@parsestack.last
+       @parsestack.pop
+       return false
+     end
+
      case ch[0]
           when ?? then readahead(2)[/^\?[#{WHSPLF}]$/o] #not needed?
-          when ?% then readahead(3)[/^%([a-ps-vyzA-PR-VX-Z]|[QqrwWx][a-zA-Z0-9])/]
+          when ?% then readahead(3)[/^%([a-pt-vyzA-PR-VX-Z]|[QqrswWx][a-zA-Z0-9])/]
           when ?< then !readahead(4)[/^<<-?['"`a-z0-9_]/i]
           else raise 'unexpected ch (#{ch}) in quote_expected?'
      #     when ?+,?-,?&,?*,?~,?! then '*&='[readahead(2)[1..1]]
@@ -1240,22 +1495,29 @@ end
    end
 
    #-----------------------------------
+   #returns false if last token was an value, true if it was an operator.
+   #returns what block yields if last token was a method name.
    #used to resolve the ambiguity of
-   # <<, %, /, ?, :, and newline in ruby
+   # <<, %, /, ?, :, and newline (among others) in ruby
    def after_nonid_op?
       case @last_operative_token
-         when MethNameToken,VarNameToken, FUNCLIKE_KEYWORDS.token_pat
+         when MethNameToken, FUNCLIKE_KEYWORDS.token_pat ,VarNameToken
+         #VarNameToken should really be left out of this case... 
+         #should be in next branch instread
+         #callers all check for last token being not a variable if they pass anything
+         #but {false} in the block
             return yield
          when StringToken, SymbolToken, NumberToken, HerePlaceholderToken,
-              %r{^(class|module|do|end|self|true|false|nil|
-                   __FILE__|__LINE__|[\})\]]|alias|(un)?def|for
+              %r{^(
+                class|module|end|self|true|false|nil|
+                __FILE__|__LINE__|[\})\]]|alias|(un)?def|for
               )$}x.token_pat
-            #do shouldn't be in above list... dunno about def/undef
-            #maybe class/module shouldn't either?  
+            #dunno about def/undef
+            #maybe class/module shouldn't he here either?  
             #for is also in NewlineToken branch, below.
             #what about rescue?
             return false
-         when /^(#{RUBYOPERATORREX}|#{INNERBOUNDINGWORDS})$/o.token_pat
+         when /^(#{RUBYOPERATORREX}|#{INNERBOUNDINGWORDS}|do)$/o.token_pat
             #regexs above must match whole string
             #assert(@last_operative_token==$&) #disabled 'cause $& is now always nil :(
             return true
@@ -1273,19 +1535,46 @@ end
       end
    end
 
+
+
+
+   #-----------------------------------
+   #returns the last context on @parsestack which isn't an ImplicitContext
+   def last_context_not_implicit
+     @parsestack.reverse_each{|ctx|
+       return ctx unless ImplicitContext===ctx
+     }
+     fail
+   end
+
+   #-----------------------------------
+   #a | has been seen. is it an operator? or a goalpost?
+   #(goalpost == delimiter of block param list)
+   #if it is a goalpost, end the BlockParamListLhsContext on
+   #the context stack, as well as any implicit contexts on top of it.
+   def conjunction_or_goalpost(ch)
+     result=quadriop(ch)
+     if result===/^|$/ and BlockParamListLhsContext===last_context_not_implicit 
+       @moretokens.push( *abort_noparens!("|"))
+       assert(BlockParamListLhsContext===@parsestack.last)
+       @parsestack.pop
+       @moretokens.push KeywordToken.new("|", input_position-1)
+       result=@moretokens.shift
+     end
+     result
+   end
+   
    #-----------------------------------
    def quadriop(ch) #match /&&?=?/ (&, &&, &=, or &&=)
       assert(%w[& * | < >].include?(ch))
-#      '&*'[ch] and qe=quote_expected?(ch) #not needed?
       result=getchar + (eat_next_if(ch)or'')
       if eat_next_if(?=)
          result << ?=
-#      elsif qe and result[/^[&*]$/] #not needed?
-#         @moretokens<<NoWsToken.new(@file.pos) #not needed?
       end
       return operator_or_methname_token(result)
    end
 
+
    #-----------------------------------
    def biop(ch) #match /%=?/ (% or %=)
       assert(ch[/^[%^~]$/])
@@ -1295,18 +1584,18 @@ end
       end
       return operator_or_methname_token( result)
    end
-
    #-----------------------------------
-   def tilde(ch) #match /~=?/ (~ or ~=)
+   def tilde(ch) #match ~
       assert(ch=='~')
       result=getchar
-#      eat_next_if(?=) ?
+#      eat_next_if(?=) ?   #ack, spppft, I'm always getting this backwards
 #         result <<?= :
          WHSPLF[nextchar.chr] ||
-           @moretokens << NoWsToken.new(@file.pos)
+           @moretokens << NoWsToken.new(input_position)
       #why is the NoWsToken necessary at this point?    
-      return operator_or_methname_token( result)
-         #result should distinguish unary ~
+      result=operator_or_methname_token result
+      result.unary=true         #result should distinguish unary ~
+      result
    end
 
    #-----------------------------------
@@ -1327,8 +1616,9 @@ end
         else #unary operator
           result=getchar
           WHSPLF[nextchar.chr] or
-            @moretokens << NoWsToken.new(@file.pos)
-          return(operator_or_methname_token result)
+            @moretokens << NoWsToken.new(input_position)
+          result=(operator_or_methname_token result)
+          result.unary=true
           #todo: result should distinguish unary+binary +-
         end
       else #binary operator
@@ -1337,45 +1627,54 @@ end
          if eat_next_if(?=)
             result << ?=
          end
-         return(operator_or_methname_token result)
+         result=(operator_or_methname_token result)
          #todo: result should distinguish unary+binary +-
       end
+      result
    end
 
    #-----------------------------------
    def equals(ch) #match /=(>|~|==?)?/ (= or == or =~ or === or =>)
-      offset=@file.pos
+      offset= input_position
       str=getchar
       assert str=='='
-      c=(eat_next_if(/^[~=>]$/)or'')
+      c=(eat_next_if(/[~=>]/)or'')
       str << c
+      result= operator_or_methname_token( str,offset)
       case c
       when '=': str<< (eat_next_if(?=)or'')
       
-      when '>': @bracestack.last.see @bracestack,:arrow
+      when '>': 
+        unless ParamListContextNoParen===@parsestack.last
+          @moretokens.unshift result
+          @moretokens.unshift( *abort_noparens!("=>"))
+          result=@moretokens.shift
+        end
+        @parsestack.last.see self,:arrow
       when '': #record local variable definitions
 
-        @bracestack.push AssignmentRhsContext.new(@linenum)
+        @parsestack.push AssignmentRhsContext.new(@linenum)
         @moretokens.unshift AssignmentRhsListStartToken.new( offset+1)
       end
-      return operator_or_methname_token( str,offset)
+      return result
    end
 
    #-----------------------------------
    def exclam(ch) #match /![~=]?/ (! or != or !~)
       assert nextchar==?!
       result=getchar
-      k=eat_next_if(/^[~=]$/)
+      k=eat_next_if(/[~=]/)
       if k
         result+=k
       else
         WHSPLF[nextchar.chr] or
-          @moretokens << NoWsToken.new(@file.pos)
+          @moretokens << NoWsToken.new(input_position)
       end
-      return KeywordToken.new(result)
+      return KeywordToken.new(result, input_position-result.size)
       #result should distinguish unary !
    end
 
+
    #-----------------------------------
    def dot(ch)
       str=''
@@ -1391,7 +1690,6 @@ end
       dot_rhs(result)
       return result
    end
-   
    #-----------------------------------
    def dot_rhs(prevtok)
       safe_recurse { |a|
@@ -1403,20 +1701,17 @@ end
       }     
    end
 
-   #-----------------------------------
-   def single_quote(ch=nil)
-      double_quote(ch)
-   end
-
   #-----------------------------------
   def back_quote(ch=nil)
-    oldpos=@file.pos
-    @last_operative_token===/^(def|::|\.)$/ and return MethNameToken.new(
-       (eat_next_if(?`) or raise "insanity"), oldpos
-    )
-    double_quote(ch)
+    if @last_operative_token===/^(def|::|\.)$/
+      oldpos= input_position
+      MethNameToken.new(eat_next_if(?`), oldpos)
+    else
+      double_quote(ch)
+    end
   end
 
+if false
    #-----------------------------------
    def comment(str)
      result=""
@@ -1441,27 +1736,30 @@ end
 
      return IgnoreToken.new(result)
    end
-
+end
    #-----------------------------------
    def open_brace(ch)
       assert((ch!='[' or !want_op_name))
       assert(@moretokens.empty?)
       lastchar=prevchar
-      ch=eat_next_if(/^[({\[]$/)or raise "lexer confusion"
-      tokch=KeywordToken.new(ch,@file.pos-1)
+      ch=eat_next_if(/[({\[]/)or raise "lexer confusion"
+      tokch=KeywordToken.new(ch, input_position-1)
+      
 
       #maybe emitting of NoWsToken can be moved into var_or_meth_name ??
       case tokch.ident
       when '['
-        #fixme: in contexts expecting an (operator) method name, we 
-        #       should match [] or []= at this point
-        @bracestack.push ListImmedContext.new(ch,@linenum)
+        # in contexts expecting an (operator) method name, we 
+        #       would want to match [] or []= at this point
+        #but control never comes this way in those cases... goes 
+        #to custom parsers for alias, undef, and def in #parse_keywords
+        tokch.set_infix! unless after_nonid_op?{WHSPLF[lastchar]}
+        @parsestack.push ListImmedContext.new(ch,@linenum)
         lasttok=last_operative_token
         #could be: lasttok===/^[a-z_]/i
-        if (VarNameToken===lasttok or MethNameToken===lasttok or
-            lasttok===FUNCLIKE_KEYWORDS) and !WHSPCHARS[lastchar]
+        if (VarNameToken===lasttok or ImplicitParamListEndToken===lasttok or MethNameToken===lasttok) and !WHSPCHARS[lastchar]
                @moretokens << (tokch)
-               tokch= NoWsToken.new(@file.pos-1)
+               tokch= NoWsToken.new(input_position-1)
         end
       when '('
         lasttok=last_operative_token
@@ -1470,19 +1768,20 @@ end
             lasttok===FUNCLIKE_KEYWORDS)
           unless WHSPCHARS[lastchar]
                @moretokens << tokch
-               tokch= NoWsToken.new(@file.pos-1)
+               tokch= NoWsToken.new(input_position-1)
           end
-          @bracestack.push ParamListContext.new(@linenum)
+          @parsestack.push ParamListContext.new(@linenum)
         else
-          @bracestack.push ParenContext.new(@linenum)
+          @parsestack.push ParenContext.new(@linenum)
         end
 
       when '{'
       #check if we are in a hash literal or string inclusion (#{}),
       #in which case below would be bad.
-      if after_nonid_op?{false}
-        @bracestack.push ListImmedContext.new(ch,@linenum) #that is, a hash
+      if after_nonid_op?{false} or @last_operative_token.has_no_block?
+        @parsestack.push ListImmedContext.new(ch,@linenum) #that is, a hash
       else
+        tokch.set_infix!
 =begin not needed now, i think
         # 'need to find matching callsite context and end it if implicit'
         lasttok=last_operative_token
@@ -1492,8 +1791,8 @@ end
         end
 =end
 
-        @localvars.start_block
-        @bracestack.push BlockContext.new(@linenum)
+        localvars.start_block
+        @parsestack.push BlockContext.new(@linenum)
         block_param_list_lookahead
       end
       end
@@ -1504,18 +1803,18 @@ end
    def close_brace(ch)
       ch==eat_next_if(/[)}\]]/) or raise "lexer confusion"
       @moretokens.concat abort_noparens!(ch)
-      @moretokens<< kw=KeywordToken.new( ch,@file.pos-1)
-      @bracestack.last.see @bracestack,:semi #hack
-      if @bracestack.empty? 
+      @parsestack.last.see self,:semi #hack
+      @moretokens<< kw=KeywordToken.new( ch, input_position-1)
+      if @parsestack.empty? 
         lexerror kw,"unmatched brace: #{ch}"
         return @moretokens.shift
       end
-      ctx=@bracestack.pop
+      ctx=@parsestack.pop
       origch,line=ctx.starter,ctx.linenum
       ch==PAIRS[origch] or
         lexerror kw,"mismatched braces: #{origch}#{ch}\n" +
                  "matching brace location", @filename, line
-      BlockContext===ctx and @localvars.end_block
+      BlockContext===ctx and localvars.end_block
       if ParamListContext==ctx.class
         assert ch==')'
         #kw.set_callsite! #not needed?
@@ -1525,29 +1824,29 @@ end
 
    #-----------------------------------
    def eof(ch=nil)
-       #this must be the very last character...
-       oldpos=@file.pos
-       assert(?\0==@file.getc)
+     #this must be the very last character...
+     oldpos= input_position
+     assert(?\0==getc)
 
-       result= "\0#{ignored_tokens(true).delete_if{|t|FileAndLineToken===t}}"
+     result= "\0#{ignored_tokens(true).delete_if{|t|FileAndLineToken===t}}"
 
-       @file.pos==@file.stat.size or
-          lexerror result,'nul character is not at the end of file'
-       @file.pos=@file.stat.size
-       return(endoffile_detected result)
+     eof? or
+        lexerror result,'nul character is not at the end of file'
+     input_position_set @file.size
+     return(endoffile_detected result)
    end
 
    #-----------------------------------
    def endoffile_detected(s='')
      @moretokens.push( *(abort_noparens!.push super(s)))
      result= @moretokens.shift
-     balanced_braces? or (lexerror result,"unbalanced braces at eof. bracestack=#{@bracestack.inspect}")
+     balanced_braces? or (lexerror result,"unbalanced braces at eof. parsestack=#{@parsestack.inspect}")
      result
    end
 
   #-----------------------------------
   def single_char_token(ch)
-    KeywordToken.new super(ch), @file.pos-1
+    KeywordToken.new super(ch), input_position-1
   end
 
   #-----------------------------------
@@ -1557,13 +1856,13 @@ end
   
   #-----------------------------------
   def semicolon(ch)
-      assert @moretokens.empty?
+    assert @moretokens.empty?
     @moretokens.push(*abort_noparens!)
-    @bracestack.last.see @bracestack,:semi
-         if ExpectDoOrNlContext===@bracestack.last #should be in context's see:semi handler
-            @bracestack.pop
-            assert @bracestack.last.starter[/^(while|until|for)$/]
-         end
+    @parsestack.last.see self,:semi
+    if ExpectDoOrNlContext===@parsestack.last #should be in context's see:semi handler
+      @parsestack.pop
+      assert @parsestack.last.starter[/^(while|until|for)$/]
+    end
     @moretokens.push single_char_token(ch)
     return @moretokens.shift
   end
@@ -1582,7 +1881,11 @@ end
 
   #-----------------------------------
   #tokenify_results_of  :identifier
-  save_offsets_in(*CHARMAPPINGS.values.uniq-[:symbol_or_op,:open_brace,:whitespace])
+  save_offsets_in(*CHARMAPPINGS.values.uniq-[
+    :symbol_or_op,:open_brace,:whitespace,:exclam,:backquote
+  
+  
+  ])
   #save_offsets_in :symbol
 
 end