reg 0.4.6

data/philosophy.txt

@@ -0,0 +1,72 @@
+
+
+That's a long story, and well worth telling. 
+
+A long time ago, I wanted a better regexp than regexp. My search ended 
+ when I found an extremely obscure language called gema (the 
+ general-purpose matcher). I'm guessing that I'm the only person to ever 
+ take gema seriously. For a time, I became the worlds foremost expert on 
+ gema. Gema is designed around the idea that all computation can be 
+ modeled as pattern and replacement. Everything in gema is pattern and 
+ replacement... essentially everything is done with regexps. I was 
+ fascinated with the idea. This seemed to me to be a much better model 
+ for most programming problems, which typically involve reading input, 
+ tranforming it in some way, and writing it out again. Conventional 
+ languages (starting with fortran, and including ruby) are based around 
+ the idea of a program being a long string of formulas. This is great 
+ for math-heavy stuff, but most programming is really about data 
+ manipulation, not math. 
+
+But there was trouble in paradise. Gema was wonderful, but weird. The 
+ syntax was cranky. The author had issued one version long ago then 
+ disappeared. Gema code was hard to read, in part because 
+ everythingwasalljammedtogether . 
+ Ifyouinsertspacestomakeitmorer eadable,itchangesthesemanticso fyourprogram. 
+ There were strange problems that I never tracked down or fully 
+ characterized. The only data-type was the string. You had to be an 
+ expert at avoiding the invisible pitfalls of the language to get 
+ anywhere. But I did get surprisingly far. I managed to coax gema into 
+ becoming a true parser, and parsing a toy language. 
+ I wanted to write a compiler in gema. Yes, the whole compiler. And 
+ parsing the toy language was already straining its capabilites. It 
+ wasn't the data model; I actually figured out how to model all other 
+ data types using strings. A match-and-replace language is actually much 
+ better suited to most compiler tasks than an algol-like formula 
+ language. 
+
+Eventually, I abandoned gema, determined to recreate it's glory in a 
+ cleaner form. It was at about this time that I discovered ruby. The 
+ successor to gema was ruma, the ruby matcher. Ruma would be basically 
+ just like gema, but without the problems. Whitespace allowed between 
+ tokens. Proper quotation mechanisms, including nested quotes. And the 
+ language used in the actions (replacements) would be full ruby, instead 
+ of gema's inadequate and crude action language. 
+
+Ruma got maybe halfway done... quite a ways, really. As part of ruma, I 
+ needed a ruby lexer to make sense of the actions. This turned out to be 
+ quite a lot harder than I had anticipated; I'm still working on that 
+ lexer. 
+
+After grinding away at the lexer for a while, dreaming of ruma in the 
+ meantime, I had a brainstorm. Ruma, like gema, was to be a string-based 
+ language. It only operated on strings. In gema, that was just fine 
+ because everything was strings and you just had to live with that. But 
+ ruby has all these other types, a real type system. Wouldn't it be nice 
+ to have those sophisticated search capabilites for other types too? 
+ Well, since I proved to myself that all data types can be converted to 
+ strings, why not convert the ruby data into strings and then match that 
+ in ruma. Of course, it would be so much nicer to just do the matching 
+ on the data in it's original form.... 
+
+The breakthrough came when I realized how malleable ruby really is. I 
+ had become accustomed to c, which I still love, but in so many ways 
+ it's so much more limited. I didn't really have to write my own parser 
+ and lexer; ruby could do it all for me. I just had to override a bunch 
+ of operators. 
+
+After that, it was simple. All I do is override the right operators, 
+ and ruby does the parsing and hands me the match expressions in 
+ already-parsed form. Reg is amazingly small in the end. Most of the 
+ effort and code went into the array matcher, but at least as much 
+ functionality is to be had from the hash and object matchers, which 
+ were trivial. 

data/reg.gemspec

@@ -0,0 +1,27 @@
+require 'rubygems'
+
+spec = Gem::Specification.new do |s|
+    s.name = 'reg'
+    s.rubyforge_project = 'reg'
+    s.version = '0.4.6'
+    s.summary = 'The reg pattern matching/replacement language'
+    s.files = %w[item_thattest.rb regbackref.rb regknows.rb regtest.rb
+                 numberset.rb regbind.rb reglogic.rb regvar.rb
+                 COPYING parser.txt regcase.rb reglookab.rb           
+                 README regcore.rb regold.rb reg.gemspec reggrid.csv            
+                 assert.rb philosophy.txt regdeferred.rb regpath.rb 
+                 regposition.rb trace.rb calc.reg reg.rb   
+                 regguide.txt regprogress.rb reghash.rb regreplace.rb
+                 forward_to.rb regarray.rb regarrayold.rb regitem_that.rb regsugar.rb]
+    s.require_path = '.'
+    s.has_rdoc = false
+    s.requirements=["none"]
+    s.add_dependency("cursor", [">= 0.9"])
+    s.author = 'Caleb Clausen'
+end
+
+
+if $0==__FILE__
+  Gem::manage_gems
+  Gem::Builder.new(spec).build
+end

data/reg.rb

@@ -0,0 +1,33 @@
+=begin copyright
+    reg - the ruby extended grammar
+    Copyright (C) 2005  Caleb Clausen
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+=end
+require 'regcore'
+require 'reglogic'
+require 'reghash'
+require 'regarray'
+#require 'regarrayold' #old bt engine
+require 'regprogress' #new bt engine
+#enable one engine or the other, but not both
+
+require 'regbackref'
+require 'regitem_that'
+require 'regknows'
+require 'regsugar'
+require 'regold'  #will go away
+Kernel.instance_eval( &Reg::TLA_pirate)
+Reg::Sugar.include!

data/regarray.rb

@@ -0,0 +1,675 @@
+=begin copyright
+    reg - the ruby extended grammar
+    Copyright (C) 2005  Caleb Clausen
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+=end
+
+require "assert"
+require "pp"
+
+module Reg
+  module Reg
+    def itemrange; 1..1 end  #default match 1 item
+
+
+    #create a (vector) Reg that will match this pattern repeatedly.
+    #(creates a Reg::Repeat.)
+    #the argument determines the number of times to match.
+    #times may be a positive integer, zero, INFINITY, or a
+    #range over any of the above. if a range, the lower
+    #end may not be INFINITY! Reg#- and Reg#+ are shortcuts
+    #for the most common cases of multiplting by a range.
+    #(at least 0 and at most INFINITY.) watch out when
+    #multiplying with zero and INFINITY (including in a
+    #range), as you can easily create a situation where
+    #the number of matches to enumerate explodes exponentionaly,
+    #or even is infinite. i won't say too much here except
+    #that these are generally the same sorts of problems you
+    #can run into with Regexps as well.
+    def *(times=0..INFINITY)
+      Repeat.new(self,times)
+    end
+
+    #repeat this pattern up to atmost times. could match
+    #0 times as the minimum number of matches here is zero.
+    def -(atmost=1)
+      self*(0..atmost)
+    end
+
+    #repeat this pattern atleast times or more
+    def +(atleast=1)
+      self*(atleast..INFINITY)
+    end
+
+  end
+
+  #--------------------------
+  module Multiple
+    def ===(other)
+      method_missing(:===, other)
+    end
+
+    def maybe_multiple(needsmult) #better name needed
+      assert( needsmult.respond_to?( :mmatch))
+      class <<needsmult
+        undef_method :mmatch
+        include Multiple
+        #alias mmatch mmatch_multiple #this doesn't work... why?
+        def mmatch(a,s) mmatch_multiple(a,s); end #have to do this instead
+      end
+      assert( needsmult.respond_to?( :mmatch))
+    end
+
+    def maybe_multiples(*args) end
+    #we're already multiple; no need to try to become multiple again
+
+    def mmatch(arr,idx) #multiple match
+      abstract
+    end
+
+    #negated Reg::Multiple's are automatically lookaheads (not implemented yet)
+    def ~
+      ~(Lookahead.new self)
+    end
+
+
+    def starts_with
+      abstract
+    end
+
+    def ends_with
+      abstract
+    end
+
+    def matches_class
+      raise 'multiple regs match no single class'
+    end
+  end
+
+  #--------------------------
+  module Backtrace
+  #  protected
+
+    def regs(ri) @regs[ri] end
+
+    def update_di(di,len); di+len; end
+    #--------------------------
+    $RegTraceEnable=$RegTraceDisable=nil
+    def trace_enabled?
+      @trace||=nil
+      $RegTraceEnable or (!$RegTraceDisable && @trace)
+    end
+
+    #--------------------------
+    def trace!
+      @trace=true
+      self
+    end
+
+    #--------------------------
+    def notrace!
+      @trace=false
+      self
+    end
+  end
+
+  #--------------------------
+  if false
+  class RR < ::Array
+    def inspect
+      [self,super].to_s
+    end
+
+    def rrflatten
+      result=[]
+      each{|i|
+        case i
+          when RR then         result +=i.rrflatten
+          when Literal then result << i.unlit
+          else                 result << i
+        end
+      }
+    end
+
+    def +(other)
+      RR[*super]
+    end
+  end
+  Result=RR
+  else
+  RR=::Array
+  end
+
+  #--------------------------
+  class MatchSet
+
+    def next_match(ary,start)
+      abstract
+    end
+
+    def deep_copy
+      abstract
+    end
+
+    def ob_state
+      instance_variables.sort.map{|i| instance_variable_get i }
+    end
+
+    def ==(other)
+      self.class==other.class and ob_state==other.ob_state
+    end
+
+  end
+
+  #--------------------------
+  class SingleRepeatMatchSet < MatchSet
+    def initialize(startcnt,stepper,endcnt)
+      endcnt==startcnt and raise 'why even make it a set, then?'
+      (endcnt-startcnt)*stepper>0 or raise "tried to make null match set"
+      assert startcnt>=0
+      assert endcnt>=0
+      @matchtimes,@stepper,@endcnt=startcnt,stepper,endcnt
+    end
+
+    def next_match(arr,idx)
+      assert @stepper == -1 #'only greedy matching implemnted for now'
+      @endcnt<=@matchtimes or return nil
+      assert @matchtimes >=0
+      result=[RR[arr[idx...idx+@matchtimes]], @matchtimes]
+      assert ::Array===result.first.first
+      @matchtimes+=@stepper
+      
+      assert @matchtimes >=-1
+
+      assert ::Array===result.first.first
+      return result
+    end
+
+    def deep_copy
+      dup
+    end
+  end
+
+ 
+  #--------------------------
+  class Repeat
+    include Reg,Backtrace,Multiple
+
+    attr :times
+
+    def max_matches; @times.end end
+
+    def regs(ri) @reg end
+
+    def initialize(reg,times)
+      Integer===times and times=times..times
+      times.exclude_end? and times=times.begin..times.end-1
+      assert times.begin <= times.end
+      assert times.begin < INFINITY
+      assert times.begin >= 0
+      assert times.end >= 0
+      if Multiple===reg
+        class<<self
+          #alias mmatch mmatch_multiple #this doesn't work... why?
+          def mmatch(a,s) mmatch_multiple(a,s); end #have to do this instead
+        end
+      else
+        assert reg.itemrange==(1..1)
+        @itemrange=times
+      end
+      @reg,@times=reg,times
+    end
+
+    def itemrange
+      defined? @itemrange and return @itemrange
+
+      i=@reg.itemrange
+      rf,rl=i.first,i.last
+      tf,tl=times.first,times.last
+      @itemrange = rf*tf ..
+          if tl==0 or rl==0
+            0
+          elsif tl==INFINITY
+            #ought to emit warnings if trouble here...
+            #rl==INFINITY and maybe trouble
+            #rf==0 and trouble
+            INFINITY
+          elsif rl==INFINITY
+            #...and here
+            #maybe trouble #... combinatorial explosion
+            INFINITY
+          else
+            rl*tl
+           end
+    end
+
+
+    def enough_matches? matchcnt
+      @times===matchcnt
+    end
+
+    def inspect
+      if @times.end==INFINITY
+        "(#{@reg.inspect})+#{@times.begin}"
+      elsif @times.begin==0
+        "(#{@reg.inspect})-#{@times.end}"
+      elsif @times.begin==@times.end
+        "(#{@reg.inspect})*#{@times.begin}"
+      else
+        "(#{@reg.inspect})*(#{@times.begin}..#{@times.end})"
+      end
+    end
+
+    def subregs; @reg end
+
+  private
+
+  end
+
+
+
+
+  #--------------------------
+  class OrMatchSet < MatchSet
+    def initialize(orreg,idx,set,firstmatch)
+      @orreg,@idx,@set,@firstmatch=orreg,idx,set,firstmatch
+      assert @firstmatch.nil? || ::Array===@firstmatch.first.first
+    end
+
+    def ob_state
+      instance_variables.map{|i| instance_variable_get i }
+    end
+
+    def ==(other)
+      OrMatchSet===other and ob_state==other.ob_state
+    end
+
+    def next_match(ary,idx)
+      if @firstmatch
+        result,@firstmatch=@firstmatch,nil
+        assert ::Array===result
+        assert ::Array===result.first.first
+        assert 2==result.size
+        assert Integer===result.last
+        return result
+      end
+      @set and result= @set.next_match(ary,idx)
+      while result.nil?
+        @idx+=1
+        @idx >= @orreg.regs.size and return nil
+        x=@orreg.regs[@idx].mmatch(ary,idx)
+        @set,result=*if MatchSet===x then [x,x.next_match] else [nil,x] end
+      end
+      a=RR[nil]*@orreg.regs.size
+      a[idx]=result[0]
+      result[0]=a
+      assert ::Array===result.first.first
+      return result
+    end
+
+    def deep_copy
+      result=OrMatchSet.new(@orreg,@idx,@set && @set.deep_copy,@firstmatch)
+      assert self==result
+      return result
+    end
+  end
+
+  #--------------------------
+  class Or
+    def mmatch(arr,start)
+      assert start <= arr.size
+      @regs.each_with_index {|reg,i|
+        reg===arr[start] and
+          return OrMatchSet.new(self,i,nil,[arr[start]])
+      } unless start == arr.size
+      return nil
+    end
+
+    def itemrange
+    if true  
+      min,max=INFINITY,0
+      @regs.each {|r|
+        min=r.itemrange.first if min>r.itemrange.first
+        max=r.itemrange.last if max<r.itemrange.last
+      }
+      return min..max
+    else
+      limits=@regs.map{|r|
+
+        i=(r.respond_to? :itemrange)? r.itemrange : 1..1
+        [i.first,i.last]
+      }.transpose
+      limits.first.sort.first .. limits.last.sort.last
+    end
+    end
+
+  private
+    def mmatch_multiple(arr,start)
+      mat=i=nil
+      @regs.each_with_index{|r,i|
+        if r.respond_to? :mmatch
+          mat=r.mmatch(arr,start) or next
+          if mat.respond_to? :next_match
+            return OrMatchSet.new(self,i,mat,mat.next_match(arr,start))
+          else
+            return OrMatchSet.new(self,i,nil,mat)
+          end
+        else
+          r===arr[start] and
+            return OrMatchSet.new(self,i,nil,[[[arr[start]]],1])
+        end
+      }
+
+      assert mat.nil?
+      return nil
+    end
+  end
+
+  #--------------------------
+  class Xor
+    def clean_result
+      huh
+    end
+
+    def itemrange
+      #min,max=INFINITY,0
+      #@regs.each {|r|
+      #  min=[min,r.itemrange.first].sort.first
+      #  max=[r.itemrange.last,max].sort.last
+      #}
+      #return min..max
+      limits=@regs.map{|r| i=r.itemrange; [i.first,i.last]}.transpose
+      limits.first.sort.first .. limits.last.sort.last
+    end
+
+  private
+=begin
+    def mmatch_multiple(arr,start)
+      mat=i=nil
+      count=0
+      @regs.each_with_index{|reg,idx|
+        if reg.respond_to? :mmatch
+          mat=reg.mmatch(arr,start) or next
+        else
+          reg===arr[start] or next
+          mat=[[arr[start]],1]
+        end
+        count==0 or return nil
+        count=1
+        assert mat
+      }
+
+      return nil unless mat
+      assert count==1
+      mat.respond_to? :next_match and return XorMatchSet.new(reg,idx,mat,huh)
+
+      a=RR[nil]*regs.size
+      a[idx]=mat[0]
+      mat[0]=a
+      assert huh
+      assert ::Array===mat.first.first
+      return mat
+    end
+=end    
+    
+    def mmatch_multiple arr, start
+      found=nil
+      @regs.each{|reg|
+        if m=reg.mmatch(arr, start)
+          return if found
+          found=m
+        end
+      }
+      return found
+    end
+
+  end
+
+  #--------------------------
+  class And
+    include Backtrace #shouldn't this be included only when needed?
+
+    def update_di(di,len) di; end
+
+
+    def clean_result
+      huh
+    end
+
+
+    def enough_matches? matchcnt
+      matchcnt==@regs.size
+    end
+
+    def itemrange
+      limits=@regs.map{|r| i=r.itemrange; [i.first,i.last]}.transpose
+      limits.first.sort.last .. limits.last.sort.last
+    end
+
+  private
+    def mmatch_multiple(arr,start)
+    #in this version, at least one of @regs is a multiple reg
+      assert( (0..arr.size).include?( start))
+      result,*bogus=huh.bt_match(arr,start,0,0,[RR[]])
+      result and AndMatchSet.new(self,result)
+    end
+  end
+
+  #--------------------------
+  class Array
+    include Reg,Backtrace
+
+    def max_matches; @regs.size end
+
+    def initialize(*regs)
+      @regs=regs
+    end
+
+    class <<self
+      alias new__nobooleans new
+      def new(*args)
+#        args.detect{|o| /^(AND|X?OR)$/.sym===o } or return new__nobooleans(*args)
+#        +[/^(AND|X?OR)$/.sym.splitter].match(args)
+        Pair===args.first and return OrderedHash.new(*args)
+        new__nobooleans(*args)
+      end
+      alias [] new
+    end
+
+    def matches_class; ::Array end
+
+    def -@ #subsequence inclusion
+      Subseq.new(*@regs)
+    end
+    
+    def +@ #cvt to Reg::Array; that what we are already....
+      self
+    end
+
+    def maybe_multiples(*args) end #never do anything for Reg::Array
+
+    def enough_matches? matchcnt
+      matchcnt==@regs.size
+    end
+
+    def +(reg)
+      #not right... + should not modify self
+      if self.class==reg.class 
+        @regs.concat reg.regs
+      else
+        super
+      end
+    end
+
+    def inspect
+      "+["+ @regs.collect{|r| r.inspect}.join(', ') +"]"
+    end
+  
+    def subregs; @regs end
+  end
+
+
+
+
+
+
+
+  #--------------------------
+  class Subseq < ::Reg::Array
+    include Multiple
+
+    def max_matches; @regs.size end
+
+    def initialize(*regs)
+      regs.each{|reg| Multiple===reg and class<<self
+        undef mmatch
+        def mmatch(a,s) mmatch_multiple(a,s) end
+      end}
+
+      @regs=regs
+    end
+
+
+    def inspect
+      super.sub( /^\+/,'-')
+    end
+    
+    def itemrange
+      #add the ranges of the individual items
+      @itemrange ||= #some caching...
+        @regs.inject(0){|sum,ob| sum+ob.begin } .. 
+          @regs.inject(0){|sum,ob| sum+ob.end }
+    end
+
+    def -@ #subsequence inclusion... that's what we are, do nothing
+      self
+    end
+    
+    def +@ #cvt to Reg::Array
+      Array.new(*@regs)
+    end
+
+  private
+
+#tla of +[], regproc{}
+assign_TLA true, :Reg=>:Array
+assign_TLA :Res=>:Subseq
+#no need to alias the constant name 'Reg', too.
+#ruby does it for us.
+  end
+
+
+
+  #--------------------------
+  class None; end
+  class <<None
+    include Reg
+    def new; self end
+
+    def *(times) 
+      times===0 ? Many[0] : self
+    end
+
+    def ~; Any;  end
+
+    def &(other); self; end
+
+    def |(other) other end
+    def ^(other) other end
+
+    def ===(other); false; end
+    def matches_class; self; end
+  end
+
+if defined? $RegAnyEnable #disabled for now -- these optimizations are broken
+
+  #--------------------------
+  class Any; end
+  class <<Any  #maybe all this can be in Object's meta-class....
+    include Reg
+    
+    #any is a singleton
+    def new;        self      end
+
+    def *(times)
+      Many.new(times)
+    end
+
+    def ~; None;  end
+
+    def &(other); other; end
+
+    def |(other); self;  end
+    def ^(other); ~other end
+
+    def ===(other); true;end
+    def matches_class; ::Object end
+  end
+
+  #--------------------------
+  class Many
+    include Reg
+    include Multiple
+
+    class <<self
+      @@RAMs={}
+      alias uncached__new new
+      def new times=0..INFINITY
+        @@RAMs[times] ||= uncached__new times
+      end
+      alias [] new
+    end
+
+    def initialize(times=0..INFINITY)
+      Integer===times and times=times..times
+      @times=times
+    end
+
+    def mmatch(arr,start)
+      left=arr.size-start
+      beg=@times.begin
+      beg<=left and 
+        SingleRepeatMatchSet.new([left,@times.end].max, -1, beg)
+    end
+
+    def subregs; Any  end
+
+    def inspect; "Any*(#{@times})"; end
+  end
+
+  #--------------------------
+  class ::Object
+    def reg
+      Any
+    end
+  end
+  OB=Any
+  OBS=Many[]
+
+else #traditional and uncomplicated version of OB and OBS  
+    OB=::Object.reg
+    OBS=OB+0  #std abbreviation for 0 or more of anything
+    def OBS.inspect
+      "OBS"
+    end
+    def OB.inspect
+      "OB"
+    end  
+end
+
+
+
+end