sequence 0.1.0

data/README.txt

@@ -0,0 +1,320 @@
+Sequence provides a unified api for access to sequential data types, like
+Strings, Arrays, Files, IOs, and Enumerations. Each sequence encapsulates
+some data and a current position within it. Some operations apply to data
+at (or relative to) the position, others are independant of position. The
+api contains operations for moving the position, reading and writing data
+(with or without moving the position) forward or backward from the current
+position or anywhere, scanning for patterns (like StringScanner, but it
+works in Files too, among others), and saving a position that will remain
+valid even after data is deleted or inserted elsewhere within the
+sequence. 
+
+There are also some utility classes for making sequences reversed or
+circular, turning one-way sequences into two-way, buffering, and making
+sequences that are subsets or aggregations of existing sequences. 
+
+Sequence is based on Eric Mahurin's Cursor library. I'd like to thank Eric
+for Cursor, without which Sequence would not have existed; my design is
+very much a derivative of his.
+
+
+Sequences always fall into one of two broad categories: string-like and
+array- like. String-like cursors contain only character data, whereas
+Array-like cursors contain objects. 
+ 
+
+
+
+Stuff that's  unworking at the moment is marked with **
+
+Reading a single element:
+When any of these operations fail (at beginning/end), nil is returned. 
+(Note: nil can also be found inside array-like cursors, so a nil result
+doesn't necessarily mean you're at eof.) A name with -back means
+backwards, -ahead/-behind means lookahead/lookbehind (pos not moved). 
+
+    #read1       #get next element and advance position
+    #readback1   #get previous element and move position back
+    #readahead1  #get next element, leaving position alone
+    #readbehind1 #get previous element, leaving position alone
+
+
+Read methods:
+These all come in forward and backward forms, and forms that can hold the
+position in place. The element sequences are passed/returned in
+Array/String like things. 
+
+A note on backwards operations: some methods move the position backward
+instead of forward. They operate on data immediately before the position
+instead of immediately after. Data is still read or processed in normal
+order. To get data in backwards order, use Sequence::Reversed. 
+
+    #read(len)     #read +len+ elements. leaving position after the data read
+    #readback(len) #read data behind current position, 
+                   #leaving position before the data read.
+    
+    #readahead(len)    #like read, but position is left alone.
+    #readbehind(len)   #like readback, but position is left alone.
+    #read!(reverse=false)    # read the remaining elements.
+
+Numeric position methods:
+The methods below deal with numeric positions (a pos) to represent the
+sequence location. A non-negative number represents the number of elements
+from the beginning. A negative number represents the location relative to
+the end. 
+
+    #pos    # number of elements from the beginning (0 is at the beginning). 
+    #pos?(p)    #  this checks to see if p is a valid numeric position.
+
+    #pos=(p)    # Set #pos to be +p+. 
+    #goto p    #go to an absolute position; identical to #pos=
+
+    #move(len)    # move len elements, relative to the current position 
+                  #and return distance moved
+    
+    #move!(reverse=false)    # move to end of the remaining elements 
+                             # and return distance moved
+    #begin!    #go to beginning
+    #end!    #go to end
+
+    #rest_size #number of data items remaining
+    #eof?    #are we at past the end of the data, 
+             #with no more data ever to arrive?
+
+Sequence::Position methods:
+
+The position methods below use a Sequence::Position to hold the position
+rather than simply a numeric position.  These position objects hold a
+#pos, which does not change when the parent sequence's position changes.
+(Also, the #pos in these objects adjust based on insertions and
+deletions.)
+
+    #position(_pos=pos) #returns a Sequence::Position to 
+                        #represent the current location.
+    #position=(p)       #C Set the position to a Position +p+ (from #position)
+    #position?(p)       #  this queries whether a particular #position +p+ 
+                        #is valid (is a child or self).
+    
+    #-(other)  #return new Position decreased by a length or 
+               #distance between 2 positions.
+    #+(len)    #Returns a new #position increased by +len+ 
+               #(positive or negative).
+
+    #succ    # Return a new #position for next location 
+             # or +nil+ if we are at the end.
+    #pred    # Return a new #position for previous location 
+             # or +nil+ if we are at the beginning.
+
+    #begin    # Return a new #position for the beginning.
+    #end    # Return a new #position for the end.
+
+    #<=>(other)    # Compare +other+ (a #position) to the current position. 
+
+
+Access the entire collection: 
+    #data,      #return the data underlying the sequence. 
+                #the type of the result depends on the sequence type
+    
+    #data_class #return Array if this sequence can contain any object, 
+                #String if it contains only characters
+    #new_data   # Return an empty String or Array, 
+                #depending on what #data_class is
+    
+    
+    #all_data  #return a String or Array 
+               #containing all the data of the sequence
+
+    #size/length    # Returns the number of elements.
+
+    #empty?    # is there any data in the sequence?
+
+    #each     # Performs each just to make this class Enumerable.  
+    
+    
+Random access: 
+    #<<(elem)             #append to the end
+
+    #slice/[](index)     # random access to sequence data like in Array/String
+    #slice/[](index,len) # random access to sequence data like in Array/String
+    #slice/[](range)     # random access to sequence data like in Array/String
+
+Modifying data:
+    #slice!              # slice and delete data
+    #[]=/modify(sliceargs,newdata)  #replace an arbitrary subsequence 
+                                    #with a different one
+
+   #modify has a number of special subcases:
+    #insert   -- len is 0 (all existing element are retained)
+    #delete   -- newdata.size is 0
+    #append   -- insert data after end  
+    #prepend  -- insert data before start
+    #push/pop -- insert/delete element(s) at end 
+    #shift/unshift -- insert/delete elements at start
+    #overwrite  -- replacedata.size == len, no shifting needed
+    
+    #subcases that use the position:
+    #(over)write  -- overwrite after current position and move ahead
+    #(over)writeback -- overwrite before current position and move back
+    #(over)writeahead/behind -- overwrite near location without moving it
+    #deletebehind/#deleteahead -- delete before or after the location ** 
+    #insertbehind/#insertahead -- insert before or after the location **
+
+
+
+
+Taken from/inspired by StringScanner:
+See the StringScanner documentation (ri StringScanner) for a description
+of these methods. I have extended StringScanner's interface to take
+Strings and character literals (Integers) as well as Regexps. 
+
+About anchors:
+
+When going forward:
+  \A ^ match current position
+  \Z $ match end of data (String, File, whatever)
+When going backward:
+  \A ^ match beginning of data
+  \Z $ match current position
+
+^ and $ also match at line edges, as usual
+
+
+My strategy is to rewrite the anchors in the regexp to make them conform
+to the desired definition. For instance, \Z is replaced with (?!), unless
+the last byte of the file is within the buffer to be compared against, in
+which case it is left alone. 
+
+To counter the speed problem, there's a cache so the same regexp doesn't
+have to be rewritten more than once. 
+
+about matchdata:
+#pre_match/#post_match may not be what you expect; they are Sequences.
+#offset contains numeric positions from the very beginning of the Sequence.
+                #anchored, forwards
+    #scan(pat)  #if pat is right after current position, 
+                #advance position and return what pat matched, else nil
+    #skip(pat)  #like scan, but returns length instead of match data
+    #check(pat) #like scan, but doesn't move position  
+    #match?(pat)#like scan, but returns length and doesn't move position
+    
+                      #unanchored, forwards
+    #scan_until(pat)  #scan for pat somewhere after position 
+                      #(not necessarily right after)
+    #skip_until(pat)  #skip til pattern somewhere after position
+    #check_until(pat) #check til pattern somewhere after position
+    #exist?(pat)      #does pat exist somewhere after position? if so where?
+
+                      #anchored, backwards
+    #scanback(pat)    #scan for pat right before pos
+    #skipback(pat)    #skip pat before pos
+    #checkback(pat)   #check pat before pos
+    #matchback?(pat)  #match pat before pos
+    
+                      #unanchored, backwards
+    #scanback_until(pat)  #scan for pat somewhere before pos
+    #skipback_until(pat)  #skip til pat somewhere before pos
+    #checkback_until(pat) #check for pat somewhere before pos
+    #existback?(pat)      #does pat exist somewhere before pos? if so where?
+    
+    #skip_literal
+    #skip_literals
+    
+    #skip_until_literal  **
+    #skip_until_literals **
+
+    #last_match
+    
+    #maxmatchlen       #query scan buffer size
+    #maxmatchlen= len  #set scan buffer size
+    
+    #split **
+    
+    
+    #index/#rindex
+
+    #stride **
+
+    #holding    #hold current position while executing a block. 
+                #The current pos is passed in.
+    #holding?   #like #holding, but position is reset only if 
+                #block returns false or nil 
+    #holding! &block    #like #holding, but block is instance_eval'd 
+                        #in the sequence.
+
+
+    #subseq(*args)    #make a new seq out of a subrange of current seq data.
+
+    #reverse    #make a new seq that reverses the order of data.
+    
+    #close    # Close the seq.  This will also close/invalidate 
+              #every child #position and derived sequence attached to 
+              #this one.
+    #closed?    # Is the seq closed?
+
+
+
+  #nearbegin(len)  #is this seq within len elements of the beginning?
+  #nearend(len)    #is this seq within len elements of the end?
+
+    #more_data?   #is there any more data in the seq?    
+    #was_data?    #has any data been seen so far, or are 
+                  #we still at the beginning?
+
+    
+    #first    #return first element of data
+    #last    #return last element of data
+
+
+
+sequence classes:
+base sequences:
+Sequence      #ancestor class
+  Indexed
+    OfArray     #over data in an Array
+    OfString    #over data in a String
+  File          #over data in a File (no insert/delete)
+  IO  (R/O)     #over data in an IO (pipe/socket/tty/whatever)
+  Enum (R/O)    #over data in an Enumeration 
+  SingleItem    #over a single scalar item
+  OfHash         (**)
+  OfObjectIvars  (**)
+  OfObjectMethods  (**)
+
+derived sequences:
+  Shifting       #saves a copy of the base sequence data 
+                 #(in another sequence) as it is read
+  Buffered (**)  #makes unidirectional sequences bidirectional 
+                 #(up to the buffer size)
+  Reversed       #reverses the order of its base sequence
+  Circular       #loops base sequence data over and over
+  SubSeq         #extracts a contiguous subset of base sequence 
+                 #data into a new seq
+  List      (**) #logically concatenates its base sequences
+  Overlay (**)   #intercepts writes to the base seq
+  Position       #an independant position within the base seq
+  
+Future:
+  Scanning  (**)
+  Splitting (**)
+  Striding (**)
+  Transforming (**)
+  Branching (**)
+  LinkedList (**)
+  DoubleLinkedList (**)
+  BinaryTree (**)
+
+  Pipe        (**)       #a bidirectional communication channel 
+                         #with sequence api
+  IndexedList   (**)
+  
+ 
+known problems:
+Some unit tests fail
+Buffered does not work at all
+Shifting's modify methods don't work reliably
+Some write operations failing for List
+No unit tests at all for array-like sequences
+(tho Reg's unit test does test OfArray at least somewhat...)
+
+Copyright (C) 2006  Caleb Clausen  
+Distributed under the terms of Ruby's license.

data/Rakefile

@@ -0,0 +1,32 @@
+# Copyright (C) 2006  Caleb Clausen
+# Distributed under the terms of Ruby's license.
+require 'rubygems'
+require 'hoe'
+require 'lib/sequence/version.rb'
+ 
+ 
+ 
+ 
+   Hoe.new("sequence", Sequence::VERSION) do |_|
+
+     _.author = "Caleb Clausen"
+     _.email = "sequence-owner @at@ inforadical .dot. net"
+     _.url = "http://sequence.rubyforge.org/"
+     _.summary = "A single api for reading and writing sequential data types."
+     _.description = <<-END
+A unified wrapper api for accessing data in Strings, Arrays, Files, IOs, 
+and Enumerations. Each sequence encapsulates some data and a current position 
+within it. There are methods for moving the position, reading and writing data
+(with or without moving the position) forward or backward from the current
+position (or anywhere at all), scanning for patterns (like StringScanner, but 
+it works in Files too, among others), and saving a position that will remain
+valid even after data is deleted or inserted elsewhere within the sequence. 
+
+There are also some utility classes for making sequences reversed or
+circular, turning one-way sequences into two-way, buffering, and making
+sequences that are subsets or aggregations of existing sequences. 
+     END
+   end
+
+   # add other tasks here
+

data/lib/assert.rb

@@ -0,0 +1,16 @@
+# Copyright (C) 2006  Caleb Clausen
+# Distributed under the terms of Ruby's license.
+
+module Kernel
+  def assert(expr,msg="assertion failed")
+    defined? $Debug and $Debug and (expr or raise msg)
+  end
+
+  @@printed={}
+  def fixme(s)
+    unless @@printed[s] 
+      @@printed[s]=1
+      defined? $Debug and $Debug and $stderr.print "FIXME: #{s}\n"
+    end
+  end
+end

data/lib/sequence/arraylike.rb

@@ -0,0 +1,57 @@
+# Copyright (C) 2006  Caleb Clausen
+# Distributed under the terms of Ruby's license.
+class Sequence
+  module ArrayLike
+    def data_class; Array end
+    def like; ArrayLike end
+
+    def scan(pat)
+        elem=nil
+        more_data? and holding?{pat===(elem=read1)} and return [elem]
+    end
+    
+    def scan_until pat
+        i=index(pat,pos) or return
+        read(i-pos)+scan(pat)
+    end
+    
+    def scanback pat
+        elem=nil
+        was_data? and holding?{pat===(elem=readback1)} and return [elem]
+    end
+
+    def scanback_until pat
+        i=rindex(pat,pos) or return
+        readback(pos-i)
+    end
+
+    #I ought to have #match and #matchback like in StringLike too
+    
+    def push(*arr)
+      append arr
+    end
+    
+    def unshift(*arr)
+      prepend arr
+    end
+    
+    def index pat,pos=0
+      pos=_normalize_pos(pos)
+      begin
+        pat===(slice pos) and return pos
+        pos+=1
+      end until pos>=size
+      nil
+    end
+    
+    def rindex pat,pos=-1
+      pos=_normalize_pos(pos)
+      begin
+        pat===(slice pos) and return pos
+        pos-=1
+      end until pos<0
+      nil
+    end
+    
+  end
+end

data/lib/sequence/buffered.rb

@@ -0,0 +1,188 @@
+# $Id$
+# Copyright (C) 2006  Caleb Clausen
+# Distributed under the terms of Ruby's license.
+
+require 'sequence'
+#require 'sequence/split'
+
+class Sequence
+# This class gives unidirectional sequences (i.e. IO pipes) some
+# bidirectional capabilities.  An input sequence (or input IO) and/or an output
+# sequence (or output IO)
+# can be specified.  The #position, #position?, and #position! methods are
+# used to control buffering.  Full sequence capability (limited by the size of the buffer
+# sequence) is accessible starting from the first #position.  When the end of
+# the buffer is reached more data is read from the input sequence (if not nil) .  When no
+# #position is outstanding, everything before the buffer sequence is written
+# to the output sequence (if not nil).  If the sequence is attempted
+# to be moved before the buffer, the output sequence is read in reverse (which
+# the output sequence may not like).
+#how much of that should remain true?
+class Buffered < Sequence
+    def initialize(input,buffer_size=1024,buffer=nil)
+        @input = input
+        huh #@input used incorrectly... it should be used kinda like a read-once data store
+            #and Buffered should have an independant position
+        @buffer_size=buffer_size
+        @buffer = buffer||@input.new_data
+#        @output_pos = output_pos
+        @buffer_pos=@pos=@input.pos
+        
+        @input.on_change_notify self  
+    end
+    
+    def change_notification
+      huh #invalidate (part of) @buffer if it overlaps the changed area
+      huh #adjust @buffer_pos as necessary
+    end
+    
+    attr_accessor :buffer_size
+    attr :pos
+    
+    # :stopdoc:
+    def new_data
+        @input.new_data
+    end
+=begin
+    protected
+    def _delete1after?
+        v0 = @buffer.delete1after?
+        v0.nil? && @input && (v0 = @input.read(1)) && (v0 = v0[0])
+        v0
+    end
+    def _delete1before?
+        v0 = @buffer.delete1before?
+        v0.nil? && @output_pos>0 && (@output_pos -= 1;v0 = @output.read(-1)) && (v0 = v0[0])
+        v0
+    end
+    def _insert1before(v)
+        if not position?
+            len = @buffer.move!(true)
+            if @output
+                value = @buffer.read(len,nil)
+                value << v
+                @output.write(value)
+            else
+                @buffer.read(len,nil) if len
+            end
+            @output_pos += (len||0)+1
+        else
+            @buffer.insert1before(v)
+        end
+        true
+    end
+    def _insert1after(v)
+        @buffer.insert1after(v)
+    end
+    public
+    def close
+        if @output
+            @buffer.move!(true)
+            value = @buffer.read!(false) and @output.write(value)
+        end
+        super
+    end
+=end 
+    def _default_maxmatchlen; @buffer_size/2 end
+
+    attr :pos
+    
+    def _pos=(pos)
+      if pos<@buffer_pos
+        @pos=@input.pos=pos  #could raise exception, if @input doesn't support #pos=
+      elsif pos<=@buffer_pos+@buffer.size
+        @pos=pos
+      else #@pos > buffer_end_pos
+        assert @buffer_pos+@buffer.size==@input.pos
+        @buffer<<@input.read(pos-@input.pos)
+        buffer_begin_ageout!
+      end
+    end
+
+    def history_mode?(pos=@pos)
+      pos<@buffer_pos+@buffer.size
+    end
+    
+    def_delegators :@input, :size
+
+    def crude_read(len)
+        assert @buffer_pos+@buffer.size==@input.pos
+        result=@input.read(len)
+        @buffer<<result
+        buffer_begin_ageout!
+        result
+    end
+
+    def crude_read_before(len)
+        assert @buffer_pos+@buffer.size==@input.pos
+        result=@input.read(len)
+        @buffer.insert(0,*result)
+        buffer_end_ageout!
+        result
+    end
+
+    def read(len)
+      if @pos<@buffer_pos
+        if @buffer_pos-@pos >= @buffer_size
+          @buffer_pos=@pos
+          @buffer=new_data
+          return crude_read(len)
+        else 
+          crude_read_before(@buffer_pos-@pos)
+          self._pos=@buffer_pos
+          
+          #fall thru
+        end
+      end
+      
+      if history_mode?
+        if history_mode?(pos+len-1)
+          result=@buffer[@pos-@buffer_pos,len]
+          @pos+=len
+        else
+          result=@buffer[@pos-@buffer.pos..0]
+          result<<crude_read(len-result.size)
+        end
+
+        result
+      else
+        crude_read len
+      end
+    end
+
+    def buffer_begin_ageout!
+      diff=@buffer.size-@buffer_size 
+      if diff>0 
+        @buffer.slice!(0,diff)
+        @buffer_pos+=diff
+      end
+    end
+    
+    def buffer_end_ageout!
+      diff=@buffer.size-@buffer_size 
+      if diff>0 
+        @buffer.slice!(-diff..-1)
+      end
+    end
+
+    def modify(*args)
+      huh "what does it mean to write to a Buffered?"
+      repldata=args.pop
+      first,len,only1=_parse_slice_args(*args)
+      first<pos-@buffer.size and huh
+      result=new_data
+      if first<pos
+        result=@buffer[huh]
+      end
+      if first+len>pos
+        huh
+      end
+      huh
+    end
+
+
+    # :startdoc:
+end
+end
+
+