bahuvrihi-external 0.3.1

data/History

@@ -0,0 +1,17 @@
+== 0.3.1 / 2009-06-29
+
+* updates to gem dependencies
+* experimentally added IO as an accepted type in Utils
+
+== 0.3.0 / 2008-10-27
+
+Major update with refactoring (ex ExtArr is now ExternalArray) 
+and greatly expanded testing. [] and []= methods all Externals 
+now comply with the Array specification in RubySpec[rubyspec.org].
+Implementation of other methods is under way.
+
+== 0.1.0 / 2007-12-10 revision 23
+
+Initial release with working [] and []= methods
+and several basic array functions for ExtInd, 
+ExtArr, and ExtArc.

data/MIT-LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2006-2008, Regents of the University of Colorado.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this
+software and associated documentation files (the "Software"), to deal in the Software
+without restriction, including without limitation the rights to use, copy, modify, merge,
+publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons
+to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or
+substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

data/README

@@ -0,0 +1,203 @@
+= External 
+
+Indexing and array-like access to data stored on disk rather than in memory.
+
+== Description
+
+External provides a way to index and access array data directly from a file
+without loading it into memory.  Indexes may be cached in memory or stored 
+on disk with the data file, in essence giving you arbitrarily large arrays.
+Externals automatically chunk and buffer methods like <tt>each</tt> so that
+the memory footprint remains low even during enumeration.  
+
+The main External classes are:
+
+* ExternalIndex   --  for formatted binary data
+* ExternalArchive --  for string data
+* ExternalArray   --  for objects (serialized as YAML)
+
+The array-like behavior of these classes is developed using modified versions
+of the RubySpec[http://rubyspec.org] specification for Array.  The idea is to 
+eventually duck-type all Array methods, including sort and collect, with 
+acceptable performance.
+
+* Rubyforge[http://rubyforge.org/projects/external]
+* Lighthouse[http://bahuvrihi.lighthouseapp.com/projects/10590-external]
+* Github[http://github.com/bahuvrihi/external/tree/master]
+
+==== Bugs/Known Issues
+
+* only a limited set of array methods are currently supported
+* currently only [] and []= are fully tested vs RubySpec
+* documentation is patchy
+
+Note also that YAML dump/load of some objects doesn't work or doesn't
+reproduce the object; such objects will not be properly stored in an
+ExternalArray.  Problematic objects include:
+
+Proc and Class:
+
+  block = lambda {}
+  YAML.load(YAML.dump(block))         # !> TypeError: allocator undefined for Proc
+  YAML.dump(Object)                   # !> TypeError: can't dump anonymous class Class
+
+Carriage returns ("\r"):
+
+  YAML.load(YAML.dump("\r"))          # => nil
+  YAML.load(YAML.dump("\r\n"))        # => ""
+  YAML.load(YAML.dump("string with \r\n inside"))  # => "string with \n inside"
+
+Chains of newlines ("\n"):
+
+  YAML.load(YAML.dump("\n"))          # => ""
+  YAML.load(YAML.dump("\n\n"))        # => ""
+  
+DateTime is loaded as Time:
+
+  YAML.load(YAML.dump(DateTime.now)).class         # => Time
+  
+== Usage
+
+=== ExternalArray
+
+ExternalArray can be initialized from data using the [] operator and used like
+an array.
+
+  a = ExternalArray['str', {'key' => 'value'}]
+  a[0]                                # => 'str'
+  a.last                              # => {'key' => 'value'}
+  a << [1,2]; a.to_a                  # => ['str', {'key' => 'value'}, [1,2]]
+
+ExternalArray serializes and stores entries to an io while building an io_index
+that tracks the start and length of each entry.  By default ExternalArray
+will serialize to a Tempfile and use an Array as the io_index:
+
+  a.io.class                          # => Tempfile
+  a.io.rewind; a.io.read              # => "--- str\n--- \nkey: value\n--- \n- 1\n- 2\n"
+  a.io_index.class                    # => Array
+  a.io_index.to_a                     # => [[0, 8], [8, 16], [24, 13]]
+
+To save this data more permanently, provide a path to <tt>close</tt>; the tempfile
+is moved to the path and a binary index file will be created:
+
+  a.close('example.yml')
+  File.read('example.yml')            # => "--- str\n--- \nkey: value\n--- \n- 1\n- 2\n"
+  
+  index = File.read('example.index')
+  index.unpack('I*')                  # => [0, 8, 8, 16, 24, 13]
+
+ExternalArray provides <tt>open</tt> to create ExternalArrays from an existing 
+file; the instance will use an index file if it exists and automatically
+reindex the data if it does not.  Manual calls to reindex may be necessary when
+you initialize an ExternalArray with <tt>new</tt> instead of <tt>open</tt>:
+
+  # use of an existing index file
+  ExternalArray.open('example.yml') do |b|
+    File.basename(b.io_index.io.path) # => 'example.index'
+    b.to_a                            # => ['str', {'key' => 'value'}, [1,2]]
+  end
+
+  # automatic reindexing
+  FileUtils.rm('example.index')
+  ExternalArray.open('example.yml') do |b|
+    b.to_a                            # => ['str', {'key' => 'value'}, [1,2]]
+  end
+  
+  # manual reindexing
+  file = File.open('example.yml')
+  c = ExternalArray.new(file)
+  
+  c.to_a                              # => []
+  c.reindex
+  c.to_a                              # => ['str', {'key' => 'value'}, [1,2]]
+
+=== ExternalArchive
+
+ExternalArchive is exactly like ExternalArray except that it only stores
+strings (ExternalArray is actually a subclass of ExternalArchive which
+dumps/loads strings).
+
+  arc = ExternalArchive["swift", "brown", "fox"]
+  arc[2]                              # => "fox"
+  arc.to_a                            # => ["swift", "brown", "fox"]
+  arc.io.rewind; arc.io.read          # => "swiftbrownfox"
+
+ExternalArchive is useful as a base for classes to access archival data.
+Here is a simple parser for FASTA[http://en.wikipedia.org/wiki/Fasta_format]
+data:
+
+  # A sample FASTA entry
+  # >gi|5524211|gb|AAD44166.1| cytochrome b [Elephas maximus maximus]
+  # LCLYTHIGRNIYYGSYLYSETWNTGIMLLLITMATAFMGYVLPWGQMSFWGATVITNLFSAIPYIGTNLV
+  # EWIWGGFSVDKATLNRFFAFHFILPFTMVALAGVHLTFLHETGSNNPLGLTSDSDKIPFHPYYTIKDFLG
+  # LLILILLLLLLALLSPDMLGDPDNHMPADPLNTPLHIKPEWYFLFAYAILRSVPNKLGGVLALFLSIVIL
+  # GLMPFLHTSKHRSMMLRPLSQALFWTLTMDLLTLTWIGSQPVEYPYTIIGQMASILYFSIILAFLPIAGX
+  # IENY
+  
+  class FastaEntry
+    attr_reader :header, :body
+    
+    def initialize(str)
+      @body = str.split(/\r?\n/)
+      @header = body.shift
+    end
+  end
+  
+  class FastaArchive < ExternalArchive
+    def str_to_entry(str); FastaEntry.new(str); end
+    def entry_to_str(entry); ([entry.header] + entry.body).join("\n"); end
+    
+    def reindex
+      reindex_by_sep('>', :entry_follows_sep => true)
+    end
+  end
+  
+  require 'open-uri'
+  fasta = FastaArchive.new open('http://external.rubyforge.org/doc/tiny_fasta.txt')
+  fasta.reindex
+  
+  fasta.length                        # => 5
+  fasta[0].body                       # => ["MEVNILAFIATTLFVLVPTAFLLIIYVKTVSQSD"]
+
+The non-redundant {NCBI protein database}[ftp://ftp.ncbi.nih.gov/blast/db/FASTA/]
+contains greater than 7 million FASTA entries in a 3.56 GB file; ExternalArchive
+is targeted at files that size, where lazy loading of data and a small memory 
+footprint are critical.
+
+=== ExternalIndex
+
+ExternalIndex provides array-like access to formatted binary data.  The index of an 
+uncached ExternalArray is an ExternalIndex configured for binary data like 'II'; two
+integers corresponding to the start position and length an entry.
+
+  index = ExternalIndex[1, 2, 3, 4, 5, 6, {:format => 'II'}]
+  index.format                        # => 'I*'
+  index.frame                         # => 2
+  index[1]                            # => [3,4]
+  index.to_a                          # => [[1,2], [3,4], [5,6]]
+
+ExternalIndex handles arbitrary packing formats, opening many possibilities:
+
+  Tempfile.new('sample.txt') do |file|
+    file << [1,2,3].pack("IQS")
+    file << [4,5,6].pack("IQS")
+    file << [7,8,9].pack("IQS")
+    file.flush
+
+    index = ExternalIndex.new(file, :format => "IQS")
+    index[1]                          # => [4,5,6]
+    index.to_a                        # => [[1,2,3], [4,5,6], [7,8,9]]
+  end
+
+== Installation
+
+External is available from RubyForge[http://rubyforge.org/projects/external].  Use:
+
+  % gem install external
+
+== Info 
+
+Copyright (c) 2006-2008, Regents of the University of Colorado.
+Developer:: {Simon Chiang}[http://bahuvrihi.wordpress.com], {Biomolecular Structure Program}[http://biomol.uchsc.edu/], {Hansen Lab}[http://hsc-proteomics.uchsc.edu/hansenlab/] 
+Support:: CU Denver School of Medicine Deans Academic Enrichment Fund
+Licence:: {MIT-Style}[link:files/MIT-LICENSE.html]

data/lib/external.rb

@@ -0,0 +1,2 @@
+$:.unshift File.expand_path(File.dirname(__FILE__))
+require 'external_array'

data/lib/external/base.rb

@@ -0,0 +1,217 @@
+# For some inexplicable reason yaml MUST be required before
+# tempfile in order for ExtArrTest::test_LSHIFT to pass.
+# Otherwise it fails with 'TypeError: allocator undefined for Proc'
+ 
+require 'yaml'
+require 'tempfile'
+
+require 'external/enumerable'
+require 'external/io'
+
+module External
+  
+  # Base provides shared IO and Array-like methods used by ExternalArchive, 
+  # ExternalArray, and ExternalIndex.
+  class Base
+    class << self
+      
+      # Initializes an instance of self with File.open(path, mode) as an io.
+      # As with File.open, the instance will be passed to the block and
+      # closed when the block returns.  If no block is given, open returns
+      # the new instance.  
+      #
+      # Nil may be provided as an fd, in which case a Tempfile will be
+      # used (in which case mode gets ignored as Tempfiles always open
+      # in 'r+' mode).
+      def open(path=nil, mode="rb", *argv)
+        begin
+          io = path == nil ? nil : File.open(path, mode) 
+          base = new(io, *argv)
+        rescue(Errno::ENOENT)
+          io.close if io
+          raise
+        end
+
+        if block_given?
+          begin
+            yield(base)
+          ensure
+            base.close
+          end
+        else
+          base
+        end
+      end
+    end
+    
+    include External::Enumerable  
+    include External::Chunkable
+    
+    # The underlying io for self.
+    attr_reader :io
+    
+    # The default tempfile basename for Base instances
+    # initialized without an io.
+    TEMPFILE_BASENAME = "external_base"
+    
+    # Creates a new instance of self with the specified io. A
+    # nil io causes initialization with a Tempfile; a string
+    # io will be converted into a StringIO.
+    def initialize(io=nil)
+      self.io = case io
+      when nil then Tempfile.new(TEMPFILE_BASENAME)
+      when String then StringIO.new(io)
+      else io
+      end
+      
+      @enumerate_to_a = true
+    end
+ 
+    # True if io is closed.
+    def closed?
+      io.closed?
+    end
+  
+    # Closes io.  If a path is specified, io will be dumped to it.  If
+    # io is a File or Tempfile, the existing file is moved (not dumped)
+    # to path.  Raises an error if path already exists and overwrite is 
+    # not specified.
+    def close(path=nil, overwrite=false)
+      result = !io.closed?
+      
+      if path 
+        if File.exists?(path) && !overwrite
+          raise ArgumentError, "already exists: #{path}"
+        end
+      
+        case io
+        when File, Tempfile
+          io.close unless io.closed?
+          FileUtils.move(io.path, path)
+        else
+          io.flush
+          io.rewind
+          File.open(path, "w") do |file|
+             file << io.read(io.default_blksize) while !io.eof?
+          end
+        end
+      end
+      
+      io.close unless io.closed?
+      result
+    end
+    
+    # Flushes the io and resets the io length.  Returns self
+    def flush
+      io.flush
+      io.reset_length
+      self
+    end
+    
+    # Returns a duplicate of self.  This can be a slow operation
+    # as it may involve copying the full contents of one large
+    # file to another.
+    def dup
+      flush
+      another.concat(self)
+    end
+    
+    # Returns another instance of self.  Must be
+    # implemented in a subclass.
+    def another
+      raise NotImplementedError
+    end
+    
+    ###########################
+    # Array methods
+    ###########################
+    
+    # Returns true if _self_ contains no elements
+    def empty?
+      length == 0
+    end
+    
+    def eql?(another)
+      self == another
+    end
+    
+    # Returns the first n entries (default 1)
+    def first(n=nil)
+      n.nil? ? self[0] : self[0,n]
+    end
+    
+    # Alias for []
+    def slice(one, two = nil)
+      self[one, two]
+    end
+    
+    # Returns self.
+    #--
+    # Warning -- errors show up when this doesn't return
+    # an Array... however to return an array with to_ary
+    # may mean converting a Base into an Array for
+    # insertions... see/modify convert_to_ary
+    def to_ary
+      self
+    end
+    
+    #
+    def inspect
+      "#<#{self.class}:#{object_id} #{ellipse_inspect(self)}>"
+    end
+    
+    protected
+    
+    # Sets io and extends the input io with Io.
+    def io=(io) # :nodoc:
+      io.extend Io unless io.kind_of?(Io)
+      @io = io
+    end
+    
+    # converts obj to an int using the <tt>to_int</tt>
+    # method, if the object responds to <tt>to_int</tt>
+    def convert_to_int(obj)  # :nodoc:
+      obj.respond_to?(:to_int) ? obj.to_int : obj
+    end
+
+    # converts obj to an array using the <tt>to_ary</tt>
+    # method, if the object responds to <tt>to_ary</tt>
+    def convert_to_ary(obj)  # :nodoc:
+      obj == nil ? [] : obj.respond_to?(:to_ary) ? obj.to_ary : [obj]
+    end
+    
+    # a more array-compliant version of Chunkable#split_range
+    def split_range(range, total=length) # :nodoc:
+      # split the range
+      start = convert_to_int(range.begin)
+      raise TypeError, "can't convert #{range.begin.class} into Integer" unless start.kind_of?(Integer)
+      start += total if start < 0
+      
+      finish = convert_to_int(range.end)
+      raise TypeError, "can't convert #{range.end.class} into Integer" unless finish.kind_of?(Integer)
+      finish += total if finish < 0
+      
+      length = finish - start
+      length -= 1 if range.exclude_end?
+      
+      [start, length]
+    end
+    
+    # helper to inspect large arrays
+    def ellipse_inspect(array) # :nodoc:
+      if array.length > 10
+        "[#{collect_join(array[0,5])} ... #{collect_join(array[-5,5])}] (length = #{array.length})"
+      else
+        "[#{collect_join(array.to_a)}]"
+      end
+    end
+    
+    # another helper to inspect large arrays
+    def collect_join(array) # :nodoc:
+      array.collect do |obj|
+        obj.inspect
+      end.join(', ')
+    end
+    
+  end
+end