external 0.1.0 → 0.3.0

data/History

@@ -1,3 +1,10 @@
+== 0.3.0 / 2008-10-27
+
+Major update with refactoring (ex ExtArr is now ExternalArray) 
+and greatly expanded testing. [] and []= methods all Externals 
+now comply with the Array specification in RubySpec[rubyspec.org].
+Implementation of other methods is under way.
+
 == 0.1.0 / 2007-12-10 revision 23
 
 Initial release with working [] and []= methods

data/MIT-LICENSE

@@ -1,6 +1,4 @@
-Copyright (c) 2006-2007, Regents of the University of Colorado.
-Developer:: Simon Chiang, Biomolecular Structure Program, Hansen Lab
-Support:: CU Denver School of Medicine Deans Academic Enrichment Fund
+Copyright (c) 2006-2008, Regents of the University of Colorado.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this
 software and associated documentation files (the "Software"), to deal in the Software

data/README

@@ -4,165 +4,200 @@ Indexing and array-like access to data stored on disk rather than in memory.
 
 == Description
 
-External provides an easy way to index files such that array-like calls can store and
-retrieve entries directly from the file without loading it into memory.  The indexes can 
-be cached for performance or stored on disk alongside the data file, in essence giving you
-arbitrarily large arrays.  
+External provides a way to index and access array data directly from a file
+without loading it into memory.  Indexes may be cached in memory or stored 
+on disk with the data file, in essence giving you arbitrarily large arrays.
+Externals automatically chunk and buffer methods like <tt>each</tt> so that
+the memory footprint remains low even during enumeration.  
 
-The main classes of external provide array-like access to the following:
-* ExtInd (External Index) -- formatted binary data
-* ExtArr (External Array) -- externally stored ruby objects
-* ExtArc (External Archive) -- externally stored string data
+The main External classes are:
 
-ExtArc is a subclass of ExtArr specialized for string archival files, formats like FASTA
-where entries are strings delimited by '>':
+* ExternalIndex   --  for formatted binary data
+* ExternalArchive --  for string data
+* ExternalArray   --  for objects (serialized as YAML)
 
-	>Q9BXQ0|Q9BXQ0_HUMAN Tissue transglutaminase (Fragment) - Homo sapiens (Human).
-	LEPFSGKALCSWSIC
-	>P02452|CO1A1_HUMAN Collagen alpha-1(I) chain - Homo sapiens (Human).
-	MFSFVDLRLLLLLAATALLTHGQEEGQVEGQDEDIPPITCVQNGLRYHDRDVWKPEPCRI
-	CVCDNGKVLCDDVICDETKNCPGAEVPEGECCPVCPDGSESPTDQETTGVEGPKGDTGPR
-	GPRGPAGPPGRDGIPGQPGLPGPPGPPGPPGPPGLGGNFAPQLSYGYDEKSTGGISVPGP
-	...
+The array-like behavior of these classes is developed using modified versions
+of the RubySpec[http://rubyspec.org] specification for Array.  The idea is to 
+eventually duck-type all Array methods, including sort and collect, with 
+acceptable performance.
 
-The array-like behavior of these classes is developed against modified versions of the 
-Array tests themselves, and often uses the exact same tests. The idea is to eventually
-duck-type all Array methods, including sort and collect, with acceptable performance.
+* Rubyforge[http://rubyforge.org/projects/external]
+* Lighthouse[http://bahuvrihi.lighthouseapp.com/projects/10590-external]
+* Github[http://github.com/bahuvrihi/external/tree/master]
 
-=== Bugs/Known Issues
+==== Bugs/Known Issues
 
 * only a limited set of array methods are currently supported
-* reindexing of ExtArr does not work for arrays containing yaml strings
-* yaml serialization/deserialization of some strings do not reproduce identical input
-  and so will not be faithfully store in ExtArr.  Carriage return string are notable:  
-  "\r", "\r\n", "string_with_\r\n_internal", as are chains of newlines: "\n", "\n\n"
-* documentation is poor at the moment
+* currently only [] and []= are fully tested vs RubySpec
+* documentation is patchy
 
---
-== Performance
-++
+Note also that YAML dump/load of some objects doesn't work or doesn't
+reproduce the object; such objects will not be properly stored in an
+ExternalArray.  Problematic objects include:
 
-== Info 
+Proc and Class:
 
-Copyright (c) 2006-2007, Regents of the University of Colorado.
-Developer:: {Simon Chiang}[http://bahuvrihi.wordpress.com], {Biomolecular Structure Program}[http://biomol.uchsc.edu/], {Hansen Lab}[http://hsc-proteomics.uchsc.edu/hansenlab/] 
-Support:: CU Denver School of Medicine Deans Academic Enrichment Fund
-Licence:: MIT-Style
+  block = lambda {}
+  YAML.load(YAML.dump(block))         # !> TypeError: allocator undefined for Proc
+  YAML.dump(Object)                   # !> TypeError: can't dump anonymous class Class
 
-== Installation
+Carriage returns ("\r"):
 
-External is available from RubyForge[http://rubyforge.org/projects/external].  Use:
+  YAML.load(YAML.dump("\r"))          # => nil
+  YAML.load(YAML.dump("\r\n"))        # => ""
+  YAML.load(YAML.dump("string with \r\n inside"))  # => "string with \n inside"
 
-  % gem install external
+Chains of newlines ("\n"):
 
-== Usage
+  YAML.load(YAML.dump("\n"))          # => ""
+  YAML.load(YAML.dump("\n\n"))        # => ""
+  
+DateTime is loaded as Time:
 
-=== ExtArr
+  YAML.load(YAML.dump(DateTime.now)).class         # => Time
+  
+== Usage
 
-ExtArr can be initialized from data using the [] operator and used as an array.
+=== ExternalArray
 
-  ea = ExtArr[1, 2.2, "cat", {:key => 'value'}]
-  ea[2]                # => "cat"
-  ea.last              # => {:key => 'value'}
-  ea << [:a, :b]
-  ea.to_a              # => [1, 2.2, "cat", {:key => 'value'}, [:a, :b]]
+ExternalArray can be initialized from data using the [] operator and used like
+an array.
 
-Behind the scenes, ExtArr serializes and stores entries on a data source (io) and builds an 
-ExtInd that tracks where each entry begins and ends.
+  a = ExternalArray['str', {'key' => 'value'}]
+  a[0]                                # => 'str'
+  a.last                              # => {'key' => 'value'}
+  a << [1,2]; a.to_a                  # => ['str', {'key' => 'value'}, [1,2]]
 
-  ea.io.class          # => Tempfile
-  ea.io.rewind
-  ea.io.read           # => "--- 1\n--- 2.2\n--- cat\n--- \n:key: value\n--- \n- :a\n- :b\n"
+ExternalArray serializes and stores entries to an io while building an io_index
+that tracks the start and length of each entry.  By default ExternalArray
+will serialize to a Tempfile and use an Array as the io_index:
 
-  ea.index.class       # => ExtInd
-  ea.index.to_a        # => [[0, 6], [6, 8], [14, 8], [22, 17], [39, 15]]
+  a.io.class                          # => Tempfile
+  a.io.rewind; a.io.read              # => "--- str\n--- \nkey: value\n--- \n- 1\n- 2\n"
+  a.io_index.class                    # => Array
+  a.io_index.to_a                     # => [[0, 8], [8, 16], [24, 13]]
 
-By default External supports File, Tempfile, and StringIO data sources.  If no data source is 
-given (as above), the external array is initialized to a Tempfile so that it will be cleaned 
-up on exit.
+To save this data more permanently, provide a path to <tt>close</tt>; the tempfile
+is moved to the path and a binary index file will be created:
 
-ExtArr can be initialized from existing data sources.  In this case, ExtArr tries to find and 
-load an existing index; if the index doesn't exist, then you have to reindex the data manually.
+  a.close('example.yml')
+  File.read('example.yml')            # => "--- str\n--- \nkey: value\n--- \n- 1\n- 2\n"
+  
+  index = File.read('example.index')
+  index.unpack('I*')                  # => [0, 8, 8, 16, 24, 13]
 
-  File.open('path/to/file.txt', "w+") do |file|
-    file << "--- 1\n--- 2.2\n--- cat\n--- \n:key: value\n--- \n- :a\n- :b\n"
-    file.flush
+ExternalArray provides <tt>open</tt> to create ExternalArrays from an existing 
+file; the instance will use an index file if it exists and automatically
+reindex the data if it does not.  Manual calls to reindex may be necessary when
+you initialize an ExternalArray with <tt>new</tt> instead of <tt>open</tt>:
 
-    index_filepath = ExtArr.default_index_filepath(file.path)
-    File.exists?(index_filepath)         # => false
-
-    ea = ExtArr.new(file)
-    ea.to_a              # => []
-    ea.reindex 
-    ea.to_a              # => [1, 2.2, "cat", {:key => 'value'}, [:a, :b]]
+  # use of an existing index file
+  ExternalArray.open('example.yml') do |b|
+    File.basename(b.io_index.io.path) # => 'example.index'
+    b.to_a                            # => ['str', {'key' => 'value'}, [1,2]]
   end
 
-ExtArr provides an open method for easy access to file data:
-
-  ExtArr.open('path/to/file.txt') do |ea|
-    # ...
+  # automatic reindexing
+  FileUtils.rm('example.index')
+  ExternalArray.open('example.yml') do |b|
+    b.to_a                            # => ['str', {'key' => 'value'}, [1,2]]
   end
-
-=== ExtArc
-
-ExtArc is a subclass of ExtArr designed for string archival files.  Rather than serialize and
-load ruby objects to and from the data file, ExtArc simply read and writes strings.  In 
-addition, ExtArc provides additional reindexing methods designed to make reindexing easy.
-
-  arc = ExtArc[">swift", ">brown", ">fox"]
-  arc[2]                # => ">fox"
-  arc.to_a              # => [">swift", ">brown", ">fox"]
-
-  arc.io.class          # => Tempfile
-  arc.io.rewind
-  arc.io.read           # => ">swift>brown>fox"
-
-  File.open('path/to/file.txt', "w+") do |file|
-    file << ">swift>brown>fox"
-    file.flush
-
-    # Reindex by a separation string
-    arc = ExtArc.new(file)
-    arc.to_a                                 # => []
-    arc.reindex_by_sep(:sep_string => ">", :entry_follows_sep => true)
-    arc.to_a                                 # => [">swift", ">brown", ">fox"]
-
-    # Reindex by scanning an entry
-    arc = ExtArc.new(file)
-    arc.to_a                                 # => []
-    arc.reindex_by_scan(/>\w*/)
-    arc.to_a                                 # => [">swift", ">brown", ">fox"]
+  
+  # manual reindexing
+  file = File.open('example.yml')
+  c = ExternalArray.new(file)
+  
+  c.to_a                              # => []
+  c.reindex
+  c.to_a                              # => ['str', {'key' => 'value'}, [1,2]]
+
+=== ExternalArchive
+
+ExternalArchive is exactly like ExternalArray except that it only stores
+strings (ExternalArray is actually a subclass of ExternalArchive which
+dumps/loads strings).
+
+  arc = ExternalArchive["swift", "brown", "fox"]
+  arc[2]                              # => "fox"
+  arc.to_a                            # => ["swift", "brown", "fox"]
+  arc.io.rewind; arc.io.read          # => "swiftbrownfox"
+
+ExternalArchive is useful as a base for classes to access archival data.
+Here is a simple parser for FASTA[http://en.wikipedia.org/wiki/Fasta_format]
+data:
+
+  # A sample FASTA entry
+  # >gi|5524211|gb|AAD44166.1| cytochrome b [Elephas maximus maximus]
+  # LCLYTHIGRNIYYGSYLYSETWNTGIMLLLITMATAFMGYVLPWGQMSFWGATVITNLFSAIPYIGTNLV
+  # EWIWGGFSVDKATLNRFFAFHFILPFTMVALAGVHLTFLHETGSNNPLGLTSDSDKIPFHPYYTIKDFLG
+  # LLILILLLLLLALLSPDMLGDPDNHMPADPLNTPLHIKPEWYFLFAYAILRSVPNKLGGVLALFLSIVIL
+  # GLMPFLHTSKHRSMMLRPLSQALFWTLTMDLLTLTWIGSQPVEYPYTIIGQMASILYFSIILAFLPIAGX
+  # IENY
+  
+  class FastaEntry
+    attr_reader :header, :body
+    
+    def initialize(str)
+      @body = str.split(/\r?\n/)
+      @header = body.shift
+    end
   end
- 
-=== ExtInd
-
-ExtInd provides array-like access to formatted binary data.  The index of ExtArr is an 
-ExtInd constructed to access data formatted as 'II'; two integers corresponding to the 
-start position and length of entries in the ExtArr data source.  For simple, repetitive 
-formats like 'II', processing is optimized to use a general format and frame.
-
-  ea = ExtArr.new
-  ea.index.class        # => ExtInd
-  index = ea.index
-
-  index.format          # => 'I*'
-  index.frame           # => 2
-  index << [1,2]
-  index << [3,4]
-  index.to_a            # => [[1,2],[3,4]]
-
-ExtInd handles arbitrary packing formats, opening many possibilites:
-
-  File.open('path/to/file', "w+") do |file|
+  
+  class FastaArchive < ExternalArchive
+    def str_to_entry(str); FastaEntry.new(str); end
+    def entry_to_str(entry); ([entry.header] + entry.body).join("\n"); end
+    
+    def reindex
+      reindex_by_sep('>', :entry_follows_sep => true)
+    end
+  end
+  
+  require 'open-uri'
+  fasta = FastaArchive.new open('http://external.rubyforge.org/doc/tiny_fasta.txt')
+  fasta.reindex
+  
+  fasta.length                        # => 5
+  fasta[0].body                       # => ["MEVNILAFIATTLFVLVPTAFLLIIYVKTVSQSD"]
+
+The non-redundant {NCBI protein database}[ftp://ftp.ncbi.nih.gov/blast/db/FASTA/]
+contains greater than 7 million FASTA entries in a 3.56 GB file; ExternalArchive
+is targeted at files that size, where lazy loading of data and a small memory 
+footprint are critical.
+
+=== ExternalIndex
+
+ExternalIndex provides array-like access to formatted binary data.  The index of an 
+uncached ExternalArray is an ExternalIndex configured for binary data like 'II'; two
+integers corresponding to the start position and length an entry.
+
+  index = ExternalIndex[1, 2, 3, 4, 5, 6, {:format => 'II'}]
+  index.format                        # => 'I*'
+  index.frame                         # => 2
+  index[1]                            # => [3,4]
+  index.to_a                          # => [[1,2], [3,4], [5,6]]
+
+ExternalIndex handles arbitrary packing formats, opening many possibilities:
+
+  Tempfile.new('sample.txt') do |file|
     file << [1,2,3].pack("IQS")
     file << [4,5,6].pack("IQS")
     file << [7,8,9].pack("IQS")
     file.flush
 
-    index = ExtInd.new(file, :format => "IQS")
-    index[1]          # => [4,5,6]
-    index.to_a        # => [[1,2,3],[4,5,6],[7,8,9]]
+    index = ExternalIndex.new(file, :format => "IQS")
+    index[1]                          # => [4,5,6]
+    index.to_a                        # => [[1,2,3], [4,5,6], [7,8,9]]
   end
 
-Note: at the moment formats must be specified longhand, ie 'III' cannot be written as 'I3',
-and the native size directives for sSiIlL are not supported.
+== Installation
+
+External is available from RubyForge[http://rubyforge.org/projects/external].  Use:
+
+  % gem install external
+
+== Info 
+
+Copyright (c) 2006-2008, Regents of the University of Colorado.
+Developer:: {Simon Chiang}[http://bahuvrihi.wordpress.com], {Biomolecular Structure Program}[http://biomol.uchsc.edu/], {Hansen Lab}[http://hsc-proteomics.uchsc.edu/hansenlab/] 
+Support:: CU Denver School of Medicine Deans Academic Enrichment Fund
+Licence:: {MIT-Style}[link:files/MIT-LICENSE.html]

data/lib/external.rb

@@ -1,3 +1,2 @@
-require 'ext_ind'
-require 'ext_arr'
-require 'ext_arc'
+$:.unshift File.expand_path(File.dirname(__FILE__))
+require 'external_array'

data/lib/external/base.rb

@@ -1,66 +1,65 @@
-require 'external/io'
-require 'external/chunkable'
-require 'external/enumerable'
+# For some inexplicable reason yaml MUST be required before
+# tempfile in order for ExtArrTest::test_LSHIFT to pass.
+# Otherwise it fails with 'TypeError: allocator undefined for Proc'
+ 
+require 'yaml'
 require 'tempfile'
 
+require 'external/enumerable'
+require 'external/io'
+
 module External
   
-  #--
-  # Base provides the basic array functionality shared by ExtArr and Index,
-  # essentially wrapping the IO functions required to access and utilized external
-  # array data with the standard array functions.  Bases can be opened with 
-  # in any of the IO modes; the capabilities of Base will be reduced accordingly
-  # (ie read-only Bases cannot write values using []=, for instance).  
-  #
-  # It is VERY IMPORTANT to realize that the underlying IO will be opened using the
-  # given mode.  The 'w' mode will overwrite all existing data; 'r+' is a safer mode
-  # for full read-write functionality.  Note that since Base actively scans over
-  # the IO, append modes essentially behaves like write, but does not overwrite existing
-  # data.
-  #
-  # To work properly, Base must be subclassed with methods:
-  # * length
-  # * io_fetch
-  #++
-  #
-  #
+  # Base provides shared IO and Array-like methods used by ExternalArchive, 
+  # ExternalArray, and ExternalIndex.
   class Base
     class << self
-      def open(fd=nil, mode="r", options={})
-        fd = File.open(fd, mode) unless fd == nil
-        ab = self.new(fd, options)
+      
+      # Initializes an instance of self with File.open(path, mode) as an io.
+      # As with File.open, the instance will be passed to the block and
+      # closed when the block returns.  If no block is given, open returns
+      # the new instance.  
+      #
+      # Nil may be provided as an fd, in which case a Tempfile will be
+      # used (in which case mode gets ignored as Tempfiles always open
+      # in 'r+' mode).
+      def open(path=nil, mode="rb", *argv)
+        path = File.open(path, mode) unless path == nil
+        base = new(path, *argv)
         
         if block_given?
           begin
-            yield(ab)
+            yield(base)
           ensure
-            ab.close
+            base.close
           end
         else
-          ab
+          base
         end
       end
     end
     
     include External::Enumerable  
     include External::Chunkable
- 
+    
+    # The underlying io for self.
     attr_reader :io
     
-    # Initializes a new Base given the file descriptor, mode and options.
-    # (see open_io for details on what io is opened for a given file descriptor)
-    #
-    # If mode contains an 's', then the Base will be initialized in strio 
-    # mode where the underlying IO will be a StringIO.  In this case the fd 
-    # will be used as the string to initialize the StringIO.
-    #
-    # Standard options for Base include:
-    # nil_value:: the value written to file for nils, and converted to nil on read
-    #             (default ' ')
-    # max_gap:: the maximum gap size used by Offset (default 10000)
-    # max_chunk_size:: the chunk size used by Offset (default 1M)
+    # The default tempfile basename for Base instances
+    # initialized without an io.
+    TEMPFILE_BASENAME = "external_base"
+    
+    # Creates a new instance of self with the specified io. A
+    # nil io causes initialization with a Tempfile; a string
+    # io will be converted into a StringIO.
     def initialize(io=nil)
-      self.io = (io.nil? ? Tempfile.new("array_base") : io)
+      self.io = case io
+      when nil then Tempfile.new(TEMPFILE_BASENAME)
+      when String then StringIO.new(io)
+      else io
+      end
+      
+      @enumerate_to_a = true
     end
  
     # True if io is closed.
@@ -68,18 +67,146 @@ module External
       io.closed?
     end
   
-    # Closes io.
-    def close
+    # Closes io.  If a path is specified, io will be dumped to it.  If
+    # io is a File or Tempfile, the existing file is moved (not dumped)
+    # to path.  Raises an error if path already exists and overwrite is 
+    # not specified.
+    def close(path=nil, overwrite=false)
+      result = !io.closed?
+      
+      if path 
+        if File.exists?(path) && !overwrite
+          raise ArgumentError, "already exists: #{path}"
+        end
+      
+        case io
+        when File, Tempfile
+          io.close unless io.closed?
+          FileUtils.move(io.path, path)
+        else
+          io.flush
+          io.rewind
+          File.open(path, "w") do |file|
+             file << io.read(io.default_blksize) while !io.eof?
+          end
+        end
+      end
+      
       io.close unless io.closed?
+      result
+    end
+    
+    # Flushes the io and resets the io length.  Returns self
+    def flush
+      io.flush
+      io.reset_length
+      self
+    end
+    
+    # Returns a duplicate of self.  This can be a slow operation
+    # as it may involve copying the full contents of one large
+    # file to another.
+    def dup
+      flush
+      another.concat(self)
+    end
+    
+    # Returns another instance of self.  Must be
+    # implemented in a subclass.
+    def another
+      raise NotImplementedError
+    end
+    
+    ###########################
+    # Array methods
+    ###########################
+    
+    # Returns true if _self_ contains no elements
+    def empty?
+      length == 0
+    end
+    
+    def eql?(another)
+      self == another
+    end
+    
+    # Returns the first n entries (default 1)
+    def first(n=nil)
+      n.nil? ? self[0] : self[0,n]
+    end
+    
+    # Alias for []
+    def slice(one, two = nil)
+      self[one, two]
+    end
+    
+    # Returns self.
+    #--
+    # Warning -- errors show up when this doesn't return
+    # an Array... however to return an array with to_ary
+    # may mean converting a Base into an Array for
+    # insertions... see/modify convert_to_ary
+    def to_ary
+      self
+    end
+    
+    #
+    def inspect
+      "#<#{self.class}:#{object_id} #{ellipse_inspect(self)}>"
     end
     
     protected
     
-    # Sets io and extends the input io with External::Position.
-    def io=(io)
-      io.extend External::IO unless io.kind_of?(External::IO)
+    # Sets io and extends the input io with Io.
+    def io=(io) # :nodoc:
+      io.extend Io unless io.kind_of?(Io)
       @io = io
     end
+    
+    # converts obj to an int using the <tt>to_int</tt>
+    # method, if the object responds to <tt>to_int</tt>
+    def convert_to_int(obj)  # :nodoc:
+      obj.respond_to?(:to_int) ? obj.to_int : obj
+    end
 
+    # converts obj to an array using the <tt>to_ary</tt>
+    # method, if the object responds to <tt>to_ary</tt>
+    def convert_to_ary(obj)  # :nodoc:
+      obj == nil ? [] : obj.respond_to?(:to_ary) ? obj.to_ary : [obj]
+    end
+    
+    # a more array-compliant version of Chunkable#split_range
+    def split_range(range, total=length) # :nodoc:
+      # split the range
+      start = convert_to_int(range.begin)
+      raise TypeError, "can't convert #{range.begin.class} into Integer" unless start.kind_of?(Integer)
+      start += total if start < 0
+      
+      finish = convert_to_int(range.end)
+      raise TypeError, "can't convert #{range.end.class} into Integer" unless finish.kind_of?(Integer)
+      finish += total if finish < 0
+      
+      length = finish - start
+      length -= 1 if range.exclude_end?
+      
+      [start, length]
+    end
+    
+    # helper to inspect large arrays
+    def ellipse_inspect(array) # :nodoc:
+      if array.length > 10
+        "[#{collect_join(array[0,5])} ... #{collect_join(array[-5,5])}] (length = #{array.length})"
+      else
+        "[#{collect_join(array.to_a)}]"
+      end
+    end
+    
+    # another helper to inspect large arrays
+    def collect_join(array) # :nodoc:
+      array.collect do |obj|
+        obj.inspect
+      end.join(', ')
+    end
+    
   end
 end