bindata 0.8.1 → 0.9.0

data/ChangeLog

@@ -1,13 +1,22 @@
 = BinData Changelog
 
--== Version 0.8.1 (2008-01-14)
+== Version 0.9.0 (2008-06-02)
+
+* Added :adjust_offset option to automatically seek to a given offset.
+* Modified #read to accept strings as well as IO streams.
+* Choice now accepts sparse arrays and hashes as :choice.
+* Added BinData::Rest to help with debugging.
+* Major internal restructuring - memory usage is much better.
+* Improved documentation
+
+== Version 0.8.1 (2008-01-14)
 
 * Reduced memory consumption.
 * Increased execution speed.
 * Deprecated BinData::Base.parameters
 * Fixed spec syntax (thanks to David Goodlad)
 
--== Version 0.8.0 (2007-10-14)
+== Version 0.8.0 (2007-10-14)
 
 * Add reserved field names to Struct.
 * Prevent warnings about method redefinition.

data/README

@@ -15,7 +15,7 @@ Do you ever find yourself writing code like this?
 It's ugly, violates DRY and feels like you're writing Perl, not Ruby.
 There is a better way.
 
-  class Rectangle < BinData::Struct
+  class Rectangle < BinData::MultiValue
     uint16le :len
     string   :name, :read_length => :len
     uint32le :width
@@ -36,7 +36,7 @@ download[http://rubyforge.org/frs/?group_id=3252] page.
 
 BinData declarations are easy to read.  Here's an example.
 
-  class MyFancyFormat < BinData::Struct
+  class MyFancyFormat < BinData::MultiValue
     stringz :comment
     uint8   :count, :check_value => lambda { (value % 2) == 0 }
     array   :some_ints, :type => :int32be, :initial_length => :count
@@ -62,7 +62,7 @@ the writing code, have a go at the reading code.
 The general format of a BinData declaration is a class containing one or more
 fields.
 
-  class MyName < BinData::Struct
+  class MyName < BinData::MultiValue
     type field_name, :param1 => "foo", :param2 => bar, ...
     ...
   end
@@ -100,7 +100,7 @@ the string contains the string's length.
 
 Here's how we'd implement the same example with BinData.
 
-  class PascalString < BinData::Struct
+  class PascalString < BinData::MultiValue
     uint8  :len,  :value => lambda { data.length }
     string :data, :read_length => :len
   end
@@ -120,7 +120,7 @@ Here's how we'd implement the same example with BinData.
 This syntax needs explaining.  Let's simplify by examining reading and
 writing separately.
 
-  class PascalStringReader < BinData::Struct
+  class PascalStringReader < BinData::MultiValue
     uint8  :len
     string :data, :read_length => :len
   end
@@ -132,7 +132,7 @@ This states that when reading the string, the initial length of the string
 Note that <tt>:read_length => :len</tt> is syntactic sugar for
 <tt>:read_length => lambda { len }</tt>, but more on that later.
 
-  class PascalStringWriter < BinData::Struct
+  class PascalStringWriter < BinData::MultiValue
     uint8  :len, :value => lambda { data.length }
     string :data
   end
@@ -152,6 +152,13 @@ length afterwards.
 These are the predefined types.  Custom types can be created by composing
 these types.
 
+BinData::String::   A sequence of bytes.
+BinData::Stringz::  A zero terminated sequence of bytes.
+
+BinData::Array::    A list of objects of the same type.
+BinData::Choice::   A choice between several objects.
+BinData::Struct::   An ordered collection of named objects.
+
 BinData::Int8::     Signed  8 bit integer.
 BinData::Int16le::  Signed 16 bit integer (little endian).
 BinData::Int16be::  Signed 16 bit integer (big endian).
@@ -173,16 +180,11 @@ BinData::FloatBe::  Single precision floating point number (big endian).
 BinData::DoubleLe:: Double precision floating point number (little endian).
 BinData::DoubleBe:: Double precision floating point number (big endian).
 
-BinData::String::   A sequence of bytes.
-BinData::Stringz::  A zero terminated sequence of bytes.
-
-BinData::Array::    A list of objects of the same type.
-BinData::Choice::   A choice between several objects.
-BinData::Struct::   An ordered collection of named objects.
+BinData::Rest::     Consumes the rest of the input stream.
 
 == Parameters
 
-  class PascalStringWriter < BinData::Struct
+  class PascalStringWriter < BinData::MultiValue
     uint8  :len, :value => lambda { data.length }
     string :data
   end
@@ -217,7 +219,7 @@ produced is independent of architecture.  Explicitly specifying the
 endianess of each numeric type can become tedious, so the following
 shortcut is provided.
 
-  class A < BinData::Struct
+  class A < BinData::MultiValue
     endian :little
 
     uint16   :a
@@ -229,7 +231,7 @@ shortcut is provided.
 
 is equivalent to:
 
-  class A < BinData::Struct
+  class A < BinData::MultiValue
     uint16le  :a
     uint32le  :b
     double_le :c
@@ -243,13 +245,47 @@ cascade to nested types, as illustrated with the array in the above example.
 
 == Creating custom types
 
-Custom types should be created by subclassing BinData::Struct.
-Ocassionally it may be useful to subclass BinData::Single.  Subclassing
-other classes may have unexpected results and is unsupported.
+Custom types should be created by subclassing BinData::MultiValue or
+BinData::SingleValue.  Ocassionally it may be useful to subclass
+BinData::Single.  Subclassing other classes may have unexpected results
+and is unsupported.
+
+Let us revisit the Pascal String example.
+
+  class PascalString < BinData::MultiValue
+    uint8  :len,  :value => lambda { data.length }
+    string :data, :read_length => :len
+  end
+
+We'd like to make PascalString a custom type that behaves like a
+BinData::Single object so we can use :initial_value etc.  Here's an
+example usage of what we'd like:
+
+  class Favourites < BinData::MultiValue
+    pascal_string :language, :initial_value => "ruby"
+    pascal_string :os,       :initial_value => "unix"
+  end
+
+  f = Favourites.new
+  f.os = "freebsd"
+  f.to_s #=> "\004ruby\007freebsd"
+
+We create this type of custom string by inheriting from BinData::SingleValue
+and implementing the #get and #set methods.
+
+  class PascalString < BinData::SingleValue
+    uint8  :len,  :value => lambda { data.length }
+    string :data, :read_length => :len
+
+    def get;   self.data; end
+    def set(v) self.data = v; end
+  end
 
+If the type we are creating represents a single value then inherit from
+BinData::SingleValue, otherwise inherit from BinData::MultiValue.
 
 == License
 
 BinData is released under the same license as Ruby.
 
-Copyright (c) 2007 Dion Mendel
+Copyright (c) 2007, 2008 Dion Mendel

data/TODO

@@ -3,3 +3,7 @@
 * Need optional parameter for fields
 
 * Should I add seek capability?
+
+* Add (Multi|Single)Value.custom_parameters
+
+* BitFields

data/examples/gzip.rb

@@ -9,12 +9,12 @@ class Gzip
   # Known compression methods
   DEFLATE = 8
 
-  class Extra < BinData::Struct
+  class Extra < BinData::MultiValue
     uint16le :len,  :length => lambda { data.length }
     string   :data, :read_length => :len
   end
 
-  class Header < BinData::Struct
+  class Header < BinData::MultiValue
     uint16le :ident,      :value => 0x8b1f, :check_value => 0x8b1f
     uint8    :compression_method, :initial_value => DEFLATE
     uint8    :flags,      :value => :calculate_flags_val,
@@ -60,7 +60,7 @@ class Gzip
     end
   end
 
-  class Footer < BinData::Struct
+  class Footer < BinData::MultiValue
     uint32le :crc32
     uint32le :uncompressed_size
   end

data/lib/bindata.rb

@@ -5,10 +5,17 @@ require 'bindata/array'
 require 'bindata/choice'
 require 'bindata/float'
 require 'bindata/int'
+require 'bindata/multi_value'
+require 'bindata/rest'
+require 'bindata/single_value'
 require 'bindata/string'
 require 'bindata/stringz'
 require 'bindata/struct'
 
+# = BinData
+# 
+# A declarative way to read and write structured binary data.
+# 
 module BinData
-  VERSION = "0.8.1"
+  VERSION = "0.9.0"
 end

data/lib/bindata/array.rb

@@ -1,15 +1,31 @@
 require 'bindata/base'
+require 'bindata/sanitize'
 
 module BinData
   # An Array is a list of data objects of the same type.
   #
   #   require 'bindata'
-  #   require 'stringio'
   #
-  #   a = BinData::Array.new(:type => :int8, :initial_length => 5)
-  #   io = StringIO.new("\x03\x04\x05\x06\x07")
-  #   a.read(io)
-  #   a.snapshot #=> [3, 4, 5, 6, 7]
+  #   data = "\x03\x04\x05\x06\x07\x08\x09"
+  #
+  #   obj = BinData::Array.new(:type => :int8, :initial_length => 6)
+  #   obj.read(data)
+  #   obj.snapshot #=> [3, 4, 5, 6, 7, 8]
+  #
+  #   obj = BinData::Array.new(:type => :int8,
+  #                            :read_until => lambda { index == 1 })
+  #   obj.read(data)
+  #   obj.snapshot #=> [3, 4]
+  #
+  #   obj = BinData::Array.new(:type => :int8,
+  #                            :read_until => lambda { element >= 6 })
+  #   obj.read(data)
+  #   obj.snapshot #=> [3, 4, 5, 6]
+  #
+  #   obj = BinData::Array.new(:type => :int8,
+  #           :read_until => lambda { array[index] + array[index - 1] == 13 })
+  #   obj.read(data)
+  #   obj.snapshot #=> [3, 4, 5, 6, 7]
   #
   # == Parameters
   #
@@ -30,7 +46,7 @@ module BinData
   #
   # Each data object in an array has the variable +index+ made available
   # to any lambda evaluated as a parameter of that data object.
-  class Array < Base
+  class Array < BinData::Base
     include Enumerable
 
     # Register this class
@@ -39,22 +55,44 @@ module BinData
     # These are the parameters used by this class.
     mandatory_parameter :type
     optional_parameters :initial_length, :read_until
+    mutually_exclusive_parameters :initial_length, :read_until
+
+    class << self
+      # Returns a sanitized +params+ that is of the form expected
+      # by #initialize.
+      def sanitize_parameters(params, endian = nil)
+        params = params.dup
+
+        unless params.has_key?(:initial_length) or params.has_key?(:read_until)
+          # ensure one of :initial_length and :read_until exists
+          params[:initial_length] = 0
+        end
+
+        if params.has_key?(:type)
+          type, el_params = params[:type]
+          klass = lookup(type, endian)
+          raise TypeError, "unknown type '#{type}' for #{self}" if klass.nil?
+          params[:type] = [klass, SanitizedParameters.new(klass, el_params, endian)]
+        end
 
-    # An empty hash shared by all instances
-    @@empty_hash = Hash.new.freeze
+        super(params, endian)
+      end
+
+      # An array has no fields.
+      def all_possible_field_names(sanitized_params)
+        []
+      end
+    end
 
     # Creates a new Array
     def initialize(params = {}, env = nil)
-      super(cleaned_params(params), env)
-      ensure_mutual_exclusion(:initial_length, :read_until)
+      super(params, env)
 
-      type, el_params = param(:type)
-      klass = klass_lookup(type)
-      raise TypeError, "unknown type '#{type}' for #{self}" if klass.nil?
+      klass, el_params = param(:type)
 
       @element_list    = nil
       @element_klass   = klass
-      @element_params  = el_params || @@empty_hash
+      @element_params  = el_params
     end
 
     # Clears the element at position +index+.  If +index+ is not given, then
@@ -126,6 +164,12 @@ module BinData
       elements.collect { |e| e.snapshot }
     end
 
+    # Returns whether this data object contains a single value.  Single
+    # value data objects respond to <tt>#value</tt> and <tt>#value=</tt>.
+    def single_value?
+      return false
+    end
+
     # An array has no fields.
     def field_names
       []
@@ -239,17 +283,5 @@ module BinData
       @element_list << element
       element
     end
-
-    # Returns a hash of cleaned +params+.  Cleaning means that param
-    # values are converted to a desired format.
-    def cleaned_params(params)
-      unless params.has_key?(:initial_length) or params.has_key?(:read_until)
-        # ensure one of :initial_length and :read_until exists
-        new_params = params.dup
-        new_params[:initial_length] = 0
-        params = new_params
-      end
-      params
-    end
   end
 end

data/lib/bindata/base.rb

@@ -1,5 +1,7 @@
 require 'bindata/lazy'
+require 'bindata/sanitize'
 require 'bindata/registry'
+require 'stringio'
 
 module BinData
   # Error raised when unexpected results occur when reading data from IO.
@@ -21,6 +23,10 @@ module BinData
   #                           is made available to any lambda assigned to
   #                           this parameter.  This parameter is only checked
   #                           before reading.
+  # [<tt>:adjust_offset</tt>] Ensures that the current IO offset is at this
+  #                           position before reading.  This is like
+  #                           <tt>:check_offset</tt>, except that it will
+  #                           adjust the IO offset instead of raising an error.
   class Base
     class << self
       # Returns the mandatory parameters used by this class.  Any given args
@@ -63,11 +69,75 @@ module BinData
       end
       alias_method :optional_parameter, :optional_parameters
 
-      # Returns both the mandatory and optional parameters used by this class.
-      def parameters
-        # warn about deprecated method - remove before releasing 1.0
-        warn "warning: #parameters is deprecated."
-        (mandatory_parameters + optional_parameters).uniq
+      # Returns the default parameters used by this class.  Any given args
+      # are appended to the parameters list.  The parameters for a class will
+      # include the parameters of its ancestors.
+      def default_parameters(params = {})
+        unless defined? @default_parameters
+          @default_parameters = {}
+          ancestors[1..-1].each do |parent|
+            if parent.respond_to?(:default_parameters)
+              @default_parameters = @default_parameters.merge(parent.default_parameters)
+            end
+          end
+        end
+        if not params.empty?
+          @default_parameters = @default_parameters.merge(params)
+        end
+        @default_parameters
+      end
+      alias_method :default_parameter, :default_parameters
+
+      # Returns the pairs of mutually exclusive parameters used by this class.
+      # Any given args are appended to the parameters list.  The parameters for
+      # a class will include the parameters of its ancestors.
+      def mutually_exclusive_parameters(*args)
+        unless defined? @mutually_exclusive_parameters
+          @mutually_exclusive_parameters = []
+          ancestors[1..-1].each do |parent|
+            if parent.respond_to?(:mutually_exclusive_parameters)
+              @mutually_exclusive_parameters.concat(parent.mutually_exclusive_parameters)
+            end
+          end
+        end
+        if not args.empty?
+          @mutually_exclusive_parameters << [args[0].to_sym, args[1].to_sym]
+        end
+        @mutually_exclusive_parameters
+      end
+
+      # Returns a list of parameters that are accepted by this object
+      def accepted_parameters
+        (mandatory_parameters + optional_parameters + default_parameters.keys).uniq
+      end
+
+      # Returns a sanitized +params+ that is of the form expected
+      # by #initialize.
+      def sanitize_parameters(params, *args)
+        params = params.dup
+
+        # add default parameters
+        default_parameters.each do |k,v|
+          params[k] = v unless params.has_key?(k)
+        end
+
+        # ensure mandatory parameters exist
+        mandatory_parameters.each do |prm|
+          if not params.has_key?(prm)
+            raise ArgumentError, "parameter ':#{prm}' must be specified " +
+                                 "in #{self}"
+          end
+        end
+
+        # ensure mutual exclusion
+        mutually_exclusive_parameters.each do |param1, param2|
+          if params.has_key?(param1) and params.has_key?(param2)
+            raise ArgumentError, "params #{param1} and #{param2} " +
+                                 "are mutually exclusive"
+          end
+        end
+
+        params
       end
 
       # Instantiates this class and reads from +io+.  For single value objects
@@ -86,19 +156,17 @@ module BinData
       private :register
 
       # Returns the class matching a previously registered +name+.
-      def lookup(name)
+      def lookup(name, endian = nil)
+        name = name.to_s
         klass = Registry.instance.lookup(name)
-        if klass.nil?
+        if klass.nil? and endian != nil
           # lookup failed so attempt endian lookup
-          if self.respond_to?(:endian) and self.endian != nil
-            name = name.to_s
-            if /^u?int\d\d?$/ =~ name
-              new_name = name + ((self.endian == :little) ? "le" : "be")
-              klass = Registry.instance.lookup(new_name)
-            elsif ["float", "double"].include?(name)
-              new_name = name + ((self.endian == :little) ? "_le" : "_be")
-              klass = Registry.instance.lookup(new_name)
-            end
+          if /^u?int\d{1,3}$/ =~ name
+            new_name = name + ((endian == :little) ? "le" : "be")
+            klass = Registry.instance.lookup(new_name)
+          elsif ["float", "double"].include?(name)
+            new_name = name + ((endian == :little) ? "_le" : "_be")
+            klass = Registry.instance.lookup(new_name)
           end
         end
         klass
@@ -106,7 +174,9 @@ module BinData
     end
 
     # Define the parameters we use in this class.
-    optional_parameters :check_offset, :readwrite
+    optional_parameters :check_offset, :adjust_offset
+    default_parameters :readwrite => true
+    mutually_exclusive_parameters :check_offset, :adjust_offset
 
     # Creates a new data object.
     #
@@ -114,68 +184,26 @@ module BinData
     # reference callable objects (methods or procs).  +env+ is the
     # environment that these callable objects are evaluated in.
     def initialize(params = {}, env = nil)
-      # all known parameters
-      mandatory = self.class.mandatory_parameters
-      optional = self.class.optional_parameters
-
-      # default :readwrite param to true if unspecified
-      if not params.has_key?(:readwrite)
-        params = params.dup
-        params[:readwrite] = true
+      unless SanitizedParameters === params
+        params = SanitizedParameters.new(self.class, params)
       end
 
-      # ensure mandatory parameters exist
-      mandatory.each do |prm|
-        if not params.has_key?(prm)
-          raise ArgumentError, "parameter ':#{prm}' must be specified " +
-                               "in #{self}"
-        end
-      end
-
-      # partition parameters into known and extra parameters
-      @params = {}
-      extra   = {}
-      params.each do |k,v|
-        k = k.to_sym
-        raise ArgumentError, "parameter :#{k} is nil in #{self}" if v.nil?
-        if mandatory.include?(k) or optional.include?(k)
-          @params[k] = v.freeze
-        else
-          extra[k] = v.freeze
-        end
-      end
+      @params = params.accepted_parameters
 
       # set up the environment
       @env             = env || LazyEvalEnv.new
-      @env.params      = extra
+      @env.params      = params.extra_parameters
       @env.data_object = self
     end
 
-    # Returns the class matching a previously registered +name+.
-    def klass_lookup(name)
-      @cache ||= {}
-      klass = @cache[name]
-      if klass.nil?
-        klass = self.class.lookup(name)
-        if klass.nil? and @env.parent_data_object != nil
-          # lookup failed so retry in the context of the parent data object
-          klass = @env.parent_data_object.klass_lookup(name)
-        end
-        @cache[name] = klass
-      end
-      klass
-    end
-
-    # Returns a list of parameters that are accepted by this object
-    def accepted_parameters
-      (self.class.mandatory_parameters + self.class.optional_parameters).uniq
-    end
-
-    # Reads data into this bin object by calling #do_read then #done_read.
+    # Reads data into this data object by calling #do_read then #done_read.
     def read(io)
+      # wrap strings in a StringIO
+      io = StringIO.new(io) if io.respond_to?(:to_str)
+
       # remove previous method to prevent warnings
       class << io
-        undef_method(:bindata_mark) if method_defined?(:bindata_mark)
+        remove_method(:bindata_mark) if method_defined?(:bindata_mark)
       end
 
       # remember the current position in the IO object
@@ -198,17 +226,19 @@ module BinData
       _write(io) if eval_param(:readwrite) != false
     end
 
+    # Returns the string representation of this data object.
+    def to_s
+      io = StringIO.new
+      write(io)
+      io.rewind
+      io.read
+    end
+
     # Returns the number of bytes it will take to write this data.
     def num_bytes(what = nil)
       (eval_param(:readwrite) != false) ? _num_bytes(what) : 0
     end
 
-    # Returns whether this data object contains a single value.  Single
-    # value data objects respond to <tt>#value</tt> and <tt>#value=</tt>.
-    def single_value?
-      respond_to? :value
-    end
-
     # Return a human readable representation of this object.
     def inspect
       snapshot.inspect
@@ -243,14 +273,6 @@ module BinData
       @params.has_key?(key.to_sym)
     end
 
-    # Raise an error if +param1+ and +param2+ are both given as params.
-    def ensure_mutual_exclusion(param1, param2)
-      if has_param?(param1) and has_param?(param2)
-        raise ArgumentError, "params #{param1} and #{param2} " +
-                             "are mutually exclusive"
-      end
-    end
-
     # Checks that the current offset of +io+ is as expected.  This should
     # be called from #do_read before performing the reading.
     def check_offset(io)
@@ -264,11 +286,32 @@ module BinData
           raise ValidityError, "offset is '#{actual_offset}' but " +
                                "expected '#{expected}'"
         end
+      elsif has_param?(:adjust_offset)
+        actual_offset = io.pos - io.bindata_mark
+        expected = eval_param(:adjust_offset)
+        if actual_offset != expected
+          begin
+            seek = expected - actual_offset
+            io.seek(seek, IO::SEEK_CUR)
+            warn "adjusting stream position by #{seek} bytes" if $VERBOSE
+          rescue
+            # could not seek so raise an error
+            raise ValidityError, "offset is '#{actual_offset}' but " +
+                                 "couldn't seek to expected '#{expected}'"
+          end
+        end
       end
     end
 
+    ###########################################################################
     # To be implemented by subclasses
 
+    # Returns a list of the names of all possible field names for an object
+    # created with +sanitized_params+.
+    def self.all_possible_field_names(sanitized_params)
+      raise NotImplementedError
+    end
+
     # Resets the internal state to that of a newly created object.
     def clear
       raise NotImplementedError
@@ -299,6 +342,12 @@ module BinData
       raise NotImplementedError
     end
 
+    # Returns whether this data object contains a single value.  Single
+    # value data objects respond to <tt>#value</tt> and <tt>#value=</tt>.
+    def single_value?
+      raise NotImplementedError
+    end
+
     # Returns a list of the names of all fields accessible through this
     # object.
     def field_names
@@ -306,9 +355,10 @@ module BinData
     end
 
     # Set visibility requirements of methods to implement
-    public :clear, :done_read, :snapshot, :field_names
+    public :clear, :done_read, :snapshot, :single_value?, :field_names
     private :_do_read, :_write, :_num_bytes
 
     # End To be implemented by subclasses
+    ###########################################################################
   end
 end