bindata 0.9.3 → 0.10.0

data/ChangeLog

@@ -1,5 +1,23 @@
 = BinData Changelog
 
+== Version
+
+* Arbitrary byte sized integers are now supported (e.g. 24bit, 808bit).
+* Renamed String :trim_value parameter to :trim_padding.
+* BinData::Array now behaves more like Ruby's Array.
+* Added debug_name
+* Added ability to trace reading
+* Primitives now behave as their value.  Calling #value is no longer needed.
+* Renamed #to_s -> #to_binary_s to avoid confusion with ruby's #to_s.
+* Added #assign as the generic way to assign values to objects.
+* Added :copy_on_change parameter to Choice.
+* Implement #offset for all objects.
+* Renamed Single -> BasePrimitive.
+* Renamed SingleValue -> Primitive.
+* Renamed MultiValue -> Record.
+* The :onlyif parameter now only applies to fields inside Structs.
+* LazyEvaluator can now supply arguments when invoking methods
+
 == Version 0.9.3 (2008-12-03)
 
 * Arrays can now :read_until => :eof

data/NEWS

@@ -0,0 +1,59 @@
+= 0.10.0
+
+There are several new features in this release.  The major ones are:
+
+* Debugging declarations is now easier with the ability to trace values when
+  reading.
+
+      BinData::trace_reading(STDERR) do
+        obj_not_quite_working_correctly.read(io)
+      end
+
+  Will result in a debugging trace written to STDERR.
+
+* Support for arbitrary sized integers and bit fields.
+
+* Struct/Array field/element access now returns a BinData object, rather than
+  the underlying Fixnum or String.  The BinData object will behave like it's
+  underlying primitive so existing code should continue to work.  This allows
+  for an improved syntax in your declarations.
+
+  If your code requires access to the underlying primitive, you can access it
+  with the #value method.
+
+There are several deprecations and one backwards incompatible change in this
+release.  Turn on $VERBOSE mode (-w command line switch) to see warnings
+regarding these deprecations.
+
+== IMPORTANT - Ruby does not warn you about this change!
+
+* The #to_s method for getting the binary string representation of a data
+  object has been renamed to #to_binary_s.  If you use this method you must
+  manually change your code or you will get strange results.
+
+== Deprecations.  Ruby will warn you of these when in $VERBOSE mode.
+
+Code using these deprecated features will still work in this release but will
+fail to work in some future release.  Change your code now.
+
+* BinData::SingleValue has been renamed to BinData::Primitive.
+
+* BinData::MultiValue has been renamed to BinData::Record.
+
+* String :trim_value parameter has been renamed to :trim_padding.
+
+* Registry.instance should be replaced with RegisteredClasses.
+
+* struct.offset_of("field") should be replaced with struct.field.offset.
+
+* struct.clear("field") should be replaced with struct.field.clear.
+
+* struct.clear?("field") should be replaced with struct.field.clear?.
+
+* struct.num_bytes("field") should be replaced with struct.field.num_bytes.
+
+* array.clear(n) should be replaced with array[n].clear.
+
+* array.clear?(n) should be replaced with array[n].clear?.
+
+* array.num_bytes(n) should be replaced with array[n].num_bytes.

data/README

@@ -15,7 +15,7 @@ Do you ever find yourself writing code like this?
 It's ugly, violates DRY and feels like you're writing Perl, not Ruby.
 There is a better way.
 
-  class Rectangle < BinData::MultiValue
+  class Rectangle < BinData::Record
     uint16le :len
     string   :name, :read_length => :len
     uint32le :width
@@ -36,7 +36,7 @@ download[http://rubyforge.org/frs/?group_id=3252] page.
 
 BinData declarations are easy to read.  Here's an example.
 
-  class MyFancyFormat < BinData::MultiValue
+  class MyFancyFormat < BinData::Record
     stringz :comment
     uint8   :count, :check_value => lambda { (value % 2) == 0 }
     array   :some_ints, :type => :int32be, :initial_length => :count
@@ -62,7 +62,7 @@ the writing code, have a go at the reading code.
 The general format of a BinData declaration is a class containing one or more
 fields.
 
-  class MyName < BinData::MultiValue
+  class MyName < BinData::Record
     type field_name, :param1 => "foo", :param2 => bar, ...
     ...
   end
@@ -72,8 +72,7 @@ or a user defined type.  For user defined types, convert the class name
 from CamelCase to lowercase underscore_style.
 
 *field_name* is the name by which you can access the data.  Use either a
-String or a Symbol.  You may specify a name as nil, but this is described
-later in the tutorial.
+String or a Symbol.
 
 Each field may have *parameters* for how to process the data.  The
 parameters are passed as a Hash using Symbols for keys.
@@ -100,7 +99,7 @@ the string contains the string's length.
 
 Here's how we'd implement the same example with BinData.
 
-  class PascalString < BinData::MultiValue
+  class PascalString < BinData::Record
     uint8  :len,  :value => lambda { data.length }
     string :data, :read_length => :len
   end
@@ -120,7 +119,7 @@ Here's how we'd implement the same example with BinData.
 This syntax needs explaining.  Let's simplify by examining reading and
 writing separately.
 
-  class PascalStringReader < BinData::MultiValue
+  class PascalStringReader < BinData::Record
     uint8  :len
     string :data, :read_length => :len
   end
@@ -132,7 +131,7 @@ This states that when reading the string, the initial length of the string
 Note that <tt>:read_length => :len</tt> is syntactic sugar for
 <tt>:read_length => lambda { len }</tt>, but more on that later.
 
-  class PascalStringWriter < BinData::MultiValue
+  class PascalStringWriter < BinData::Record
     uint8  :len, :value => lambda { data.length }
     string :data
   end
@@ -194,7 +193,7 @@ BinData::Rest::     Consumes the rest of the input stream.
 
 == Parameters
 
-  class PascalStringWriter < BinData::MultiValue
+  class PascalStringWriter < BinData::Record
     uint8  :len, :value => lambda { data.length }
     string :data
   end
@@ -229,7 +228,7 @@ produced is independent of architecture.  Explicitly specifying the
 endianess of each numeric type can become tedious, so the following
 shortcut is provided.
 
-  class A < BinData::MultiValue
+  class A < BinData::Record
     endian :little
 
     uint16   :a
@@ -241,7 +240,7 @@ shortcut is provided.
 
 is equivalent to:
 
-  class A < BinData::MultiValue
+  class A < BinData::Record
     uint16le  :a
     uint32le  :b
     double_le :c
@@ -255,35 +254,35 @@ cascade to nested types, as illustrated with the array in the above example.
 
 == Creating custom types
 
-Custom types should be created by subclassing BinData::MultiValue or
-BinData::SingleValue.  Ocassionally it may be useful to subclass
-BinData::Single.  Subclassing other classes may have unexpected results
+Custom types should be created by subclassing BinData::Record or
+BinData::Primitive.  Ocassionally it may be useful to subclass
+BinData::BasePrimitive.  Subclassing other classes may have unexpected results
 and is unsupported.
 
 Let us revisit the Pascal String example.
 
-  class PascalString < BinData::MultiValue
+  class PascalString < BinData::Record
     uint8  :len,  :value => lambda { data.length }
     string :data, :read_length => :len
   end
 
 We'd like to make PascalString a custom type that behaves like a
-BinData::Single object so we can use :initial_value etc.  Here's an
+BinData::BasePrimitive object so we can use :initial_value etc.  Here's an
 example usage of what we'd like:
 
-  class Favourites < BinData::MultiValue
+  class Favourites < BinData::Record
     pascal_string :language, :initial_value => "ruby"
     pascal_string :os,       :initial_value => "unix"
   end
 
   f = Favourites.new
   f.os = "freebsd"
-  f.to_s #=> "\004ruby\007freebsd"
+  f.to_binary_s #=> "\004ruby\007freebsd"
 
-We create this type of custom string by inheriting from BinData::SingleValue
+We create this type of custom string by inheriting from BinData::Primitive
 and implementing the #get and #set methods.
 
-  class PascalString < BinData::SingleValue
+  class PascalString < BinData::Primitive
     uint8  :len,  :value => lambda { data.length }
     string :data, :read_length => :len
 
@@ -291,11 +290,11 @@ and implementing the #get and #set methods.
     def set(v) self.data = v; end
   end
 
-If the type we are creating represents a single value then inherit from
-BinData::SingleValue, otherwise inherit from BinData::MultiValue.
+If the type we are creating represents a primitive value then inherit from
+BinData::Primitive, otherwise inherit from BinData::Record.
 
 == License
 
 BinData is released under the same license as Ruby.
 
-Copyright (c) 2007, 2008 Dion Mendel
+Copyright (c) 2007 - 2009 Dion Mendel

data/TODO

@@ -1,20 +1,26 @@
-* Think how offset_of should work.
+* Write a Rakefile
 
-    + perhaps #offset_of and #abs_offset_of methods?
-    + needed for struct and array
+* Registry should auto create integers and bits. (e.g. 24bit int, 191bit int)
 
-* Add ability to determine name of a data object.
-    e.g.  obj.a[4].c
+* Improve speed of Sanitizer for recursive records
+    rework it so that sanitizing is done during definition (before initialising).
+
+* Write a detailed tutorial (probably as a web page).
+
+* Need more examples.
+
+* Need wrapper to be able to define new wrapped classes - with parameters
+
+-----------------------------------------------------------------------------
+define_wrapped_class("MyInt", :uint32be, :initial_value => 3) #=> MyInt
+define_wrapped_class("ByteArray", :array, :type => :uint8) #=> ByteArray
+define_wrapped_class("MyInt", :uint32be, {:initial_value => :mult}, [], [], {:mult => 3}) #=> MyInt
+          # mandatory, optional, default
+-----------------------------------------------------------------------------
 
-  This will be used when throwing exceptions to aid debugging.
 
-* Using the above names, add tracing capability when reading.
 
-* Clean up Array to make it as close to ruby Array as possible.
 
-* Need more examples.
 
-* Refactor test cases.
-    add more comprehensive integration tests to serve as examples
+refactor snapshot -> _snapshot et al
 
-* Need better documentation.

data/examples/gzip.rb

@@ -9,14 +9,14 @@ class Gzip
   # Known compression methods
   DEFLATE = 8
 
-  class Extra < BinData::MultiValue
+  class Extra < BinData::Record
     endian :little
 
     uint16 :len,  :length => lambda { data.length }
     string :data, :read_length => :len
   end
 
-  class Header < BinData::MultiValue
+  class Header < BinData::Record
     endian :little
 
     uint16  :ident,      :value => 0x8b1f, :check_value => 0x8b1f
@@ -45,7 +45,7 @@ class Gzip
     uint16  :crc16,      :onlyif => lambda { fcrc16.nonzero? }
   end
 
-  class Footer < BinData::MultiValue
+  class Footer < BinData::Record
     endian :little
 
     uint32 :crc32
@@ -64,7 +64,7 @@ class Gzip
   def_delegators :@footer, :crc32, :uncompressed_size
 
   def mtime
-    Time.at(@header.mtime)
+    Time.at(@header.mtime.snapshot)
   end
 
   def mtime=(tm)

data/lib/bindata.rb

@@ -6,17 +6,18 @@ require 'bindata/bits'
 require 'bindata/choice'
 require 'bindata/float'
 require 'bindata/int'
-require 'bindata/multi_value'
+require 'bindata/primitive'
+require 'bindata/record'
 require 'bindata/rest'
-require 'bindata/single_value'
 require 'bindata/string'
 require 'bindata/stringz'
 require 'bindata/struct'
+require 'bindata/trace'
 
 # = BinData
 # 
 # A declarative way to read and write structured binary data.
 # 
 module BinData
-  VERSION = "0.9.3"
+  VERSION = "0.10.0"
 end

data/lib/bindata/array.rb

@@ -55,30 +55,26 @@ module BinData
   class Array < BinData::Base
     include Enumerable
 
-    # Register this class
     register(self.name, self)
 
-    # These are the parameters used by this class.
-    bindata_mandatory_parameter :type
-    bindata_optional_parameters :initial_length, :read_until
-    bindata_mutually_exclusive_parameters :initial_length, :read_until
+    mandatory_parameter :type
+    optional_parameters :initial_length, :read_until
+    mutually_exclusive_parameters :initial_length, :read_until
 
     class << self
-      # Ensures that +params+ is of the form expected by #initialize.
+
       def sanitize_parameters!(sanitizer, params)
         unless params.has_key?(:initial_length) or params.has_key?(:read_until)
           # ensure one of :initial_length and :read_until exists
           params[:initial_length] = 0
         end
 
-        if params.has_key?(:read_length)
-          warn ":read_length is not used with arrays.  You probably want to change this to :initial_length"
-        end
+        warn_replacement_parameter(params, :read_length, :initial_length)
 
         if params.has_key?(:type)
           type, el_params = params[:type]
-          klass = sanitizer.lookup_klass(type)
-          sanitized_params = sanitizer.sanitize_params(klass, el_params)
+          klass = sanitizer.lookup_class(type)
+          sanitized_params = sanitizer.sanitized_params(klass, el_params)
           params[:type] = [klass, sanitized_params]
         end
 
@@ -86,27 +82,26 @@ module BinData
       end
     end
 
-    # Creates a new Array
     def initialize(params = {}, parent = nil)
       super(params, parent)
 
-      klass, el_params = no_eval_param(:type)
+      el_class, el_params = get_parameter(:type)
 
       @element_list    = nil
-      @element_klass   = klass
+      @element_class   = el_class
       @element_params  = el_params
     end
 
     # Returns if the element at position +index+ is clear?.  If +index+
-    # is not given, then returns whether all fields are clear.
+    # is not given, then returns whether all elements are clear.
     def clear?(index = nil)
-      if @element_list.nil?
-        true
-      elsif index.nil?
-        elements.each { |f| return false if not f.clear? }
-        true
+      if index.nil?
+        @element_list.nil? or elements.inject(true) { |all_clear, f| all_clear and f.clear? }
+      elsif index < elements.length
+        warn "'obj.clear?(n)' is deprecated.  Replacing with 'obj[n].clear?'"
+        elements[index].clear?
       else
-        (index < elements.length) ? elements[index].clear? : true
+        true
       end
     end
 
@@ -114,105 +109,116 @@ module BinData
     # the internal state of the array is reset to that of a newly created
     # object.
     def clear(index = nil)
-      if @element_list.nil?
-        # do nothing as the array is already clear
-      elsif index.nil?
+      if index.nil?
         @element_list = nil
       elsif index < elements.length
+        warn "'obj.clear(n)' is deprecated.  Replacing with 'obj[n].clear'"
         elements[index].clear
       end
     end
 
-    # Returns whether this data object contains a single value.  Single
-    # value data objects respond to <tt>#value</tt> and <tt>#value=</tt>.
-    def single_value?
-      return false
+    # Returns the first index of +obj+ in self.
+    #
+    # a = BinData::String.new; a.value = "a"
+    # b = BinData::String.new; b.value = "b"
+    # c = BinData::String.new; c.value = "c"
+    #
+    # arr = BinData::Array.new(:type => :string)
+    # arr.push(a, b, c)
+    #
+    # arr.find_index("b") #=> 1
+    # arr.find_index(c) #=> 2
+    #
+    def find_index(obj)
+      elements.find_index(obj)
     end
+    alias_method :index, :find_index
 
-    # To be called after calling #do_read.
-    def done_read
-      elements.each { |f| f.done_read }
+    # Returns the first index of +obj+ in self.
+    #
+    # Uses equal? for the comparator.
+    def find_index_of(obj)
+      elements.find_index { |el| el.equal?(obj) }
     end
 
-    # Appends a new element to the end of the array.  If the array contains
-    # single_values then the +value+ may be provided to the call.
-    # Returns the appended object, or value in the case of single_values.
-    def append(value = nil)
-      # TODO: deprecate #append as it can be replaced with #push
-      append_new_element
-      self[-1] = value unless value.nil?
-      self.last
+    def push(*args)
+      insert(-1, *args)
+      self
     end
+    alias_method :<<, :push
 
-    def index(obj)
-      # TODO handle single values
-      elements.index(obj)
+    def unshift(*args)
+      insert(0, *args)
+      self
     end
 
-    # Pushes the given object(s) on to the end of this array. 
-    # This expression returns the array itself, so several appends may 
-    # be chained together.
-    def push(*args)
-      args.each do |arg|
-        if @element_klass == arg.class
-          # TODO: need to modify arg.env to add_variable(:index) and
-          # to link arg.env to self.env
-          elements.push(arg)
-        else
-          append(arg)
-        end
-      end
+    def concat(array)
+      insert(-1, *array.to_ary)
       self
     end
 
-    # Returns the element at +index+.  If the element is a single_value
-    # then the value of the element is returned instead.
-    def [](*args)
-      if args.length == 1 and ::Integer === args[0]
-        # extend array automatically
-        while args[0] >= elements.length
-          append_new_element
-        end
+    def insert(index, *objs)
+      extend_array(index - 1)
+      elements.insert(index, *to_storage_formats(objs))
+      self
+    end
+
+    def append(value = nil)
+      warn "#append is deprecated, use push or slice instead"
+      if value.nil?
+        slice(length)
+      else
+        push(value)
       end
+      self.last
+    end
 
-      data = elements[*args]
-      if args.length > 1 or ::Range === args[0]
-        data.collect { |el| (el && el.single_value?) ? el.value : el }
+    # Returns the element at +index+.
+    def [](arg1, arg2 = nil)
+      if arg1.respond_to?(:to_int) and arg2.nil?
+        slice_index(arg1.to_int)
+      elsif arg1.respond_to?(:to_int) and arg2.respond_to?(:to_int)
+        slice_start_length(arg1.to_int, arg2.to_int)
+      elsif arg1.is_a?(Range) and arg2.nil?
+        slice_range(arg1)
       else
-        (data && data.single_value?) ? data.value : data
+        raise TypeError, "can't convert #{arg1} into Integer" unless arg1.respond_to?(:to_int)
+        raise TypeError, "can't convert #{arg2} into Integer" unless arg2.respond_to?(:to_int)
       end
     end
     alias_method :slice, :[]
 
-    # Sets the element at +index+.  If the element is a single_value
-    # then the value of the element is set instead.
-    def []=(index, value)
-      # extend array automatically
-      while index >= elements.length
-        append_new_element
-      end
+    def slice_index(index)
+      extend_array(index)
+      at(index)
+    end
 
-      obj = elements[index]
-      unless obj.single_value?
-        # TODO: allow setting objects, not just values
-        raise NoMethodError, "undefined method `[]=' for #{self}", caller
-      end
-      obj.value = value
+    def slice_start_length(start, length)
+      elements[start, length]
     end
 
-    # Iterate over each element in the array.  If the elements are
-    # single_values then the values of the elements are iterated instead.
-    def each
-      elements.each do |el|
-        yield(el.single_value? ? el.value : el)
-      end
+    def slice_range(range)
+      elements[range]
+    end
+    private :slice_index, :slice_start_length, :slice_range
+
+    # Returns the element at +index+.  Unlike +slice+, if +index+ is out
+    # of range the array will not be automatically extended.
+    def at(index)
+      elements[index]
+    end
+
+    # Sets the element at +index+.
+    def []=(index, value)
+      extend_array(index)
+      elements[index].assign(value)
     end
 
     # Returns the first element, or the first +n+ elements, of the array.
     # If the array is empty, the first form returns nil, and the second
     # form returns an empty array.
     def first(n = nil)
-      if n.nil? and elements.empty?
+      if n.nil? and empty?
         # explicitly return nil as arrays grow automatically
         nil
       elsif n.nil?
@@ -247,87 +253,151 @@ module BinData
 
     # Allow this object to be used in array context.
     def to_ary
-      snapshot
+      collect { |el| el }
+    end
+
+    # Iterate over each element in the array.
+    def each
+      elements.each { |el| yield el }
+    end
+
+    def debug_name_of(child)
+      index = find_index_of(child)
+      "#{debug_name}[#{index}]"
+    end
+
+    def offset_of(child)
+      index = find_index_of(child)
+      sum = sum_num_bytes_below_index(index)
+
+      child_offset = (::Integer === child.do_num_bytes) ? sum.ceil : sum.floor
+
+      offset + child_offset
     end
 
     #---------------
     private
 
-    # Reads the values for all fields in this object from +io+.
+    def extend_array(max_index)
+      max_length = max_index + 1
+      while elements.length < max_length
+        append_new_element
+      end
+    end
+
+    def to_storage_formats(els)
+      els.collect { |el| to_storage_format(el) }
+    end
+
+    def to_storage_format(obj)
+      element = new_element
+      element.assign(obj)
+      element
+    end
+
     def _do_read(io)
-      if has_param?(:initial_length)
+      if has_parameter?(:initial_length)
         elements.each { |f| f.do_read(io) }
-      elsif has_param?(:read_until)
-        if no_eval_param(:read_until) == :eof
-          @element_list = nil
-          loop do
-            element = append_new_element
-            begin
-              element.do_read(io)
-            rescue
-              @element_list.pop
-              break
-            end
-          end
-        else
-          @element_list = nil
-          loop do
-            element = append_new_element
-            element.do_read(io)
-            variables = { :index => self.length - 1, :element => self.last,
-                          :array => self }
-            finished = eval_param(:read_until, variables)
-            break if finished
-          end
+      elsif has_parameter?(:read_until)
+        read_until(io)
+      end
+    end
+
+    def read_until(io)
+      if get_parameter(:read_until) == :eof
+        read_until_eof(io)
+      else
+        read_until_condition(io)
+      end
+    end
+
+    def read_until_eof(io)
+      finished = false
+      while not finished
+        element = append_new_element
+        begin
+          element.do_read(io)
+        rescue
+          elements.pop
+          finished = true
         end
       end
     end
 
-    # Writes the values for all fields in this object to +io+.
+    def read_until_condition(io)
+      finished = false
+      while not finished
+        element = append_new_element
+        element.do_read(io)
+        variables = { :index => self.length - 1, :element => self.last,
+                      :array => self }
+        finished = eval_parameter(:read_until, variables)
+      end
+    end
+
+    def _done_read
+      elements.each { |f| f.done_read }
+    end
+
     def _do_write(io)
       elements.each { |f| f.do_write(io) }
     end
 
-    # Returns the number of bytes it will take to write the element at
-    # +index+.  If +index+, then returns the number of bytes required
-    # to write all fields.
     def _do_num_bytes(index)
       if index.nil?
-        (elements.inject(0) { |sum, f| sum + f.do_num_bytes }).ceil
-      else
+        sum_num_bytes_for_all_elements.ceil
+      elsif index < elements.length
+        warn "'obj.num_bytes(n)' is deprecated.  Replacing with 'obj[n].num_bytes'"
         elements[index].do_num_bytes
+      else
+        0
       end
     end
 
-    # Returns a snapshot of the data in this array.
+    def _assign(array)
+      raise ArgumentError, "can't set a nil value for #{debug_name}" if array.nil?
+
+      @element_list = to_storage_formats(array.to_ary)
+    end
+
     def _snapshot
       elements.collect { |e| e.snapshot }
     end
 
-    # Returns the list of all elements in the array.  The elements
-    # will be instantiated on the first call to this method.
     def elements
       if @element_list.nil?
         @element_list = []
-        if has_param?(:initial_length)
-          # create the desired number of instances
-          eval_param(:initial_length).times do
-            append_new_element
+        if has_parameter?(:initial_length)
+          eval_parameter(:initial_length).times do
+            @element_list << new_element
           end
         end
       end
       @element_list
     end
 
-    # Creates a new element and appends it to the end of @element_list.
-    # Returns the newly created element
     def append_new_element
-      # ensure @element_list is initialised
-      elements()
-
-      element = @element_klass.new(@element_params, self)
-      @element_list << element
+      element = new_element
+      elements << element
       element
     end
+
+    def new_element
+      @element_class.new(@element_params, self)
+    end
+
+    def sum_num_bytes_for_all_elements
+      sum_num_bytes_below_index(length)
+    end
+
+    def sum_num_bytes_below_index(index)
+      sum = 0
+      (0...index).each do |i|
+        nbytes = elements[i].do_num_bytes
+        sum = ((::Integer === nbytes) ? sum.ceil : sum) + nbytes
+      end
+
+      sum
+    end
   end
 end