bindata 1.3.1 → 1.4.0

data/manual.md

@@ -40,6 +40,8 @@ manipulating.
 It supports all the common datatypes that are found in structured binary
 data. Support for dependent and variable length fields is built in. 
 
+Last updated: 2011-06-14
+
 ## License
 
 BinData is released under the same license as Ruby.
@@ -698,9 +700,11 @@ Three of the fields have parameters.
 ## Strings
 
 BinData supports two types of strings - fixed size and zero terminated.
-Strings are treated as a sequence of 8bit bytes.  This is the same as
-strings in Ruby 1.8.  The issue of character encoding is ignored by
-BinData.
+Strings are treated internally as a sequence of 8bit bytes.  This is the
+same as strings in Ruby 1.8.  BinData fully supports Ruby 1.9 string
+encodings.  See this [FAQ
+entry](#im_using_ruby_19_how_do_i_use_string_encodings_with_bindata) for
+details.
 
 ### Fixed Sized Strings
 
@@ -834,6 +838,9 @@ and implement the following three methods:
 
 :   The ruby value that a clear object should return.
 
+If you wish to access parameters from inside these methods, you can
+use `eval_parameter(key)`.
+
 Here is an example of a big integer implementation.
 
     # A custom big integer format.  Binary format is:
@@ -842,7 +849,6 @@ Here is an example of a big integer implementation.
     #             positive form of the integer.  The upper bit of each byte
     #             is set when there are more bytes in the stream.
     class BigInteger < BinData::BasePrimitive
-      register_self
 
       def value_to_binary_string(value)
         negative = (value < 0) ? 1 : 0
@@ -1105,6 +1111,169 @@ Examples
 
 # Advanced Topics
 
+## Debugging
+
+BinData includes several features to make it easier to debug
+declarations.
+
+### Tracing
+
+BinData has the ability to trace the results of reading a data
+structure.
+
+    class A < BinData::Record
+      int8  :a
+      bit4  :b
+      bit2  :c
+      array :d, :initial_length => 6, :type => :bit1
+    end
+
+    BinData::trace_reading do
+      A.read("\373\225\220")
+    end
+{:ruby}
+
+Results in the following being written to `STDERR`.
+
+    obj.a => -5
+    obj.b => 9
+    obj.c => 1
+    obj.d[0] => 0
+    obj.d[1] => 1
+    obj.d[2] => 1
+    obj.d[3] => 0
+    obj.d[4] => 0
+    obj.d[5] => 1
+{:ruby}
+
+### Rest
+
+The rest keyword will consume the input stream from the current position
+to the end of the stream.
+
+    class A < BinData::Record
+      string :a, :read_length => 5
+      rest   :rest
+    end
+
+    obj = A.read("abcdefghij")
+    obj.a #=> "abcde"
+    obj.rest #=" "fghij"
+{:ruby}
+
+### Hidden fields
+
+The typical way to view the contents of a BinData record is to call
+`#snapshot` or `#inspect`.  This gives all fields and their values.  The
+`hide` keyword can be used to prevent certain fields from appearing in
+this output.  This removes clutter and allows the developer to focus on
+what they are currently interested in.
+
+    class Testing < BinData::Record
+      hide :a, :b
+      string :a, :read_length => 10
+      string :b, :read_length => 10
+      string :c, :read_length => 10
+    end
+
+    obj = Testing.read(("a" * 10) + ("b" * 10) + ("c" * 10))
+    obj.snapshot #=> {"c"=>"cccccccccc"}
+    obj.to_binary_s #=> "aaaaaaaaaabbbbbbbbbbcccccccccc"
+{:ruby}
+
+## Parameterizing User Defined Types
+
+All BinData types have parameters that allow the behaviour of an object
+to be specified at initialization time.  User defined types may also
+specify parameters.  There are two types of parameters: mandatory and
+default.
+
+### Mandatory Parameters
+
+Mandatory parameters must be specified when creating an instance of the
+type.
+
+    class Polygon < BinData::Record
+      mandatory_parameter :num_edges
+
+      uint8 :num, :value => lambda { vertices.length }
+      array :vertices, :initial_length => :num_edges do
+        int8 :x
+        int8 :y
+      end
+    end
+
+    triangle = Polygon.new
+        #=> raises ArgumentError: parameter 'num_edges' must be specified in Polygon
+
+    triangle = Polygon.new(:num_edges => 3)
+    triangle.snapshot #=> {"num" => 3, "vertices" =>
+                             [{"x"=>0, "y"=>0}, {"x"=>0, "y"=>0}, {"x"=>0, "y"=>0}]}
+{:ruby}
+
+### Default Parameters
+
+Default parameters are optional.  These parameters have a default value
+that may be overridden when an instance of the type is created.
+
+    class Phrase < BinData::Primitive
+      default_parameter :number => "three"
+      default_parameter :adjective => "blind"
+      default_parameter :noun => "mice"
+
+      stringz :a, :initial_value => :number
+      stringz :b, :initial_value => :adjective
+      stringz :c, :initial_value => :noun
+
+      def get; "#{a} #{b} #{c}"; end
+      def set(v)
+        if /(.*) (.*) (.*)/ =~ v
+          self.a, self.b, self.c = $1, $2, $3
+        end
+      end
+    end
+
+    obj = Phrase.new(:number => "two", :adjective => "deaf")
+    obj.to_s #=> "two deaf mice"
+{:ruby}
+
+## Extending existing Types
+
+Sometimes you wish to create a new type that is simply an existing type
+with some predefined parameters.  Examples could be an array with a
+specified type, or an integer with an initial value.
+
+This can be achieved by subclassing the existing type and providing
+default parameters.  These parameters can of course be overridden at
+initialisation time.
+
+Here we define an array that contains big endian 16 bit integers.  The
+array has a preferred initial length.
+
+    class IntArray < BinData::Array
+      default_parameters :type => :uint16be, :initial_length => 5
+    end
+
+    arr = IntArray.new
+    arr.size #=> 5
+{:ruby}
+
+The initial length can be overridden at initialisation time.
+
+    arr = IntArray.new(:initial_length => 8)
+    arr.size #=> 8
+{:ruby}
+
+We can also use the block form syntax:
+
+    class IntArray < BinData::Array
+      endian :big
+      default_parameter :initial_length => 5
+
+      uint16
+    end
+{:ruby}
+
 ## Skipping over unused data
 
 Some structures contain binary data that is irrelevant to your purposes.  
@@ -1176,155 +1345,86 @@ versions of `string` and `int16le`.
     c.to_binary_s #=> "\377\377\377\377\377"
 {:ruby}
 
-## Wrappers
-
-Sometimes you wish to create a new type that is simply an existing type
-with some predefined parameters.  Examples could be an array with a
-specified type, or an integer with an initial value.
-
-This can be achieved with a wrapper.  A wrapper creates a new type based
-on an existing type which has predefined parameters.  These parameters
-can of course be overridden at initialisation time.
-
-Here we define an array that contains big endian 16 bit integers.  The
-array has a preferred initial length.
-
-    class IntArray < BinData::Wrapper
-      endian :big
-      array :type => :uint16, :initial_length => 5
-    end
-
-    arr = IntArray.new
-    arr.size #=> 5
-{:ruby}
-
-The initial length can be overridden at initialisation time.
-
-    arr = IntArray.new(:initial_length => 8)
-    arr.size #=> 8
-{:ruby}
-
-## Parameterizing User Defined Types
+---------------------------------------------------------------------------
 
-All BinData types have parameters that allow the behaviour of an object
-to be specified at initialization time.  User defined types may also
-specify parameters.  There are two types of parameters: mandatory and
-default.
+# FAQ
 
-### Mandatory Parameters
+## I'm using Ruby 1.9.  How do I use string encodings with BinData?
 
-Mandatory parameters must be specified when creating an instance of the
-type.  The `:type` parameter of `Array` is an example of a mandatory
-type.
+BinData will internally use 8bit binary strings to represent the data.
+You do not need to worry about converting between encodings.
 
-    class IntArray < BinData::Wrapper
-      mandatory_parameter :byte_count
+If you wish BinData to present string data in a specific encoding, you
+can override `#snapshot` as illustrated below:
 
-      array :type => :uint16be, :initial_length => lambda { byte_count / 2 }
+    class UTF8String < BinData::String
+      def snapshot
+        super.force_encoding('UTF-8')
+      end
     end
 
-    arr = IntArray.new
-        #=> raises ArgumentError: parameter 'byte_count' must be specified in IntArray
-
-    arr = IntArray.new(:byte_count => 12)
-    arr.snapshot #=> [0, 0, 0, 0, 0, 0]
+    str = UTF8String.new("\xC3\x85\xC3\x84\xC3\x96")
+    str #=> "ÅÄÖ"
+    str.to_binary_s #=> "\xC3\x85\xC3\x84\xC3\x96"
 {:ruby}
 
-### Default Parameters
-
-Default parameters are optional.  These parameters have a default value
-that may be overridden when an instance of the type is created.
-
-    class Phrase < BinData::Primitive
-      default_parameter :number => "three"
-      default_parameter :adjective => "blind"
-      default_parameter :noun => "mice"
+## How do I speed up initialization?
 
-      stringz :a, :initial_value => :number
-      stringz :b, :initial_value => :adjective
-      stringz :c, :initial_value => :noun
+I'm doing this and it's slow.
 
-      def get; "#{a} #{b} #{c}"; end
-      def set(v)
-        if /(.*) (.*) (.*)/ =~ v
-          self.a, self.b, self.c = $1, $2, $3
-        end
-      end
+    999.times do |i|
+      foo = Foo.new(:bar => "baz")
+      ...
     end
-
-    obj = Phrase.new(:number => "two", :adjective => "deaf")
-    obj.to_s #=> "two deaf mice"
 {:ruby}
 
-## Debugging
-
-BinData includes several features to make it easier to debug
-declarations.
-
-### Tracing
-
-BinData has the ability to trace the results of reading a data
-structure.
+BinData is optimized to be declarative.  For imperative use, the
+above naïve approach will be slow.  Below are faster alternatives.
 
-    class A < BinData::Record
-      int8  :a
-      bit4  :b
-      bit2  :c
-      array :d, :initial_length => 6, :type => :bit1
-    end
+The fastest approach is to reuse objects by calling `#clear` instead of
+instantiating more objects.
 
-    BinData::trace_reading do
-      A.read("\373\225\220")
+    foo = Foo.new(:bar => "baz")
+    999.times do
+      foo.clear
+      ...
     end
 {:ruby}
 
-Results in the following being written to `STDERR`.
+If you can't reuse objects, then consider the prototype pattern.
 
-    obj.a => -5
-    obj.b => 9
-    obj.c => 1
-    obj.d[0] => 0
-    obj.d[1] => 1
-    obj.d[2] => 1
-    obj.d[3] => 0
-    obj.d[4] => 0
-    obj.d[5] => 1
+    prototype = Foo.new(:bar => "baz")
+    999.times do
+      foo = prototype.new
+      ...
+    end
 {:ruby}
 
-### Rest
+The prefered approach is to be declarative.
 
-The rest keyword will consume the input stream from the current position
-to the end of the stream.
+    class FooList < BinData::Array
+      default_parameter :initial_length => 999
 
-    class A < BinData::Record
-      string :a, :read_length => 5
-      rest   :rest
+      foo :bar => "baz"
     end
 
-    obj = A.read("abcdefghij")
-    obj.a #=> "abcde"
-    obj.rest #=" "fghij"
+    array = FooList.new
+    array.each { ... }
 {:ruby}
 
-### Hidden fields
-
-The typical way to view the contents of a BinData record is to call
-`#snapshot` or `#inspect`.  This gives all fields and their values.  The
-`hide` keyword can be used to prevent certain fields from appearing in
-this output.  This removes clutter and allows the developer to focus on
-what they are currently interested in.
+## How do I model this complex nested format?
 
-    class Testing < BinData::Record
-      hide :a, :b
-      string :a, :read_length => 10
-      string :b, :read_length => 10
-      string :c, :read_length => 10
-    end
+A common pattern in file formats and network protocols is
+[type-length-value](http://en.wikipedia.org/wiki/Type-length-value).  The
+`type` field specifies how to interpret the `value`.  This gives a way to
+dynamically structure the data format.  An example is the TCP/IP protocol
+suite.  An IP datagram can contain a nested TCP, UDP or other packet type as
+decided by the `protocol` field.
 
-    obj = Testing.read(("a" * 10) + ("b" * 10) + ("c" * 10))
-    obj.snapshot #=> {"c"=>"cccccccccc"}
-    obj.to_binary_s #=> "aaaaaaaaaabbbbbbbbbbcccccccccc"
-{:ruby}
+Modelling this structure can be difficult when the nesting is recursive, e.g.
+IP tunneling.  Here is an example of the simplest possible recursive TLV structure,
+a [list that can contains atoms or other
+lists](http://bindata.rubyforge.org/svn/trunk/examples/list.rb).
 
 ---------------------------------------------------------------------------
 

data/spec/array_spec.rb

@@ -296,3 +296,23 @@ describe BinData::Array, "nested within an Array" do
     subject.should == [ [4], [5, 6], [7, 8, 9] ]
   end
 end
+
+describe BinData::Array, "subclassed" do
+  class IntArray < BinData::Array
+    endian :big
+    default_parameter :initial_element_value => 0
+
+    uint16 :initial_value => :initial_element_value
+  end
+
+  it "should forward parameters" do
+    subject = IntArray.new(:initial_length => 7)
+    subject.length.should == 7
+  end
+
+  it "should be able to override default parameters" do
+    subject = IntArray.new(:initial_length => 3, :initial_element_value => 5)
+    subject.to_binary_s.should == "\x00\x05\x00\x05\x00\x05"
+  end
+end
+

data/spec/base_primitive_spec.rb

@@ -5,6 +5,16 @@ require File.expand_path(File.join(File.dirname(__FILE__), "example"))
 require 'bindata/base_primitive'
 require 'bindata/io'
 
+describe BinData::BasePrimitive do
+  let(:r) { BinData::RegisteredClasses }
+
+  it "should not be registered" do
+    lambda {
+      r.lookup("BasePrimitive")
+    }.should raise_error(BinData::UnRegisteredTypeError)
+  end
+end
+
 describe BinData::BasePrimitive, "all subclasses" do
   class SubClassOfBasePrimitive < BinData::BasePrimitive
     expose_methods_for_testing