bindata 1.2.2 → 1.3.1

data/ChangeLog

@@ -1,5 +1,17 @@
 = BinData Changelog
 
+== Version 1.3.0 (2011-01-25)
+
+* BinData objects can now assign values when instantiating.
+* Improved support for bit-based alignment.
+* Updated reference manual.
+* Added examples for declaring recursive structures.
+* Objects deriving from BinData::Base should no longer override #initialize.
+* Added BinData::Base(#new, #initialize_instance) to speed up
+  instantiation of multiple objects.
+* Updated specs to rspec-1.3.0
+* BinData::Struct.hide now expects symbols instead of strings.
+
 == Version 1.2.2 (2010-12-14)
 
 * Added Base.bindata_name method.

data/NEWS

@@ -1,3 +1,56 @@
+= 1.3.0
+
+== New features
+
+You can now assign values to BinData objects when instantiating them.
+
+Previously:
+    obj = BinData::Stringz.new(:max_length => 10)
+    obj.assign("abc")
+
+Now:
+    obj = BinData::Stringz.new("abc", :max_length => 10)
+
+== Backwards incompatible changes
+
+This version makes some backwards incompatible changes for more advanced
+users of BinData.
+
+This only affects you if you have created your own custom types by
+subclassing BinData::Base or BinData::BasePrimitive.
+
+All instance variables must now be initialized in #initialize_instance.
+Implementing #initialize is no longer allowed.
+
+Run your existing code in $VERBOSE mode ("ruby -w"), and BinData will
+inform you of any changes you need to make to your code.
+
+If your code previously looked like:
+
+    class MyType < BinData::Base
+      register_self
+
+      def initialize(parameters = {}, parent = nil)
+        super
+
+        @var1 = ...
+        @var2 = ...
+      end
+      ...
+    end
+
+You must change it to look like:
+
+    class MyType < BinData::Base
+      register_self
+
+      def initialize_instance
+        @var1 = ...
+        @var2 = ...
+      end
+      ...
+    end
+
 = 1.0.0
 
 Version 1.0.0 removes all deprecated features.  If you are upgrading from a

data/Rakefile

@@ -7,7 +7,8 @@ CURRENT_VERSION = BinData::VERSION
 
 PKG_FILES = FileList[
   "[A-Z]*",
-  "{examples,spec,lib}/**/*.rb",
+  "examples/**/*",
+  "{spec,lib}/**/*.rb",
   "tasks/**/*.rake",
   "setup.rb",
   "manual.haml",

data/examples/NBT.txt

@@ -0,0 +1,149 @@
+Named Binary Tag specification
+
+NBT (Named Binary Tag) is a tag based binary format designed to carry large amounts of binary data with smaller amounts of additional data.
+An NBT file consists of a single GZIPped Named Tag of type TAG_Compound.
+
+A Named Tag has the following format:
+
+    byte tagType
+    TAG_String name
+    [payload]
+    
+The tagType is a single byte defining the contents of the payload of the tag.
+
+The name is a descriptive name, and can be anything (eg "cat", "banana", "Hello World!"). It has nothing to do with the tagType.
+The purpose for this name is to name tags so parsing is easier and can be made to only look for certain recognized tag names.
+Exception: If tagType is TAG_End, the name is skipped and assumed to be "".
+
+The [payload] varies by tagType.
+
+Note that ONLY Named Tags carry the name and tagType data. Explicitly identified Tags (such as TAG_String above) only contains the payload. 
+
+
+The tag types and respective payloads are:
+
+    TYPE: 0  NAME: TAG_End
+    Payload: None.
+    Note:    This tag is used to mark the end of a list.
+             Cannot be named! If type 0 appears where a Named Tag is expected, the name is assumed to be "".
+             (In other words, this Tag is always just a single 0 byte when named, and nothing in all other cases)
+    
+    TYPE: 1  NAME: TAG_Byte
+    Payload: A single signed byte (8 bits)
+
+    TYPE: 2  NAME: TAG_Short
+    Payload: A signed short (16 bits, big endian)
+
+    TYPE: 3  NAME: TAG_Int
+    Payload: A signed short (32 bits, big endian)
+
+    TYPE: 4  NAME: TAG_Long
+    Payload: A signed long (64 bits, big endian)
+
+    TYPE: 5  NAME: TAG_Float
+    Payload: A floating point value (32 bits, big endian, IEEE 754-2008, binary32)
+
+    TYPE: 6  NAME: TAG_Double
+    Payload: A floating point value (64 bits, big endian, IEEE 754-2008, binary64)
+    
+    TYPE: 7  NAME: TAG_Byte_Array
+    Payload: TAG_Int length 
+             An array of bytes of unspecified format. The length of this array is <length> bytes
+
+    TYPE: 8  NAME: TAG_String
+    Payload: TAG_Short length 
+             An array of bytes defining a string in UTF-8 format. The length of this array is <length> bytes
+
+    TYPE: 9  NAME: TAG_List
+    Payload: TAG_Byte tagId
+             TAG_Int length
+             A sequential list of Tags (not Named Tags), of type <typeId>. The length of this array is <length> Tags
+    Notes:   All tags share the same type.
+             
+    TYPE: 10 NAME: TAG_Compound
+    Payload: A sequential list of Named Tags. This array keeps going until a TAG_End is found.
+             TAG_End end
+    Notes:   If there's a nested TAG_Compound within this tag, that one will also have a TAG_End, so simply reading until the next TAG_End will not work.
+             The names of the named tags have to be unique within each TAG_Compound
+             The order of the tags is not guaranteed.
+             
+             
+
+
+
+Decoding example:
+(Use http://www.minecraft.net/docs/test.nbt to test your implementation)
+
+
+First we start by reading a Named Tag.
+After unzipping the stream, the first byte is a 10. That means the tag is a TAG_Compound (as expected by the specification).
+
+The next two bytes are 0 and 11, meaning the name string consists of 11 UTF-8 characters. In this case, they happen to be "hello world".
+That means our root tag is named "hello world". We can now move on to the payload.
+
+From the specification, we see that TAG_Compound consists of a series of Named Tags, so we read another byte to find the tagType.
+It happens to be an 8. The name is 4 letters long, and happens to be "name". Type 8 is TAG_String, meaning we read another two bytes to get the length,
+then read that many bytes to get the contents. In this case, it's "Bananrama".
+
+So now we know the TAG_Compound contains a TAG_String named "name" with the content "Bananrama"
+
+We move on to reading the next Named Tag, and get a 0. This is TAG_End, which always has an implied name of "". That means that the list of entries
+in the TAG_Compound is over, and indeed all of the NBT file.
+
+So we ended up with this:
+
+	TAG_Compound("hello world"): 1 entries
+	{
+	   TAG_String("name"): Bananrama
+	}
+
+
+
+For a slightly longer test, download http://www.minecraft.net/docs/bigtest.nbt
+You should end up with this:
+
+	TAG_Compound("Level"): 11 entries
+	{
+	   TAG_Short("shortTest"): 32767
+	   TAG_Long("longTest"): 9223372036854775807
+	   TAG_Float("floatTest"): 0.49823147
+	   TAG_String("stringTest"): HELLO WORLD THIS IS A TEST STRING ���!
+	   TAG_Int("intTest"): 2147483647
+	   TAG_Compound("nested compound test"): 2 entries
+	   {
+	      TAG_Compound("ham"): 2 entries
+	      {
+	         TAG_String("name"): Hampus
+	         TAG_Float("value"): 0.75
+	      }
+	      TAG_Compound("egg"): 2 entries
+	      {
+	         TAG_String("name"): Eggbert
+	         TAG_Float("value"): 0.5
+	      }
+	   }
+	   TAG_List("listTest (long)"): 5 entries of type TAG_Long
+	   {
+	      TAG_Long: 11
+	      TAG_Long: 12
+	      TAG_Long: 13
+	      TAG_Long: 14
+	      TAG_Long: 15
+	   }
+	   TAG_Byte("byteTest"): 127
+	   TAG_List("listTest (compound)"): 2 entries of type TAG_Compound
+	   {
+	      TAG_Compound: 2 entries
+	      {
+	         TAG_String("name"): Compound tag #0
+	         TAG_Long("created-on"): 1264099775885
+	      }
+	      TAG_Compound: 2 entries
+	      {
+	         TAG_String("name"): Compound tag #1
+	         TAG_Long("created-on"): 1264099775885
+	      }
+	   }
+	   TAG_Byte_Array("byteArrayTest (the first 1000 values of (n*n*255+n*7)%100, starting with n=0 (0, 62, 34, 16, 8, ...))"): [1000 bytes]
+	   TAG_Double("doubleTest"): 0.4931287132182315
+	}

data/examples/ip_address.rb

@@ -16,8 +16,7 @@ class IPAddr < BinData::Primitive
   end
 end
 
-ip = IPAddr.new
-ip.value = "127.0.0.1"
+ip = IPAddr.new("127.0.0.1")
 
 puts "human readable value:  #{ip}"                     #=> 127.0.0.1
 puts "binary representation: #{ip.to_binary_s.inspect}" #=> "\177\000\000\001"

data/examples/list.rb

@@ -0,0 +1,124 @@
+require 'bindata'
+
+# An example of a recursively defined data format.
+#
+# This format is used to describe atoms and lists.
+# It is recursive because lists can contain other lists.
+#
+# Atoms - contain a single integer
+# Lists - contain a mixture of atoms and lists
+#
+# The binary representation is:
+#
+# Atoms - A single byte 'a' followed by an int32 containing the value.
+# Lists - A single byte 'l' followed by an int32 denoting the number of
+#         items in the list.  This is followed by all the items in the list.
+#
+# All integers are big endian.
+#
+#
+# A first attempt at a declaration would be:
+#
+#     class Atom < BinData::Record
+#       string  :tag, :length => 1, :check_value => 'a'
+#       int32be :val
+#     end
+#
+#     class List < BinData::Record
+#       string  :tag,  :length => 1, :check_value => 'l'
+#       int32be :num,  :value => lambda { vals.length }
+#       array   :vals, :initial_length => :num do
+#         choice :selection => ??? do
+#           atom
+#           list
+#         end
+#       end
+#     end
+#
+# Notice how we get stuck on attemping to write a declaration for
+# the contents of the list.  We can't determine if the list item is
+# an atom or list because we haven't read it yet.  It appears that
+# we can't proceed.
+#
+# The cause of the problem is that the tag identifying the type is
+# coupled with that type.
+#
+# The solution is to decouple the tag from the type.  We introduce a
+# new type 'Term' that is a thin container around the tag plus the
+# type (atom or list).
+#
+# The declaration then becomes:
+#
+#     class Term < BinData::Record; end # forward declaration
+#     
+#     class Atom < BinData::Wrapper
+#       int32be
+#     end
+#     
+#     class List < BinData::Record
+#       int32be :num,  :value => lambda { vals.length }
+#       array   :vals, :type => :term, :initial_length => :num
+#     end
+#     
+#     class Term < BinData::Record
+#       string :tag, :length => 1
+#       choice :term, :selection => :tag do
+#         atom 'a'
+#         list 'l'
+#       end
+#     end
+
+
+class Term < BinData::Record; end  # Forward declaration
+
+class Atom < BinData::Wrapper
+  int32be
+
+  def decode
+    snapshot
+  end
+
+  def self.encode(val)
+    Atom.new(val)
+  end
+end
+
+class List < BinData::Record
+  int32be :num,  :value => lambda { vals.length }
+  array   :vals, :initial_length => :num, :type => :term
+
+  def decode
+    vals.collect { |v| v.decode }
+  end
+
+  def self.encode(val)
+    List.new(:vals => val.collect { |v| Term.encode(v) })
+  end
+end
+
+class Term < BinData::Record
+  string :tag, :length => 1
+  choice :term, :selection => :tag do
+    atom 'a'
+    list 'l'
+  end
+
+  def decode
+    term.decode
+  end
+
+  def self.encode(val)
+    if Fixnum === val
+      Term.new(:tag => 'a', :term => Atom.encode(val))
+    else
+      Term.new(:tag => 'l', :term => List.encode(val))
+    end
+  end
+end
+
+
+p Term.encode(4)
+p Term.encode(4).decode
+puts
+p Term.encode([1, [2, 3], 4])
+p Term.encode([1, [2, 3], 4]).decode

data/examples/nbt.rb

@@ -0,0 +1,195 @@
+require 'bindata'
+
+# An example reader for Minecraft's NBT format.
+# http://www.minecraft.net/docs/NBT.txt
+#
+# This is an example of how to write a BinData
+# declaration for a recursively defined file format.
+module Nbt
+
+  TAG_NAMES = {
+     0 => "End",
+     1 => "Byte",
+     2 => "Short",
+     3 => "Int",
+     4 => "Long",
+     5 => "Float",
+     6 => "Double",
+     7 => "Byte_Array",
+     8 => "String",
+     9 => "List",
+    10 => "Compound"
+  }
+
+  # NBT.txt line 25
+  class TagEnd < BinData::Primitive
+    def get; ""; end
+    def set(v); end
+
+    def to_formatted_s(indent = 0); to_s; end
+  end
+
+  # NBT.txt line 31
+  class TagByte < BinData::Wrapper
+    int8
+
+    def to_formatted_s(indent = 0); to_s; end
+  end
+
+  # NBT.txt line 34
+  class TagShort < BinData::Wrapper
+    int16be
+
+    def to_formatted_s(indent = 0); to_s; end
+  end
+
+  # NBT.txt line 37
+  class TagInt < BinData::Wrapper
+    int32be
+
+    def to_formatted_s(indent = 0); to_s; end
+  end
+
+  # NBT.txt line 40
+  class TagLong < BinData::Wrapper
+    int64be
+
+    def to_formatted_s(indent = 0); to_s; end
+  end
+
+  # NBT.txt line 43
+  class TagFloat < BinData::Wrapper
+    float_be
+
+    def to_formatted_s(indent = 0); to_s; end
+  end
+
+  # NBT.txt line 46
+  class TagDouble < BinData::Wrapper
+    double_be
+
+    def to_formatted_s(indent = 0); to_s; end
+  end
+
+  # NBT.txt line 49
+  class TagByteArray < BinData::Record
+    int32be :len, :value => lambda { data.length }
+    string  :data, :read_length => :len
+
+    def to_formatted_s(indent = 0)
+      "[#{len} bytes]"
+    end
+  end
+
+  # NBT.txt line 53
+  class TagString < BinData::Primitive
+    int16be :len, :value => lambda { data.length }
+    string  :data, :read_length => :len
+
+    def get
+      self.data
+    end
+
+    def set(v)
+      self.data = v
+    end
+
+    def to_formatted_s(indent = 0); to_s; end
+  end
+
+  ## Payload is the most important class to understand.
+  ## This abstraction allows recursive formats.
+  ## eg. lists can contain lists can contain lists. 
+
+  # Forward references used by Payload
+  class TagCompound < BinData::Record; end
+  class TagList < BinData::Record; end
+
+  # NBT.txt line 10
+  class Payload < BinData::Wrapper
+    mandatory_parameter :tag_selector
+    choice :selection => :tag_selector do
+      tag_end        0
+      tag_byte       1
+      tag_short      2
+      tag_int        3
+      tag_long       4
+      tag_float      5
+      tag_double     6
+      tag_byte_array 7
+      tag_string     8
+      tag_list       9
+      tag_compound   10
+    end
+  end
+
+  # NBT.txt line 6, 27
+  class NamedTag < BinData::Record
+    int8 :tag_id
+    tag_string :name,    :onlyif => :not_end_tag?
+    payload    :payload, :onlyif => :not_end_tag?, :tag_selector => :tag_id
+
+    def not_end_tag?
+      tag_id != 0
+    end
+
+    def to_formatted_s(indent = 0)
+      "  " * indent +
+      "TAG_#{TAG_NAMES[tag_id]}(\"#{name}\"): " +
+      payload.to_formatted_s(indent) + "\n"
+    end
+  end
+
+  # NBT.txt line 57
+  class TagList < BinData::Record
+    int8 :tag_id
+    int32be :len, :value => lambda { data.length }
+    array :data, :initial_length => :len do
+      payload :tag_selector => :tag_id
+    end
+
+    def to_formatted_s(indent = 0)
+      pre = "  " * indent
+      tag_type = "TAG_#{TAG_NAMES[tag_id]}"
+
+      "#{len} entries of type #{tag_type}\n" +
+      pre + "{\n" + 
+        data.collect { |el| "  #{pre}#{tag_type}: #{el.to_formatted_s(indent + 1)}\n" }.join("") +
+      pre + "}"
+    end
+  end
+
+  # NBT.txt line 63
+  class TagCompound < BinData::Record
+    array :data, :read_until => lambda { element.tag_id == 0 } do
+      named_tag
+    end
+
+    def to_formatted_s(indent = 0)
+      pre = "  " * indent
+      "#{data.length - 1} entries\n" +
+      pre + "{\n" + 
+        data[0..-2].collect { |el| el.to_formatted_s(indent + 1) }.join("") +
+      pre + "}"
+    end
+  end
+
+  # NBT.txt line 3
+  class Nbt < BinData::Wrapper
+    named_tag
+
+    def self.read(io)
+      require 'zlib'
+      super(Zlib::GzipReader.new(io))
+    end
+  end
+end
+
+if $0 == __FILE__
+  require 'stringio'
+
+  bigtest_nbt = StringIO.new "\037\213\b\000\000\000\000\000\000\003\355T\317O\032A\024~\302\002\313\226\202\261\304\020c\314\253\265\204\245\333\315B\021\211\261\210\026,\232\r\032\330\2501\206\270+\303\202.\273fw\260\361\324K{lz\353?\323#\177C\317\275\366\277\240\303/{i\317\275\3602\311\367\346\275o\346{o&y\002\004TrO,\016x\313\261M\215x\364\343pb>\b{\035\307\245\223\030\017\202G\335\356\204\002b\265\242\252\307xv\\W\313\250U\017\e\310\326\036j\225\206\206\r\255~X{\217\203\317\203O\203o\317\003\020n[\216>\276\2458Ld\375\020\352\332t\246\#@\334f.i\341\265\323\273s\372v\v)\333\v\340\357\350=\0368[\357\021\bV\365\336]\337\v@\340^\267\372d\267\004\000\214ALs\306\bUL\323 .}\244\300\310\302\020\263\272\336X\vS\243\356D\216E\0030\261'S\214L\361\351\024\243S\214\205\341\331\237\343\263\362D\201\245|3\335\330\273\307\252u\023_(\034\b\327.\321Y?\257\035\e`!Y\337\372\361\005\376\301\316\374\235\275\000\274\361@\311\370\205B@F\376\236\353\352\017\223:h\207`\273\35327\243(\n\216\273\365\320ic\312N\333\351\354\346\346+;\275%\276dI\t=\252\273\224\375\030~\350\322\016\332o\025L\261h>+\341\233\234\204\231\274\204\005\teY\026E\000\377/(\256/\362\302\262\244.\035 wZ;\271\214\312\347)\337QA\311\026\265\305m\241*\255,\3051\177\272z\222\216^\235_\370\022\005#\e\321\366\267w\252\315\225r\274\236\337X]K\227\256\222\027\271D\320\200\310\372>\277\263\334T\313\aun\243\266vY\222\223\251\334QP\231k\3145\346\032\377W#\bB\313\351\e\326x\302\354\376\374z\373}x\323\204\337\324\362\244\373\b\006\000\000"
+
+  nbt = Nbt::Nbt.read(bigtest_nbt)
+  puts nbt.to_formatted_s
+end