rstruct 0.1.0

data/README.rdoc

@@ -0,0 +1,137 @@
+= rstruct
+
+Rstruct is yet another ruby binary struct parser/builder framework based around
+a declarative syntax. Rstruct is designed with simplicity in mind and with a
+a slightly less active focus on performance. 
+
+The goal of Rstruct is is to provide a system that lets you define structures
+once, and use them in many ways as you see fit. Kind-of like... C structs in
+system/API header files.
+
+The structure declaration syntax emulates C structures syntax, although Rstruct
+is not intended to be able to parse C structures from C header files. (howewver
+an addon could conceivably be built to do so with minimal fuss)
+
+Multiple approaches to parsing and building are supported and more can be added
+with minimal fuss. For example, you might not wish to use the same interface to
+build or parse a structure on an IO object as you would a String object. In
+other words, it's nice to have something that can work easily with streamed IO
+as well as buffers.
+
+Rstruct was written because the authors wanted something comparable to C
+structures without having a strong need for extra 'magic' in parsing or
+building structures. While there exist numerous options in this space, they
+all seem to have suffered from a combination of performance and interface issues
+which limit their potential (and in some cases, spotty maintenance). Having, 
+tried pretty much all of these alternatives (and even contributed to a few) 
+in previous projects, the author still decided to write rstruct.
+
+That said, the author does not claim Rstruct to be superior to any others, it's
+just another approach. There are several other excellent binary structure
+library options out there such as BinData, BitStruct, or Ruckus. Some of these
+support variable length structures which, at this time, Rstruct does not. If
+you are looking for something to parse variable length data automatically, you
+are may be better off checking out one of these alterntives. 
+
+== Installation
+
+  (sudo)? gem install rstruct
+
+== Synopsis
+Here is a trivial example defining and packing a raw structure
+
+  require 'rstruct'
+  extend Rstruct::ClassMethods
+
+  example = struct(:example) {
+    uint32 :a
+    uint32 :b
+    uint32 :c
+  }.instance
+
+  example.a = 0x41
+  example.b = 0x42
+  example.c = 0x43
+
+  raw = example.write()
+  # => "A\x00\x00\x00B\x00\x00\x00C\x00\x00\x00" # on a little endian machine
+  # => "\x00\x00\x00A\x00\x00\x00B\x00\x00\x00C" # on a big endian machine
+
+
+Here is a fully functional Rstruct parser example using Apple's FAT file structure.
+
+  # To compare the structs to their C counterparts, see:
+  #   http://fxr.watson.org/fxr/source/EXTERNAL_HEADERS/mach-o/fat.h?v=xnu-1228
+
+  require 'rstruct'
+  extend Rstruct::ClassMethods
+
+  FAT_MAGIC = 0xcafebabe
+  FAT_CIGAM = 0xbebafeca
+  
+  struct(:fat_header) {
+    uint32be  :magic;      # FAT_MAGIC
+    uint32be  :nfat_arch;  # number of structs that follow
+  }
+
+  typedef :uint32be, :cpu_type_t
+  typedef :uint32be, :cpu_subtype_t
+
+  struct(:fat_arch) {
+    cpu_type_t     :cputype    # cpu specifier (int)
+    cpu_subtype_t  :cpusubtype # machine specifier (int)
+    uint32be       :offset     # file offset to this object file
+    uint32be       :size       # size of this object file
+    uint32be       :align      # alignment as a power of 2
+  }
+
+  # a basic helper to produce textual dumps of fields
+  def dump(struct)
+    struct.each_pair.map {|k,v| "  #{k} = 0x%0.8x" % v}
+  end
+
+  File.open('ls.from_a_mac','r') do |f|
+    # Read and dump the FAT header from the file
+    head = get_type(:fat_header).read(f)
+    puts "FAT header:", dump(head)
+    puts
+
+    # Read and dump the architectures after the header
+
+    fat_arch = get_type(:fat_arch)
+    (head.nfat_arch).times do |i|
+      arch = fat_arch.read(f) # note, it reads to a seperate object on each arch
+      puts "  Architecture #{i}:"
+      puts "  " << dump(arch).join("\n  ")
+      puts
+    end
+  end
+  
+.. which should produce output something like the following:
+
+  FAT header:
+    magic = 0xcafebabe
+    nfat_arch = 0x00000002
+
+    Architecture 0:
+      cputype = 0x01000007
+      cpusubtype = 0x80000003
+      offset = 0x00001000
+      size = 0x00009ab0
+      align = 0x0000000c
+
+    Architecture 1:
+      cputype = 0x00000007
+      cpusubtype = 0x00000003
+      offset = 0x0000b000
+      size = 0x00008b30
+      align = 0x0000000c
+
+
+Please refer to rdoc, the samples/ directory, and unit tests for more information.
+
+== Copyright
+
+Copyright (c) 2011 Eric Monti. See LICENSE.txt for
+further details.
+

data/Rakefile

@@ -0,0 +1,50 @@
+require 'rubygems'
+require 'bundler'
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+require 'rake'
+
+require 'jeweler'
+Jeweler::Tasks.new do |gem|
+  # gem is a Gem::Specification... see http://docs.rubygems.org/read/chapter/20 for more options
+  gem.name = "rstruct"
+  gem.homepage = "http://github.com/emonti/rstruct"
+  gem.license = "GPL"
+  gem.summary = %Q{A library for working with Ruby binary structures in a way similar to c-structs}
+  gem.description = %Q{A library for working with Ruby binary structures in a way similar to c-structs}
+  gem.email = "esmonti@gmail.com"
+  gem.authors = ["Eric Monti"]
+  # Include your dependencies below. Runtime dependencies are required when using your gem,
+  # and development dependencies are only needed for development (ie running rake tasks, tests, etc)
+  #  gem.add_runtime_dependency 'jabber4r', '> 0.1'
+  #  gem.add_development_dependency 'rspec', '> 1.2.3'
+end
+Jeweler::RubygemsDotOrgTasks.new
+
+require 'rspec/core'
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |spec|
+  spec.pattern = FileList['spec/**/*_spec.rb']
+end
+
+RSpec::Core::RakeTask.new(:rcov) do |spec|
+  spec.pattern = 'spec/**/*_spec.rb'
+  spec.rcov = true
+end
+
+task :default => :spec
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  version = File.exist?('VERSION') ? File.read('VERSION') : ""
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "rstruct #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/VERSION

@@ -0,0 +1 @@
+0.1.0

data/lib/rstruct.rb

@@ -0,0 +1,70 @@
+require 'stringio'
+require 'rstruct/registry'
+require 'rstruct/types'
+require 'rstruct/structure'
+require 'rstruct/struct_builder'
+
+module Rstruct
+  module ClassMethods
+
+    # Take the following C struct for example:
+    #
+    #   struct mach_header {
+    #     uint32_t      magic;      /* mach magic number identifier */
+    #     cpu_type_t    cputype;    /* cpu specifier */
+    #     cpu_subtype_t cpusubtype; /* machine specifier */
+    #     uint32_t      filetype;   /* type of file */
+    #     uint32_t      ncmds;      /* number of load commands */
+    #     uint32_t      sizeofcmds; /* the size of all the load commands */
+    #     uint32_t      flags;      /* flags */
+    #   }
+    #
+    # Which might be defined with rstruct as follows:
+    #
+    #   extend(Rstruct::ClassMethods)
+    #
+    #   typedef :uint32_t, :cpu_type_t
+    #   typedef :uint32_t, :cpu_subtype_t
+    #
+    #   struct(:mach_header) {
+    #     uint32_t      :magic      # mach magic number identifier
+    #     cpu_type_t    :cputype    # cpu specifier
+    #     cpu_subtype_t :cpusubtype # machine specifier
+    #     uint32_t      :filetype   # type of file
+    #     uint32_t      :ncmds      # number of load commands
+    #     uint32_t      :sizeofcmds # the size of all the load commands
+    #     uint32_t      :flags      # flags
+    #   }
+    #
+    def struct(name, opts={},&block)
+      Rstruct::Structure.new(name, opts, &block)
+    end
+
+    def typedef(p, t, opts={})
+      reg = opts[:registry] || default_registry
+      reg.typedef(p,t,opts)
+    end
+
+    def sizeof(typ, reg=nil)
+      reg ||= default_registry
+      if t=reg[typ]
+        t.sizeof
+      else
+        raise(InvalidTypeError, "unknown type: #{typ}")
+      end
+    end
+
+    # Returns the default Rstruct registry
+    def default_registry
+      Registry::DEFAULT_REGISTRY
+    end
+
+    def get_type(typ, reg=Registry::DEFAULT_REGISTRY)
+      reg[typ]
+    end
+
+  end
+
+  extend(ClassMethods)
+end
+

data/lib/rstruct/base_types.rb

@@ -0,0 +1,5 @@
+require 'rstruct/registry'
+
+require 'rstruct/base_types/type.rb'
+require 'rstruct/base_types/packed_type.rb'
+require 'rstruct/base_types/container_type.rb'

data/lib/rstruct/base_types/container_type.rb

@@ -0,0 +1,102 @@
+require 'rstruct/base_types/type'
+
+module Rstruct
+
+  module ContainerMixins
+    def rstruct_type
+      @rstruct_type
+    end
+
+    def rstruct_type=(val)
+      if @rstruct_type
+        raise(ArgumentError, "Can't override the rstruct_type once it is set")
+      else
+        @rstruct_type = val
+      end
+    end
+
+    def write(dst=nil, pvals=nil)
+      if dst.is_a?(String)
+        l = dst.size
+        dst = StringIO.new(dst)
+        dst.pos = l
+      elsif dst.nil?
+        dst = StringIO.new
+      end
+
+      typ ||= self.rstruct_type
+
+
+      vals = (pvals.respond_to?(:values) ? pvals.values : pvals)
+      vals ||= self.values
+
+      opos = dst.pos
+      typ.fields.each_with_index do |f, i|
+        fldval = vals[i]
+        if fldval.respond_to?(:write)
+          fldval.write(dst, fldval)
+        else
+          dst.write(f.typ.pack_value(fldval, self))
+        end
+      end
+      if dst.is_a?(StringIO) and pvals.nil?
+        dst.pos = opos
+        return(dst.read)
+      else
+        return dst.pos - opos
+      end
+    end
+  end
+
+  class ContainerType < Type
+    include Packable
+
+    def initialize(*args, &block)
+      @countainer = true
+      super(*args, &block)
+    end
+
+    def groupable?
+      self.fields.find {|f| not f.groupable? }.nil?
+    end
+
+    def format
+      self.fields.map do |f| 
+        if f.groupable?
+          f.format 
+        else
+          return nil
+        end
+      end.join
+    end
+
+    def sizeof
+      self.fields.inject(0) do |s,v| 
+        if vs=v.typ.sizeof
+          s+=vs
+        else
+          return nil
+        end
+      end
+    end
+
+    def field_names
+      @field_names ||= self.fields.map{|f| f.name }
+    end
+
+    def field_types 
+      @field_types ||= self.fields.map{|f| f.typ }
+    end
+
+    def read(raw, obj=nil)
+      raw = StringIO.new(raw) if raw.is_a?(String)
+      obj = self.instance()
+      fields.each do |f|
+        obj[f.name] = f.read(raw, obj)
+      end
+      return obj
+    end
+
+  end
+
+end

data/lib/rstruct/base_types/packed_type.rb

@@ -0,0 +1,78 @@
+require 'rstruct/base_types/type'
+
+module Rstruct
+  class PackError < StandardError
+  end
+
+  class ReadError < StandardError
+  end
+
+  module Packable
+    private
+      # sets up a callback for packing a value to raw data
+      # for this type
+      def on_pack(&block)
+        @pack_cb = block
+      end
+
+      # sets up a callback for unpacking data from a string for
+      # this type
+      def on_unpack(&block)
+        @unpack_cb = block
+      end
+
+    public
+      # Called when parsing. While you can override this in subclasses,
+      # in general it is probably better to use the 'on_unpack' method
+      # to define a proc to handle unpacking for special cases.
+      def read(raw, predecessors=nil)
+        if raw.respond_to?(:read)
+          raw = raw.read(self.sizeof())
+        end
+        if raw.size < self.sizeof()
+          raise(ReadError, "Expected #{self.sizeof} bytes, but only got #{raw.size} bytes")
+        end
+
+        vals = 
+          if @unpack_cb
+            @unpack_cb.call(raw, predecessors)
+          else
+            raw.unpack(self.format)
+          end
+        return(self.claim_value(vals, predecessors))
+      end
+
+      # Called when composing raw data. While you can override this in
+      # subclasses, in general it is probably better to use the 'on_pack'
+      # method to define a proc to handle packing for special cases.
+      def pack_value(val, obj=nil)
+        begin
+          if @pack_cb
+            @pack_cb.call(val, obj)
+          else
+            varray = val.is_a?(Array) ? val : [val]
+            varray.pack(self.format)
+          end
+        rescue => e
+          raise(PackError, "Error packing #{val.inspect} as type #{self.name.inspect} -- #{e.class} -> #{e}")
+        end
+      end
+  end
+
+  class PackedType < Type
+    include Packable
+    attr_reader :size, :format
+
+    def initialize(name, size, format, opts={}, &block)
+      @size = size
+      @format = format
+      @groupable = true
+      super(name, opts, &block)
+    end
+
+    def instance(val=nil)
+      val
+    end
+  end
+end
+