byteinterpreter 1.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: b0eb3381f96d234f733f66860614a9690253ea1a182560555cc438cfaff3bc31
+  data.tar.gz: 90025a8f011378e986a058348ddc8c638196fdc5850d83c602dc84069f53eefe
+SHA512:
+  metadata.gz: 13d197967fb4e723bdc5500d69901fcacc234f583a4fa3dbf4cc0b6ed76b5ec44582864af580b1b96e21e017603eee0ea862c333627772d2a4c398b7d431e4cf
+  data.tar.gz: f2aed83bbb6a1f9316c7c59af66f49b1e588ba5b51bccbfe39d41e7f216d9691b8148b2aab662dcf1ff38b20b89be90d49a1dac46b0a81bc7b6eea47017306ae

data/LICENSE.md

@@ -0,0 +1,7 @@
+Copyright 2018 Michael K Gremillion II
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,181 @@
+ByteInterpreter is a tool to interpret binary data in a fixed-length data
+structure into another format, or to encode data from another format into that
+same fixed-length data structure.
+
+## Introduction
+ByteInterpreter was made to assist with modifying an old Dreamcast-era RPG, by
+translating some of its content (spells, abilities, etc) from binary data into
+a human-readable format. Since the potential applications for this tool are more
+broad than just making mods for a singular videogame, I decided to spin it off
+into its own little tool for others to use.
+
+## Scope
+ByteInterpreter isn't overly-ambitious -- it's just there to read and write
+bytes for *fixed-length* data structures. That "fixed-length" bit is important
+-- there are no plans for ByteInterpreter to support data structures with
+variable length values. In the future I may expand ByteInterpreter's ability to
+cope with variable-length data structures, but only once it works the best it
+can with fixed-length structures.
+
+Right now, it can interpret binary data in four sizes - 8-bit, 16-bit, 32-bit,
+and 64-bit. It can also read strings in any arbitrary size. In the near future,
+I'd like to support reading bit fields and to handle pointers appropriately.
+
+## Installation
+ByteInterpreter can be installed with the following command -
+```
+gem install byteinterpreter
+```
+Afterwards, it's just a matter of requiring it in your code -
+```ruby
+require 'byteinterpreter'
+```
+
+## Examples
+In this example, we're trying to mod an old RPG named *Last Illusion VII*. We
+have the binary file that contains the game's spells, and we know the format
+for those spells is something like:
+1. Unsigned 8-bit integer - Spell element
+2. String, 20 characters - Spell name
+3. Unsigned 16-bit integer - Spell damage
+4. String, 50 characters - Spell description
+5. Signed 8-bit integer - Spell speed
+
+So how do we extract the information from this binary file?
+
+### Reading bit by bit
+The simplest way is to make a new instance of ByteInterpreter, load the binary
+file, and read the information bit by bit with `#interpret_bytes` and
+`#interpret_string`.
+```ruby
+require 'byteinterpreter'
+
+f = File.open("SPELLS.BIN", "rb")
+bi = ByteInterpreter.new(stream: f)
+
+# Interpret the first spell's element
+element = bi.interpret_bytes(size: 1, signed: false)
+# => 1
+
+# Interpret the first spell's name
+name = bi.interpret_string(size: 20)
+# => "Fireball           "
+```
+These two methods can read most simple structures in binary files. Take care
+to read values as `signed` when appropriate! For instance, in our example format,
+spell speed is signed, so we'd read it thus:
+```ruby
+bi.interpret_bytes(size: 1, signed: true)
+# => -50
+```
+Technically speaking, nothing truly horrid will happen if you forget, but your
+data will be inaccurate.
+
+The other method of reading bytes from a binary file is to use Instructions.
+
+### Using Instructions
+In order to use Instructions, you must first load them. As of writing this
+README, ByteInterpreter only knows how to load Instructions from JSON. You can
+load JSON instructions with the `#load_instructions` method:
+```ruby
+bi.load_instructions(type: :json, filename: "spell_instructions.json")
+```
+The expected format of a JSON file for Instructions is an array containing
+objects, one object for each field we'll be reading. The objects should each
+have the following attributes: `"key"`, `"type"`, `"size"`, and `"signed"`.
+* `key` - The name of the field we're reading (e.g. "Element" or "Name")
+* `type` - The type of the field, either "bin" for binary values or "str" for
+  strings.
+* `size` - How many bytes the field is. Binary values can only be 1, 2, 4, or
+  8 bytes, but string values can be as short or long as you wish.
+* `signed` - For binary values, if the resulting integer is signed or not.
+  Totally ignored for strings.
+
+So a JSON Instructions file for our example might look like this:
+```json
+[
+  {
+    "key": "element", "type": "bin", "size": 1, "signed": false
+  },
+  {
+    "key": "name", "type": "str", "size": 20, "signed": false
+  },
+  {
+    "key": "power", "type": "bin", "size": 2, "signed": false
+  },
+  {
+    "key": "description", "type": "str", "size": 50, "signed": false
+  },
+  {
+    "key": "speed", "type": "bin", "size": 1, "signed": true
+  }
+]
+```
+If your Instructions are incorrectly formatted, never fear: ByteInterpreter
+will raise a `ValidationError` and let you know what you did wrong. Also note:
+it's very important that the objects in the array are in the same order as the
+fields in the structure you're reading, as it will read the binary file in the
+exact same order as specified in the JSON array. The validation methods won't
+catch this, as ByteInterpreter has no idea what format your file is supposed to
+be in.
+
+Once you load these instructions, using them is very easy. We use the
+`#interpret_from_instructions` method, which takes a block and passes the
+key and interpreted value of each individual instruction to the given block.
+```ruby
+spell = {}
+bi.load_instructions(type: :json, filename: "spell_instructions.json")
+bi.interpret_from_instructions do |key, value|
+  spell[key] = value
+end
+
+spell.inspect
+# => {:element => 1, :name => "Fireball           ", etc.
+```
+_**Note:** You may have noticed in the examples, but ByteInterpreter will
+preserve any whitespace given in strings. It makes no assumptions about what
+you want to do with the interpreted string, so if you want to remove any
+whitespace you will have to call `#chomp` yourself._
+
+Reading the entire spell file would just involve wrapping your call to
+`#interpret_from_instructions` inside a loop of some sort:
+```ruby
+spellbook = Array.new
+30.times do
+  spell = {}
+  bi.interpret_from_instructions do |key, value|
+    spell[key] = value
+  end
+  spellbook.push(spell)
+end
+```
+
+### Writing bytes
+Writing bytes is almost as easy as reading them.
+```ruby
+f = File.new("NEW_SPELLS.BIN", "wb")
+bi = ByteInterpreter.new(stream: f)
+
+bi.encode_bytes(value: 1, size: 1, signed: false)
+bi.encode_string(value: "Fireball", size: 20)
+```
+These methods work about how you expect them to. `#encode_bytes` makes no
+attempt to ensure the value you're passing to it actually fits into the given
+size in bytes, so you should probably do some checking before encoding.
+Thankfully, `#encode_strings` is friendlier and will either pad or truncate
+your string to fit the appropriate size.
+
+And of course, encoding works with instructions as well.
+```ruby
+new_spell = {element: 1, name: "Fireball", etc.}
+bi.encode_from_instructions(values: new_spell)
+```
+`#encode_from_instructions` takes only one argument, a `Hash` with keys
+matching each field key in the Instructions file, and values paired with those
+keys that match the type for that field in the Instructions file. It is
+**very** important that your given `Hash` contains one key for each field;
+otherwise `#encode_from_instructions` will attempt to read a nonexistant key.
+
+If you'd like to write your own Instructions, see the
+[instructions.rb file](./lib/byte_interpreter/instructions.rb), it has a fairly
+in-depth explanation on doing so.

data/lib/byteinterpreter/instructions.rb

@@ -0,0 +1,110 @@
+# frozen_string_literal: true
+
+require "json"
+
+class ByteInterpreter
+  ##
+  # The Instructions class represents a collection of ordered operations to
+  # perform on an IO stream. This class is used by ByteInterpreter to either
+  # interpret or encode bytes in a rigid, structured way.
+  #
+  # At the most basic level, Instructions are just an Array filled with Hashes,
+  # each Hash having exactly four keys -- :key, :type, :size, and :signed. Each
+  # key has requirements for its value:
+  # - :key -- Must be a value easily convertible to a Symbol object.
+  # - :type -- Must match one of the elements in the constant VALID_TYPES.
+  # - :size -- For binary types ("bin"), must match one of the elements in the
+  #   constant VALID_BIN_SIZES. String types ("str") must be a
+  #   positive Integer.
+  # - :signed -- For binary types ("bin"), must be the +true+ or +false+
+  #   literals. String types ("str") ignore this value completely.
+  #
+  # Writing your own method for loading instructions is fairly simple. The
+  # method must call #add_field in the desired order of instruction execution,
+  # passing to it a Hash that conforms to the requirements above. The method
+  # must also be named +load_from_type+, where +type+ is what will be
+  # passed into ByteInterpreter#load_instructions. See #load_from_json for an
+  # example of this.
+  class Instructions
+    ##
+    # Raised by instruction validation methods.
+    class ValidationError < StandardError
+    end
+
+    ##
+    # Valid values for the :type key in the instructions Hash.
+    VALID_TYPES = %w[bin str].freeze
+
+    ##
+    # Valid values for binary types for the :size key in the instructions Hash.
+    VALID_BIN_SIZES = [1, 2, 4, 8].freeze
+
+    ##
+    # Keys that are in every properly-formatted instructions Hash.
+    FIELD_NAMES = %i[key type size signed].freeze
+
+    ##
+    # Creates a blank Instructions object.
+    def initialize
+      @data = []
+    end
+
+    ##
+    # Passes the given block to the internal Array's #each method.
+    def each(&block)
+      @data.each(&block)
+    end
+
+    ##
+    # Clears all loaded instructions.
+    def clear
+      @data.clear
+    end
+
+    ##
+    # Adds the given Hash to the end of the list of instructions, and validates
+    # it.
+    # @param field [Hash] A properly-formatted instructions Hash. See the
+    #   documentation for this class on the appropriate format for this Hash.
+    # @return [void]
+    def add_field(field:)
+      @data.push(field.select { |k, _v| FIELD_NAMES.include? k })
+      validate_field(field: @data.last)
+    end
+
+    ##
+    # Loads instructions from a JSON file. The JSON file should contain a
+    # top-level array, with each element being an object with the appropriate
+    # keys and values. Keys are automatically converted from strings to
+    # symbols, but boolean values are not converted from strings to literals.
+    # @param filename [String] The filename of the JSON file to load,
+    #   **including** the filetype extension, if any.
+    # @return [void]
+    def load_from_json(filename:)
+      json_fields = JSON.parse(File.open(filename, "rt", &:read), symbolize_names: true)
+      json_fields.each do |field|
+        add_field(field: field)
+      end
+    end
+
+    ##
+    # Validates a given Hash to ensure it conforms to the instruction format.
+    # @param field [Hash] The Hash object to evaluate.
+    # @return [Boolean]
+    # @raise [ValidationError] if the Hash does not conform to the instruction
+    #   format
+    def validate_field(field:)
+      unless VALID_TYPES.include? field[:type]
+        raise ValidationError, "Illegal type defined at key \"#{field[:key]}\": #{field[:type]}.
+        Valid types are #{VALID_TYPES}."
+      end
+
+      if (field[:type] == "bin") && !VALID_BIN_SIZES.include?(field[:size])
+        raise ValidationError, "Illegal size defined for binary field at key \"#{field[:key]}\": #{field[:size]}.
+        Valid sizes for binary values are #{VALID_BIN_SIZES}."
+      end
+
+      true
+    end
+  end
+end

data/lib/byteinterpreter.rb

@@ -0,0 +1,227 @@
+# frozen_string_literal: true
+
+require_relative "byteinterpreter/instructions.rb"
+
+##
+# The ByteInterpreter is a tool used to extract bytes and strings from a binary
+# file, and also to encode bytes and strings into a binary file. It can also
+# take a series of instructions to extract or encode data in an ordinal manner,
+# suitable for writing binary files with rigid structure size requirements.
+class ByteInterpreter
+  ##
+  # Reads the endian mode being used by the interpreter.
+  attr_reader :endian_mode
+
+  ##
+  # Creates and sets up a new ByteInterpreter.
+  # @param endian [:little, :big, nil] Default for this value is nil. The
+  #   endian mode that will be used by the interpreter for reading/writing
+  #   bytes. If nil is specified, the interpreter will assume machine-native
+  #   endianness.
+  # @param stream [#read, #write] The IO stream, or IO-like object, that the
+  #   interpreter will perform operations on. The interpreter will not open or
+  #   close the stream for you, and assumes you have already changed the
+  #   position to the appropriate offset for its operations. The interpreter
+  #   also assumes you have opened the stream as binary (as opposed to text),
+  #   and for the appropriate operations (read/write).
+  def initialize(endian: nil, stream:)
+    @endian_mode = endian
+    @instructions = nil
+    @iostream = stream
+  end
+
+  ##
+  # Changes the stream being used by the interpreter for operations.
+  # @param new_stream [#read, #write] The IO stream, or IO-like object, that
+  #   the interpreter will perform operations on. See #new for what is expected
+  #   of this stream.
+  # @return [void]
+  def iostream=(new_stream)
+    raise ArgumentError "Object given is not stream-like." unless stream_like?(obj: new_stream)
+    @iostream = new_stream
+  end
+
+  ##
+  # Reads a set number of bytes, interprets them into an integer, and returns
+  # the result.
+  # @param size [1, 2, 4, 8] The number of bytes to read from the stream.
+  #   ByteInterpreter can only interpret 8-, 16-, 32-, and 64-bit values at
+  #   this time, so this parameter is limited to just a few numbers.
+  # @param signed [Boolean] Default for this value is false. Set this to
+  #   true if the bytes being read can be negative or positive.
+  # @return [Integer] the interpreted byte
+  def interpret_bytes(size: 2, signed: false)
+    bytes = @iostream.read(size)
+    directive = build_directive(size: size, signed: signed)
+
+    bytes.unpack(directive).first
+  end
+
+  ##
+  # Reads a set number of bytes, interprets them into a string, and returns the
+  # result.
+  # @param size [Integer] The number of bytes to read from the stream. Unlike
+  #   #interpret_bytes, this size can be any positive integer.
+  # @return [String] the interpreted string
+  def interpret_string(size:)
+    @iostream.read(size)
+  end
+
+  ##
+  # Writes a set number of bytes, encoded from the given value.
+  # @param value [Integer] The value to encode and write.
+  # @param size [1,2,4,8] The size of the value in bytes.
+  # @param signed [Boolean] Set this to true if the bytes being written can be
+  #   negative and positive, false otherwise.
+  # @return [void]
+  # @note The interpreter makes no attempt to ensure that your +value+ fits
+  #   into +size+ bytes. To avoid unintended behavior, you should validate your
+  #   input into this method.
+  def encode_bytes(value:, size:, signed:)
+    value = Array(value) unless value.respond_to? :pack
+    @iostream.write(value.pack(build_directive(size: size, signed: signed)))
+  end
+
+  ##
+  # Writes a string into a given number of bytes.
+  # @param value [String] The value to write to the stream.
+  # @param size [Integer] The size of the value in bytes. Unlike
+  #   #encode_bytes, this size can be any positive integer.
+  # @return [void]
+  # @note If +value+ is smaller than +size+, the interpreter will pad +value+
+  #   with 0x20 to fill the remaining space. Even so, care should be taken to
+  #   validate your input to this method, especially if you want to handle
+  #   strings that are larger than +size+, or want to handle size differences
+  #   differently than this method.
+  def encode_string(value:, size:)
+    @iostream.write(value.slice(0, size).ljust(size, "\x20"))
+  end
+
+  ##
+  # Loads instructions from a file for structured, ordinal operations.
+  # @param type [Symbol] The type of the file that holds the instructions.
+  #   This argument **must** have a corresponding method in the
+  #   ByteInterpreter::Instructions class, named +load_from_type+, replacing
+  #   +type+ with the actual name of the filetype.
+  # @param filename [String] The filename of the instructions to load.
+  # @return [void]
+  # @note ByteInterpreter comes with only one +type+ built-in: JSON.
+  def load_instructions(type:, filename:)
+    @instructions = Instructions.new if @instructions.nil?
+    @instructions.clear
+    @instructions.send("load_from_" + type.to_s, filename: filename)
+  end
+
+  ##
+  # Uses the loaded instructions (you did call #load_instructions first,
+  # right?) to interpret bytes and strings from the stream, passing them as
+  # arguments to the given block.
+  # @yieldparam key [Symbol] The key of the interpreted data. Typically used to
+  #   set variables in the calling object.
+  # @yieldparam value [Integer, String] The value of the interpreted data.
+  # @return [Integer] the combined size, in bytes, of the operation
+  def interpret_from_instructions
+    struct_size = 0
+    @instructions.each do |field|
+      if field[:type] == "bin"
+        value = interpret_bytes(size: field[:size], signed: field[:signed])
+      elsif field[:type] == "str"
+        value = interpret_string(size: field[:size])
+      end
+
+      struct_size += field[:size]
+
+      yield field[:key], value
+    end
+
+    struct_size
+  end
+
+  ##
+  # Uses the loaded instructions (you did call #load_instructions first,
+  # right?) to encode the given values into bytes and strings, and write them
+  # to the stream.
+  #
+  # This method encodes and writes bytes in the order of the loaded
+  # instructions; this means it will seek each key from the given Hash, instead
+  # of seeking around the file and writing in whatever order the Hash may be
+  # in.
+  # @param values [Hash] The values to read and encode. This Hash **must**
+  #   have keys that match *all* keys from the loaded instructions.
+  # @return [Integer] the combined size, in bytes, of the operation
+  def encode_from_instructions(values:)
+    struct_size = 0
+    @instructions.each do |field|
+      key = field[:key].to_sym
+
+      if field[:type] == "bin"
+        encode_bytes(value: values[key], size: field[:size], signed: field[:signed])
+      elsif field[:type] == "str"
+        encode_string(value: values[key], size: field[:size])
+      end
+
+      struct_size += field[:size]
+    end
+
+    struct_size
+  end
+
+  private
+
+  ##
+  # This constant maps byte lengths to their respective Strings for the
+  # directives in Array#pack and String#unpack.
+  DIRECTIVE_SIZES = { 1 => "C", 2 => "S", 4 => "L", 8 => "Q" }.freeze
+
+  ##
+  # Uses DIRECTIVE_SIZES to translate a byte length to a usable String.
+  # @param size [1,2,4,8] The byte length to translate.
+  # @raise [ArgumentError] if +size+ is not 1, 2, 4, or 8.
+  # @return [String] the translated directive String.
+  def determine_directive_letter(size:)
+    raise ArgumentError "Invalid size argument (#{size})." unless DIRECTIVE_SIZES.key?(size)
+    DIRECTIVE_SIZES[size].dup
+  end
+
+  ##
+  # Returns the glyph for the set endianness, for use in building the directive
+  # String.
+  # @return [String] if endian_mode is non-nil
+  # @return [nil] if endian_mode is nil
+  def determine_endian_glyph
+    case endian_mode
+    when :little
+      "<"
+    when :big
+      ">"
+    else
+      ""
+    end
+  end
+
+  ##
+  # Builds a directive String, fit for use in Array#pack and String#unpack.
+  # @param size [1,2,4,8] The size to translate into a directive String.
+  # @param signed [Boolean] Set this to true if the bytes being written can be
+  #   negative and positive, false otherwise.
+  # @return [String] the built directive String
+  def build_directive(size:, signed:)
+    directive = determine_directive_letter(size: size)
+    directive.downcase! if signed
+
+    directive += determine_endian_glyph if "SsLlQqJjIi".include?(directive)
+
+    directive
+  end
+
+  ##
+  # Checks if the given object is stream-like -- that is, responds to #read
+  # and #write.
+  # @param obj [Object] The object to test.
+  # @return [Boolean]
+  # @note For fun, consider making an inverse of this method named
+  #   "illiterate?"
+  def stream_like?(obj:)
+    obj.respond_to?(:read) && obj.respond_to?(:write)
+  end
+end

metadata

@@ -0,0 +1,51 @@
+--- !ruby/object:Gem::Specification
+name: byteinterpreter
+version: !ruby/object:Gem::Version
+  version: 1.1.0
+platform: ruby
+authors:
+- Michael K Gremillion
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2018-09-13 00:00:00.000000000 Z
+dependencies: []
+description: |
+  The ByteInterpreter is a tool to interpret bytes from and encode bytes to
+  binary files. It can either do this piecemeal, or via a set of rigid,
+  ordered instructions that define a data structure.
+email: 
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE.md
+- README.md
+- lib/byteinterpreter.rb
+- lib/byteinterpreter/instructions.rb
+homepage: https://github.com/mkgremillion/ByteInterpreter
+licenses:
+- MIT
+metadata:
+  source_code_uri: https://github.com/mkgremillion/ByteInterpreter
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.7.6
+signing_key: 
+specification_version: 4
+summary: A tool to interpret bytes from and encode bytes to binary files.
+test_files: []