nwn-lib 0.4.5 → 0.4.6

data/BINARIES

@@ -43,6 +43,6 @@ Usage is simple: just pass all old .yml files to the script, it will convert the
 in-place (read: make a backup!).
 
 This uses nwn-gff-convert and nwn-gff, so all environment variables are taken into
-consideration. You will want to point NWN_LIB_INFER_DATA_FILE to something useful.
+consideration.
 
 Backup and testing is ADVISED!

data/CHANGELOG

@@ -164,4 +164,42 @@ Bernhard Stoeckner (14):
       Scripting: ask: fix return value of hash and arrays
       TwoDA: add disclaimer for Cache references
       TwoDA: add Table#[], Table#[]=
-
+      0.4.5-rel
+
+=== 0.4.6
+Bernhard Stoeckner <elven@swordcoast.net> (26):
+      bin/nwn-dsl: always print backtraces
+      tests: add rspec stub for later test introduction
+      Field: complete valid_for?, add checks in readers, add rspec
+      Field: #path prints proper list indices
+      ResourceManager: generic implementation as per bioware docs
+      TwoDA: read 2da header with arbitary spaces between magic and version
+      TwoDA: make id mismatch parsing more robust
+      TwoDA: nwn does not fill in missing row IDs
+      TwoDA: fix typo preventing #to_2da with NWN_LIB_2DA_NEWLINE absent
+      TwoDA: only print to $stderr when -d/$DEBUG is set
+      TwoDA: do not raise error on misshaped ID column, add documentation
+      TwoDA: do not compact multiple nonquoted cells in last column
+      TwoDA: add specs
+      Erf: skip over lid and strz to read locstrs correctly
+      Erf: seek to offsets instead of depending on constants, handle IOErrors
+      Erf: add specs
+      Tlk: fix reading last strref in a tlk
+      Tlk: add specs
+      Cexolocstr: add specs
+      Struct: .new accepts a block
+      Struct: add specs
+      Gff: add specs
+      feature kill: remove type/value inferring, NWN_LIB_INFER_*
+      Revert "add env setting NWN_LIB_FILTER_EMPTY_EXOLOCSTR"
+      Rakefile/rdoc: use mislav-hanna if available
+      update all documentation to reflect current situation
+      0.4.6-rel
+
+      This release changes a few features; most prominent of all is the
+      infer mechanism - it is removed completely for sanity reasons.
+
+      Also, quite a few bugfixes found their way into this release, thanks
+      to the newly-added specs.
+
+      Please browse the commit log for a full list of changes.

data/CHEATSHEET

@@ -1,9 +1,23 @@
 ==== Convert all item files in the current directory to yaml and back
 
- for x in *.uti; do
-     nwn-gff-convert -i"$x" -o"$x.yml" -lg -ky
- done
+  for x in *.uti; do
+    nwn-gff -i"$x" -o"$x.yml"
+  done
 
- for x in *.uti.yml; do
-     nwn-gff-convert -i"$x" -o"$(basename $x .yml)" -ly -kg
- done
+  for x in *.uti.yml; do
+    nwn-gff -i"$x" -o"$(basename $x .yml)"
+  done
+
+==== A Makefile that rebuilds only changed resources
+
+This can be done with a GNU Makefile too, with the added benefit that
+it will keep track of changes automagically and only rebuild those resources
+that you have changed (saving you a lot of time).
+
+  yml = $(wildcard *.yml)
+  gff = $(basename $(yml))
+
+  all: $(gff)
+
+  % : %.yml
+      nwn-gff -i $< -o $@

data/README

@@ -4,36 +4,41 @@ This package provides a library for reading, changing, and writing common file f
 
 They should work with NWN2 just as well, since the file format specifications did not change.
 
-=== Upgrade from 0.3.6 to 0.4.x
 
-With the release of 0.4.0, the API changed significantly. Previous yaml dumps made with 0.3.x are INCOMPATIBLE, and so are all scripts.
-I can't help you with your API bindings, but for your YAML dumps, a converter script has been provided (see BINARIES).
+=== Features of nwn-lib
 
-=== A word on version numbers
+==== nwn-lib reads and writes the following file formats (hopefully bug-free!), both with an API and useful command-line tools:
 
-Please consider all version releases below 1.0.0 to be unstable, even though the code is fully functional - further API changes may not be avoidable between major revisions - such as is the case with 0.3 -> 0.4.
+* GFF 3.2 (are, git, gic, dlg, itp, ifo, jrl, fac, ssf, ut*, among others)
+* ERF (mod, hak, erf, among others)
+* 2DA V2.0
+* TLK
 
-=== Features of nwn-lib
+==== nwn-lib can read and write the following representations of GFF data:
 
-* a feature-complete parser and generator of valid GFF V3.2 data
-* shell scripts and tools to simplify your life (see BINARIES)
 * kivinen-style ("gffprint.pl") presentation of data
 * yaml presentation of data
 * ruby-native marshalling of gff data
+
+==== Also in the box:
+
+* shell scripts and tools to simplify your life (see BINARIES)
 * extensive developer API
 * a powerful get-out-of-my-way scripting system for data transformation (see SCRIPTING)
-* guessing of field-types and -values for shorter plaintext data dumps (see TYPE_AND_VALUE_INFERRING)
 
-Also in the box:
 
-* a CHEATSHEET showing off some cool tricks
-* some gadgetry for working with 2da files
+=== Upgrade from 0.3.6 to 0.4.x
+
+With the release of 0.4.0, the API changed significantly. Previous yaml dumps made with 0.3.x are INCOMPATIBLE, and so are all scripts.
+I can't help you with your API bindings, but for your YAML dumps, a converter script has been provided (see BINARIES).
+
 
 === Attention Unicode/UTF-users
 
 ruby 1.8 does not support character sets natively, and as such nwn-gff-irb will <b>FAIL</b> to encode non-standard characters properly on non-latin1 shells.
 This will be worked around in a future release until the release of ruby 1.9, which will provide native charset support.
 
+
 === Quickstart
 
 To use it, simply install the gem (available on rubyforge):
@@ -45,4 +50,16 @@ And do the following in a script of your own devising:
   require 'rubygems'
   require 'nwn/all'
 
+Also, Read BINARIES.
+
 For nwn-lib scripts, see SCRIPTING.
+
+For using the developer API, I suggest you start reading NWN::Gff::Struct.
+
+=== Latest code & developer contact
+
+The latest source is available through git[http://git.swordcoast.net/?p=nwn/nwn-lib.git;a=summary].
+
+All stable releases are tagged properly.
+
+You can reach me via email[mailto:elven@swordcoast.net].

data/Rakefile

@@ -1,7 +1,7 @@
 require "rake"
 require "rake/clean"
 require "rake/gempackagetask"
-require "rake/rdoctask"
+require "hanna/rdoctask" rescue require "rake/rdoctask"
 require "fileutils"
 include FileUtils
 
@@ -9,13 +9,13 @@ include FileUtils
 # Configuration
 ##############################################################################
 NAME = "nwn-lib"
-VERS = "0.4.5"
+VERS = "0.4.6"
 CLEAN.include ["**/.*.sw?", "pkg", ".config", "rdoc", "coverage"]
 RDOC_OPTS = ["--quiet", "--line-numbers", "--inline-source", '--title', \
   'nwn-lib: a ruby library for accessing NWN resource files', \
   '--main', 'README']
 
-DOCS = ["README", "BINARIES", "DATA_STRUCTURES", "SCRIPTING", "SETTINGS", "TYPE_VALUE_INFERRING", "CHEATSHEET", "CHANGELOG", "COPYING"]
+DOCS = ["README", "BINARIES", "SCRIPTING", "SETTINGS", "CHEATSHEET", "CHANGELOG", "COPYING"]
 
 Rake::RDocTask.new do |rdoc|
   rdoc.rdoc_dir = "rdoc"

data/SETTINGS

@@ -1,82 +1,16 @@
 There are various environment variables you can configure to adjust various parts of nwn-lib.
 
-<b>All of them are optional!</b>
+<b>All of them are optional, and, in fact, can waste your time!</b>
 
 Under linux, just add them to your shell environment (usually .bashrc), like so:
 
-  export NWN_LIB_INFER_DATA_FILE=/usr/lib/ruby/gems/1.8/gems/nwn-lib-0.4.0/data/gff-common-nwn1.yaml
+  export NWN_LIB_FILTER_EMPTY_EXOLOCSTR=1
 
 
-== NWN_LIB_INFER_DATA_FILE
-
-The path to the type/value inferring data file. See TYPE_VALUE_INFERRING for a definition.
-
-Please be advised that inferring support for YAML is <b>EXPERIMENTAL</b> and may result in
-<b>module and gff file corruption</b>.
-
-== NWN_LIB_DONT_COMPACT_FIELDS
-
-Set to non-nil ("1" will do) to prevent nwn-lib from compacting scalar fields into a more-readable
-(but still fully parseable) format, if type inferring data is available.
-
-=== Without compacting
-  ItemList:
-    value:
-    - !nwn-lib.elv.es,2008-12/struct
-      __data_type: UTC/ItemList
-      __struct_id: 0
-      InventoryRes: {value: herb053}
-  PaletteID: {value: 6}
-  SkillList: [{value: 0}, {value: 0}, {value: 0}, {value: 0}, ...
-
-=== With compacting
-  ItemList:
-  - !nwn-lib.elv.es,2008-12/struct
-    __data_type: UTC/ItemList
-    InventoryRes: herb053
-  PaletteID: 6
-  SkillList: [0, 0, 0, 0, 0, 0, 2, 0, 0, 0 ...
-
-
-== NWN_LIB_DONT_COMPACT_LIST_STRUCTS
-
-Setting this to non-nil ("1" will do) will prevent nwn-lib from compacting lists that contain
-structs, if type inferring data is available.
-
-=== Without compacting
-  SkillList:
-  - !nwn-lib.elv.es,2008-12/struct {__data_type: UTC/SkillList, Rank: 0}
-  - !nwn-lib.elv.es,2008-12/struct {__data_type: UTC/SkillList, Rank: 0}
-  - !nwn-lib.elv.es,2008-12/struct {__data_type: UTC/SkillList, Rank: 0}
-  - !nwn-lib.elv.es,2008-12/struct {__data_type: UTC/SkillList, Rank: 0}
-  - !nwn-lib.elv.es,2008-12/struct {__data_type: UTC/SkillList, Rank: 0}
-  - !nwn-lib.elv.es,2008-12/struct {__data_type: UTC/SkillList, Rank: 0}
-  - !nwn-lib.elv.es,2008-12/struct {__data_type: UTC/SkillList, Rank: 2}
-  - !nwn-lib.elv.es,2008-12/struct {__data_type: UTC/SkillList, Rank: 0}
-  ...
-
-=== With compacting
-  SkillList: [0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, ...
-
-
-== NWN_LIB_CLEAR_KNOWN_VALUES
-
-Set to non-nil ("1" will do) to make nwn-lib omit fields that resolve to default values
-as configured in $NWN_LIB_INFER_DATA_FILE.
-
-The default is to not omit known-value data fields.
-
-== NWN_LIB_FILTER_EMPTY_EXOLOCSTR
-
-Set to non-nil ("1" will do) to make nwn-lib filter out empty exolocstr fields from
-input files on reading.
-
-The default is to keep them as-is.
-
 == NWN_LIB_2DA_LOCATION
 
 Set to a path containing all 2da files to initialize the 2da cache. This is needed for
-most interactive helpers and a few type infer gizmos.
+most interactive helpers and a few gizmos, and, of course, for TwoDA::Cache.
 
 == NWN_LIB_TWODA_NEWLINE
 

data/bin/nwn-dsl

@@ -6,9 +6,6 @@ require 'nwn/all'
 $backtrace = false
 OptionParser.new do |o|
   o.banner = "Usage: nwn-dsl [options] <script> -- [arguments to script]"
-  o.on "-b", "--backtrace", "Show a backtrace on error" do
-    $backtrace = true
-  end
 end.parse!
 
 fail "Not enough arguments (try -h)." unless ARGV.size > 0
@@ -19,9 +16,7 @@ begin
   NWN::Gff::Scripting.run_script(IO.read($SCRIPT), nil, ARGV)
 rescue => e
   $stderr.puts e.message
-  if $backtrace
-    $stderr.puts ""
-    $stderr.puts "  " + e.backtrace.join("\n")
-  end
+  $stderr.puts ""
+  $stderr.puts "  " + e.backtrace.join("\n")
   exit 1
 end

data/lib/nwn/all.rb

@@ -1,4 +1,5 @@
 require 'nwn/twoda'
+require 'nwn/res'
 require 'nwn/gff'
 require 'nwn/tlk'
 require 'nwn/erf'

data/lib/nwn/erf.rb

@@ -4,123 +4,10 @@ module NWN
   module Erf
     ValidTypes = %w{ERF HAK MOD}
 
-    Extensions = {
-      'res' => 0,
-      'bmp' => 1,
-      'mve' => 2,
-      'tga' => 3,
-      'wav' => 4,
-      'wfx' => 5,
-      'plt' => 6,
-      'ini' => 7,
-      'mp3' => 8,
-      'mpg' => 9,
-      'txt' => 10,
-      'plh' => 2000,
-      'tex' => 2001,
-      'mdl' => 2002,
-      'thg' => 2003,
-      'fnt' => 2005,
-      'lua' => 2007,
-      'slt' => 2008,
-      'nss' => 2009,
-      'ncs' => 2010,
-      'mod' => 2011,
-      'are' => 2012,
-      'set' => 2013,
-      'ifo' => 2014,
-      'bic' => 2015,
-      'wok' => 2016,
-      '2da' => 2017,
-      'tlk' => 2018,
-      'txi' => 2022,
-      'git' => 2023,
-      'bti' => 2024,
-      'uti' => 2025,
-      'btc' => 2026,
-      'utc' => 2027,
-      'dlg' => 2029,
-      'itp' => 2030,
-      'btt' => 2031,
-      'utt' => 2032,
-      'dds' => 2033,
-      'bts' => 2034,
-      'uts' => 2035,
-      'ltr' => 2036,
-      'gff' => 2037,
-      'fac' => 2038,
-      'bte' => 2039,
-      'ute' => 2040,
-      'btd' => 2041,
-      'utd' => 2042,
-      'btp' => 2043,
-      'utp' => 2044,
-      'dft' => 2045,
-      'gic' => 2046,
-      'gui' => 2047,
-      'css' => 2048,
-      'ccs' => 2049,
-      'btm' => 2050,
-      'utm' => 2051,
-      'dwk' => 2052,
-      'pwk' => 2053,
-      'btg' => 2054,
-      'utg' => 2055,
-      'jrl' => 2056,
-      'sav' => 2057,
-      'utw' => 2058,
-      '4pc' => 2059,
-      'ssf' => 2060,
-      'hak' => 2061,
-      'nwm' => 2062,
-      'bik' => 2063,
-      'ndb' => 2064,
-      'ptm' => 2065,
-      'ptt' => 2066,
-      'bak' => 2067,
-      'osc' => 3000,
-      'usc' => 3001,
-      'trn' => 3002,
-      'utr' => 3003,
-      'uen' => 3004,
-      'ult' => 3005,
-      'sef' => 3006,
-      'pfx' => 3007,
-      'cam' => 3008,
-      'lfx' => 3009,
-      'bfx' => 3010,
-      'upe' => 3011,
-      'ros' => 3012,
-      'rst' => 3013,
-      'ifx' => 3014,
-      'pfb' => 3015,
-      'zip' => 3016,
-      'wmp' => 3017,
-      'bbx' => 3018,
-      'tfx' => 3019,
-      'wlk' => 3020,
-      'xml' => 3021,
-      'scc' => 3022,
-      'ptx' => 3033,
-      'ltx' => 3034,
-      'trx' => 3035,
-      'mdb' => 4000,
-      'mda' => 4001,
-      'spt' => 4002,
-      'gr2' => 4003,
-      'fxa' => 4004,
-      'fxe' => 4005,
-      'jpg' => 4007,
-      'pwc' => 4008,
-      'ids' => 9996,
-      'erf' => 9997,
-      'bif' => 9998,
-      'key' => 9999,
-    }.freeze
+    # This reads and writes NWN::Resources::Container
+    # as valid ERF binary data.
+    class Erf < NWN::Resources::Container
 
-    class Erf
-
-      attr_accessor :content
       attr_accessor :localized_strings
       attr_accessor :description_str_ref
 
@@ -131,74 +18,16 @@ module NWN
 
       # Create a new Erf object, optionally reading a existing file from +io+.
       def initialize io = nil
-        @content = []
+        super()
         @localized_strings = {}
         @io = io
         @file_type, @file_version = "ERF", "V1.0"
         @year = Time.now.year
         @description_str_ref = 0xffffffff
-        @day_of_year = Time.now.yday # strftime("%j").to_i
+        @day_of_year = Time.now.yday
         read_from io if io
       end
 
-      def add filename
-        @content << ContentObject.new_from(filename)
-      end
-
-      def has?(filename)
-        base = File.basename(filename)
-        @content.each {|f|
-          return true if f.filename.downcase == base.downcase
-        }
-        return false
-      end
-
-      class ContentObject
-        attr_accessor :resref
-        attr_accessor :res_type
-        attr_accessor :io
-        attr_accessor :offset
-        attr_accessor :size_override
-
-        def self.new_from filename
-          stat = File.stat(filename)
-          base = File.basename(filename).split(".")[0..-2].join(".")
-          ext = File.extname(filename)[1..-1]
-          res_type = NWN::Erf::Extensions[ext] or raise ArgumentError,
-            "Not a valid extension: #{ext.inspect} (while packing #{filename})"
-
-          ContentObject.new(base, res_type, filename, 0, stat.size)
-        end
-
-        def initialize resref, res_type, io = nil, offset = nil, size = nil
-          @resref, @res_type = resref, res_type
-          @io, @offset = io, offset
-          @size_override = size
-        end
-
-        # Get the size in bytes of this object.
-        def size
-          @size_override || (@io.is_a?(IO) ? @io.stat.size : File.stat(@io).size)
-        end
-
-        def get
-          if @io.is_a?(IO)
-            @io.seek(@offset) if @offset
-            @io.read(self.size)
-          else
-            IO.read(@io)
-          end
-        end
-
-        def filename
-          @resref + "." + self.extension
-        end
-
-        def extension
-          NWN::Erf::Extensions.index(@res_type)
-        end
-      end
-
     private
 
       def read_from io
@@ -219,42 +48,40 @@ module NWN
 
         if @file_version == "V1.0"
           @filename_length = 16
-#        elsif version == "V1.1"
-#          @filename_length = 32
         else
           raise IOError, "Invalid erf version: #{@file_version}"
         end
 
-        raise IOError, "key list not after locstr list" unless
-          offset_to_keys == offset_to_locstr + locstr_size
-
-        raise IOError, "Offset to locstr list is not after header" if
-          offset_to_locstr != 160
-
+        @io.seek(offset_to_locstr)
         locstr = @io.read(locstr_size)
-        raise IOError, "Cannot read locstr list" unless
-          locstr.size == locstr_size
 
         for lstr in 0...locstr_count do
+          raise IOError, "locstr table does not contain enough entries or locstr_size is too small" if
+            locstr.nil? || locstr.size < 8
+
           lid, strsz = locstr.unpack("V V")
-          str = locstr.unpack("a#{strsz}")[0]
-          $stderr.puts "Expected string size does not match actual string size (want: #{strsz}, got #{str.size} of #{str.inspect})" if
-            strsz != str.size
+          str = locstr.unpack("x8 a#{strsz}")[0]
+          raise IOError,
+            "Expected locstr size does not match actual string size (want: #{strsz}, got #{str.size} of #{str.inspect})" if
+              strsz != str.size
+
           @localized_strings[lid] = str
           locstr = locstr[8 + str.size .. -1]
-          raise IOError, "locstr table does not contain enough entries (want: #{locstr_count}, got: #{lstr + 1})" if locstr.nil? &&
-            lstr + 1 < locstr_count
         end
 
         keylist_entry_size = @filename_length + 4 + 2 + 2
+        @io.seek(offset_to_keys)
         keylist = @io.read(keylist_entry_size * entry_count)
+        raise IOError, "keylist too short" if keylist.size != keylist_entry_size * entry_count
         keylist = keylist.unpack("A16 V v v" * entry_count)
         keylist.each_slice(4) {|resref, res_id, res_type, unused|
-          @content << ContentObject.new(resref, res_type, @io)
+          @content << NWN::Resources::ContentObject.new(resref, res_type, @io)
         }
 
         resourcelist_entry_size = 4 + 4
+        @io.seek(offset_to_res)
         resourcelist = @io.read(resourcelist_entry_size * entry_count)
+        raise IOError, "resource list too short" if resourcelist.size != resourcelist_entry_size * entry_count
         resourcelist = resourcelist.unpack("I I" * entry_count)
         _index = -1
         resourcelist.each_slice(2) {|offset, size|