ruby-ole 1.2.7 → 1.2.8

data/ChangeLog

@@ -1,3 +1,8 @@
+== 1.2.8 / 2008-10-08
+
+- check in the new fixes to the mbat support.
+- update README to be a bit more useful.
+
 == 1.2.7 / 2008-08-12
 
 - Prepare Ole::Types::PropertySet for write support.

data/README

@@ -1,22 +1,104 @@
 = Introduction
 
-For now, see the docs for the Ole::Storage class.
+The ruby-ole library provides a variety of functions primarily for
+working with OLE2 structured storage files, such as those produced by
+Microsoft Office - eg *.doc, *.msg etc.
+
+= Example Usage
+
+Here are some examples of how to use the library functionality,
+categorised roughly by purpose.
+
+1. Reading and writing files within an OLE container
+
+   The recommended way to manipulate the contents is via the
+   "file_system" API, whereby you use Ole::Storage instance methods
+   similar to the regular File and Dir class methods.
+
+     ole = Ole::Storage.open('oleWithDirs.ole', 'rb+')
+     p ole.dir.entries('.') # => [".", "..", "dir1", "dir2", "file1"]
+     p ole.file.read('file1')[0, 25] # => "this is the entry 'file1'"
+     ole.dir.mkdir('newdir')
+
+2. Accessing OLE meta data
+
+   Some convenience functions are provided for (currently read only)
+   access to OLE property sets and other sources of meta data.
+
+     ole = Ole::Storage.open('test_word_95.doc')
+     p ole.meta_data.file_format # => "MSWordDoc"
+     p ole.meta_data.mime_type # => "application/msword"
+     p ole.meta_data.doc_author.split.first # => "Charles"
+
+3. Raw access to underlying OLE internals
+
+   This is probably of little interest to most developers using the
+   library, but for some use cases you may need to drop down to the
+   lower level API on which the "file_system" API is constructed,
+   which exposes more of the format details.
+
+   <tt>Ole::Storage</tt> files can have multiple files with the same name,
+   or with a slash in the name, and other things that are probably
+   strictly invalid. This API is the only way to access those files.
+
+   You can access the header object directly:
+
+     p ole.header.num_sbat # => 1
+     p ole.header.magic.unpack('H*') # => ["d0cf11e0a1b11ae1"]
+
+   You can directly access the array of all Dirent objects,
+   including the root:
+
+     p ole.dirents.length # => 5
+     puts ole.root.to_tree
+     # =>
+     - #<Dirent:"Root Entry">
+       |- #<Dirent:"\001Ole" size=20 data="\001\000\000\002\000...">
+       |- #<Dirent:"\001CompObj" size=98 data="\001\000\376\377\003...">
+       |- #<Dirent:"WordDocument" size=2574 data="\334\245e\000-...">
+       \- #<Dirent:"\005SummaryInformation" size=54788 data="\376\377\000\000\001...">
+
+   You can access (through RangesIO methods, or by using the
+   relevant Dirent and AllocationTable methods) information like where within
+   the container a stream is located (these are offset/length pairs):
+
+     p ole.root["\001CompObj"].open { |io| io.ranges } # => [[0, 64], [64, 34]]
+
+See the documentation for each class for more details.
+
+= Thanks
+
+* The code contained in this project was initially based on chicago's libole
+  (source available at http://prdownloads.sf.net/chicago/ole.tgz).
+
+* It was later augmented with some corrections by inspecting pole, and (purely
+  for header definitions) gsf.
+
+* The property set parsing code came from the apache java project POIFS.
+
+* The excellent idea for using a pseudo file system style interface by providing
+  #file and #dir methods which mimic File and Dir, was borrowed (along with almost
+  unchanged tests!) from Thomas Sondergaard's rubyzip.
 
 = TODO
 
-== 1.2.8
+== 1.2.9
 
-* fix property sets a bit more. see TODO in Ole::Storage::MetaData
+* add buffering to rangesio so that performance for small reads and writes
+  isn't so awful. maybe try and remove the bottlenecks of unbuffered first
+  with more profiling, then implement the buffering on top of that.
 * fix mode strings - like truncate when using 'w+', supporting append
   'a+' modes etc. done?
 * make ranges io obey readable vs writeable modes.
 * more RangesIO completion. ie, doesn't support #<< at the moment.
-* ability to zero out padding and unused blocks
-* case insensitive mode for ole/file_system?
 
 == 1.3.1
 
-* fix this README :). maybe move todo out, and put something useful here.
+* fix property sets a bit more. see TODO in Ole::Storage::MetaData
+* ability to zero out padding and unused blocks
+* case insensitive mode for ole/file_system?
+* better tests for mbat support.
+* further doc cleanup
 
 == Longer term
 
@@ -25,3 +107,4 @@ For now, see the docs for the Ole::Storage class.
   ole implementations (maybe perl's, and poifs) just to check its in the
   ballpark, with no remaining silly bottlenecks.
 * supposedly vba does something weird to ole files. test that.
+

data/lib/ole/ranges_io.rb

@@ -217,8 +217,9 @@ end
 # this subclass of ranges io explicitly ignores the truncate part of 'w' modes.
 # only really needed for the allocation table writes etc. maybe just use explicit modes
 # for those
-# better yet write a test that breaks before I fix it.
-class RangesIONonResizeable < RangesIO
+# better yet write a test that breaks before I fix it. added nodoc for the 
+# time being.
+class RangesIONonResizeable < RangesIO # :nodoc:
 	def initialize io, mode='r', params={}
 		mode, params = 'r', mode if Hash === mode
 		flags = IO::Mode.new(mode).flags & ~IO::TRUNC

data/lib/ole/storage/base.rb

@@ -5,54 +5,8 @@ require 'ole/types'
 require 'ole/ranges_io'
 
 module Ole # :nodoc:
-	# 
-	# = Introduction
 	#
-	# <tt>Ole::Storage</tt> is a class intended to abstract away details of the
-	# access to OLE2 structured storage files, such as those produced by
-	# Microsoft Office, eg *.doc, *.msg etc.
-	#
-	# = Usage
-	#
-	# Usage should be fairly straight forward:
-	#
-	#   # get the parent ole storage object
-	#   ole = Ole::Storage.open 'myfile.msg', 'r+'
-	#   # => #<Ole::Storage io=#<File:myfile.msg> root=#<Dirent:"Root Entry">>
-	#   # read some data
-	#   ole.root[1].read 4
-	#   # => "\001\000\376\377"
-	#   # get the top level root object and output a tree structure for
-	#   # debugging
-	#   puts ole.root.to_tree
-	#   # =>
-	#   - #<Dirent:"Root Entry" size=3840 time="2006-11-03T00:52:53Z">
-	#     |- #<Dirent:"__nameid_version1.0" size=0 time="2006-11-03T00:52:53Z">
-	#     |  |- #<Dirent:"__substg1.0_00020102" size=16 data="CCAGAAAAAADAAA...">
-	#     ...
-	#     |- #<Dirent:"__substg1.0_8002001E" size=4 data="MTEuMA==">
-	#     |- #<Dirent:"__properties_version1.0" size=800 data="AAAAAAAAAAABAA...">
-	#     \- #<Dirent:"__recip_version1.0_#00000000" size=0 time="2006-11-03T00:52:53Z">
-	#        |- #<Dirent:"__substg1.0_0FF60102" size=4 data="AAAAAA==">
-	#   	 ...
-	#   # write some data, and finish up (note that open is 'r+', so this overwrites
-	#   # but doesn't truncate)
-	#   ole.root["\001CompObj"].open { |f| f.write "blah blah" }
-	#   ole.close
-	#
-	# = Thanks
-	#
-	# * The code contained in this project was initially based on chicago's libole
-	#   (source available at http://prdownloads.sf.net/chicago/ole.tgz).
-	#
-	# * It was later augmented with some corrections by inspecting pole, and (purely
-	#   for header definitions) gsf.
-	#
-	# * The property set parsing code came from the apache java project POIFS.
-	#
-	# * The excellent idea for using a pseudo file system style interface by providing
-	#   #file and #dir methods which mimic File and Dir, was borrowed (along with almost
-	#   unchanged tests!) from Thomas Sondergaard's rubyzip.
+	# This class is the primary way the user interacts with an OLE storage file.
 	#
 	# = TODO
 	#
@@ -67,7 +21,7 @@ module Ole # :nodoc:
 		class FormatError < StandardError # :nodoc:
 		end
 
-		VERSION = '1.2.7'
+		VERSION = '1.2.8'
 
 		# options used at creation time
 		attr_reader :params
@@ -81,8 +35,8 @@ module Ole # :nodoc:
 		# Low level internals, you probably shouldn't need to mess with these
 		attr_reader :header, :bbat, :sbat, :sb_file
 
-		# maybe include an option hash, and allow :close_parent => true, to be more general.
-		# +arg+ should be either a file, or an +IO+ object, and needs to be seekable.
+		# +arg+ should be either a filename, or an +IO+ object, and needs to be seekable.
+		# +mode+ is optional, and should be a regular mode string.
 		def initialize arg, mode=nil, params={}
 			params, mode = mode, nil if Hash === mode
 			params = {:update_timestamps => true}.merge(params)
@@ -118,6 +72,8 @@ module Ole # :nodoc:
 			@io.size > 0 ? load : clear
 		end
 
+		# somewhat similar to File.open, the open class method allows a block form where
+		# the Ole::Storage object is automatically closed on completion of the block.
 		def self.open arg, mode=nil, params={}
 			ole = new arg, mode, params
 			if block_given?
@@ -150,8 +106,13 @@ module Ole # :nodoc:
 
 			# create an empty bbat.
 			@bbat = AllocationTable::Big.new self
-			mbat_blocks = (0...@header.num_mbat).map { |i| i + @header.mbat_start }
-			bbat_chain = (header_block[Header::SIZE..-1] + @bbat.read(mbat_blocks)).unpack 'V*'
+			bbat_chain = header_block[Header::SIZE..-1].unpack 'V*'
+			mbat_block = @header.mbat_start
+			@header.num_mbat.times do
+				blocks = @bbat.read([mbat_block]).unpack 'V*'
+				mbat_block = blocks.pop
+				bbat_chain += blocks
+			end
 			# am i using num_bat in the right way?
 			@bbat.load @bbat.read(bbat_chain[0, @header.num_bat])
 	
@@ -268,7 +229,7 @@ module Ole # :nodoc:
 				# mbat must remain contiguous.
 				bbat_data_len = ((@bbat.length + num_mbat_blocks) * 4 / @bbat.block_size.to_f).ceil * @bbat.block_size
 				# now storing the excess mbat blocks also increases the size of the bbat:
-				new_num_mbat_blocks = ([bbat_data_len / @bbat.block_size - 109, 0].max * 4 / @bbat.block_size.to_f).ceil
+				new_num_mbat_blocks = ([bbat_data_len / @bbat.block_size - 109, 0].max * 4 / (@bbat.block_size.to_f - 4)).ceil
 				if new_num_mbat_blocks != num_mbat_blocks
 					# need more space for the mbat.
 					num_mbat_blocks = new_num_mbat_blocks
@@ -284,13 +245,16 @@ module Ole # :nodoc:
 			# now extract the info we want:
 			ranges = io.ranges
 			bbat_chain = @bbat.chain io.first_block
-			# the extra mbat data is a set of contiguous blocks at the end
 			io.close
 			bbat_chain.each { |b| @bbat[b] = AllocationTable::BAT }
 			# tack on the mbat stuff
-			@header.mbat_start = @bbat.length # need to record this here before tacking on the mbat
 			@header.num_bat = bbat_chain.length
-			num_mbat_blocks.times { @bbat << AllocationTable::META_BAT }
+			mbat_blocks = (0...num_mbat_blocks).map do
+				block = @bbat.free_block
+				@bbat[block] = AllocationTable::META_BAT
+				block
+			end
+			@header.mbat_start = mbat_blocks.first || AllocationTable::EOC
 
 			# now finally write the bbat, using a not resizable io.
 			# the mode here will be 'r', which allows write atm. 
@@ -299,15 +263,17 @@ module Ole # :nodoc:
 			# this is the mbat. pad it out.
 			bbat_chain += [AllocationTable::AVAIL] * [109 - bbat_chain.length, 0].max
 			@header.num_mbat = num_mbat_blocks
-			if num_mbat_blocks == 0
-				@header.mbat_start = AllocationTable::EOC
-			else
+			if num_mbat_blocks != 0
 				# write out the mbat blocks now. first of all, where are they going to be?
 				mbat_data = bbat_chain[109..-1]
-				q = @bbat.block_size / 4
-				mbat_data += [AllocationTable::AVAIL] *((mbat_data.length / q.to_f).ceil * q - mbat_data.length)
-				ranges = @bbat.ranges((0...num_mbat_blocks).map { |i| @header.mbat_start + i })
-				RangesIO.open(@io, :ranges => ranges) { |f| f.write mbat_data.pack('V*') }
+				# expand the mbat_data to include the linked list forward pointers.
+				mbat_data = mbat_data.to_enum(:each_slice, @bbat.block_size / 4 - 1).to_a.
+					zip(mbat_blocks[1..-1] + [nil]).map { |a, b| b ? a + [b] : a }
+				# pad out the last one.
+				mbat_data.last.push(*([AllocationTable::AVAIL] * (@bbat.block_size / 4 - mbat_data.last.length)))
+				RangesIO.open @io, :ranges => @bbat.ranges(mbat_blocks) do |f|
+					f.write mbat_data.flatten.pack('V*')
+				end
 			end
 
 			# now seek back and write the header out
@@ -561,7 +527,7 @@ module Ole # :nodoc:
 				length - 1
 			end
 
-			# must return first_block
+			# must return first_block. modifies +blocks+ in place
 			def resize_chain blocks, size
 				new_num_blocks = (size / block_size.to_f).ceil
 				old_num_blocks = blocks.length

data/lib/ole/storage/file_system.rb

@@ -3,32 +3,11 @@
 #
 # This file intends to provide file system-like api support, a la <tt>zip/zipfilesystem</tt>.
 #
-# Ideally, this will be the recommended interface, allowing Ole::Storage, Dir, and
-# Zip::ZipFile to be used exchangably. It should be possible to write recursive copy using
-# the plain api, such that you can copy dirs/files agnostically between any of ole docs, dirs,
-# and zip files.
-#
-# = Usage
-#
-# Currently you can do something like the following:
-#
-#   Ole::Storage.open 'test.doc' do |ole|
-#     ole.dir.entries '/'         # => [".", "..", "\001Ole", "1Table", "\001CompObj", ...]
-#     ole.file.read "\001CompObj" # => "\001\000\376\377\003\n\000\000\377\377..."
-#   end
-#
-# = Notes
-#
-# <tt>Ole::Storage</tt> files can have multiple files with the same name,
-# or with / in the name, and other things that are probably invalid anyway.
-# This API is unable to access those files, but of course the core, low-
-# level API can.
-#
-# need to implement some more IO functions on RangesIO, like #puts, #print
-# etc, like AbstractOutputStream from zipfile.
-#
 # = TODO
 # 
+# - need to implement some more IO functions on RangesIO, like #puts, #print
+#   etc, like AbstractOutputStream from zipfile.
+#
 # - check Dir.mkdir, and File.open, and File.rename, to add in filename 
 #   length checks (max 32 / 31 or something).
 #   do the automatic truncation, and add in any necessary warnings.

data/lib/ole/storage/meta_data.rb

@@ -1,6 +1,6 @@
 require 'ole/types/property_set'
 
-module Ole	
+module Ole
 	class Storage
 		#
 		# The MetaData class is designed to be high level interface to all the
@@ -35,33 +35,37 @@ module Ole
 			FORMAT_MAP = {
 				'MSWordDoc' => :doc
 			}
-			
+
 			CLSID_EXCEL97 = Types::Clsid.parse "{00020820-0000-0000-c000-000000000046}"
 			CLSID_EXCEL95 = Types::Clsid.parse "{00020810-0000-0000-c000-000000000046}"
 			CLSID_WORD97  = Types::Clsid.parse "{00020906-0000-0000-c000-000000000046}"
 			CLSID_WORD95  = Types::Clsid.parse "{00020900-0000-0000-c000-000000000046}"
-      
-      CLSID_MAP = {
-      	CLSID_EXCEL97 => :xls,
-      	CLSID_EXCEL95 => :xls,
-      	CLSID_WORD97  => :doc,
-      	CLSID_WORD95  => :doc
-    	}
+
+			CLSID_MAP = {
+				CLSID_EXCEL97 => :xls,
+				CLSID_EXCEL95 => :xls,
+				CLSID_WORD97  => :doc,
+				CLSID_WORD95  => :doc
+			}
 
 			MIME_TYPES = {
 				:xls => 'application/vnd.ms-excel',
 				:doc => 'application/msword',
 				:ppt => 'application/vnd.ms-powerpoint',
-				:msg => 'application/vnd.ms-outlook' # not registered at IANA, but sdeems most common usage
+				# not registered at IANA, but seems most common usage
+				:msg => 'application/vnd.ms-outlook',
+				# this is my default fallback option. also not registered at IANA.
+				# file(1)'s default is application/msword, which is useless...
+				nil  => 'application/x-ole-storage'
 			}
-	
+
 			def initialize ole
 				@ole = ole
 			end
 
 			# i'm thinking of making file_format and mime_type available through
 			# #[], #each, and #to_h also, as calculated meta data (not assignable)
-			
+
 			def comp_obj
 				return {} unless dirent = @ole.root["\001CompObj"]
 				data = dirent.read
@@ -70,7 +74,7 @@ module Ole
 				# byte_order: 0xffe
 				# windows_version: 0x00000a03 (win31 apparently)
 				# marker: 0xffffffff
-				compobj_version, byte_order, windows_version, marker, clsid = 
+				compobj_version, byte_order, windows_version, marker, clsid =
 					data.unpack("vvVVa#{Types::Clsid::SIZE}")
 				strings = []
 				i = 28
@@ -84,7 +88,7 @@ module Ole
 				{:username => strings[0], :file_format => strings[1], :unknown => strings[2..-1]}
 			end
 			private :comp_obj
-			
+
 			def file_format
 				comp_obj[:file_format]
 			end
@@ -93,7 +97,7 @@ module Ole
 				# based on the CompObj stream contents
 				type = FORMAT_MAP[file_format]
 				return MIME_TYPES[type] if type
-				
+
 				# based on the root clsid
 				type = CLSID_MAP[Types::Clsid.load(@ole.root.clsid)]
 				return MIME_TYPES[type] if type
@@ -103,19 +107,21 @@ module Ole
 				return MIME_TYPES[:msg] if has_file['__nameid_version1.0'] or has_file['__properties_version1.0']
 				return MIME_TYPES[:doc] if has_file['worddocument'] or has_file['document']
 				return MIME_TYPES[:xls] if has_file['workbook'] or has_file['book']
+
+				MIME_TYPES[nil]
 			end
-	
+
 			def [] key
 				pair = Types::PropertySet::PROPERTY_MAP[key.to_s] or return nil
 				file = FILE_MAP[pair.first] or return nil
 				dirent = @ole.root[file] or return nil
 				dirent.open { |io| return Types::PropertySet.new(io)[key] }
 			end
-			
+
 			def []= key, value
 				raise NotImplementedError, 'meta data writes not implemented'
 			end
-			
+
 			def each(&block)
 				FILE_MAP.values.each do |file|
 					dirent = @ole.root[file] or next
@@ -126,7 +132,7 @@ module Ole
 			def to_h
 				inject({}) { |hash, (name, value)| hash.update name.to_sym => value }
 			end
-			
+
 			def method_missing name, *args, &block
 				return super unless args.empty?
 				pair = Types::PropertySet::PROPERTY_MAP[name.to_s] or return super

data/lib/ole/support.rb

@@ -89,7 +89,7 @@ end
 # breadth first iteration holds its own copy of the children around.
 #
 # Main methods are #recursive, and #to_tree
-module RecursivelyEnumerable
+module RecursivelyEnumerable # :nodoc:
 	def each_recursive_depth_first(&block)
 		each_child do |child|
 			yield child

data/test/test_meta_data.rb

@@ -32,7 +32,7 @@ class TestMetaData < Test::Unit::TestCase
 		
 		ole.root.clsid = 0.chr * Ole::Types::Clsid::SIZE
 		assert_equal nil, ole.meta_data.file_format
-		assert_equal nil, ole.meta_data.mime_type
+		assert_equal 'application/x-ole-storage', ole.meta_data.mime_type
 		
 		ole.file.open('Book', 'w') { |f| }
 		assert_equal 'application/vnd.ms-excel', ole.meta_data.mime_type

data/test/test_storage.rb

@@ -173,7 +173,8 @@ class TestStorageRead < Test::Unit::TestCase
 
 	def test_dirent
 		dirent = @ole.root.children.first
-		assert_equal '#<Dirent:"\001Ole" size=20 data="\001\000\000\002\000...">', dirent.inspect
+		assert_equal "\001Ole", dirent.name
+		assert_equal 20, dirent.size
 		assert_equal '#<Dirent:"Root Entry">', @ole.root.inspect
 		
 		# exercise Dirent#[]. note that if you use a number, you get the Struct

metadata

@@ -1,15 +1,15 @@
 --- !ruby/object:Gem::Specification 
 name: ruby-ole
 version: !ruby/object:Gem::Version 
-  version: 1.2.7
-platform: ""
+  version: 1.2.8
+platform: ruby
 authors: 
 - Charles Lowe
 autorequire: 
 bindir: bin
 cert_chain: []
 
-date: 2008-08-12 00:00:00 +10:00
+date: 2008-10-08 00:00:00 +11:00
 default_executable: 
 dependencies: []
 
@@ -27,29 +27,29 @@ files:
 - ChangeLog
 - data/propids.yaml
 - bin/oletool
+- lib/ole/support.rb
 - lib/ole/base.rb
+- lib/ole/storage.rb
 - lib/ole/file_system.rb
 - lib/ole/ranges_io.rb
 - lib/ole/storage/base.rb
 - lib/ole/storage/file_system.rb
 - lib/ole/storage/meta_data.rb
-- lib/ole/storage.rb
-- lib/ole/support.rb
+- lib/ole/types.rb
 - lib/ole/types/base.rb
 - lib/ole/types/property_set.rb
-- lib/ole/types.rb
+- test/test_types.rb
 - test/test_filesystem.rb
-- test/test_mbat.rb
+- test/test_support.rb
+- test/test_storage.rb
 - test/test_meta_data.rb
-- test/test_property_set.rb
 - test/test_ranges_io.rb
-- test/test_storage.rb
-- test/test_support.rb
-- test/test_types.rb
+- test/test_property_set.rb
+- test/test_mbat.rb
 - test/test.doc
 - test/test_word_6.doc
-- test/test_word_95.doc
 - test/test_word_97.doc
+- test/test_word_95.doc
 - test/oleWithDirs.ole
 - test/test_SummaryInformation
 has_rdoc: true
@@ -79,16 +79,16 @@ required_rubygems_version: !ruby/object:Gem::Requirement
 requirements: []
 
 rubyforge_project: ruby-ole
-rubygems_version: 0.9.5
+rubygems_version: 1.2.0
 signing_key: 
 specification_version: 2
 summary: Ruby OLE library.
 test_files: 
+- test/test_types.rb
 - test/test_filesystem.rb
-- test/test_mbat.rb
+- test/test_support.rb
+- test/test_storage.rb
 - test/test_meta_data.rb
-- test/test_property_set.rb
 - test/test_ranges_io.rb
-- test/test_storage.rb
-- test/test_support.rb
-- test/test_types.rb
+- test/test_property_set.rb
+- test/test_mbat.rb