ruby-ole 1.2.9 → 1.2.10

data/ChangeLog

@@ -1,3 +1,13 @@
+== 1.2.10 / 2009-07-20
+
+- Mostly more performance enhancements, significantly faster for
+  certain operations.
+- Using lots of files is faster due to new hash lookup for dirents by name.
+- Writes of many files are faster now too as Dirent & FileTime serialization
+  has been improved.
+- Certain operations from the filesystem api have been profiled and sped up.
+- Don't use syswrite on jruby to avoid the buffered stream warnings.
+
 == 1.2.9 / 2009-07-14
 
 - Lots of performance enhancements for RangesIO.

data/README

@@ -82,8 +82,9 @@ See the documentation for each class for more details.
 
 = TODO
 
-== 1.2.9
+== 1.2.11
 
+* internal api cleanup
 * add buffering to rangesio so that performance for small reads and writes
   isn't so awful. maybe try and remove the bottlenecks of unbuffered first
   with more profiling, then implement the buffering on top of that.
@@ -98,6 +99,7 @@ See the documentation for each class for more details.
 
 == 1.3.1
 
+* case insensitive open mode would be nice
 * fix property sets a bit more. see TODO in Ole::Storage::MetaData
 * ability to zero out padding and unused blocks
 * case insensitive mode for ole/file_system?

data/Rakefile

@@ -73,7 +73,7 @@ end
 
 Rake::GemPackageTask.new(spec) do |t|
 	t.gem_spec = spec
-	t.need_tar = true
+	t.need_tar = false
 	t.need_zip = false
 	t.package_dir = 'build'
 end
@@ -114,96 +114,3 @@ task :benchmark do
 	end
 end
 
-=begin
-
-1.2.1:
-
-      user     system      total        real
-write_1mb_1x5 73.920000   8.400000  82.320000 ( 91.893138)
-
-revision 17 (speed up AllocationTable#free_block by using
-@sparse attribute, and using Array#index otherwise):
-
-      user     system      total        real
-write_1mb_1x5 57.910000   6.190000  64.100000 ( 66.207993)
-write_1mb_2x5266.310000  31.750000 298.060000 (305.877203)
-
-add in extra resize_chain fix (return blocks to avoid calling
-AllocationTable#chain twice):
-
-      user     system      total        real
-write_1mb_1x5 43.140000   5.480000  48.620000 ( 51.835942)
-
-add in RangesIOResizeable fix (cache @blocks, to avoid calling
-AllocationTable#chain at all when resizing now, just pass it
-to AllocationTable#resize_chain):
-
-      user     system      total        real
-write_1mb_1x5 29.770000   5.180000  34.950000 ( 39.916747)
-
-40 seconds is still a really long time to write out 5 megs.
-of course, this is all with a 1_000 byte block size, which is
-a very small wite. upping this to 100_000 bytes:
-
-      user     system      total        real
-write_1mb_1x5  0.540000   0.130000   0.670000 (  1.051862)
-
-so it seems that that makes a massive difference. so i really
-need buffering in RangesIO if I don't want it to really hurt
-for small writes, as all the resize code is kind of expensive.
-
-one of the costly things at the moment, is RangesIO#offset_and_size,
-which is called for each write, and re-finds which range we are in.
-that should obviously be changed, to a fixed one that is invalidated
-on seeks. buffering would hide that problem to some extent, but i
-should fix it anyway.
-
-re-running the original 1.2.1 with 100_000 byte block size:
-
-      user     system      total        real
-write_1mb_1x5 15.590000   2.230000  17.820000 ( 18.704910)
-
-so there the really badly non-linear AllocationTable#resize_chain is
-being felt. 
-
-back to current working copy, running full benchmark:
-
-      user     system      total        real
-write_1mb_1x5  0.530000   0.150000   0.680000 (  0.708919)
-write_1mb_2x5227.940000  31.260000 259.200000 (270.200960)
-
-not surprisingly, the second case hasn't been helped much by the fixes
-so far, as they only really help multiple resizes and writes for a file.
-this could be pain in the new file system code - potentially searching
-through Dirent#children at creation time.
-
-to test, i'll profile creating 1_000 files, without writing anything:
-
-      user     system      total        real
-write_1mb_2x5 16.990000   1.830000  18.820000 ( 19.900568)
-
-hmmm, so thats not all of it. maybe its the initial chain calls, etc?
-writing 1 byte:
-
-      user     system      total        real
-write_1mb_1x5  0.520000   0.120000   0.640000 (  0.660638)
-write_1mb_2x5 19.810000   2.280000  22.090000 ( 22.696214)
-
-weird.
-
-100 bytes:
-
-      user     system      total        real
-write_1mb_1x5  0.560000   0.140000   0.700000 (  1.424974)
-write_1mb_2x5 22.940000   2.840000  25.780000 ( 26.556346)
-
-500 bytes:
-
-      user     system      total        real
-write_1mb_1x5  0.530000   0.150000   0.680000 (  1.139738)
-write_1mb_2x5 77.260000  10.130000  87.390000 ( 91.671086)
-
-what happens there? very strange. 
-
-=end
-

data/lib/ole/storage/base.rb

@@ -21,7 +21,7 @@ module Ole # :nodoc:
 		class FormatError < StandardError # :nodoc:
 		end
 
-		VERSION = '1.2.9'
+		VERSION = '1.2.10'
 
 		# options used at creation time
 		attr_reader :params
@@ -180,7 +180,7 @@ module Ole # :nodoc:
 
 			# serialize the dirents using the bbat
 			RangesIOResizeable.open @bbat, 'w', :first_block => @header.dirent_start do |io|
-				@dirents.each { |dirent| io.write dirent.to_s }
+				io.write @dirents.map { |dirent| dirent.to_s }.join
 				padding = (io.size / @bbat.block_size.to_f).ceil * @bbat.block_size - io.size
 				io.write 0.chr * padding
 				@header.dirent_start = io.first_block
@@ -203,7 +203,7 @@ module Ole # :nodoc:
 			@bbat.map! do |b|
 				b == AllocationTable::BAT || b == AllocationTable::META_BAT ? AllocationTable::AVAIL : b
 			end
-	
+
 			# currently we use a loop. this could be better, but basically,
 			# the act of writing out the bat, itself requires blocks which get
 			# recorded in the bat.
@@ -482,7 +482,7 @@ module Ole # :nodoc:
 			# The blocks are Big or Small blocks depending on the table type.
 			def blocks_to_ranges chain, size=nil
 				# truncate the chain if required
-				chain = chain[0...(size.to_f / block_size).ceil] if size
+				chain = chain[0, (size.to_f / block_size).ceil] if size
 				# convert chain to ranges of the block size
 				ranges = chain.map { |i| [block_size * i, block_size] }
 				# truncate final range if required
@@ -526,8 +526,8 @@ module Ole # :nodoc:
 			def free_block
 				if @sparse
 					i = index(AVAIL) and return i
+					@sparse = false
 				end
-				@sparse = false
 				push AVAIL
 				length - 1
 			end
@@ -565,8 +565,14 @@ module Ole # :nodoc:
 				end
 
 				# Big blocks are kind of -1 based, in order to not clash with the header.
-				def blocks_to_ranges blocks, size
-					super blocks.map { |b| b + 1 }, size
+				def blocks_to_ranges chain, size=nil
+					#super chain.map { |b| b + 1 }, size
+					# duplicated from AllocationTable#blocks_to_ranges to avoid chain.map
+					# which was decent part of benchmark profile
+					chain = chain[0, (size.to_f / block_size).ceil] if size
+					ranges = chain.map { |i| [block_size * (i + 1), block_size] }
+					ranges.last[1] -= (ranges.length * block_size - size) if ranges.last and size
+					ranges
 				end
 			end
 
@@ -711,13 +717,21 @@ module Ole # :nodoc:
 				AllocationTable::EOC, 0, 0.chr * 4
 			]
 
-			# i think its just used by the tree building
-			attr_accessor :idx
 			# This returns all the children of this +Dirent+. It is filled in
 			# when the tree structure is recreated.
-			attr_accessor :children
-			attr_accessor :name
+			attr_reader :children
+			attr_reader :name
 			attr_reader :ole, :type, :create_time, :modify_time
+			attr_reader :parent
+
+			# i think its just used by the tree building
+			attr_accessor :idx
+
+			# these are for internal use and are used for faster lookup.
+			attr_reader :name_lookup
+			attr_writer :parent
+			protected :name_lookup, :parent=
+
 			def initialize ole, values=DEFAULT, params={}
 				@ole = ole				
 				values, params = DEFAULT, values if Hash === values
@@ -737,44 +751,46 @@ module Ole # :nodoc:
 
 				# further extra type specific stuff
 				if file?
-					default_time = @ole.params[:update_timestamps] ? Time.now : nil
+					default_time = @ole.params[:update_timestamps] ? Types::FileTime.now : nil
 					@create_time ||= default_time
 					@modify_time ||= default_time
 					@create_time = Types::Variant.load(Types::VT_FILETIME, create_time_str) if create_time_str
 					@modify_time = Types::Variant.load(Types::VT_FILETIME, create_time_str) if modify_time_str
 					@children = nil
+					@name_lookup = nil
 				else
 					@create_time = nil
 					@modify_time = nil
 					self.size = 0 unless @type == :root
 					@children = []
+					@name_lookup = {}
 				end
+
+				@parent = nil
 				
 				# to silence warnings. used for tree building at load time
 				# only.
 				@idx = nil
 			end
 
+			def name= name
+				if @parent
+					map = @parent.instance_variable_get :@name_lookup
+					map.delete @name
+					map[name] = self
+				end
+				@name = name
+			end
+
+			def children= children
+				@children = []
+				children.each { |child| self << child }
+			end
+
 			def open mode='r'
 				raise Errno::EISDIR unless file?
 				io = RangesIOMigrateable.new self, mode
-				# TODO work on the mode string stuff a bit more.
-				# maybe let the io object know about the mode, so it can refuse
-				# to work for read/write appropriately. maybe redefine all unusable
-				# methods using singleton class to throw errors.
-				# for now, i just want to implement truncation on use of 'w'. later,
-				# i need to do 'a' etc.
-				case mode
-				when 'r', 'r+'
-					# as i don't enforce reading/writing, nothing changes here. kind of
-					# need to enforce tt if i want modify times to work better.
-					@modify_time = Time.now if mode == 'r+'
-				when 'w'
-					@modify_time = Time.now
-				#	io.truncate 0
-				#else
-				#	raise NotImplementedError, "unsupported mode - #{mode.inspect}"
-				end
+				@modify_time = Types::FileTime.now if io.mode.writeable?
 				if block_given?
 					begin   yield io
 					ensure; io.close
@@ -798,7 +814,7 @@ module Ole # :nodoc:
 
 			# maybe need some options regarding case sensitivity.
 			def / name
-				children.find { |child| name === child.name }
+				@name_lookup[name]
 			end
 
 			def [] idx
@@ -887,11 +903,23 @@ module Ole # :nodoc:
 				str + '>'
 			end
 
-			def delete child
+			def << child
+				child.parent = self
+				@name_lookup[child.name] = child
+				@children << child
+			end
+
+			# remove the Dirent +child+ from the children array, truncating the data
+			# by default.
+			def delete child, truncate=true
 				# remove from our child array, so that on reflatten and re-creation of @dirents, it will be gone
-				raise ArgumentError, "#{child.inspect} not a child of #{self.inspect}" unless @children.delete child
+				unless @children.delete(child)
+					raise ArgumentError, "#{child.inspect} not a child of #{self.inspect}"
+				end
+				@name_lookup.delete(child.name)
+				child.parent = nil
 				# free our blocks
-				child.open { |io| io.truncate 0 }
+				child.open { |io| io.truncate 0 } if child.file?
 			end
 
 			def self.copy src, dst
@@ -902,7 +930,7 @@ module Ole # :nodoc:
 				if src.dir?
 					src.children.each do |src_child|
 						dst_child = Dirent.new dst.ole, :type => src_child.type
-						dst.children << dst_child
+						dst << dst_child
 						Dirent.copy src_child, dst_child
 					end
 				else

data/lib/ole/storage/file_system.rb

@@ -45,12 +45,13 @@ module Ole # :nodoc:
 		# (change it)
 		def dirent_from_path path
 			dirent = @root
-			path = file.expand_path path
-			path = path.sub(/^\/*/, '').sub(/\/*$/, '').split(/\/+/)
+			path = file.expand_path(path).split('/')
 			until path.empty?
+				part = path.shift
+				next if part.empty?
 				return nil if dirent.file?
-				return nil unless dirent = dirent/path.shift
-			end
+				return nil unless dirent = dirent/part
+			end			
 			dirent
 		end
 
@@ -107,14 +108,20 @@ module Ole # :nodoc:
 			end
 
 			def expand_path path
-				# get the raw stored pwd value (its blank for root)
-				pwd = @ole.dir.instance_variable_get :@pwd
-				# its only absolute if it starts with a '/'
-				path = "#{pwd}/#{path}" unless path =~ /^\//
+				# its already absolute if it starts with a '/'
+				unless path =~ /^\//
+					# get the raw stored pwd value (its blank for root)
+					pwd = @ole.dir.instance_variable_get :@pwd
+					path = "#{pwd}/#{path}"
+				end
 				# at this point its already absolute. we use File.expand_path
 				# just for the .. and . handling
 				# No longer use RUBY_PLATFORM =~ /win/ as it matches darwin. better way?
-				File.expand_path(path)[File::ALT_SEPARATOR == "\\" ? (2..-1) : (0..-1)]
+				if File::ALT_SEPARATOR != "\\"
+					File.expand_path(path)
+				else
+					File.expand_path(path)[2..-1]
+				end
 			end
 
 			# +orig_path+ is just so that we can use the requested path
@@ -122,7 +129,7 @@ module Ole # :nodoc:
 			def dirent_from_path path, orig_path=nil
 				orig_path ||= path
 				dirent = @ole.dirent_from_path path
-				raise Errno::ENOENT,  orig_path unless dirent
+				raise Errno::ENOENT, orig_path unless dirent
 				raise Errno::EISDIR, orig_path if dirent.dir?
 				dirent
 			end
@@ -152,7 +159,7 @@ module Ole # :nodoc:
 						# a get_parent_dirent function.
 						parent_path, basename = File.split expand_path(path)
 						parent = @ole.dir.send :dirent_from_path, parent_path, path
-						parent.children << dirent = Dirent.new(@ole, :type => :file, :name => basename)
+						parent << dirent = Dirent.new(@ole, :type => :file, :name => basename)
 					end
 				else
 					dirent = dirent_from_path path
@@ -211,31 +218,20 @@ module Ole # :nodoc:
 					1 + 1
 				end
 				# reparent the dirent
-				from_parent_path, from_basename = File.split expand_path(from_path)
 				to_parent_path, to_basename = File.split expand_path(to_path)
-				from_parent = @ole.dir.send :dirent_from_path, from_parent_path, from_path
+				from_parent = dirent.parent
 				to_parent = @ole.dir.send :dirent_from_path, to_parent_path, to_path
-				from_parent.children.delete dirent
+				from_parent.delete dirent, false
 				# and also change its name
 				dirent.name = to_basename
-				to_parent.children << dirent
+				to_parent << dirent
 				0
 			end
 
-			# crappy copy from Dir.
 			def unlink(*paths)
 				paths.each do |path|
-					dirent = @ole.dirent_from_path path
-					# i think we should free all of our blocks from the
-					# allocation table.
-					# i think if you run repack, all free blocks should get zeroed,
-					# but currently the original data is there unmodified.
-					open(path) { |f| f.truncate 0 }
-					# remove ourself from our parent, so we won't be part of the dir
-					# tree at save time.
-					parent_path, basename = File.split expand_path(path)
-					parent = @ole.dir.send :dirent_from_path, parent_path, path
-					parent.children.delete dirent
+					dirent = dirent_from_path path
+					dirent.parent.delete dirent
 				end
 				paths.length # hmmm. as per ::File ?
 			end
@@ -243,15 +239,17 @@ module Ole # :nodoc:
 		end
 
 		#
-		# an *instance* of this class is supposed to provide similar methods
+		# An *instance* of this class is supposed to provide similar methods
 		# to the class methods of Dir itself.
 		#
-		# pretty complete. like zip/zipfilesystem's implementation, i provide
+		# Fairly complete - like zip/zipfilesystem's implementation, i provide
 		# everything except chroot and glob. glob could be done with a glob
-		# to regex regex, and then simply match in the entries array... although
-		# recursive glob complicates that somewhat.
+		# to regex conversion, and then simply match in the entries array...
+		# although recursive glob complicates that somewhat.
+		#
+		# Dir.chroot, Dir.glob, Dir.[], and Dir.tmpdir is the complete list of
+		# methods still missing.
 		#
-		# Dir.chroot, Dir.glob, Dir.[], and Dir.tmpdir is the complete list.
 		class DirClass
 			def initialize ole
 				@ole = ole
@@ -271,11 +269,8 @@ module Ole # :nodoc:
 
 			def open path
 				dir = Dir.new path, entries(path)
-				if block_given?
-					yield dir
-				else
-					dir
-				end
+				return dir unless block_given?
+				yield dir
 			end
 
 			# as for file, explicit alias to inhibit block
@@ -286,17 +281,14 @@ module Ole # :nodoc:
 			# pwd is always stored without the trailing slash. we handle
 			# the root case here
 			def pwd
-				if @pwd.empty?
-					'/'
-				else
-					@pwd
-				end
+				return '/' if @pwd.empty?
+				@pwd
 			end
 			alias getwd :pwd
 
 			def chdir orig_path
 				# make path absolute, squeeze slashes, and remove trailing slash
-				path = @ole.file.expand_path(orig_path).gsub(/\/+/, '/').sub(/\/$/, '')
+				path = @ole.file.expand_path(orig_path).squeeze('/').sub(/\/$/, '')
 				# this is just for the side effects of the exceptions if invalid
 				dirent_from_path path, orig_path
 				if block_given?
@@ -331,10 +323,7 @@ module Ole # :nodoc:
 				entries(path).each(&block)
 			end
 
-			# there are some other important ones, like:
-			# chroot (!), glob etc etc. for now, i think
 			def mkdir path
-				# as for rmdir below:
 				parent_path, basename = File.split @ole.file.expand_path(path)
 				# note that we will complain about the full path despite accessing
 				# the parent path. this is consistent with ::Dir
@@ -342,29 +331,14 @@ module Ole # :nodoc:
 				# now, we first should ensure that it doesn't already exist
 				# either as a file or a directory.
 				raise Errno::EEXIST, path if parent/basename
-				parent.children << Dirent.new(@ole, :type => :dir, :name => basename)
+				parent << Dirent.new(@ole, :type => :dir, :name => basename)
 				0
 			end
 
 			def rmdir path
 				dirent = dirent_from_path path
 				raise Errno::ENOTEMPTY, path unless dirent.children.empty?
-
-				# now delete it, how to do that? the canonical representation that is
-				# maintained is the root tree, and the children array. we must remove it
-				# from the children array.
-				# we need the parent then. this sucks but anyway:
-				# we need to split the path. but before we can do that, we need
-				# to expand it first. eg. say we need the parent to unlink
-				# a/b/../c. the parent should be a, not a/b/.., or a/b.
-				parent_path, basename = File.split @ole.file.expand_path(path)
-				# this shouldn't be able to fail if the above didn't
-				parent = dirent_from_path parent_path
-				# note that the way this currently works, on save and repack time this will get
-				# reflected. to work properly, ie to make a difference now it would have to re-write
-				# the dirent. i think that Ole::Storage#close will handle that. and maybe include a
-				# #repack.
-				parent.children.delete dirent
+				dirent.parent.delete dirent
 				0 # hmmm. as per ::Dir ?
 			end
 			alias delete :rmdir
@@ -373,7 +347,6 @@ module Ole # :nodoc:
 			# note that there is nothing remotely ole specific about
 			# this class. it simply provides the dir like sequential access
 			# methods on top of an array.
-			# hmm, doesn't throw the IOError's on use of a closed directory...
 			class Dir
 				include Enumerable
 
@@ -408,14 +381,12 @@ module Ole # :nodoc:
 					raise IOError if @closed
 					@pos = [[0, pos].max, @entries.length].min
 				end
+				alias tell :pos
+				alias seek :pos=
 
 				def rewind
-					raise IOError if @closed
-					@pos = 0
+					seek 0
 				end
-
-				alias tell :pos
-				alias seek :pos=
 			end
 		end
 	end

data/lib/ole/types/base.rb

@@ -57,8 +57,34 @@ module Ole # :nodoc:
 		# for VT_FILETIME
 		class FileTime < DateTime
 			SIZE = 8
+
+			# DateTime.new is slow... faster version for FileTime
+			def self.new year, month, day, hour=0, min=0, sec=0, usec=0
+				# DateTime will remove leap and leap-leap seconds
+				sec = 59 if sec > 59
+				if month <= 2
+					month += 12
+					year  -= 1
+				end
+				y   = year + 4800
+				m   = month - 3
+				jd  = day + (153 * m + 2) / 5 + 365 * y + y / 4 - y / 100 + y / 400 - 32045
+				fr  = hour / 24.0 + min / 1440.0 + sec / 86400.0
+				new! jd + fr - 0.5, 0, ITALY
+			end
+
+			def self.from_time time
+				new(*(time.to_a[0, 6].reverse + [time.usec]))
+			end
+
+			def self.now
+				from_time Time.now
+			end
+
 			EPOCH = new 1601, 1, 1
 
+			#def initialize year, month, day, hour, min, sec
+
 			# Create a +DateTime+ object from a struct +FILETIME+
 			# (http://msdn2.microsoft.com/en-us/library/ms724284.aspx).
 			#
@@ -68,27 +94,26 @@ module Ole # :nodoc:
 				low, high = str.to_s.unpack 'V2'
 				# we ignore these, without even warning about it
 				return nil if low == 0 and high == 0
-				# switched to rational, and fixed the off by 1 second error i sometimes got.
-				# time = EPOCH + (high * (1 << 32) + low) / 1e7 / 86400 rescue return
-				# use const_get to ensure we can return anything which subclasses this (VT_DATE?)
-				const_get('EPOCH') + Rational(high * (1 << 32) + low, 1e7.to_i * 86400) rescue return
-				# extra sanity check...
-				#unless (1800...2100) === time.year
-				#	Log.warn "ignoring unlikely time value #{time.to_s}"
-				#	return nil
-				#end
-				#time
+				# the + 0.00001 here stinks a bit...
+				seconds = (high * (1 << 32) + low) / 1e7 + 0.00001
+				EPOCH + seconds / 86400 rescue return
 			end
-			
+
 			# +time+ should be able to be either a Time, Date, or DateTime.
 			def self.dump time
-				# i think i'll convert whatever i get to be a datetime, because of
-				# the covered range.
 				return 0.chr * SIZE unless time
-				time = time.send(:to_datetime) if Time === time
-				# don't bother to use const_get here
-				bignum = (time - EPOCH) * 86400 * 1e7.to_i
-				high, low = bignum.divmod 1 << 32
+				# convert whatever is given to be a datetime, to handle the large range
+				case time
+				when Date # this includes DateTime & FileTime
+				when Time
+					time = from_time time
+				else
+					raise ArgumentError, 'unknown time argument - %p' % [time]
+				end
+				# round to milliseconds (throwing away nanosecond precision) to
+				# compensate for using Float-based DateTime
+				nanoseconds = ((time - EPOCH).to_f * 864000000).round * 1000
+				high, low = nanoseconds.divmod 1 << 32
 				[low, high].pack 'V2'
 			end
 			

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: ruby-ole
 version: !ruby/object:Gem::Version 
-  version: 1.2.9
+  version: 1.2.10
 platform: ruby
 authors: 
 - Charles Lowe
@@ -9,7 +9,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2009-07-14 00:00:00 +10:00
+date: 2009-07-20 00:00:00 +10:00
 default_executable: 
 dependencies: []