ruby-msg 1.3.1 → 1.4.0

data/README

@@ -1,121 +1,116 @@
-#summary ruby-msg - A library for reading Outlook msg files, and for converting them to RFC2822 emails.
+= Introduction
 
-= Introduction =
+Generally, the goal of the project is the conversion of .msg files
+into proper rfc2822 emails, independent of outlook, or any platform
+dependencies etc. In fact its currently pure ruby, so it should be
+easy to get started with.
 
-Generally, the goal of the project is the conversion of .msg files into proper rfc2822
-emails, independent of outlook, or any platform dependencies etc. 
-In fact its currently pure ruby, so it should be easy to get started with.
+There's also work-in-progess pst support (unfortunately outlook 97
+only currently), based on libpst, making this project more of a general
+ruby mapi message store conversion library now (though some significant
+cleaning up has to happen first).
 
-It draws on `msgconvert.pl`, but tries to take a cleaner and more complete approach.
-Neither are complete yet, however, but I think that this project provides a clean foundation upon which to work on a good converter for msg files for use in outlook migrations etc.
+It draws on <tt>msgconvert.pl</tt>, but tries to take a cleaner and
+more complete approach. Neither are complete yet, however, but I think
+that this project provides a clean foundation upon which to work on
+a good converter for msg files for use in outlook migrations etc.
 
 I am happy to accept patches, give commit bits etc.
 
 Please let me know how it works for you, any feedback would be welcomed.
 
-= Usage =
-
-Higher level access to the msg, can be had through the top level data accessors.
-
-{{{
-require 'msg'
-
-msg = Msg.load open(filename)
-
-# access to the 3 main data stores, if you want to poke with the msg
-# internals
-msg.recipients
-# => [#<Recipient:'\'Marley, Bob\' <bob.marley@gmail.com>'>]
-msg.attachments
-# => [#<Attachment filename='blah1.tif'>, #<Attachment filename='blah2.tif'>]
-msg.properties
-# => #<Properties ... normalized_subject='Testing' ... 
-# creation_time=#<DateTime: 2454042.45074714,0,2299161> ...>
-}}}
-
-To completely abstract away all msg peculiarities, convert the msg to a mime object.
-The message as a whole, and some of its main parts support conversion to mime objects.
-
-{{{
-msg.attachments.first.to_mime
-# => #<Mime content_type='application/octet-stream'>
-mime = msg.to_mime
-puts mime.to_tree
-# =>
-- #<Mime content_type='multipart/mixed'>
-  |- #<Mime content_type='multipart/alternative'>
-  |  |- #<Mime content_type='text/plain'>
-  |  \- #<Mime content_type='text/html'>
-  |- #<Mime content_type='application/octet-stream'>
-  \- #<Mime content_type='application/octet-stream'>
-
-# convert mime object to serialised form,
-# inclusive of attachments etc. (not ideal in memory, but its wip).
-puts mime.to_s
-}}}
-
-You can also access the underlying ole object, and see all the gory details of how msgs are serialised:
-
-{{{
-puts msg.ole.root.to_tree
-# =>
-- #<OleDir:"Root Entry" size=3840 time="2006-11-03T00:52:53Z">
-  |- #<OleDir:"__nameid_version1.0" size=0 time="2006-11-03T00:52:53Z">
-  |  |- #<OleDir:"__substg1.0_00020102" size=16 data="CCAGAAAAAADAAA...">
-  |  |- #<OleDir:"__substg1.0_00030102" size=64 data="DoUAAAYAAABShQ...">
-  |  |- #<OleDir:"__substg1.0_00040102" size=0 data="">
-  |  |- #<OleDir:"__substg1.0_10010102" size=16 data="UoUAAAYAAQAQhQ...">
-  |  |- #<OleDir:"__substg1.0_10090102" size=8 data="GIUAAAYABgA=">
-  |  |- #<OleDir:"__substg1.0_100A0102" size=8 data="BoUAAAYABwA=">
-  |  |- #<OleDir:"__substg1.0_100F0102" size=8 data="A4UAAAYABAA=">
-  |  |- #<OleDir:"__substg1.0_10110102" size=8 data="AYUAAAYAAwA=">
-  |  |- #<OleDir:"__substg1.0_10120102" size=8 data="DoUAAAYAAAA=">
-  |  \- #<OleDir:"__substg1.0_101E0102" size=8 data="VIUAAAYAAgA=">
-  |- #<OleDir:"__substg1.0_001A001E" size=8 data="SVBNLk5vdGU=">
-  ...
-  |- #<OleDir:"__substg1.0_8002001E" size=4 data="MTEuMA==">
-  |- #<OleDir:"__properties_version1.0" size=800 data="AAAAAAAAAAABAA...">
-  \- #<OleDir:"__recip_version1.0_#00000000" size=0 time="2006-11-03T00:52:53Z">
-
-     |- #<OleDir:"__substg1.0_0FF60102" size=4 data="AAAAAA==">
-     |- #<OleDir:"__substg1.0_3001001E" size=4 data="YXNkZg==">
-     |- #<OleDir:"__substg1.0_5FF6001E" size=4 data="YXNkZg==">
-     \- #<OleDir:"__properties_version1.0" size=152 data="AAAAAAAAAAAeAA...">
-}}}
-
-= Further Details =
-
-Named properties have recently been implemented, and Msg::Properties now allows associated guids. Keys are represented by Msg::Properties::Key, which contains the relevant code.
-
-You can now write code like:
-{{{
-props = msg.properties
-
-props[0x0037] # access subject by mapi code
-props[0x0037, Msg::Properties::PS_MAPI] # equivalent, with explicit GUID.
-key = Msg::Properties::Key.new 0x0037 # => 0x0037
-props[key] # same again
-
-# keys support being converted to symbols, and then use a symbolic lookup
-key.to_sym # => :subject
-props[:subject] # as above
-props.subject   # still good
-}}}
-
-Under the hood, there is complete support for named properties:
-{{{
-# to get the categories as set by outlook
-props['Keywords', Msg::Properties::PS_PUBLIC_STRINGS]
-# => ["Business", "Competition", "Favorites"]
-
-# and as a fallback, the symbolic lookup will automatically use named properties,
-# which can be seen:
-props.resolve :keywords
-# => #<Key {00020329-0000-0000-c000-000000000046}/"Keywords">
-
-# which allows this to work:
-props.keywords # as above
-}}}
-
-With some more work, the property storage model should be able to reach feature
-completion.
+= Features
+
+Broad features of the project:
+
+* Can be used as a general msg library, where conversion to and working
+  on a standard format doesn't make sense.
+
+* Supports conversion of msg files to standard formats, like rfc2822
+  emails, vCards, etc.
+
+* Well commented, and easily extended.
+
+* Most key .msg structures are understood, and the only the parsing
+  code should require minor tweaks. Most of remaining work is in achieving
+  high-fidelity conversion to standards formats (see [TODO]).
+
+Features of the lower-level msg handling:
+
+* Supports both types of property storage (large ones in +substg+
+  files, and small ones in the +properties+ file).
+
+* Complete support for named properties in different GUID namespaces.
+
+* Support for mapping property codes to symbolic names, with many
+  included.
+
+* RTF decompression support included, as well as HTML extraction from
+  RTF where appropriate (both in pure ruby, see <tt>lib/msg/rtf.rb</tt>)
+
+* Initial RTF converter, for providing a readable body when only RTF
+  exists (needs work)
+
+* Initial support for handling embedded ole files, converting nested
+  .msg files to message/rfc822 attachments, and serializing others
+  as ole file attachments (allows you to view embedded excel for example).
+
+= Usage
+
+At the command line, it is simple to convert individual msg files
+to .eml, or to convert a batch to an mbox format file. See help for
+details:
+
+  msgtool -c some_email.msg > some_email.eml
+  msgtool -m *.msg > mbox
+
+There is also a fairly complete and easy to use high level library
+access:
+
+  require 'msg'
+  
+  msg = Msg.open filename
+  
+  # access to the 3 main data stores, if you want to poke with the msg
+  # internals
+  msg.recipients
+  # => [#<Recipient:'\'Marley, Bob\' <bob.marley@gmail.com>'>]
+  msg.attachments
+  # => [#<Attachment filename='blah1.tif'>, #<Attachment filename='blah2.tif'>]
+  msg.properties
+  # => #<Properties ... normalized_subject='Testing' ... 
+  # creation_time=#<DateTime: 2454042.45074714,0,2299161> ...>
+
+To completely abstract away all msg peculiarities, convert the msg
+to a mime object. The message as a whole, and some of its main parts
+support conversion to mime objects.
+
+  msg.attachments.first.to_mime
+  # => #<Mime content_type='application/octet-stream'>
+  mime = msg.to_mime
+  puts mime.to_tree
+  # =>
+  - #<Mime content_type='multipart/mixed'>
+    |- #<Mime content_type='multipart/alternative'>
+    |  |- #<Mime content_type='text/plain'>
+    |  \- #<Mime content_type='text/html'>
+    |- #<Mime content_type='application/octet-stream'>
+    \- #<Mime content_type='application/octet-stream'>
+  
+  # convert mime object to serialised form,
+  # inclusive of attachments etc. (not ideal in memory, but its wip).
+  puts mime.to_s
+
+= Other
+
+For more information, see
+
+* TODO
+
+* MsgDetails[http://code.google.com/p/ruby-msg/wiki/MsgDetails]
+
+* OleDetails[http://code.google.com/p/ruby-ole/wiki/OleDetails]
+
+* msgconv[http://www.matijs.net/software/msgconv/], the original
+  perl converter.
+

data/Rakefile

@@ -8,55 +8,69 @@ require 'fileutils'
 
 $:.unshift 'lib'
 
-require 'msg'
+require 'mapi/msg'
 
 PKG_NAME = 'ruby-msg'
-PKG_VERSION = Msg::VERSION
+PKG_VERSION = Mapi::VERSION
 
 task :default => [:test]
 
 Rake::TestTask.new(:test) do |t|
-	t.test_files = FileList["test/test_*.rb"]
-	t.warning = true
+	t.test_files = FileList["test/test_*.rb"] - ['test/test_pst.rb']
+	t.warning = false
 	t.verbose = true
 end
 
-# RDocTask wasn't working for me
-desc 'Build the rdoc HTML Files'
-task :rdoc do
-	system "rdoc -S -N --main Msg --tab-width 2 --title '#{PKG_NAME} documentation' lib"
+begin
+	require 'rcov/rcovtask'
+	# NOTE: this will not do anything until you add some tests
+	desc "Create a cross-referenced code coverage report"
+	Rcov::RcovTask.new do |t|
+		t.test_files = FileList['test/test*.rb']
+		t.ruby_opts << "-Ilib" # in order to use this rcov
+		t.rcov_opts << "--xrefs"  # comment to disable cross-references
+		t.rcov_opts << "--exclude /usr/local/lib/site_ruby"
+		t.verbose = true
+	end
+rescue LoadError
+	# Rcov not available
+end
+
+Rake::RDocTask.new do |t|
+	t.rdoc_dir = 'doc'
+	t.title    = "#{PKG_NAME} documentation"
+	t.options += %w[--main README --line-numbers --inline-source --tab-width 2]
+	t.rdoc_files.include 'lib/**/*.rb'
+	t.rdoc_files.include 'README'
 end
 
 spec = Gem::Specification.new do |s|
-	s.name = PKG_NAME
-	s.version = PKG_VERSION
-	s.summary = %q{Ruby Msg library.}
+	s.name        = PKG_NAME
+	s.version     = PKG_VERSION
+	s.summary     = %q{Ruby Msg library.}
 	s.description = %q{A library for reading Outlook msg files, and for converting them to RFC2822 emails.}
-	s.authors = ["Charles Lowe"]
-	s.email = %q{aquasync@gmail.com}
-	s.homepage = %q{http://code.google.com/p/ruby-msg}
-	#s.rubyforge_project = %q{ruby-msg}
-
-	s.executables = ['msgtool']
-	s.files  = Dir.glob('data/*.yaml') + ['Rakefile', 'README', 'FIXES']
-	s.files += Dir.glob("lib/**/*.rb")
-	s.files += Dir.glob("test/test_*.rb")
-	s.files += Dir.glob("bin/*")
+	s.authors     = ["Charles Lowe"]
+	s.email       = %q{aquasync@gmail.com}
+	s.homepage    = %q{http://code.google.com/p/ruby-msg}
+	s.rubyforge_project = %q{ruby-msg}
+
+	s.executables = ['mapitool']
+	s.files       = FileList['data/*.yaml', 'Rakefile', 'README', 'FIXES']
+	s.files      += FileList['lib/**/*.rb', 'test/test_*.rb', 'bin/*']
 	
-	s.has_rdoc = true
-	s.rdoc_options += ['--main', 'Msg',
+	s.has_rdoc    = true
+	s.extra_rdoc_files = ['README']
+	s.rdoc_options += ['--main', 'README',
 					   '--title', "#{PKG_NAME} documentation",
 					   '--tab-width', '2']
 
-
-	s.autorequire = 'msg'
-
-	s.add_dependency 'ruby-ole', '>=1.2.1'
+	s.add_dependency 'ruby-ole', '>=1.2.4'
+	s.add_dependency 'vpim', '>=0.360'
 end
 
 Rake::GemPackageTask.new(spec) do |p|
 	p.gem_spec = spec
-	p.need_tar = true
+	p.need_tar = false #true
 	p.need_zip = false
 	p.package_dir = 'build'
 end

data/bin/mapitool

@@ -0,0 +1,195 @@
+#! /usr/bin/ruby
+
+$:.unshift File.dirname(__FILE__) + '/../lib'
+
+require 'optparse'
+require 'rubygems'
+require 'mapi/msg'
+require 'mapi/pst'
+require 'mapi/convert'
+require 'time'
+
+class Mapitool
+	attr_reader :files, :opts
+	def initialize files, opts
+		@files, @opts = files, opts
+		seen_pst = false
+		raise ArgumentError, 'Must specify 1 or more input files.' if files.empty?
+		files.map! do |f|
+			ext = File.extname(f.downcase)[1..-1]
+			raise ArgumentError, 'Unsupported file type - %s' % f unless ext =~ /^(msg|pst)$/
+			raise ArgumentError, 'Expermiental pst support not enabled' if ext == 'pst' and !opts[:enable_pst]
+			[ext.to_sym, f]
+		end
+		if dir = opts[:output_dir]
+			Dir.mkdir(dir) unless File.directory?(dir)
+		end
+	end
+
+	def each_message(&block)
+		files.each do |format, filename|
+			if format == :pst
+				if filter_path = opts[:filter_path]
+					filter_path = filter_path.tr("\\", '/').gsub(/\/+/, '/').sub(/^\//, '').sub(/\/$/, '')
+				end
+				open filename do |io|
+					pst = Mapi::Pst.new io
+					pst.each do |message|
+						next unless message.type == :message
+						if filter_path
+							next unless message.path =~ /^#{Regexp.quote filter_path}(\/|$)/i
+						end
+						yield message
+					end
+				end
+			else
+				Mapi::Msg.open filename, &block
+			end
+		end
+	end
+
+	def run
+		each_message(&method(:process_message))
+	end
+
+	def make_unique filename
+		@map ||= {}
+		return @map[filename] if !opts[:individual] and @map[filename]
+		try = filename
+		i = 1
+		try = filename.gsub(/(\.[^.]+)$/, ".#{i += 1}\\1") while File.exist?(try)
+		@map[filename] = try
+		try
+	end
+
+	def process_message message
+		# TODO make this more informative
+		mime_type = message.mime_type
+		return unless pair = Mapi::Message::CONVERSION_MAP[mime_type]
+
+		combined_map = {
+			'eml' => 'Mail.mbox',
+			'vcf' => 'Contacts.vcf',
+			'txt' => 'Posts.txt'
+		}
+
+		# TODO handle merged mode, pst, etc etc...
+		case message
+		when Mapi::Msg
+			if opts[:individual]
+				filename = message.root.ole.io.path.gsub(/msg$/i, pair.last)
+			else
+				filename = combined_map[pair.last] or raise NotImplementedError
+			end
+		when Mapi::Pst::Item
+			if opts[:individual]
+				filename = "#{message.subject.tr ' ', '_'}.#{pair.last}".gsub(/[^A-Za-z0-9.()\[\]{}-]/, '_')
+			else
+				filename = combined_map[pair.last] or raise NotImplementedError
+				filename = (message.path.tr(' /', '_.').gsub(/[^A-Za-z0-9.()\[\]{}-]/, '_') + '.' + File.extname(filename)).squeeze('.')
+			end
+			dir = File.dirname(message.instance_variable_get(:@desc).pst.io.path)
+			filename = File.join dir, filename
+		else
+			raise
+		end
+
+		if dir = opts[:output_dir]
+			filename = File.join dir, File.basename(filename)
+		end
+
+		filename = make_unique filename
+
+		write_message = proc do |f|
+			data = message.send(pair.first).to_s
+			if !opts[:individual] and pair.last == 'eml'
+				# we do the append > style mbox quoting (mboxrd i think its called), as it
+				# is the only one that can be robuslty un-quoted. evolution doesn't use this!
+				f.puts "From mapitool@localhost #{Time.now.rfc2822}"
+				#munge_headers mime, opts
+				data.each do |line|
+					if line =~ /^>*From /o
+						f.print '>' + line
+					else
+						f.print line
+					end
+				end
+			else
+				f.write data
+			end
+		end
+
+		if opts[:stdout]
+			write_message[STDOUT]
+		else
+			open filename, 'a', &write_message
+		end
+	end
+
+	def munge_headers mime, opts
+		opts[:header_defaults].each do |s|
+			key, val = s.match(/(.*?):\s+(.*)/)[1..-1]
+			mime.headers[key] = [val] if mime.headers[key].empty?
+		end
+	end
+end
+
+def mapitool
+	opts = {:verbose => false, :action => :convert, :header_defaults => []}
+	op = OptionParser.new do |op|
+		op.banner = "Usage: mapitool [options] [files]"
+		#op.separator ''
+		#op.on('-c', '--convert', 'Convert input files (default)') { opts[:action] = :convert }
+		op.separator ''
+		op.on('-o', '--output-dir DIR', 'Put all output files in DIR') { |d| opts[:output_dir] = d }
+		op.on('-i', '--[no-]individual', 'Do not combine converted files') { |i| opts[:individual] = i }
+		op.on('-s', '--stdout', 'Write all data to stdout') { opts[:stdout] = true }
+		op.on('-f', '--filter-path PATH', 'Only process pst items in PATH') { |path| opts[:filter_path] = path }
+		op.on(      '--enable-pst', 'Turn on experimental PST support') { opts[:enable_pst] = true }
+		#op.on('-d', '--header-default STR', 'Provide a default value for top level mail header') { |hd| opts[:header_defaults] << hd }
+		# --enable-pst
+		op.separator ''
+		op.on('-v', '--[no-]verbose', 'Run verbosely') { |v| opts[:verbose] = v }
+		op.on_tail('-h', '--help', 'Show this message') { puts op; exit }
+	end
+
+	files = op.parse ARGV
+
+	# for windows. see issue #2
+	STDOUT.binmode
+
+	Mapi::Log.level = Ole::Log.level = opts[:verbose] ? Logger::WARN : Logger::FATAL
+
+	tool = begin
+		Mapitool.new(files, opts)
+	rescue ArgumentError
+		puts $!
+		puts op
+		exit 1
+	end
+	
+	tool.run
+end
+
+mapitool
+
+__END__
+
+mapitool [options] [files]
+
+files is a list of *.msg & *.pst files.
+
+one of the options should be some sort of path filter to apply to pst items.
+
+--filter-path=
+--filter-type=eml,vcf
+
+with that out of the way, the entire list of files can be converted into a
+list of items (with meta data about the source).
+
+--convert
+--[no-]separate one output file per item or combined output
+--stdout
+--output-dir=.
+
+