ruby-msg-nx 0.3.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: ecb72e2d9f9b305ba77303b4c31252fe421758e4b89e1d00b8249bcda268aaea
+  data.tar.gz: 248766c857e0b52131ab2bbaf0ed346468a856cb315abe85d76f5a3929d3fb1a
+SHA512:
+  metadata.gz: 00a1d0b03495afe9e2183ba3804c0c6235e57c6e9840abd23140ea4a36f5a1472ea5eeb4d99cd16f420fa801de4255f284a6c5d25842f80d93b56a2ef612d3a4
+  data.tar.gz: 48c8ca8a4be497c011a2529eb6a1460ee24d7c8a3ba87939f40576598d989335aa79e8460cdaefaa4cfa8b1902e7392343661846e4ad040616f2c3c5e1de3680

data/COPYING

@@ -0,0 +1,20 @@
+Copyright (c) 2007-2014 Charles Lowe
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+

data/ChangeLog

@@ -0,0 +1,108 @@
+== 0.1.0 / 2021-12-22
+
+- Rebooting.
+
+== 1.5.2 / 2014-08-20
+
+- Move mime.rb file to avoid conflicts with mime_types gem (github #7,
+  blerins).
+- Minor fix to mapitool for ruby >= 1.9.
+- Alway require mapi/convert (indirect fix for missed step in README,
+  github #6).
+- Various minor cleanups.
+
+== 1.5.1 / 2012-07-03
+
+- Fix handling of different body types (issue #14). Was breaking on
+  files without RTF content since 8933c26e, and also failing on files
+  where PR_BODY_HTML was a string rather than a stream.
+- Move classes from RTF into Mapi::RTF (github #4).
+
+== 1.5.0 / 2011-05-18
+
+- Fixes for ruby 1.9.
+- Move Mime into the Mapi module namespace (crowbot).
+- Use ascii regex flag to avoid unicode probs (crowbot).
+
+== 1.4.0 / 2008-10-12
+
+- Initial simple msg test case.
+- Update TODO, stripping out all the redundant ole stuff.
+- Fix property set guids to use the new Ole::Types::Clsid type.
+- Add block form of Msg.open
+- Fix file requires for running tests individually.
+- Update pst RangesIO subclasses for changes in ruby-ole.
+- Merge initial pst reading code (converted from libpst).
+- Pretty big pst refactoring, adding initial outlook 2003 pst support.
+- Flesh out move to mapi to clean up the way pst hijacks the msg
+  classes currently.
+- Add a ChangeLog :).
+- Update README, by converting Home.wiki with wiki2rdoc converter.
+- Separate out generic mapi object code from msg code, and separate out
+  conversion code.
+- Add decent set of Mapi and Msg unit tests, approaching ~55% code coverage,
+  not including pst.
+- Add TMail note conversion alternative, to eventually allow removal of
+  custom Mime class.
+- Expose experimental pst support through renamed mapitool program.
+
+== 1.3.1 / 2007-08-21
+
+- Add fix for issue #2, and #4.
+- Move ole code to ruby-ole project, and depend on it.
+
+== 1.2.17 / 2007-05-13
+
+(This was last release before splitting out ruby-ole. subsequent bug fix
+point releases 1-3 were made directly on the gem, not reflected in the
+repository, though the fixes were also forward-ported.)
+
+- Update Ole::Storage backend, finalising api for split to separate 
+  library.
+
+== 1.2.16 / 2007-04-28
+
+- Some minor fixes to msg parser.
+- Extending RTF and body conversion support.
+- Initial look at possible wmf conversion for embedded images.
+- Add initial cli converter tool
+- Add rdoc to ole/storage, and msg/properties
+- Add streaming IO support to Ole::Storage, and use it in Msg::Properties
+- Updates to test cases
+- Add README, and update TODO
+- Convert rtf support tools in c to small ruby class.
+- Merge preliminary write support for Ole::Storage, as well as preliminary
+  filesystem api.
+
+== 1.2.13 / 2007-01-22
+
+- Nested msg support
+
+== 1.2.10 / 2007-01-21
+
+- Add initial vcard support.
+- Implement a named properties map, for vcard conversion.
+- Add orderedhash to Mime for keeping header order
+- Fix line endings in lib/mime
+- First released version
+
+== <= 1.2.9 / 2007-01-11..2007-01-19
+
+(Haven't bothered to note exact versions and dates - nothing here was released.
+can look at history of lib/msg.rb to see exact VERSION at each commit.)
+
+- Merged most of the named property work.
+- Added some test files.
+- Update svn:ignore, to exclude test messages and ole files which I can't
+  release. Need to get some clean files for use in test cases.
+  Also excluding source to the mapitags files for the moment.
+  A lot of it is not redistributable
+- Added a converter to extract embedded html in rtf. Downloaded somewhere,
+  source unknown.
+- Minor fix to ole/storage.rb, after new OleDir#type behaviour
+- Imported support.rb, replacing previously required std.rb
+- Added initial support for parsing times in Msg::Properties.
+- Imported some rtf decompression code and minor updates.
+- Cleaned up the ole class a bit
+- Fixed OleDir#data method using sb_blocks map (see POLE).
+

data/Home.md

@@ -0,0 +1,133 @@
+# Introduction
+
+Generally, the goal of the project is to enable the conversion of
+msg and pst files into standards based formats, without reliance on
+outlook, or any platform dependencies. In fact its currently **pure
+ruby**, so it should be easy to get running.
+
+It is targeted at people who want to migrate their PIM data from outlook,
+converting msg and pst files into rfc2822 emails, vCard contacts,
+iCalendar appointments etc. However, it also aims to be a fairly complete
+mapi message store manipulation library, providing a sane model for
+(currently read-only) access to msg and pst files (message stores).
+
+I am happy to accept patches, give commit bits etc.
+
+Please let me know how it works for you, any feedback would be welcomed.
+
+# Features
+
+Broad features of the project:
+
+* Can be used as a general mapi library, where conversion to and working
+  on a standard format doesn't make sense.
+
+* Supports conversion of messages to standard formats, like rfc2822
+  emails, vCard, etc.
+
+* Well commented, and easily extended.
+
+* Basic RTF converter, for providing a readable body when only RTF
+  exists (needs work)
+
+* RTF decompression support included, as well as HTML extraction from
+  RTF where appropriate (both in pure ruby, see `lib/mapi/rtf.rb`)
+
+* Support for mapping property codes to symbolic names, with many
+  included.
+
+Features of the msg format message store:
+
+* Most key .msg structures are understood, and the only the parsing
+  code should require minor tweaks. Most of remaining work is in achieving
+  high-fidelity conversion to standards formats (see `TODO`).
+
+* Supports both types of property storage (large ones in +substg+
+  files, and small ones in the +properties+ file.
+
+* Complete support for named properties in different GUID namespaces.
+
+* Initial support for handling embedded ole files, converting nested
+  .msg files to message/rfc822 attachments, and serializing others
+  as ole file attachments (allows you to view embedded excel for example).
+
+Features of the pst format message store:
+
+* Handles both Outlook 1997 & 2003 format pst files, both with no-
+  and "compressible-" encryption.
+
+* Understanding of the file format is still very superficial.
+
+# Usage
+
+At the command line, it is simple to convert individual msg or pst
+files to .eml, or to convert a batch to an mbox format file. See mapitool
+help for details:
+
+```sh
+mapitool -si some_email.msg > some_email.eml
+mapitool -s *.msg > mbox
+```
+
+There is also a fairly complete and easy to use high level library
+access:
+
+```
+require 'mapi/msg'
+
+msg = Mapi::Msg.open filename
+
+# access to the 3 main data stores, if you want to poke with the msg
+# internals
+msg.recipients
+# => [#<Recipient:'\'Marley, Bob\' <bob.marley@gmail.com>'>]
+msg.attachments
+# => [#<Attachment filename='blah1.tif'>, #<Attachment filename='blah2.tif'>]
+msg.properties
+# => #<Properties ... normalized_subject='Testing' ... 
+# creation_time=#<DateTime: 2454042.45074714,0,2299161> ...>
+```
+
+To completely abstract away all msg peculiarities, convert the msg
+to a mime object. The message as a whole, and some of its main parts
+support conversion to mime objects.
+
+```
+msg.attachments.first.to_mime
+# => #<Mime content_type='application/octet-stream'>
+mime = msg.to_mime
+puts mime.to_tree
+# =>
+- #<Mime content_type='multipart/mixed'>
+  |- #<Mime content_type='multipart/alternative'>
+  |  |- #<Mime content_type='text/plain'>
+  |  \- #<Mime content_type='text/html'>
+  |- #<Mime content_type='application/octet-stream'>
+  \- #<Mime content_type='application/octet-stream'>
+
+# convert mime object to serialised form,
+# inclusive of attachments etc. (not ideal in memory, but its wip).
+puts mime.to_s
+```
+
+# Thanks
+
+* The initial implementation of parsing msg files was based primarily
+  on [msgconvert.pl](http://www.matijs.net/software/msgconv/).
+
+* The basis for the outlook 97 pst file was the source to +libpst+.
+
+* The code for rtf decompression was implemented by inspecting the
+  algorithm used in the +JTNEF+ project.
+
+# Other
+
+For more information, see
+
+* [TODO](https://github.com/aquasync/ruby-msg/wiki/TODO)
+
+* [MsgDetails](https://github.com/aquasync/ruby-msg/wiki/MsgDetails)
+
+* [PstDetails](https://github.com/aquasync/ruby-msg/wiki/PstDetails)
+
+* [OleDetails](https://github.com/aquasync/ruby-msg/wiki/OleDetails)

data/Rakefile

@@ -0,0 +1,52 @@
+require 'rubygems'
+require 'rake/testtask'
+
+require 'rbconfig'
+require 'fileutils'
+
+spec = eval File.read('ruby-msg.gemspec')
+
+task :default => [:test]
+
+Rake::TestTask.new do |t|
+	t.test_files = FileList["test/test_*.rb"]
+	t.warning = false
+	t.verbose = true
+end
+
+begin
+	Rake::TestTask.new(:coverage) do |t|
+		t.test_files = FileList["test/test_*.rb"]
+		t.warning = false
+		t.verbose = true
+		t.ruby_opts = ['-rsimplecov -e "SimpleCov.start; load(ARGV.shift)"']
+	end
+rescue LoadError
+	# SimpleCov not available
+end
+
+begin
+	require 'rdoc/task'
+	RDoc::Task.new do |t|
+		t.rdoc_dir = 'doc'
+		t.rdoc_files.include 'lib/**/*.rb'
+		t.rdoc_files.include 'README', 'ChangeLog'
+		t.title    = "#{PKG_NAME} documentation"
+		t.options += %w[--line-numbers --inline-source --tab-width 2]
+		t.main	   = 'README'
+	end
+rescue LoadError
+	# RDoc not available or too old (<2.4.2)
+end
+
+begin
+	require 'rubygems/package_task'
+	Gem::PackageTask.new(spec) do |t|
+		t.need_tar = true
+		t.need_zip = false
+		t.package_dir = 'build'
+	end
+rescue LoadError
+	# RubyGems too old (<1.3.2)
+end
+

data/bin/mapitool

@@ -0,0 +1,204 @@
+#! /usr/bin/ruby
+
+$:.unshift File.dirname(__FILE__) + '/../lib'
+
+require 'optparse'
+require 'rubygems'
+require 'mapi/msg'
+require 'mapi/pst'
+require 'mapi/helper'
+require 'time'
+
+class Mapitool
+	attr_reader :files
+	attr_reader :opts
+
+	#FILTER_NON_FILE_NAME = /[^A-Za-z0-9.()\[\]{}-]/
+	FILTER_NON_FILE_NAME = /[\x00-0x1f"\/\\\?\*\|\<\>\:]/
+
+	def initialize files, opts
+		@files, @opts = files, opts
+		seen_pst = false
+		raise ArgumentError, 'Must specify 1 or more input files.' if files.empty?
+		@helper = Mapi::Helper.new opts[:ansi_encoding], opts[:to_unicode]
+		files.map! do |f|
+			ext = File.extname(f.downcase)[1..-1]
+			raise ArgumentError, 'Unsupported file type - %s' % f unless ext =~ /^(msg|pst|ost)$/
+			raise ArgumentError, 'Expermiental pst support not enabled' if /^(pst|ost)$/.match(ext) and !opts[:enable_pst]
+			[ext.to_sym, f]
+		end
+		if dir = opts[:output_dir]
+			Dir.mkdir(dir) unless File.directory?(dir)
+		end
+	end
+
+	def each_message(&block)
+		files.each do |format, filename|
+			if [:pst, :ost].include? format
+				if filter_path = opts[:filter_path]
+					filter_path = filter_path.tr("\\", '/').gsub(/\/+/, '/').sub(/^\//, '').sub(/\/$/, '')
+				end
+				open filename do |io|
+					pst = Mapi::Pst.new io, @helper
+					pst.each do |message|
+						next unless message.type == :message
+						if filter_path
+							next unless message.path =~ /^#{Regexp.quote filter_path}(\/|$)/i
+						end
+						yield message
+					end
+				end
+			else
+				Mapi::Msg.open filename, nil, @helper, &block
+			end
+		end
+	end
+
+	def run
+		each_message(&method(:process_message))
+	end
+
+	def make_unique filename
+		@map ||= {}
+		return @map[filename] if !opts[:individual] and @map[filename]
+		try = filename
+		i = 1
+		try = filename.gsub(/(\.[^.]+)$/, ".#{i += 1}\\1") while File.exist?(try)
+		@map[filename] = try
+		try
+	end
+
+	def process_message message
+		# TODO make this more informative
+		mime_type = message.mime_type
+		return unless pair = Mapi::Message::CONVERSION_MAP[mime_type]
+
+		combined_map = {
+			'eml' => 'Mail.mbox',
+			'vcf' => 'Contacts.vcf',
+			'txt' => 'Posts.txt'
+		}
+
+		# TODO handle merged mode, pst, etc etc...
+		case message
+		when Mapi::Msg
+			if opts[:individual]
+				filename = message.root.ole.io.path.gsub(/msg$/i, pair.last)
+			else
+				filename = combined_map[pair.last] or raise NotImplementedError
+			end
+		when Mapi::Pst::Item
+			if opts[:individual]
+				filename = "#{message.subject.tr ' ', '_'}.#{pair.last}".gsub(FILTER_NON_FILE_NAME, '_')
+			else
+				filename = combined_map[pair.last] or raise NotImplementedError
+				filename = (message.path.tr(' /', '_.').gsub(FILTER_NON_FILE_NAME, '_') + '.' + File.extname(filename)).squeeze('.')
+			end
+			dir = File.dirname(message.instance_variable_get(:@node).pst.io.path)
+			filename = File.join dir, filename
+		else
+			raise
+		end
+
+		if dir = opts[:output_dir]
+			filename = File.join dir, File.basename(filename)
+		end
+
+		filename = make_unique filename
+
+		write_message = proc do |f|
+			data = message.send(pair.first).to_s
+			if !opts[:individual] and pair.last == 'eml'
+				# we do the append > style mbox quoting (mboxrd i think its called), as it
+				# is the only one that can be robuslty un-quoted. evolution doesn't use this!
+				f.puts "From mapitool@localhost #{Time.now.rfc2822}"
+				#munge_headers mime, opts
+				data.lines.each do |line|
+					if line =~ /^>*From /o
+						f.print '>' + line
+					else
+						f.print line
+					end
+				end
+			else
+				f.write data
+			end
+		end
+
+		if opts[:stdout]
+			write_message[STDOUT]
+		else
+			# Using binary mode. On Windows "\r\n" will become "\r\n\r\n" or such.
+			open filename, 'ab', &write_message
+		end
+	end
+
+	def munge_headers mime, opts
+		opts[:header_defaults].each do |s|
+			key, val = s.match(/(.*?):\s+(.*)/)[1..-1]
+			mime.headers[key] = [val] if mime.headers[key].empty?
+		end
+	end
+end
+
+def mapitool
+	opts = {:verbose => false, :action => :convert, :header_defaults => []}
+	op = OptionParser.new do |op|
+		op.banner = "Usage: mapitool [options] [files]"
+		#op.separator ''
+		#op.on('-c', '--convert', 'Convert input files (default)') { opts[:action] = :convert }
+		op.separator ''
+		op.on('-o', '--output-dir DIR', 'Put all output files in DIR') { |d| opts[:output_dir] = d }
+		op.on('-i', '--[no-]individual', 'Do not combine converted files') { |i| opts[:individual] = i }
+		op.on('-s', '--stdout', 'Write all data to stdout') { opts[:stdout] = true }
+		op.on('-f', '--filter-path PATH', 'Only process pst items in PATH') { |path| opts[:filter_path] = path }
+		op.on(      '--enable-pst', 'Turn on experimental PST support') { opts[:enable_pst] = true }
+		op.on('-e', '--ansi-encoding CHARSET', 'Use this text to charset for non Unicode text') { |charset| opts[:ansi_encoding] = charset }
+		op.on('-u', '--to-unicode', 'Convert ansi text to unicode') { opts[:to_unicode] = true }
+		#op.on('-d', '--header-default STR', 'Provide a default value for top level mail header') { |hd| opts[:header_defaults] << hd }
+		# --enable-pst
+		op.separator ''
+		op.on('-v', '--[no-]verbose', 'Run verbosely') { |v| opts[:verbose] = v }
+		op.on_tail('-h', '--help', 'Show this message') { puts op; exit }
+	end
+
+	files = op.parse ARGV
+
+	# for windows. see issue #2
+	STDOUT.binmode
+
+	Mapi::Log.level = Ole::Log.level = opts[:verbose] ? Logger::WARN : Logger::FATAL
+
+	tool = begin
+		Mapitool.new(files, opts)
+	rescue ArgumentError
+		puts $!
+		puts op
+		exit 1
+	end
+	
+	tool.run
+end
+
+mapitool
+
+__END__
+
+mapitool [options] [files]
+
+files is a list of *.msg & *.pst files.
+
+one of the options should be some sort of path filter to apply to pst items.
+
+--filter-path=
+--filter-type=eml,vcf
+
+with that out of the way, the entire list of files can be converted into a
+list of items (with meta data about the source).
+
+--convert
+--[no-]separate one output file per item or combined output
+--stdout
+--output-dir=.
+
+