mail_runner 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: c867be342a949392aa028eab45d263043929f6a7
-  data.tar.gz: 3f1eef044f8e09e8783e9b9de6938a7f0dd2248b
+  metadata.gz: ea440f11b72d87c1092c0d42120549bffbe0c997
+  data.tar.gz: 9ec7d59c90a08b354ccf904e3f21dec2372f9865
 SHA512:
-  metadata.gz: 2f3766091b2c171e155f6138b6f17df1a82387262e283f4caa5b46d8b64a4fb294bbb9b80bbade7b9049f826c8a61a8c080cf0ac1c00a9f75fc8603e03a0695f
-  data.tar.gz: 7a03821b4edace1182ba1e4c32f3801147c46333635e3964be3ff228e346a9961258cbb7157724bf4c19a190bb8ba3a4daeb1a277103c812d83eb895145704de
+  metadata.gz: a262d9ee7a41d399ac11ef6849ec6fe87dcb606b2ac53a423e916e87ede61120915f6d19c3918ee569af4d6342ab7caabcbf22ee968c18e089aa1cd6ba4e376c
+  data.tar.gz: fdb1fc94033e3cf60b00264700184eba923973aee4c93eacfdcc42d96e6b28fc69ec030fd69cf59849ac7d80e8bde43a9e9c85a6b3a88c4c8bb5f2d5d3e2ff6a

data/README.md

@@ -1,4 +1,7 @@
 #mail_runner
+
+[![Gem Version](https://badge.fury.io/rb/mail_runner.svg)](https://badge.fury.io/rb/mail_runner) [![Code Climate](https://codeclimate.com/github/kert-io/mail_runner/badges/gpa.svg)](https://codeclimate.com/github/kert-io/mail_runner)
+
 **WORK IN PROGRESS**. See Roadmap below for features not yet completed.
 
 MailRunner acts as your mailman picking up your email from an [MTA](https://en.wikipedia.org/wiki/Message_transfer_agent), such as Postfix, and then delivering it directly to your app sending each email object as json to webhook.  You can tell it to deliver locally or send it to any active webhook making it a functional mailserver for several apps.
@@ -105,28 +108,151 @@ keys | Value
 If for any reason, mailrunner is not able to deliver the mail to the specified webhook, it will add it to the mailrunner mail queue to process later. The usual reason this occurs is the webhook is unresponsive, it returns an error code or the server is down. Mailrunner will intermittently test the server if this occurs and once it is working properly, it will process the queue.  If mail is not being delivered, you can check the mailrunner log for details on what is happening on mailrunners end.
 
 ## Additional Options
-####Daemonize
+###Daemonize
  Use the `-d ` flag to turn mailrunner into a [daemon](https://en.wikipedia.org/wiki/Daemon_(computing)) & keep it running in the the background.  When running as a daemon, be sure to set the logfile path using the ` -L ` flag.
 
-####Logging
-  Mailrunner wil output all logging info to STDOUT if no logfile path is set.  To set a logfile path, use the `-L path/to/logfile.log ` flag followed by the absolute path to the logfile location.   If the file doesn't exist, it will be created, but the directory path must still be valid.
+###Logging
+  Mailrunner wil output all logging info to STDOUT if no logfile path is set.  
+
+  To set a logfile path, use the `-L path/to/logfile.log ` flag followed by the absolute path to the logfile location.   If the file doesn't exist, it will be created, but the directory path must still be valid.
+
+  Mailrunner logging is designed for leaving it running continuously in the background.  So the log doesn't become unusable, logs will automatically archive themselves in the same directory and start a new log each week.  
   
-####archive
-####Config
+###Archive
+Alongside delivery to any webhook hook, Mailrunner also supports archiving your emails. Beyond archival purposes, this is super convenient for chasing down bugs, feature testing, and system error recovery. Mailrunner can archive locally or to a cloud storage device.  Mailrunner currently supports AWS & Rackspace. 
+
+Use the Config file option when using the archive feature to simplify passing archive parameters to mailrunner.  Archive option notes are included directly in the [sample config file](https://gist.github.com/kert-io/3d8d24d048dd25801b7f)
+
+**File Naming**
+All archived messages are saved as json and use the unique message-id as the file name. (NOTE: All illegal file system characters have been replaced with underscores; check the message-id in the message header section when seeking the actual id.) 
+
+When looking for an archived copy of a message, you must locate it by the message-id.  To do so, run the id through the same name-scrubbing process used during the archive process: `/[#<$+%>!&*?=\/:@]/` are replaced with '_'.
+
+###Config File
 
 Mailrunner can also be launched with a config file storing all defaults in one place.  When using a config file, you can launch a mailrunner instance with `mailrunner -c /path/to/config.yml` leaving off all typically required flags.  [sample config file](https://gist.github.com/kert-io/3d8d24d048dd25801b7f)
 
 When using a config file you can set your defaults in the config file but still override them for one-off instances using flags.  The instance will launch according to the config file, but override  only the options passed manually with each flag.
 
-# Other usage Scenarios
-####dtach
-####Monit
+## Other usage Scenarios
+###Monit
+	NOTE: before setting up Monit, test manually with desired config file settings. 
+
+  I prefer to use [Monit](https://mmonit.com/monit/) to manage my worker bots.  Why Monit?  Because I like working with native linux config files when working at the system level. (Many others try to blend conventions and I find it leads to holes in seeing your system and its bugs completely.) 
+  
+  Monit will monitor your daemonized processes, restart them in case of failure and dutifully send you a notification or email with each action.  It makes it possible to pretty much start and walk away. (psst, it does a ton more too!!)
+  
+  You can tweak mailrunner to run with your preferred monitor, but I offer the monit setup here:
 
+**step 1 - Configure Upstart to manage mailrunner at the system service level**
+  
+  While monit monitors, it uses Upstart control processes directly. [Upstart](http://upstart.ubuntu.com) is the native linux process for starting and stopping processes and keeping track of all system processes. 
+
+* navigate to init folder where all Upstart config files are stored. Create a new .conf file for mailrunner & open it with text editor:
+      
+ ```sh
+ cd /etc/init
+ sudo touch mailrunner.conf
+ sudo vim mailrunner.conf
+ ```
+ 
+* Paste the following inside the .conf file. Substitute your username into the setuid & HOME variables in the User Variables section to suit your deployment and save. 
+	* **setuid** - login name of user account you used when installing mailrunner. 
+	* **HOME** - Note this is to locate your locally installed gems using rbenv. You can also modify uncomment th global install setup included as well.
+
+	```sh
+	#   sudo start mailrunner
+	#   sudo stop mailrunner
+	#   sudo status mailrunner
+	#
+	#   or use the service command:
+	#   sudo service mailrunner {start,stop,restart,status}
+	
+	description "Upstart control over MailRunner bots. From ruby gem mail_runner"
+	
+	# no "start on", we don't want to automatically start
+	stop on (runlevel [06])
+	
+	##########################################
+	# change to match your deployment user
+	setuid username
+	env HOME="/home/username"
+	############################################
+	script
+	# this script runs in /bin/sh by default
+	# respawn as bash so we can source in rbenv
+	exec /bin/bash <<'EOT'
+	
+	# pull in system rbenv
+	# source /etc/profile.d/rbenv.sh
+	# or 
+	# pull in user installed rbenv
+	  export PATH="$HOME/.rbenv/bin:$PATH"
+	  eval "$(rbenv init -)"
+	  export PATH="$HOME/.rbenv/plugins/ruby-build/bin:$PATH"
+	
+	#launch mailrunner
+	  exec mail_runner -c $HOME/mailrunner_config.yml
+	
+	EOT
+	end script
+	```
+       
+   * You can now start and stop mailrunner with the the following system level commands:
+   
+   ```sh
+   sudo start mailrunner
+   sudo stop mailrunner
+   sudo restart mailrunner
+   ```
+
+   * NOTE: You must use the Mailrunner Config file to use Upstart and Monit.  See Mailrunner config section above.  This Upstart config assumes you store the mailrunner_config.yml file in your home directory.
+   
+**step 2 - Install and set up Monit to monitor & manage Mailrunner**
+
+* Install Monit. see this [gist]() for guidance
+
+With Monit installed, it is easy to set it up to monitor Mailrunner. Monit stores the config files for each process it monitors in the conf.d folder.
+
+* navigate to conf.d folder. Create a new conf file for mailrunner & open it with text editor:
+
+	```sh
+	cd /etc/monit/conf.d
+	sudo touch mailrunner
+	sudo vim mailrunner
+	```
+* Paste the following inside the .conf file & save. You can modify the triggers to suit your workload and resource allocation, but best to just let it run and watch it for a while and fine tune it for you setup.
+
+	```sh
+	 # mailrunner
+  	check process mailrunner
+   	with matching mail_runner
+   	
+   	start program = "/bin/bash -c 'start mailrunner'"
+   	stop program = "/bin/bash -c 'stop mailrunner'"
+   	
+   	#process monitor triggers
+    if cpu is greater than 10% for 2 cycles then alert
+    if mem is greater than 3% for 1 cycles then restart
+    if 3 restarts within 5 cycles then timeout
+	```
+	
+* reload monit settings to pull in your new config
+
+	```sh
+	sudo monit reload
+	```
+* Start mailrunner using monit.  Check status to see it initializing
+
+	```sh
+	sudo monit start mailrunner
+	sudo monit status
+	```
+*  Monit has a ton more features like alerts & notifications.  I highly recommend setting these up and monit will let you know whenever it needs your attention.  Now, relax;-)
+	
+	
 #Roadmap
-* Archiving
-* Run from Config file
 * Single bot managing several mailboxes and webhooks
-* Run bots from config file
 * test server
 
 

data/lib/mail_runner/archive_manager.rb

@@ -0,0 +1,107 @@
+module MailRunner
+  module ArchivistBot
+
+    def self.add_to_archive_stack(msg, archive)
+      que_packet = [msg, archive]
+      $redis.lpush("archive_stack", que_packet.to_json)
+      $logger.info("Archivist") { "#add_to_archive_stack:: email added to queue for processing later." }
+    end
+
+    def self.stack_height
+      return $redis.llen("archive_stack")
+    end
+
+    def self.archive_stack
+      $logger.info("Archivist") { "Stack Height:: #{stack_height}"}
+      while stack_height > 0
+        $logger.info("Archivist") { "#archive_stack:: Processing stack item:: #{stack_height}"}
+        msg, archive = pop_from_stack
+
+        if archive["destination"] == 'local'
+          deliver_to_local_archive(msg, archive)
+        else
+          deliver_to_cloud_archive(msg, archive)
+        end
+      end
+    end
+
+    def self.pop_from_stack
+      key, que_packet = $redis.blpop("archive_stack", :timeout => 5) #timeout needed for MockRedis in Testing Env.
+      $logger.info("Archivist") { "#item popped from stack for processing"}
+      data = JSON::parse(que_packet)
+      msg = data[0]
+      archive = data[1]
+      return msg, archive
+    end
+
+    def self.deliver_to_local_archive(msg, archive_opts)
+      begin
+        filename = name_msg(msg)
+        target = File.open("#{archive_opts["local_archive"]}/#{filename}", 'w')
+        target.write(msg)
+        target.close()
+        $logger.info("Archivist") { "#Message archived to local"}
+      rescue => e
+        $logger.error("Archivist") { "#deliver_to_local_archive:: failed: #{e.inspect}"}
+        add_to_archive_stack(msg,archive_opts)
+      end
+    end
+
+    def self.deliver_to_cloud_archive(msg, archive_opts)
+      attempts = 0
+      begin
+        archive = establish_archive_link(archive_opts)
+        filename = name_msg(msg)
+        mimetype = "application/json"
+
+        saved = archive.files.create :key => filename, :body => msg, :Content_type => mimetype
+        $logger.info("Archivist") { "#Message archived to cloud"}
+        return saved #explicit for testing
+      rescue => e
+        $logger.error("Archivist") { "#deliver_to_cloud_archive:: attempt #{attempts} failed: #{e.inspect}"}
+        attempts += 1
+        retry unless attempts > 1
+        unless saved
+          $logger.info("Archivist") { "Too many failed attempts, restacking msg."}
+          add_to_archive_stack(msg,archive_opts)
+        end
+      end
+    end
+
+    def self.name_msg(msg)
+      msg_id = JSON.parse(msg)[0]["msg"]["headers"]["Message-ID"]
+      return "#{msg_id.gsub(/[#<$+%>!&*?=\/:@]/,'_')}.json"#clean illegal charaters
+    end
+
+    def self.establish_archive_link(archive_opts)
+      service = new_storage_object(archive_opts)
+      dir_name = archive_opts["directory"]
+      archive_directory = service.directories.new :key => dir_name
+      return archive_directory
+    end
+
+    def self.new_storage_object(archive_opts)
+      options = format_options(archive_opts)
+      Fog::Storage.new(options)
+    end
+
+    def self.format_options(archive_opts)
+      provider = archive_opts["provider"].downcase
+
+      options = {
+        :provider => archive_opts["provider"] 
+      }
+      if provider == "rackspace"
+        options[:"#{provider}_username"] = archive_opts["username"]     
+        options[:"#{provider}_api_key"] = archive_opts["api_key"]
+        options[:"#{provider}_region"] = archive_opts["region"]
+      elsif provider == "aws"
+        options[:"#{provider}_access_key_id"] = archive_opts["api_key"]
+        options[:"#{provider}_secret_access_key"] = archive_opts["secret_key"]
+      end
+
+      return options
+    end
+    
+  end
+end

data/lib/mail_runner/bot_helpers/helpers.rb

@@ -7,8 +7,14 @@ module BotHelpers
   	def self.print_monitoring_started_msg(bot)
   		$logger.info("Helpers") { "mailbox: #{bot.mailbox}" }
   		$logger.info("Helpers") { "path: #{bot.webhook}"}
-  		$logger.info("Helpers") {"archive: #{bot.archive}"}
-  		puts "Getter Bot is on the Job!"
+      unless bot.archive.nil?
+        if bot.archive[:destination] == 'cloud'
+  		  $logger.info("Helpers") {"archive: #{bot.archive[:provider]} :: #{bot.archive[:directory]}"}
+        else
+          $logger.info("Helpers") {"archive: #{bot.archive[:destination]} :: #{bot.archive[:local_archive]}"}
+        end
+      end
+  		puts "Getter Bot is on the Job!" 
   	end
 
     def self.convert_raw_mail_to_json(mail)

data/lib/mail_runner/bot_helpers/tests.rb

@@ -1,14 +1,10 @@
 module BotHelpers
 	module Tests  #series of initial command validation tests on launch. 
-  	def self.all_args_included?(args)
+  	
+    def self.all_args_included?(args)
     	if args[:mailbox].nil? or args[:webhook].nil?
     		raise ArgumentError, 'You must include mailbox & webhook minimum. Archive argument is optional. Add -h to see help.' 
   		end
-  		#if args.size > 3
-    	#	raise ArgumentError, 'You can only include mailbox, webhook & Archive argument. 3 max! Add -h to see help.' 
-  		#end
-	  		#?test format of mailbox?
-	  		#? test valid webhook format?
   	end
   	
   	def self.test_mailbox	(path)
@@ -21,7 +17,7 @@ module BotHelpers
     	begin
     		response = RestClient.head url
         MailRunner.manager_bot.update_webhook_status("live")
-    	rescue 
+    	rescue  
     		raise ArgumentError, "ERROR: \nMake sure the server is running and the webhook exists.\nNOTE:  Server must respond to http HEAD method.\nSee README.md for proper setup.\n"
     	end
     	unless response.code == 200
@@ -38,6 +34,30 @@ module BotHelpers
       end
     end
 
-  end
+    def self.test_archive(a_set)
+      if a_set[:destination] == 'local'
+        test_local_archive(a_set)
+      elsif a_set[:destination] == 'cloud'
+        test_cloud_archive_connection(a_set)
+      else
+        raise ArgumentError, "ERROR: Archive destination setting invalid."
+      end
+    end
+
+    def self.test_local_archive(a_set)
+      unless File.directory?(a_set[:local_archive])
+        raise ArgumentError, "ERROR: Invalid local archive path."
+      end
+    end
+
+    def self.test_cloud_archive_connection(a_set)
+      a_set = JSON.parse(a_set.to_json) #Must parse as json, similar to redis queues. Stringifies symbol keys.
+      begin
+        response = MailRunner::ArchivistBot.establish_archive_link(a_set)
+      rescue => e
+       raise ArgumentError, "ERROR: Archive connection failed. Check your archive config options or disable."
+      end
+    end
 
+  end
 end

data/lib/mail_runner/cli.rb

@@ -26,7 +26,7 @@ module MailRunner
           opts[:webhook] = arg
         end
 
-        o.on '-a', '--archive', "Set to true id you want mail archived." do |arg|
+        o.on '-a', '--archive', "Use config file to setup archive option." do |arg|
           opts[:archive] = arg
         end
 
@@ -93,23 +93,6 @@ module MailRunner
     def self.set_globals
       MailRunner.set_globals
     end
-
-    def self.banner
-      %q{
-               m,
-               `$b
-          .ss,  $$:         .,d$
-          `$$P,d$P'    .,md$P"'
-           ,$$$$$bmmd$$$P^'
-         .d$$$$$$$$$$P'
-         $$^' `"^$$$'       ____  _     _      _    _
-         $:     ,$$:       / ___|(_) __| | ___| | _(_) __ _
-         `b     :$$        \___ \| |/ _` |/ _ \ |/ / |/ _` |
-                $$:         ___) | | (_| |  __/   <| | (_| |
-                $$         |____/|_|\__,_|\___|_|\_\_|\__, |
-              .d$$                                       |_|
-      }
-    end
 	end
 end
 

data/lib/mail_runner/head_manager_bot.rb

@@ -3,6 +3,7 @@ require 'json'
 require 'base64'
 require 'rest-client'
 require 'redis'
+require 'fog'
 
 module MailRunner
 
@@ -24,7 +25,7 @@ module MailRunner
     def initialize
       @mailbox = nil
       @webhook = nil
-      @archive = false
+      @archive = nil
       @webhook_status = nil
     end
 
@@ -39,7 +40,7 @@ module MailRunner
     	
     	@mailbox = "/var/mail/#{opts[:mailbox]}"
     	@webhook = opts[:webhook]
-    	@archive = opts[:archive] == "true" ? true : false 
+    	@archive = opts[:archive].nil? ? nil : opts[:archive] 
     end
 
     def update_webhook_status(status)
@@ -47,8 +48,9 @@ module MailRunner
     end
 
     def test_options
-    	BotHelpers::Tests.test_mailbox(self.mailbox)
-    	BotHelpers::Tests.test_webhook(self.webhook)
+      BotHelpers::Tests.test_mailbox(self.mailbox)
+      BotHelpers::Tests.test_webhook(self.webhook)
+      BotHelpers::Tests.test_archive(self.archive) unless self.archive.nil?
     end
 
     def run
@@ -64,10 +66,15 @@ module MailRunner
     			delegate_delayed_queue_processing
     		end
 
+        if archive_stack?
+          delegate_archive_stack_processing
+        end
+
     		sleep 5
     	end
     end
 
+#### Delegation Methods ####
 
     def inbound_manager
     	MailRunner::InboundManagerBot
@@ -97,6 +104,22 @@ module MailRunner
     		puts msg.inspect
     	end
     end
+
+    def archivist
+      MailRunner::ArchivistBot
+    end
+
+    def archive_stack?
+      archivist.stack_height > 0
+    end
+
+    def delegate_archive_stack_processing
+      begin
+        archivist.archive_stack
+      rescue Exception => msg 
+        puts msg.inspect
+      end
+    end
 		
 	end
 end

data/lib/mail_runner/inbound_manager_bot.rb

@@ -2,7 +2,7 @@ module MailRunner
 
 	module InboundManagerBot
     
-    def self.process_inbound(mailbox, webhook, archive)
+    def self.process_inbound(mailbox, webhook, archive=nil)
       raw_mail = get_mail(mailbox)
       
       unless raw_mail.nil?
@@ -13,15 +13,18 @@ module MailRunner
           json_packet = BotHelpers::Helpers.convert_raw_mail_to_json(mail)
           
           deliver_mail(webhook, json_packet)
-          
-          if archive == true && queued.nil?
-            puts "we will archive email.\n"
+
+          #Mail archived regardless of delivery errors.
+          #Delayed mail processed by queue manager, so only archived once on initial pickup. 
+          if archive   
+            MailRunner::ArchivistBot.add_to_archive_stack(json_packet, archive)
           end
-          
         end
       end
     end
 
+###Delegation Methods###
+
     def self.get_mail(mailbox)
       BotHelpers::Runner.get_contents_from_mailbox(mailbox) 
     end
@@ -40,6 +43,6 @@ module MailRunner
       end
     end
 
-  end
 
+  end
 end

data/lib/mail_runner/version.rb

@@ -1,3 +1,3 @@
 module MailRunner
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

data/mail_runner.gemspec

@@ -24,6 +24,7 @@ Gem::Specification.new do |s|
   s.add_dependency "mail", "~> 2.5.4"
   s.add_dependency "json", "~> 1.8.1"
   s.add_dependency "rest-client", "~> 1.8.0"
+  s.add_dependency "fog", "~> 1.23.0"
 
 
   s.add_development_dependency 'rake', "~> 10.3.2"

data/test/helper.rb

@@ -5,6 +5,7 @@ require 'minitest/spec'
 require "minitest/reporters"
 require 'minitest-spec-context'
 require 'mock_redis'
+require 'fog'
 require_relative '../lib/mail_runner.rb'
 
 #Use Mock Redis during testing - Can't use any blocking methods!!

data/test/test_archivist.rb

@@ -0,0 +1,90 @@
+#####################
+
+
+
+######################
+require_relative 'helper'
+
+class TestArchivist < Minitest::Test
+	
+	####### Top Level Call Tests ##########
+	describe"Archivist::stack_height" do
+		before do
+			@archivist = MailRunner::ArchivistBot
+		end
+		it "returns an integer" do
+			assert_instance_of Fixnum, @archivist.stack_height
+		end
+	end
+
+	describe "Archivist::deliver_to_local_archive" do
+		before do
+			`mkdir /tmp/local_test_archive`
+			$redis.del("archive_stack") #zero list
+			@archivist = MailRunner::ArchivistBot
+			@filename = "_1447427485391-46840973-67af09f8-7a989c0d_mixmax.com_.json"
+			@msg = File.read("#{Dir[File.dirname(__FILE__)][0] + '/test_assets/' + @filename}")
+			@archive_opts = {"local_archive" => '/tmp/local_test_archive'}
+		end
+		after do
+			`rm -rf /tmp/local_test_archive`
+		end
+		context 'when local path is valid' do
+			it "successfully adds msg to archive" do
+				assert_silent { @archivist.deliver_to_local_archive(@msg, @archive_opts) }
+				assert_equal true, File.file?("#{@archive_opts["local_archive"]}" + "/" + "#{@filename}")
+			end
+		end
+
+		context 'when local path is invalid' do
+			it "adds msg to archive_stack" do
+				@archivist.deliver_to_local_archive(@msg, "/home/invalid_path")
+				assert_equal 1, $redis.llen('archive_stack')
+			end
+		end
+	end
+
+	describe "Archivist::deliver_to_cloud_archive" do
+		before do
+			$redis.del("archive_stack") #zero list
+      @archive_opts = JSON.parse(File.read("#{Dir[File.dirname(__FILE__)][0] + '/test_assets/archive_opts.json'}"))[0]
+			@msg = File.read("#{Dir[File.dirname(__FILE__)][0] + '/test_assets/_1447427485391-46840973-67af09f8-7a989c0d_mixmax.com_.json'}")
+			@archivist = MailRunner::ArchivistBot
+			#set for Mocking only. 
+			Fog.mock!
+			#Must create fake directory in mock stub
+			service = @archivist.new_storage_object(@archive_opts)
+			service.directories.create :key => 'raw_msg_archive'
+		end
+
+		context "when cloud archive credentials valid" do
+			it "successfully adds msg to archive" do
+				assert_instance_of Fog::Storage::Rackspace::File, @archivist.deliver_to_cloud_archive(@msg, @archive_opts)
+				assert_equal 0, $redis.llen('archive_stack') #doesn't hit stack
+			end
+		end
+
+		context "When cloud archive credentials invalid" do
+			before do
+				@archive_opts["api_key"] = nil
+				@archivist.deliver_to_cloud_archive(@msg, @archive_opts)
+			end
+			it "adds msg to archive_stack" do
+				assert_equal 1, $redis.llen('archive_stack')
+			end
+		end
+	end
+
+	describe 'Archivist::establish_archive_link' do
+		it 'Test included under: test BotHelpers:: Tests :: archive_tests' do
+		end
+	end
+
+	describe 'Archivist::format_options' do
+		it 'Test included under: test BotHelpers:: Tests :: archive_tests' do
+		end
+	end
+
+ ########################################
+
+end

data/test/test_assets/_1447427485391-46840973-67af09f8-7a989c0d_mixmax.com_.json

@@ -0,0 +1 @@
+[{"msg":{"raw_msg":"Return-Path: <Coming Soon_account@gmail.com>\r\nReceived: from Coming Soon.domain.co (localhost [127.0.0.1]) by mail.Coming Soon.domain.co (Postfix) with ESMTP id A11E62202F2 for <test@Coming Soon.domain.co>; Fri, 13 Nov 2015 10:11:33 -0500\r\nReceived: by mail.Coming Soon.domain.co (Postfix, from userid 1001) id 886C222199E; Fri, 13 Nov 2015 10:11:33 -0500\r\nReceived: from mail-qk0-f175.google.com (mail-qk0-f175.google.com [209.85.220.175]) by mail.Coming Soon.domain.co (Postfix) with ESMTPS id EBFDB22F2 for <test@Coming Soon.domain.co>; Fri, 13 Nov 2015 10:11:26 -0500\r\nReceived: by qkas77 with SMTP id s77so494528qka.0 for <test@Coming Soon.domain.co>; Fri, 13 Nov 2015 07:11:26 -0800\r\nReceived: from [127.0.0.1] (ec2-54-85-60-240.compute-1.amazonaws.com. [54.85.60.240]) by smtp.gmail.com with ESMTPSA id 31sm5383325qgy.13.2015.11.13.07.11.25 for <test@Coming Soon.domain.co> (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Nov 2015 07:11:25 -0800\r\nDate: Fri, 13 Nov 2015 15:11:25 +0000\r\nFrom: Coming Soon_account@gmail.com>\r\nTo: test <test@Coming Soon.domain.co>\r\nMessage-ID: <1447427485391-46840973-67af09f8-7a989c0d@mixmax.com>\r\nSubject: Local Archive\r\nMime-Version: 1.0\r\nContent-Type: multipart/alternative;\r\n boundary=\"----sinikael-?=_1-14474274854200.5982260059099644\";\r\n charset=UTF-8\r\nContent-Transfer-Encoding: 7bit\r\nX-Original-To: test@Coming Soon.domain.co\r\nDelivered-To: talkt@Coming Soon.domain.co\r\nAuthentication-Results: mail.Coming Soon.domain.co; dkim=pass reason=\"2048-bit key;\r\n unprotected key\" header.d=gmail.com header.i=@gmail.com header.b=bkC0SRlC;\r\n dkim-adsp=pass; dkim-atps=neutral\r\nX-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dev.fake.co\r\nX-Spam-Level: \r\nX-Spam-Status: No, score=0.8 required=5.0 tests=FREEMAIL_FROM,\r\n HTML_IMAGE_RATIO_02,HTML_MESSAGE,MIME_HTML_MOSTLY,RCVD_IN_MSPIKE_H2,\r\n T_DKIM_INVALID,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0\r\nDKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;\r\n h=content-type:from:to:subject:message-id:date:mime-version;\r\n bh=OvFWnVTuQzHtF6tDRv5J3zWMNR04gofxsqdKPQBrJOA=;\r\n b=bkC0SRlCOMx0TAfP1Xo6YxpFkUtwEO/cSPiZUE9MNXyKEx6EgMOkHpNJoNuwUrQDZx\r\n NkfERwLbmeP8qTmikvrbBnsDDg6QWQX05T99B5QSf7LOJ511eorgvrnP5rPrFdhDq4EK\r\n JwjTKDvFuCSHtx3WUlvo0Gw9rqrQRGNRjUTNSU1Rc5oRIZZdspPuFVkL7lsSnVGx0j5B\r\n Jh2Qq93bnh1FOTJ+xKCIRGSGuYcohkumwV8J5dmaebTPkUv56wj+E5T3mkLFN8zPiKz1\r\n kFlKq1CsB4XcFnSs48LEbswsAhXstUYbYFtZqwXWrDohvBIoLMlRN2gdGanMXP2iRA64 XCrg==\r\nX-Received: by 10.140.16.21 with SMTP id 21mr22446073qga.87.1447427486385;\r\n Fri, 13 Nov 2015 07:11:26 -0800 (PST)\r\nX-Mailer: Mixmax (mixmax.com)\r\nX-Virus-Scanned: ClamAV using ClamSMTP\r\n\r\n\r\n------sinikael-?=_1-14474274854200.5982260059099644\r\nContent-Type: text/plain;\r\n charset=UTF-8\r\nContent-Transfer-Encoding: quoted-printable\r\n\r\nLocal Arhive\r\n\r\n\r\nSent with Mixmax=\r\n\r\n------sinikael-?=_1-14474274854200.5982260059099644\r\nContent-Type: text/html;\r\n charset=UTF-8\r\nContent-Transfer-Encoding: quoted-printable\r\n\r\n\r\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://ww=\r\nw.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\r\n\r\n<html xmlns=3D\"http://www.w3.org/1999/xhtml\" xmlns:v=3D\"urn:schemas-micro=\r\nsoft-com:vml\" xmlns:o=3D\"urn:schemas-microsoft-com:office:office\">\r\n\r\n  <head>\r\n    <meta name=3D\"viewport\" content=3D\"width=3Ddevice-width, initial-scal=\r\ne=3D1.0\">\r\n    =\r\n\r\n    =\r\n\r\n    <!--[if gte mso 9]>\r\n    <xml>\r\n      <o:OfficeDocumentSettings>\r\n        <o:AllowPNG/>\r\n        <o:PixelsPerInch>96</o:PixelsPerInch>\r\n     </o:OfficeDocumentSettings>\r\n    </xml>\r\n    <![endif]-->\r\n    =\r\n\r\n    =\r\n\r\n    <style type=3D\"text/css\">* a:hover{cursor:pointer;}</style>\r\n    =\r\n\r\n\r\n    <style>body {-webkit-animation:bugfix infinite 1s;}@-webkit-keyframes=\r\n bugfix {from {position:relative;}to {position:relative;}}</style>\r\n\r\n  </head>\r\n\r\n  <body yahoo=3D\"fix\" style=3D\"word-wrap:normal;  word-break:break-word;\"=\r\n>\r\n\r\n    =\r\n\r\n    <!--[if (gte mso 9)|(IE)]>\r\n      <style>a,body {font-family:'Calibri',Arial,sans-serif;}img {border:=\r\nnone !important;-ms-interpolation-mode:bicubic;}td {mso-line-height-rule:=\r\nexactly !important;}table.border-outer {margin-bottom:15px !important;}.b=\r\norder-middle,.border-inner {width:100%;}td.mso-card-inner table {border-c=\r\nollapse:collapse !important;mso-table-lspace:0pt;mso-table-rspace:0pt;ver=\r\ntical-align:top;}.border-outer,.border-middle,.border-inner {border:none =\r\n!important;}.mso-border-outer,.mso-border-middle,.mso-border-inner {paddi=\r\nng:1px;}.mso-border-outer {background-color:rgb(245,255,255);}.mso-border=\r\n-middle {background-color:rgb(223,246,255);}.mso-border-inner {background=\r\n-color:rgb(153,176,225);}.preview-card {margin-bottom:0 !important;paddin=\r\ng:0 !important;}.outlook-only {display:block !important;max-height:none !=\r\nimportant;overflow:visible !important;}.outlook-com-only {display:none !i=\r\nmportant;font-size:0 !important;}#mso-one-whole {width:100% !important;}<=\r\n/style>\r\n    <![endif]-->\r\n    =\r\n\r\n    =\r\n\r\n    <style>a {word-wrap:normal;word-break:break-word;}@media only screen =\r\nand (max-width:600px) {.container[not-yahoo] {-webkit-text-size-adjust:no=\r\nne !important;}.container[not-yahoo],.container[not-yahoo] .palm-one-whol=\r\ne {width:100% !important;min-width:100% !important;}.container[not-yahoo]=\r\n .palm-one-half {width:50% !important;min-width:50% !important;box-sizing=\r\n:border-box;}blockquote table[lang=3D\"container\"],blockquote table[lang=3D=\r\n\"container\"] div,blockquote table[lang=3D\"container\"] table {width:auto !=\r\nimportant;min-width:0 !important;position:relative !important;}img {max-w=\r\nidth:100%;}[class=3D\"border-outer\"],[class=3D\"border-middle\"],[class=3D\"b=\r\norder-inner\"],[title=3D\"separator\"],[class=3D\"inner\"] {width:100% !import=\r\nant;}[class=3D\"innercell\"] {padding:8px !important;}.palm-block {display:=\r\nblock !important;}.container[not-yahoo] td.palm-one-whole {display:inline=\r\n-block !important;padding:0;}.container[not-yahoo] td.palm-one-whole:firs=\r\nt-child:not(:only-child) {margin-bottom:16px;}td.hostname {padding-top:3p=\r\nx !important;}}@media only screen and (min-width:601px) {.preview-card {m=\r\nax-width:600px !important;}}@media only screen and (min-device-width :320=\r\npx) and (max-device-width :568px),only screen and (min-device-width :768p=\r\nx) and (max-device-width :1024px),only screen and (max-device-width:640px=\r\n),only screen and (max-device-width:667px),only screen and (max-width:480=\r\npx){table[class=3D\"container\"] {width:100% !important;min-width:100% !imp=\r\nortant;}.p,.small,li,font[size=3D\"2\"],font[size=3D\"3\"] {font-size:17px !i=\r\nmportant;line-height:1.5 !important;}audio {margin-bottom:10px;}.backgrou=\r\nnd-contain {background-size:contain;}}@media only screen and (min-device-=\r\nwidth :320px) and (max-device-width :568px),only screen and (min-device-w=\r\nidth :768px) and (max-device-width :1024px),only screen and (min-device-w=\r\nidth :1224px)  {.container[not-yahoo] .message-wrapper {padding-top:6px;}=\r\n.container[not-yahoo] .apple-only[style] {display:block !important;max-he=\r\night:none !important;line-height:normal !important;overflow:visible !impo=\r\nrtant;height:auto !important;width:100% !important;position:relative !imp=\r\nortant;}.ExternalClass .ecxapple-only {display:none !important;}.containe=\r\nr[not-yahoo] .no-apple {display:none !important;}.container[not-yahoo] .n=\r\no-apple {display:block;}.container[not-yahoo] form {font-size:inherit;}.c=\r\nontainer[not-yahoo] form input[type=3D\"text\"] {height:43px;padding-left:4=\r\npx !important;}.container[not-yahoo] form button:hover {cursor:pointer;}@=\r\nmedia only screen and (min-device-width :1224px)  {.apple-mail-form {disp=\r\nlay:block !important;background-color:white !important;}}}</style>\r\n    =\r\n\r\n    =\r\n\r\n    <style>.ExternalClass {width:100%;}.ExternalClass .outlook-com-button=\r\n {display:block;}.ExternalClass button {height:auto;}.ExternalClass .outl=\r\nook-com-hidden {display:none !important;}.ExternalClass .outlook-com-only=\r\n {display:block !important;max-height:none !important;line-height:normal =\r\n!important;overflow:visible !important;height:auto !important;width:100% =\r\n!important;position:relative !important;}</style>\r\n    =\r\n\r\n\r\n    <table class=3D\"container\" lang=3D\"container\" not-yahoo=3D\"fix\" borde=\r\nr=3D\"0\" cellpadding=3D\"0\" cellspacing=3D\"0\" valign=3D\"top\" style=3D\"width=\r\n:100%;  margin-top:6px;\">\r\n      <tr>\r\n        <td valign=3D\"top\" class=3D\"message-wrapper\" style=3D\"line-height=\r\n: 1.31;  color: #222;  font-family: arial, sans-serif;\">\r\n\r\n          <!--[if mso]><table  border=3D\"0\" cellpadding=3D\"0\" cellspacing=\r\n=3D\"0\" valign=3D\"top\" style=3D\"border-collapse:separate;\"><tr><td valign=3D=\r\n\"top\"><![endif]-->\r\n            <div>Local Arhive</div><div><br></div><div><br></div><div><br=\r\n></div><div>\r\n  <div class=3D\"signature\" style=3D\"color:#999999;\">\r\n    Sent with <a target=3D\"_blank\" href=3D\"https://mixmax.com/s/LDQhPf8Ft=\r\n5nm5ccBu?utm_source=3Dmixmax&amp;utm_medium=3Demail&amp;utm_campaign=3Dsi=\r\ngnature_link&amp;utm_content=3Dsent_with_mixmax\">Mixmax</a>\r\n    =\r\n\r\n  </div>\r\n  =\r\n\r\n</div><img width=3D\"0\" height=3D\"0\" style=3D\"border:0; width:0px; height:=\r\n0px;\" src=3D\"https://app.mixmax.com/api/track/v2/eoHCRmlPtUqTlCTRt/i02bj5=\r\nCbpFWbnBEdy9GcrN2byBnI/gIvNmL1JXdrF2cuQ3cvBHQ0NXZ09VdyV3ahNnI/ISe0lmb11Wb=\r\nvNEI0NXZ09VdyV3ahNnI\" alt=3D\"\">\r\n          <!--[if mso]></td></tr></table><![endif]-->\r\n        </td>\r\n      </tr>\r\n    </table>\r\n\r\n  </body>\r\n</html>\r\n\r\n------sinikael-?=_1-14474274854200.5982260059099644--\r\n","headers":{"Received":"from [127.0.0.1] (ec2-54-85-60-240.compute-1.amazonaws.com. [54.85.60.240]) by smtp.gmail.com with ESMTPSA id 31sm5383325qgy.13.2015.11.13.07.11.25 for <fake_test@post.fake.co> (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Nov 2015 07:11:25 -0800","Date":"Fri, 13 Nov 2015 15:11:25 +0000","From":"Fake <fake_account@gmail.com>","To":"fake_test Community <fake_test@post.fake.co>","Message-ID":"<1447427485391-46840973-67af09f8-7a989c0d@mixmax.com>","Subject":"Local Archive","Mime-Version":"1.0","Content-Type":"multipart/alternative;","X-Original-To":"fake_test@post.fake.co","Delivered-To":"talkpost@dev.fake.co","Authentication-Results":"mail.dev.fake.co; dkim=pass reason=\"2048-bit key;","X-Spam-Checker-Version":"SpamAssassin 3.4.0 (2014-02-07) on dev.fake.co","X-Spam-Level":"","X-Spam-Status":"No, score=0.8 required=5.0 tests=FREEMAIL_FROM,","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;","h=content-type":"from:to:subject:message-id:date:mime-version;","X-Received":"by 10.140.16.21 with SMTP id 21mr22446073qga.87.1447427486385;","Fri, 13 Nov 2015 07":"11:26 -0800 (PST)","X-Mailer":"Mixmax (mixmax.com)","X-Virus-Scanned":"ClamAV using ClamSMTP"},"from_email":"fake_account@gmail.com","from_name":"K H","to":"test Community <test@Coming Soon.domain.co>","email":"test@post.fake.co","subject":"Local Archive","tags":"","sender":null,"spam_report":"spam report","text":"Local Arhive\n\n\nSent with Mixmax","html":"\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\n\n<html xmlns=\"http://www.w3.org/1999/xhtml\" xmlns:v=\"urn:schemas-microsoft-com:vml\" xmlns:o=\"urn:schemas-microsoft-com:office:office\">\n\n  <head>\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n    \n    \n    <!--[if gte mso 9]>\n    <xml>\n      <o:OfficeDocumentSettings>\n        <o:AllowPNG/>\n        <o:PixelsPerInch>96</o:PixelsPerInch>\n     </o:OfficeDocumentSettings>\n    </xml>\n    <![endif]-->\n    \n    \n    <style type=\"text/css\">* a:hover{cursor:pointer;}</style>\n    \n\n    <style>body {-webkit-animation:bugfix infinite 1s;}@-webkit-keyframes bugfix {from {position:relative;}to {position:relative;}}</style>\n\n  </head>\n\n  <body yahoo=\"fix\" style=\"word-wrap:normal;  word-break:break-word;\">\n\n    \n    <!--[if (gte mso 9)|(IE)]>\n      <style>a,body {font-family:'Calibri',Arial,sans-serif;}img {border:none !important;-ms-interpolation-mode:bicubic;}td {mso-line-height-rule:exactly !important;}table.border-outer {margin-bottom:15px !important;}.border-middle,.border-inner {width:100%;}td.mso-card-inner table {border-collapse:collapse !important;mso-table-lspace:0pt;mso-table-rspace:0pt;vertical-align:top;}.border-outer,.border-middle,.border-inner {border:none !important;}.mso-border-outer,.mso-border-middle,.mso-border-inner {padding:1px;}.mso-border-outer {background-color:rgb(245,255,255);}.mso-border-middle {background-color:rgb(223,246,255);}.mso-border-inner {background-color:rgb(153,176,225);}.preview-card {margin-bottom:0 !important;padding:0 !important;}.outlook-only {display:block !important;max-height:none !important;overflow:visible !important;}.outlook-com-only {display:none !important;font-size:0 !important;}#mso-one-whole {width:100% !important;}</style>\n    <![endif]-->\n    \n    \n    <style>a {word-wrap:normal;word-break:break-word;}@media only screen and (max-width:600px) {.container[not-yahoo] {-webkit-text-size-adjust:none !important;}.container[not-yahoo],.container[not-yahoo] .palm-one-whole {width:100% !important;min-width:100% !important;}.container[not-yahoo] .palm-one-half {width:50% !important;min-width:50% !important;box-sizing:border-box;}blockquote table[lang=\"container\"],blockquote table[lang=\"container\"] div,blockquote table[lang=\"container\"] table {width:auto !important;min-width:0 !important;position:relative !important;}img {max-width:100%;}[class=\"border-outer\"],[class=\"border-middle\"],[class=\"border-inner\"],[title=\"separator\"],[class=\"inner\"] {width:100% !important;}[class=\"innercell\"] {padding:8px !important;}.palm-block {display:block !important;}.container[not-yahoo] td.palm-one-whole {display:inline-block !important;padding:0;}.container[not-yahoo] td.palm-one-whole:first-child:not(:only-child) {margin-bottom:16px;}td.hostname {padding-top:3px !important;}}@media only screen and (min-width:601px) {.preview-card {max-width:600px !important;}}@media only screen and (min-device-width :320px) and (max-device-width :568px),only screen and (min-device-width :768px) and (max-device-width :1024px),only screen and (max-device-width:640px),only screen and (max-device-width:667px),only screen and (max-width:480px){table[class=\"container\"] {width:100% !important;min-width:100% !important;}.p,.small,li,font[size=\"2\"],font[size=\"3\"] {font-size:17px !important;line-height:1.5 !important;}audio {margin-bottom:10px;}.background-contain {background-size:contain;}}@media only screen and (min-device-width :320px) and (max-device-width :568px),only screen and (min-device-width :768px) and (max-device-width :1024px),only screen and (min-device-width :1224px)  {.container[not-yahoo] .message-wrapper {padding-top:6px;}.container[not-yahoo] .apple-only[style] {display:block !important;max-height:none !important;line-height:normal !important;overflow:visible !important;height:auto !important;width:100% !important;position:relative !important;}.ExternalClass .ecxapple-only {display:none !important;}.container[not-yahoo] .no-apple {display:none !important;}.container[not-yahoo] .no-apple {display:block;}.container[not-yahoo] form {font-size:inherit;}.container[not-yahoo] form input[type=\"text\"] {height:43px;padding-left:4px !important;}.container[not-yahoo] form button:hover {cursor:pointer;}@media only screen and (min-device-width :1224px)  {.apple-mail-form {display:block !important;background-color:white !important;}}}</style>\n    \n    \n    <style>.ExternalClass {width:100%;}.ExternalClass .outlook-com-button {display:block;}.ExternalClass button {height:auto;}.ExternalClass .outlook-com-hidden {display:none !important;}.ExternalClass .outlook-com-only {display:block !important;max-height:none !important;line-height:normal !important;overflow:visible !important;height:auto !important;width:100% !important;position:relative !important;}</style>\n    \n\n    <table class=\"container\" lang=\"container\" not-yahoo=\"fix\" border=\"0\" cellpadding=\"0\" cellspacing=\"0\" valign=\"top\" style=\"width:100%;  margin-top:6px;\">\n      <tr>\n        <td valign=\"top\" class=\"message-wrapper\" style=\"line-height: 1.31;  color: #222;  font-family: arial, sans-serif;\">\n\n          <!--[if mso]><table  border=\"0\" cellpadding=\"0\" cellspacing=\"0\" valign=\"top\" style=\"border-collapse:separate;\"><tr><td valign=\"top\"><![endif]-->\n            <div>Local Arhive</div><div><br></div><div><br></div><div><br></div><div>\n  <div class=\"signature\" style=\"color:#999999;\">\n    Sent with <a target=\"_blank\" href=\"https://mixmax.com/s/LDQhPf8Ft5nm5ccBu?utm_source=mixmax&amp;utm_medium=email&amp;utm_campaign=signature_link&amp;utm_content=sent_with_mixmax\">Mixmax</a>\n    \n  </div>\n  \n</div><img width=\"0\" height=\"0\" style=\"border:0; width:0px; height:0px;\" src=\"https://app.mixmax.com/api/track/v2/eoHCRmlPtUqTlCTRt/i02bj5CbpFWbnBEdy9GcrN2byBnI/gIvNmL1JXdrF2cuQ3cvBHQ0NXZ09VdyV3ahNnI/ISe0lmb11WbvNEI0NXZ09VdyV3ahNnI\" alt=\"\">\n          <!--[if mso]></td></tr></table><![endif]-->\n        </td>\n      </tr>\n    </table>\n\n  </body>\n</html>\n"}}]

data/test/test_assets/archive_opts.json

@@ -0,0 +1,7 @@
+[{
+	"destination": "cloud",
+	"provider": "Rackspace",
+  "username": "test_username",             
+	"api_key": "RACKSPACE__API_KEY",                                       
+	"directory": "raw_msg_archive" 
+}]

data/test/test_assets/test_config.yml

@@ -4,16 +4,19 @@
 #absolute means '~/' must be written as '/home/username/...'
 ################################################
 
-:mailbox: test_mailbox #mbox file name
-:webhook: localhost:4000/test_hook
-:archive:
-  :provider: rackspace
-  :username: name
-  :api_key: key
-  :region: :iad
+:mailbox: test_mailbox        #mbox file name
+:webhook: localhost:4000/test_hook  #or any remote or local url endpoint
 :logfile: /home/user/mailrunner.log 
 :daemon:  false
-:verbose: false #run in debug mode - adds a few more notes to help track down bugs.
-
-
+:verbose: false           #run in debug mode - adds a few more notes to help track down bugs.
 
+#comment out all below to skip archive feature.
+:archive:
+  :destination: local               #local or cloud
+  :local_archive: /path/to/local_archive #path/to/archive_directory #only if destination set to local. folder must exist prior to launching!!
+#  :provider: rackspace              #Rackspace, AWS
+#  :username: my_name                #Rackspace only 
+#  :api_key: my_api_key                #rackspace 'api_key' or aws 'access_key_id'
+# :secret_key: some_key             #aws only - 'secret_access_key'
+#  :region: :iad                     #must be set for aws & Rackspace
+#  :directory: raw_msg_archive       #name of cloud directory

data/test/test_assets/test_config_archive_cloud.yml

@@ -0,0 +1,22 @@
+# Mailrunner config file. 
+#To use the config file, just add teh -c flag when launching mailrunner.
+# mailrunner -c /absolute/path_to/config.yml
+#absolute means '~/' must be written as '/home/username/...'
+################################################
+
+:mailbox: test_mailbox        #mbox file name
+:webhook: localhost:4000/test_hook  #or any remote or local url endpoint
+:logfile: /home/user/mailrunner.log 
+:daemon:  false
+:verbose: false           #run in debug mode - adds a few more notes to help track down bugs.
+
+#comment out all below to skip archive feature.
+:archive:
+  :destination: cloud               #local or cloud
+# :local_archive: /path/to/local_archive #path/to/archive_directory #only if destination set to local. folder must exist prior to launching!!
+  :provider: rackspace              #Rackspace, AWS
+  :username: my_name                #Rackspace only 
+  :api_key: my_api_key                #rackspace 'api_key' or aws 'access_key_id'
+# :secret_key: some_key             #aws only - 'secret_access_key'
+  :region: :iad                     #must be set for aws & Rackspace
+  :directory: raw_msg_archive       #name of cloud directory

data/test/test_bot_helpers.rb

@@ -44,6 +44,73 @@ class TestBotHelpers < Minitest::Test
 			assert_raises(ArgumentError) {@test.test_webhook("http://127.0.0.1:4000/faulty_hook")}
 		end
 	end
+
+	describe "test BotHelpers:: Tests :: archive_tests" do
+		context "method : test_archive" do
+			before do
+				@arch_opts ={:destination => 'random'}
+				@test = BotHelpers::Tests
+			end
+			it "raises argument error if destination not cloud or local" do
+				assert_raises(ArgumentError) {@test.test_archive(@arch_opts)}
+			end
+		end
+
+		context "method : test_local_archive" do
+			before do
+				`mkdir /tmp/temp_archive`
+				@arch_opts = {
+					:destination =>  'local',
+					:local_archive => '/tmp/temp_archive'
+				}
+				@test = BotHelpers::Tests
+			end
+			after do 
+				`rm -rf /tmp/temp_archive`
+			end
+
+			it "is silent if directory exists" do
+				assert_silent {@test.test_local_archive(@arch_opts)}
+			end
+
+			it "raises error if directory doesn't exist" do 
+				@arch_opts[:local_archive] = '/tmp/other_archive'
+				assert_raises(ArgumentError) {@test.test_local_archive(@arch_opts)}
+			end
+
+		end
+
+		context "method : test_cloud_archive_connection" do
+			before do
+				require '~/.apikeys'
+				#Set to test live Rackspace account. pulls in keys from require location above. 
+				#to test live, make .apikeys file in home directory and add keys there to keep out of git.
+				#to test only stub, uncomment #Fog.mock! & change username & api_key in arch_opts
+				@test = BotHelpers::Tests
+				#Fog.mock!
+				@arch_opts = {
+					:destination => 'cloud',
+					:provider => 'Rackspace',
+  				:username => "#{RACKSPACE_USERNAME}",             
+			 	 	:api_key => "#{RACKSPACE__API_KEY}",             
+					:secret_key => "sdsd34d334",                             
+					:directory => 'raw_msg_archive' 
+				}
+			end
+			it "successfully connects to cloud archive" do
+				assert_silent {@test.test_cloud_archive_connection(@arch_opts)}
+			end
+
+			it "raises error if fails connect to cloud archive" do
+				@arch_opts[:api_key] = nil
+				assert_raises(ArgumentError) {@test.test_cloud_archive_connection(@arch_opts)}
+			end
+
+		end
+
+	end
+
+
  ########################################
 
 

data/test/test_cli.rb

@@ -135,5 +135,39 @@ class TestCLI< Minitest::Test
       assert_equal '/home/user/flag.log', @options[:logfile]
     end
   end
+
+  describe 'local archive setup via config' do
+    before do
+      @cli = MailRunner::CLI
+      @options = @cli.parse_options(['mailrunner', '-c', './test/test_assets/test_config.yml'])
+    end
+    
+    it 'sets destination to local' do
+      assert_equal 'local',  @options[:archive][:destination]
+    end
+
+    it 'sets local archive path' do
+      assert_equal '/path/to/local_archive', @options[:archive][:local_archive]
+    end
+  end
+
+  describe 'remote archive setup via config' do
+    before do
+      @cli = MailRunner::CLI
+      @options = @cli.parse_options(['mailrunner', '-c', './test/test_assets/test_config_archive_cloud.yml'])
+    end
+    
+    it 'sets destination to cloud' do
+      assert_equal 'cloud',  @options[:archive][:destination]
+    end
+
+    it 'sets all cloud destination params' do
+      assert_equal 'rackspace', @options[:archive][:provider]
+      assert_equal 'my_name', @options[:archive][:username]
+      assert_equal 'my_api_key', @options[:archive][:api_key]
+      assert_equal :iad, @options[:archive][:region]
+      assert_equal 'raw_msg_archive', @options[:archive][:directory]
+    end
+  end
 	
-end
+end

data/test/{test_mail_runner.rb → test_head_manager_bot.rb}

@@ -1,16 +1,16 @@
 #####################
 # Several tests require a live server behind the test webhook.  
-# will cause sveral fails if not. Must be Post method.
+# will cause several fails if not. Must be Post method.
 
 
 
 ######################
 require_relative 'helper'
 
-class TestMailRunnerManager < Minitest::Test
+class TestHeadManager < Minitest::Test
 	
 	####### Top Level Call Tests ##########
-	describe 'MailRunner Manager initialize' do
+	describe 'MR Manager::MailRunner Manager initialize' do
 		before do
 			@bot = MailRunner.initialize_manager_bot
 		end
@@ -19,14 +19,14 @@ class TestMailRunnerManager < Minitest::Test
 			assert_instance_of MailRunner::ManagerBot, @bot 
 			assert_equal nil, @bot.mailbox
 			assert_equal nil, @bot.webhook 
-			assert_equal false, @bot.archive
+			assert_equal nil, @bot.archive
 		end
 
 		it "includes BotHelpers" do
 		end
 	end
 
-	describe "Parse Options method" do
+	describe "MR Manager::Parse Options method" do
 		before do
 			@bot = MailRunner.initialize_manager_bot
 			@opts = {:mailbox => "box_name", :webhook => "webhook/path"}
@@ -36,17 +36,17 @@ class TestMailRunnerManager < Minitest::Test
 			@bot.parse_options(@opts)
 			assert_equal "/var/mail/box_name", @bot.mailbox
 			assert_equal "webhook/path", @bot.webhook
-			assert_equal false, @bot.archive
+			assert_equal nil, @bot.archive
 		end
 
 		it "archive = true if passed archive argument" do
-			@opts[:archive] = "true"
+			@opts[:archive] = true
 			@bot.parse_options(@opts)
 			assert_equal true, @bot.archive
 		end
 	end
 
-	describe "Run Method" do
+	describe "MR Manager::Run Method" do
 		before do
 			@bot = MailRunner.initialize_manager_bot
 			@bot.mailbox = "/var/mail/root" #app 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: mail_runner
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Kert Heinecke
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-11-07 00:00:00.000000000 Z
+date: 2015-11-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: mail
@@ -52,6 +52,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 1.8.0
+- !ruby/object:Gem::Dependency
+  name: fog
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.23.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.23.0
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
@@ -164,6 +178,7 @@ files:
 - Rakefile
 - bin/mail_runner
 - lib/mail_runner.rb
+- lib/mail_runner/archive_manager.rb
 - lib/mail_runner/bot_helpers/helpers.rb
 - lib/mail_runner/bot_helpers/runner.rb
 - lib/mail_runner/bot_helpers/tests.rb
@@ -176,16 +191,20 @@ files:
 - lib/mail_runner/version.rb
 - mail_runner.gemspec
 - test/helper.rb
+- test/test_archivist.rb
+- test/test_assets/_1447427485391-46840973-67af09f8-7a989c0d_mixmax.com_.json
+- test/test_assets/archive_opts.json
 - test/test_assets/empty.txt
 - test/test_assets/inline_attachments.email
 - test/test_assets/test.email
 - test/test_assets/test.json
 - test/test_assets/test_config.yml
+- test/test_assets/test_config_archive_cloud.yml
 - test/test_assets/with_attachments.email
 - test/test_bot_helpers.rb
 - test/test_cli.rb
+- test/test_head_manager_bot.rb
 - test/test_inbound_manager_bot.rb
-- test/test_mail_runner.rb
 - test/test_queue_manager_bot.rb
 homepage: https://github.com/kert-io/mailRunner
 licenses: