mail_runner 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: c867be342a949392aa028eab45d263043929f6a7
+  data.tar.gz: 3f1eef044f8e09e8783e9b9de6938a7f0dd2248b
+SHA512:
+  metadata.gz: 2f3766091b2c171e155f6138b6f17df1a82387262e283f4caa5b46d8b64a4fb294bbb9b80bbade7b9049f826c8a61a8c080cf0ac1c00a9f75fc8603e03a0695f
+  data.tar.gz: 7a03821b4edace1182ba1e4c32f3801147c46333635e3964be3ff228e346a9961258cbb7157724bf4c19a190bb8ba3a4daeb1a277103c812d83eb895145704de

data/.gitignore

@@ -0,0 +1,11 @@
+# Ignore bundler config
+.DS_Store
+/.bundle
+
+# Ignore the default SQLite database.
+
+
+# Ignore all logfiles and tempfiles.
+Gemfile.lock
+*.gem
+*.do_not_track

data/README.md

@@ -0,0 +1,149 @@
+#mail_runner
+**WORK IN PROGRESS**. See Roadmap below for features not yet completed.
+
+MailRunner acts as your mailman picking up your email from an [MTA](https://en.wikipedia.org/wiki/Message_transfer_agent), such as Postfix, and then delivering it directly to your app sending each email object as json to webhook.  You can tell it to deliver locally or send it to any active webhook making it a functional mailserver for several apps.
+
+MailRunner, although packaged as a gem, only provides a [CLI](https://en.wikipedia.org/wiki/Command-line_interface).  You can launch one or several mailrunner bots via the CLI, daemonize them to run permanently or manage them using a process manager such as monit.  This also means, you can use it as a standalone ruby mail service alongside apps in any other language.
+
+###Note: 
+Mailrunner is designed for a very specific use case.  If you are looking for a gem to pull email from an existing regular old email account, via pop or imap, this is not the gem for you (check out [Mailroom](https://github.com/tpitale/mail_room)).  If you want a super simple setup that allows you to expose a number of different email addresses for recieving mail in your app, and the ability to suport new ones automatically, then MailRunner is for you.
+
+###Requirements 
+* Postfix MTA setup as to recieve email.  Basic setup instructions here. 
+
+###Installation
+``` 
+gem install mailrunner
+```
+
+#Usage
+Mailrunner is built with a CLI that is used to launch each bot. When in doubt about commands, add the -h flag. 
+```
+mailrunner -h
+```
+
+the basic commands
+
+```
+Usage: mail_runner [options]
+    -m, --mailbox MAILBOX            Name of Mailbox to watch
+    -w, --webhook URL                Complete url of webhook to deliver mail to
+    -a, --archive                    Set to true id you want mail archived.
+    -d, --daemon                     Daemonize process. Be sure to add logfile path.
+    -L, --logfile                    Absolute path to log file.
+    -c, --config                     Path to YAML config file.
+    -v, --verbose                    Logger runs in debug mode.
+```
+
+The mailbox and webhook options are required; all others are optional.  
+
+### basics
+1 . To launch a basic mailrunner bot
+
+```
+mailrunner -m mailbox\_name -w http://127.0.0.1:3000/some_webhook_for_mail
+```
+
+* the mailbox_name is a registered account with postfix: i.e. a system account.  
+	* Best to create these as users with no login privileges and no home directory.  You don't have to do it this way, but one less security issue to worry about and it's sole purpose is a mail pass-through anyway.  [Setup Instructions]().
+
+* the webhook is any webhook. 
+	* The example is a local path, but you can easily use any valid url.  The url MUST accept both the PUT and the HEAD http methods.  The HEAD is used to verify the url upon launching the bot. A simple HEAD example in sintra:
+
+```
+head '/some_webhook_for_mail' do
+  status 200
+end
+
+```
+
+2 . Mail is delivered as a standard POST request with a single parameter: mail_runner_event. Each email is a json encoded array with a single item containing a hash with key "msg".   Just recieve it, parse it and ...
+
+```
+post '/some_webhook_for_mail' do
+  raw_message = params[:mail_runner_envelope]
+  mail = JSON.parse(raw_message)[0]["msg"]
+```
+
+From there you can call all variables of the mail object with:
+```
+mail["key"]
+```
+
+### Format of the mail object
+The format of the decoded json object is a:
+
+keys | Value 
+ --- | ------ 
+**raw_msg**|	the full content of the email received.
+**headers**| an array of the headers received for the email: ‘Dkim-Signature’, ‘Date’, ‘Message-Id’, etc.
+**from_email**|	from email address 
+**from_name**|	from name 
+**to**|	all recipients
+**email**|	email address where email was received
+**subject**|	the subject line 
+**tags**|	any tags applied
+**sender**|	the Mandrill sender of the message
+**text**|	text version of the email body
+**html**|	HTML version of the email body
+**attachments**|	an array of any attachments. If there are no attachments, key is omitted. See format below.
+**images**|	an array of any images. If there are no attachments, key is omitted. See format below.	
+**spam_report**|  -Coming Soon.  Requires postfix installation with clamAV.
+
+ 
+
+When images or attachments are included, each one will contain the following hash:
+
+keys | Value 
+ --- | ------ 
+**name**| file name
+**type**|	MIME type
+**content**| raw content of file
+**base64**|	boolean - Base64 encoded?
+
+###Delayed Queue
+If for any reason, mailrunner is not able to deliver the mail to the specified webhook, it will add it to the mailrunner mail queue to process later. The usual reason this occurs is the webhook is unresponsive, it returns an error code or the server is down. Mailrunner will intermittently test the server if this occurs and once it is working properly, it will process the queue.  If mail is not being delivered, you can check the mailrunner log for details on what is happening on mailrunners end.
+
+## Additional Options
+####Daemonize
+ Use the `-d ` flag to turn mailrunner into a [daemon](https://en.wikipedia.org/wiki/Daemon_(computing)) & keep it running in the the background.  When running as a daemon, be sure to set the logfile path using the ` -L ` flag.
+
+####Logging
+  Mailrunner wil output all logging info to STDOUT if no logfile path is set.  To set a logfile path, use the `-L path/to/logfile.log ` flag followed by the absolute path to the logfile location.   If the file doesn't exist, it will be created, but the directory path must still be valid.
+  
+####archive
+####Config
+
+Mailrunner can also be launched with a config file storing all defaults in one place.  When using a config file, you can launch a mailrunner instance with `mailrunner -c /path/to/config.yml` leaving off all typically required flags.  [sample config file](https://gist.github.com/kert-io/3d8d24d048dd25801b7f)
+
+When using a config file you can set your defaults in the config file but still override them for one-off instances using flags.  The instance will launch according to the config file, but override  only the options passed manually with each flag.
+
+# Other usage Scenarios
+####dtach
+####Monit
+
+#Roadmap
+* Archiving
+* Run from Config file
+* Single bot managing several mailboxes and webhooks
+* Run bots from config file
+* test server
+
+
+#Testing
+#Other helpful setup links
+* Installing Postfix with spamassasin & clamav
+* Setting up Postfix to work with mailrunner
+* Setting up Monit to manage your mailrunner processes
+
+#License
+(The MIT License)
+
+Copyright (c) 2015 Kert Heinecke
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+

data/Rakefile

@@ -0,0 +1,9 @@
+# file: Rakefile
+require 'rake/testtask'
+
+Rake::TestTask.new do |task|
+  task.libs << %w(test lib)
+  task.pattern = 'test/test_*.rb'
+end
+
+task :default => :test

data/bin/mail_runner

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+#require 'mail_runner' 
+require_relative '../lib/mail_runner.rb' # use during dev to avoid rebuilding gem
+
+MailRunner::CLI.start(ARGV)

data/lib/mail_runner/bot_helpers/helpers.rb

@@ -0,0 +1,95 @@
+module BotHelpers
+  module Helpers
+    #Based on Mbox protocol. maildir not yet supported.
+    #? in ?\.?[a-zA-Z]{2,4} makes compatible with local mail and internal process logging.
+    # working test case: http://rubular.com/r/N5iHbxk1q1
+    REGEX_POSTFIX_MESSAGE_DELIMITER =/^From\W\b[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+?\.?[a-zA-Z]{2,4}\b\W{1,}[a-zA-Z]{3}\W{1,}\w{3}\W{1,}\d{1,2}\W\d{2}:\d{2}:\d{2}\W\d{4}\n/
+  	def self.print_monitoring_started_msg(bot)
+  		$logger.info("Helpers") { "mailbox: #{bot.mailbox}" }
+  		$logger.info("Helpers") { "path: #{bot.webhook}"}
+  		$logger.info("Helpers") {"archive: #{bot.archive}"}
+  		puts "Getter Bot is on the Job!"
+  	end
+
+    def self.convert_raw_mail_to_json(mail)
+      mail_array = []
+      header = parse_header(mail.header)
+      from = parse_from(header[:From])
+      attachments = parse_attachments(mail.attachments)
+      
+      msg = {
+        :raw_msg => mail.to_s, 
+        :headers => header,
+        :from_email => from[1],
+        :from_name => from[0],
+        :to => header[:To],
+        :email => header[:'X-Original-To'],
+        :subject => header[:Subject],
+        :tags => '',
+        :sender => header[:Sender],
+        :spam_report => 'spam report'
+      }
+
+      msg[:text] = mail.text_part.decoded unless mail.text_part.nil?
+      msg[:html] = mail.html_part.decoded unless mail.html_part.nil?
+      #omitted unless attachments 
+      msg[:attachments] = attachments[:a_array] unless attachments[:a_array].empty?
+      msg[:images] = attachments[:i_array] unless attachments[:i_array].empty?
+
+
+      hash = {
+        :msg => msg
+      }
+
+      mail_array << hash
+      return  mail_array.to_json
+    end
+
+
+    def self.parse_header(header_contents)
+      hash = {}
+      header = header_contents.to_s.split(/\r\n/)
+      header.each do |h|
+        parts = h.split(/:/, 2)
+        unless parts[0].nil? or parts[1].nil?
+          key = parts[0].strip.to_sym
+          next if key =~ /Return-Path/i
+          value = parts[1].strip
+          hash[key] = parts[1].strip
+        end
+      end
+      return hash
+    end
+
+    def self.parse_from(from)
+      from = from.split(/</)
+      from[0] = from[0].strip
+      from[1] = from[1].strip.gsub(/>/,'')
+      return from
+    end
+
+    def self.parse_attachments(attachments)
+      a_array = []
+      i_array = []
+      attachments.each do |att|
+        disposition = att.content_disposition.split(/;/)[0]
+        item = {
+          :name => att.filename,
+          :type => att.content_type.split(/;/)[0],
+          :content => Base64.encode64(att.body.decoded),
+          :base64 => true,
+          :attachment_id => att.content_id
+        }
+        if disposition == 'inline'
+          i_array << item
+        else
+          a_array << item
+        end
+      end
+      return {:a_array => a_array, :i_array => i_array}
+    end
+
+  end
+
+end
+

data/lib/mail_runner/bot_helpers/runner.rb

@@ -0,0 +1,53 @@
+module BotHelpers
+
+	module Runner
+    def self.get_contents_from_mailbox(mailbox)
+      unless File.zero?(mailbox)
+
+        file = File.open(mailbox, 'r+')
+        file.flock(File::LOCK_EX)     #lock file so no other process uses it while open. NOTE: lock before read, so other locks can finish executing.
+        raw_contents = file.read      # read contents to var so we can release lock and process later
+        file.truncate(0)              #clear mbox file once read
+        file.close                    #close file to release lock prior to processing contents.
+      
+        #porcess Mail content into array of individual messages
+        raw_mail = raw_contents.split(BotHelpers::Helpers::REGEX_POSTFIX_MESSAGE_DELIMITER)
+        unless raw_mail.size <= 1
+          raw_mail.shift #remove empty from split
+        end
+
+        $logger.info("Runner") { "#get_contents_from_mailbox:: #{raw_mail.size} Mail messagess retrieved"}
+        return raw_mail
+      end
+    end
+
+    def self.post_to_hook(webhook, parcel)
+      begin
+      	response = RestClient.post webhook, :mail_runner_envelope => parcel, :content_type => :json, :accept => :json
+        
+        $mad_statter.incr_stat("mail delivered")
+        $logger.info("Runner") { 
+          "#post_to_hook::response code:#{response.code}\n" + 
+          "\tEmail from: #{JSON.parse(parcel)[0]['msg']['from_email']}  to: #{JSON.parse(parcel)[0]['msg']['email']}\n" + 
+          "\tPosted to: #{webhook}"
+        }
+
+        $logger.debug("Runner") { "#post_to_hook::response header:#{response.headers}"}
+        
+        MailRunner.manager_bot.update_webhook_status("live")
+      rescue 
+        $logger.error("Runner") { "#post_to_hook::ABORT: Server appears to be down. Make sure the server is running."}
+        MailRunner.manager_bot.update_webhook_status("down")
+    		raise ArgumentError
+    	end
+      
+      unless response.code == 200
+    		$logger.error("Runner") { "#post_to_hook::ABORT: Invalid Webhook. Response not 200. NOTE, Must respond to http HEAD method."}
+        raise
+    	end
+      return response
+    end
+    
+  end
+
+end

data/lib/mail_runner/bot_helpers/tests.rb

@@ -0,0 +1,43 @@
+module BotHelpers
+	module Tests  #series of initial command validation tests on launch. 
+  	def self.all_args_included?(args)
+    	if args[:mailbox].nil? or args[:webhook].nil?
+    		raise ArgumentError, 'You must include mailbox & webhook minimum. Archive argument is optional. Add -h to see help.' 
+  		end
+  		#if args.size > 3
+    	#	raise ArgumentError, 'You can only include mailbox, webhook & Archive argument. 3 max! Add -h to see help.' 
+  		#end
+	  		#?test format of mailbox?
+	  		#? test valid webhook format?
+  	end
+  	
+  	def self.test_mailbox	(path)
+    	unless File.file?(path)
+    		raise ArgumentError, 'ERROR: Mailbox not valid' 
+    	end
+    end
+
+    def self.test_webhook(url) 
+    	begin
+    		response = RestClient.head url
+        MailRunner.manager_bot.update_webhook_status("live")
+    	rescue 
+    		raise ArgumentError, "ERROR: \nMake sure the server is running and the webhook exists.\nNOTE:  Server must respond to http HEAD method.\nSee README.md for proper setup.\n"
+    	end
+    	unless response.code == 200
+    		raise ArgumentError, "ERROR: Invalid Webhook. NOTE, Must respond to http HEAD method."
+    	end
+    end
+
+    def self.soft_test_webhook(url) 
+      begin
+        response = RestClient.head url
+        MailRunner.manager_bot.update_webhook_status("live")
+        $logger.info("ManagerBot") {"webhook status: live"}
+      rescue 
+      end
+    end
+
+  end
+
+end

data/lib/mail_runner/cli.rb

@@ -0,0 +1,115 @@
+require 'yaml'
+require 'optparse'
+module MailRunner
+	class CLI
+		
+
+    def self.start(args)
+      options = parse_options(args)
+      initialize_logger(options)
+      set_globals
+      
+      @bot = initialize_manager_bot(options)#run first to make sure it runs bot tests prior to daemonizing a process.
+      daemonize if options[:daemon] && options[:daemon] == true
+      @bot.run
+    end
+
+
+		def self.parse_options(argv)
+      opts = {}
+      @parser = OptionParser.new do |o|
+        o.on '-m', '--mailbox MAILBOX', "Name of Mailbox to watch" do |arg|
+          opts[:mailbox] = arg
+        end
+
+        o.on '-w', '--webhook URL', "Complete url of webhook to deliver mail to" do |arg|
+          opts[:webhook] = arg
+        end
+
+        o.on '-a', '--archive', "Set to true id you want mail archived." do |arg|
+          opts[:archive] = arg
+        end
+
+        o.on '-d', '--daemon', "Daemonize process. Be sure to add logfile path." do |arg|
+          opts[:daemon] = arg
+        end
+
+        o.on '-L', '--logfile PATH', "Absolute path to log file." do |arg|
+          opts[:logfile] = arg
+        end
+
+        o.on '-c', '--config PATH', "Path to YAML config file." do |arg|
+          opts[:config] = arg
+        end
+
+        o.on '-v', '--verbose', "Logger runs in debug mode." do |arg|
+          opts[:verbose] = arg
+        end
+      end
+      @parser.parse!(argv)
+      if opts[:config]
+        opts = parse_config(opts[:config]).merge(opts)
+      end
+      opts
+    end
+
+    def self.parse_config(cf_path)
+      YAML.load_file(cf_path)
+    end
+
+		def self.initialize_manager_bot(opts)
+    	begin
+			  bot = MailRunner.initialize_manager_bot
+			  bot.verify_and_set(opts)
+			rescue => e
+			  puts e.message
+			  exit 1
+			end
+      return bot
+	  end
+
+    def self.daemonize
+      #eq_to Process.daemon
+      exit if fork
+      Process.setsid
+      exit if fork
+      Dir.chdir "/" 
+      STDIN.reopen "/dev/null"
+      STDOUT.reopen "/dev/null", "a" 
+      STDERR.reopen "/dev/null", "a" 
+    end
+
+    def self.initialize_logger(options)
+      begin
+        MailRunner::Logging.initialize_logger(options[:logfile]) if options[:logfile]
+        MailRunner::Logging.add_log_file_section_header if options[:logfile]
+        MailRunner::Logging.logger.level = ::Logger::DEBUG if options[:verbose]
+      rescue => e #primarily to alert invald log path in case of daemon
+        puts e.message
+        exit 1
+      end
+    end
+
+    def self.set_globals
+      MailRunner.set_globals
+    end
+
+    def self.banner
+      %q{
+               m,
+               `$b
+          .ss,  $$:         .,d$
+          `$$P,d$P'    .,md$P"'
+           ,$$$$$bmmd$$$P^'
+         .d$$$$$$$$$$P'
+         $$^' `"^$$$'       ____  _     _      _    _
+         $:     ,$$:       / ___|(_) __| | ___| | _(_) __ _
+         `b     :$$        \___ \| |/ _` |/ _ \ |/ / |/ _` |
+                $$:         ___) | | (_| |  __/   <| | (_| |
+                $$         |____/|_|\__,_|\___|_|\_\_|\__, |
+              .d$$                                       |_|
+      }
+    end
+	end
+end
+

data/lib/mail_runner/head_manager_bot.rb

@@ -0,0 +1,102 @@
+require 'mail'
+require 'json'
+require 'base64'
+require 'rest-client'
+require 'redis'
+
+module MailRunner
+
+	class ManagerBot 
+
+    # loads all BotHelper Modules & then extends class.
+    Dir[File.dirname(__FILE__) + '/bot_helpers/*.rb'].each {|file| require file }
+
+    extend BotHelpers
+
+    #used for testing
+    attr_accessor :mailbox
+    attr_accessor :webhook
+    attr_accessor :archive
+    attr_accessor :webhook_status
+
+    
+   
+    def initialize
+      @mailbox = nil
+      @webhook = nil
+      @archive = false
+      @webhook_status = nil
+    end
+
+
+    def verify_and_set(opts)
+      parse_options(opts)
+      test_options       
+    end
+
+    def parse_options(opts)
+    	BotHelpers::Tests.all_args_included?(opts)
+    	
+    	@mailbox = "/var/mail/#{opts[:mailbox]}"
+    	@webhook = opts[:webhook]
+    	@archive = opts[:archive] == "true" ? true : false 
+    end
+
+    def update_webhook_status(status)
+    	@webhook_status = status
+    end
+
+    def test_options
+    	BotHelpers::Tests.test_mailbox(self.mailbox)
+    	BotHelpers::Tests.test_webhook(self.webhook)
+    end
+
+    def run
+    	BotHelpers::Helpers.print_monitoring_started_msg(self)
+    	$mad_statter.incr_stat("runner launched")
+    	while true 
+
+    		delegate_inbound_processing
+    		
+    		if webhook_status == "down"
+    			BotHelpers::Tests.soft_test_webhook(self.webhook)
+    		elsif delayed_queue?
+    			delegate_delayed_queue_processing
+    		end
+
+    		sleep 5
+    	end
+    end
+
+
+    def inbound_manager
+    	MailRunner::InboundManagerBot
+    end
+
+    def delegate_inbound_processing
+    	begin
+    		inbound_manager.process_inbound(mailbox, webhook, archive)
+    	rescue Exception => msg 
+    		puts msg.inspect
+    	end
+    end
+
+
+    def queue_manager
+    	MailRunner::QueueManagerBot
+    end
+
+    def delayed_queue?
+    	queue_manager.queue_length > 0
+    end
+
+    def delegate_delayed_queue_processing
+    	begin
+    		queue_manager.process_queue
+    	rescue Exception => msg 
+    		puts msg.inspect
+    	end
+    end
+		
+	end
+end

data/lib/mail_runner/inbound_manager_bot.rb

@@ -0,0 +1,45 @@
+module MailRunner
+
+	module InboundManagerBot
+    
+    def self.process_inbound(mailbox, webhook, archive)
+      raw_mail = get_mail(mailbox)
+      
+      unless raw_mail.nil?
+        raw_mail.each do |raw_msg|    
+
+          mail = read_mail(raw_msg)
+
+          json_packet = BotHelpers::Helpers.convert_raw_mail_to_json(mail)
+          
+          deliver_mail(webhook, json_packet)
+          
+          if archive == true && queued.nil?
+            puts "we will archive email.\n"
+          end
+          
+        end
+      end
+    end
+
+    def self.get_mail(mailbox)
+      BotHelpers::Runner.get_contents_from_mailbox(mailbox) 
+    end
+    
+    def self.read_mail(raw_msg)
+      Mail.read_from_string(raw_msg)
+    end
+
+    def self.deliver_mail(webhook, json_packet)
+      begin
+        BotHelpers::Runner.post_to_hook(webhook, json_packet)
+      rescue Exception => msg 
+        #interrupt exception here, so rest of inbound mail can be processed and added to queue.
+        #otherwise, it will be lost.
+        queued = MailRunner::QueueManagerBot.add_to_mail_queue(webhook, json_packet)
+      end
+    end
+
+  end
+
+end

data/lib/mail_runner/logging.rb

@@ -0,0 +1,33 @@
+require 'time'
+require 'logger'
+
+module MailRunner
+	module Logging 
+
+		def self.initialize_logger(log_target = STDOUT)
+	    oldlogger = defined?(@logger) ? @logger : nil
+	    @logger = Logger.new(log_target, 'weekly')
+	    @logger.level = Logger::INFO
+	    @logger.formatter = proc do |severity, datetime, progname, msg|
+		    date_format = datetime.strftime("%b %d %H:%M:%S ")    
+		    "#{date_format} #{severity} (#{progname}): #{msg}\n"
+			end
+
+
+	    oldlogger.close if oldlogger && !$TESTING # don't want to close testing's STDOUT logging
+	    @logger
+	  end
+
+    def self.logger
+      defined?(@logger) ? @logger : initialize_logger
+    end
+
+    def self.logger=(log)
+      @logger = (log ? log : Logger.new('/dev/null'))
+    end 
+
+    def self.add_log_file_section_header
+	    @logger.info{"\n\nInitiate LogFile :: Session #{$redis.get("MR::sessions").to_i + 1} :: #{Time.now}\n########################################################"}
+	  end
+  end
+end