net-receiver 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: bde94c119d7f2b94cb8d95a4781311477aa87fc5
+  data.tar.gz: b9ee22f2760943cbc37c366446f3021fda514570
+SHA512:
+  metadata.gz: f903d9f64e4f4307ab1a447050f5bca81f14bc953810f79ac3941a8a1745205a4f808f23162e79f911d91d3a547960ea9098686780ac382bd5189fec6e562e0c
+  data.tar.gz: 619c8e9a7f8f993dee32714fdabe595b37dba33d7268389a81697914c8f0caf4c85b754e52702f3a648afda7f16557a86f4af4ed503fb3bcd0c25432f43b2164

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+# v1.0
+This is the initial load of the gem. If you find a problem, report it to me at mjwelchphd@gmail.com, or FORK the library, fix the bug, then add a Pull Request.

data/README.md

@@ -0,0 +1,340 @@
+# Net::Receiver
+
+# EXPERIMENTAL EMAIL RECEIVER
+
+This is an experimental email receiver which is currently (v1.0.0) working, but may be revised in ways that I can't predict at the moment. All I can guarantee is that _if_ the interface changes, it won't be major.
+
+Currently, I'm using this as the base for an MTA written in Ruby. There's no intention of replacing Exim, Courier, or Postfix (or any other) existing MTA. The reason I'm doing this is because I need an MTA which has capabilities the standard MTAs don't offer. In other words, I need features not previously anticipated by the makers of those standard MTAs (and why would they have anticipated something I would think up in the future?)
+
+That being said, If you use this for anything, and want me to make special changes that don't interfere with my purpose, email me at mjwelchphd@gmail.com, and I'll work with you as best I can.
+
+# General
+
+This gem sits on top of my net-server gem, and receives standard emails. It has only a few checks that it makes on the incoming email, leaving the specialized checks up to you. You can change it's behavior by overriding base methods, and adding your own programming; don't worry, I'll show you how. Your Ruby overrides are the same as witing a configuration file for a standard MTA.
+
+This document describes the interface, and provides a sample program, so you can see how it works in every detail. The sample program is stored in the `example` directory.
+
+## What This Does
+
+This gem received the connection from net-server, receives the email by carrying on a conversation in SMTP with the sender, and finally delivers the finished email to you. It logs stuff, if the log is enabled, and, at the moment, I have debugging in it to write to the terminal, so that I can debug more easily. Those `puts` will go away in the future.
+
+## TODO!
+
+There's still stuff I need to do. A few notes in the source are prefaced with TODO! to make them easy to find. To be truthful, I'm not sure of what I may need to add, if much, in the future. I've written the receiving part of my MTA on top of this gem, so I believe that this gem is 99% complete, for what it was intended.
+
+I also need to add a method to convert the email to the format used by the Net/* standard library classes.
+
+However, I may also move the code to do `dig`, test for live servers, and other handy stuff into here, which will help you in your project.
+
+# A Sample Program
+
+Here's a sample program to demonstrate how the interface works. The complete source is in the `example` directory, so you can actually run it and see what happens. Look for the notations `#(1), #(2)` and so forth.  These reference the notes below.
+
+```ruby
+#! /usr/bin/ruby #(1)
+
+## For testing ...
+# put in your website address in place of www.example.com
+# swaks -s www.example.com:2000 -t coco@smith.com -f jamie@glock.com --ehlo example.com
+# swaks -tls -s www.example.com:2000 -t coco@smith.com -f jamie@glock.com --ehlo example.com
+
+require 'net/receiver' #(2)
+require 'logger'
+require 'sequel'
+require 'yaml'
+require 'pretty_inspect'
+
+class Receiver < Net::ReceiverCore #(3)
+
+  def password(username) #(4)
+#    DB[:mailboxes].where(:email=>username).first[:passwd]
+  end
+
+  def received(mail) #(5)
+    puts "--> *99* #{mail.pretty_inspect}"
+  end
+
+end
+
+# Open the log #(6)
+LOG = Logger::new('log/test3.log', 'daily')
+LOG.formatter = proc do |severity, datetime, progname, msg|
+  pname = if progname then '('+progname+') ' else nil end
+  "#{datetime.strftime("%Y-%m-%d %H:%M:%S")} [#{severity}] #{pname}#{msg}\n"
+end
+
+# Open the database #(7)
+if ['dev','live'].index(ENV['MODE']).nil?
+  msg = "Environmental variable MODE not set properly--must be dev or live"
+  LOG.fatal(msg)
+  puts msg
+  exit(1)
+end
+host = YAML.load_file("./database.yml")[ENV['MODE']]
+DB = Sequel.connect(host)
+LOG.info("Database \"#{host['database']}\" opened")
+
+# Start the server #(8)
+options = {
+  :server_name=>"www.example.com",
+  :private_key=>"server.key",
+  :certificate=>"server.crt",
+  :listening_ports=>['2000','2001'],
+  :ehlo_validation_check=>true
+}
+Net::Server.new(options).start
+```
+Here's the breakdown (see references like #(1), etc.):
+
+  1. This line makes `test3` self executing.
+  2. There are requires for `net/receiver`. It will require `net/server`, so you don't have to do that. It requires `logger` to demonstrate how to open a logger file; `sequel` because I use Sequel in my programming; `yaml` for reading the database 'yaml' file in the project (you'll have to change all this database stuff to your liking); and `pretty_inspect` which makes it easier to see what's coming out (you can install the gem for that).
+  3. Define your receiver like this. The base in `net-receiver` is called `ReceiverCore` in order that you can derive class `Receiver` from it.
+  4. In order to use authorization (only PLAIN supported at this time), you need this code to provide the password for a user.
+  5. The received email is delivered to the `received` method. See the example of what gets delivered below.
+  6. This is how to open a log file. The name LOG is used because it is traditional.
+  7. This is how you open a Sequel/MySQL database. You may remove this code if you don't use Sequel to read passwords from the database.
+  8. The last part is the server start code. Look at "https://github.com/mjwelchphd/net-server" documentation for more information.
+  
+# How It Works
+
+The class ReceiverCode uses a table named Patterns to guide the receiving process. For each line of the table, the number on the left is a 'level', i.e., STARTTLS (level 2) cannot come before EHLO or HELO (level 1). The pattern describing the value is next: the input value on the command line must match this pattern. The method which handles the command is last.
+
+As each line is read on the communications channel, it is matched up with this table, and if all is well, the method is called to deal with it.
+
+The `send_text` and `recv_text` methods are complex because they have to handle conditions like the client slamming the communications channel shut, and so forth.
+
+The method `psych_value` is used with MAIL FROM and RCPT TO commands to validate and investigate the email addresses given.
+
+  1. It checks that there is a legitimate address, with or with a preceeding name.
+  2. It breaks the address up into a local-part and a domain.
+  3. If the option is selected, it tests for the legal usage of dots (".") in the name, and legal characters which net-receiver defines as 
+    - uppercase and lowercase English letters (a-z, A-Z)
+    - digits 0 to 9
+    - characters ! # $ % & ' * + - / = ? ^ _ ` { | } ~
+    - dots, which must not be first or last character, and must not appear two or more times consecutively
+  4. If the option is selected, it does a Dig MX lookup, followed by a Dig A (IP) lookup if the MX was successful. This is helpful to determine if the sender's domain is legitimate.
+
+The main method is the `receive` method (which is called by net-server when a connection is requested). Receive uses the aforementioned table to read the commands and process them. It also allocates an 'item of mail' structure to put it's findings in.
+
+If `@mail[:prohibited] gets set to `true`, the loop will terminate and the connection will be closed. This is mainly for shutting down spammers who make large numbers of calls in a short period of time (DDOS attacks).
+
+Any method can `raise Quit` in order to terminate the reception also.
+
+When the main loop terminates, for whatever reason, the email will be delivered to your `received` method. When your `received` method terminates, the process is cleaned up and terminated.
+
+Methods that begin with `do_` are the methods with do any generalized processing of the commands. Typically, they create a key in the item-of-mail and all the data for that command is stored in that hash. There are very few validations of the incoming data because that is the job of your method overrides (to be described below).
+
+Methods that are named the same as the commands, i.e., `connect`, `ehlo`, `quit`, `auth`, `expn`, `help`, `noop`, `rset`, `vfry`, `mail_from`, `rcpt_to`, and `data` deliver the default response back to the `receive` method.
+
+You can change their behavior by overriding them like this example:
+
+```ruby
+class Receiver < Net::ReceiverCore
+
+    def mail_from(from)
+      return "556 5.7.27 Traffic on port #{@options[:submission_port]} must be authenticated" \
+        if !@mail[:authenticated]
+      return "556 5.7.27 Traffic on port #{@options[:submission_port]} must be encrypted" \
+        if !@mail[:encrypted]
+      super
+    end
+
+end
+```
+
+In this example, `def mail_from` overrides the method of the same name in ReceiverCore. It tests :authenticated and :encrypted, and if there is an error, it returns the error message; if not, it performs `super` which returns the default message. Don't forget to call `super`.
+
+# Start Options for Server
+  
+  Option | Default | Description
+  --- | --- | ---
+  :server_name | "example.com" | This name is only used in error messages.
+  :listening_ports | ["25","486","587"] | An array of one or more ports to listen on.
+  :private_key | Internal key | The key for encrypting/decrypting the data when in TLS mode.
+  :certificate | Internal self-signed certificate | The certificate for encrypting/decrypting the data when in TLS mode. This may be your own self-signed certificate, or one you purchase from a Certificate Authority, or you can become a Certificate Authority and sign your own.
+  :user_name | nil | This name is the user name to which each process will be switched after it is created. If it is nil, the ownership of the process will not be changed after creation. If you are using a port less than 1024, you must start the server as root, and the user name and group name of the process _must be_ specified.
+  :group_name | nil | This name is the group name to which each process will be switched after it is created.
+  :working_directory | the current path | The location of the program running the server.
+  :pid_file | "pid" | The PID of the server will be stored in this file.
+  :daemon | false | If this option is true, the server will be started as a daemon.
+
+# Start Options for Receiver
+  
+  Option | Default | Description
+  --- | --- | ---
+  :ehlo_validation_check | false | This makes `receiver` test the domain name given on the EHLO or HELO line.
+  :sender_character_check | true | This makes `receiver` test for legal characters on the MAIL FROM address.
+  :recipient_character_check | false | This makes `receiver`test for legal characters on the RCPT TO address.
+  :sender_mx_check | true | Tries to obtain the MX name and IP from the DNS for MAIL FROM.
+  :recipient_mx_check | false | Tries to obtain the MX name and IP from the DNS for RCPT TO.
+  :max_failed_msgs_per_period | 3 | I use this to say, "after 3 failed attempts, lock out the sender for s short period of time (10 minutes in my case)."
+
+I may add more defaults to this list in the future, but I'll try to make them generalized, so they fit anyone's need.
+
+# The Structure That Comes Out
+
+Here is a sample structure for an authenticated email.
+
+```text
+{
+  :local_port=>"2001",
+  :local_hostname=>"mail.example.com",
+  :remote_port=>"38436",
+  :remote_hostname=>"cpe-107-185-187-182.socal.res.rr.com",
+  :remote_ip=>"::ffff:107.185.187.182",
+  :id=>"ODZM0W-PRPAYD-49",
+  :time=>"2016-09-24 02:38:56 +0000",
+  :accepted=>true,
+  :prohibited=>false,
+  :encrypted=>true,
+  :authenticated=>"admin@example.com",
+  :connect=>{
+    :value=>"::ffff:107.185.187.182",
+    :domain=>nil
+  },
+  :ehlo=>{
+    :value=>"mail.example.com",
+    :rip=>"23.253.107.107",
+    :fip=>"23.253.107.107",
+    :domain=>"mail.example.com"
+  },
+  :mailfrom=>{
+    :accepted=>true,
+    :value=>"<admin@example.com>",
+    :name=>"",
+    :url=>"admin@example.com",
+    :local_part=>"admin",
+    :domain=>"example.com",
+    :bad_characters=>false,
+    :wrong_dot_usage=>false,
+    :ip=>"23.253.107.107",
+    :mxs=>[
+      "mail.example.com"
+    ],
+    :ips=>[
+      "23.253.107.107"
+    ]
+  },
+  :rcptto=>[
+    {
+      :accepted=>true,
+      :value=>"<coco@example.com>",
+      :name=>"",
+      :url=>"coco@example.com",
+      :local_part=>"coco",
+      :domain=>"example.com"
+    }
+  ],
+  :data=>{
+    :accepted=>true,
+    :value=>"",
+    :text=>[
+      "Date: Fri, 23 Sep 2016 19:38:55 -0700",
+      "To: coco@example.com",
+      "From: admin@example.com",
+      "Subject: test Fri, 23 Sep 2016 19:38:55 -0700",
+      "X-Mailer: swaks v20130209.0 jetmore.org/john/code/swaks/",
+      "",
+      "This is a test mailing",
+      "",
+      "."
+    ],
+    :headers=>{
+      :date=>"Fri, 23 Sep 2016 19:38:55 -0700",
+      :to=>"coco@example.com",
+      :from=>"admin@example.com",
+      :subject=>"test Fri, 23 Sep 2016 19:38:55 -0700",
+      :x_mailer=>"swaks v20130209.0 jetmore.org/john/code/swaks/"
+    }
+  }
+}
+```
+
+## Format of Delivered Mail
+
+### Global Values
+
+|Symbol |Description |
+|:--- |:--- |
+| :local_port | This is the port on your machine that the user connected to. |
+| :local_hostname | This is the `hostname` of your machine. |
+| :remote_port | This is the port on the remote machine that originated the connection. |
+| :remote_hostname | This is the `hostname` of the remote machine. |
+| :remote_ip | This is the IP of the remote machine. |
+| :id | This is the Message ID generated by `Receiver`. Note that there should alread be header `Message-ID` in the email, but if not, this one can be inserted. |
+| :time | This is the time the conncetion was made. |
+| :accepted | This is a true/false which indicates whether the email should be accepted. |
+| :prohibited | If you set this flag, `Receiver` will treat the email as spam, and the sender IP as a spammer. |
+| :encrypted | This true/false indicates whether or not a STARTTLS was completed. |
+| :authenticated | This value is nil or the email address  of the authenticated entity. |
+
+### CONNECT Values
+
+|Symbol |Description |
+|:--- |:--- |
+| :value | This is the remote IP (taken from the value above). |
+| :domain | If a domain can be discovered for the remote IP, it will be here. |
+
+### EHLO Values
+
+|Symbol |Description |
+|:--- |:--- |
+| :value | This is the raw data supplied on the EHLO line. |
+| :rip | This is the reverse IP, if any, obtained by looking up the value. |
+| :fip | The reverse IP is used to get the MX, which is then looked up to get this forward IP. |
+| :domain | This is the MX value obtained from looking up the reverse IP. |
+
+### MAIL FROM Values
+
+|Symbol |Description |
+|:--- |:--- |
+| :accepted | This true/false value indicates if the MAIL FROM value appears to be acceptable. |
+| :value | This is the raw data presented on the MAIL FROM line. |
+| :name | If a name preceeded the email address, it is put here. |
+| :url | This is the "pure" email address in the MAIL FROM statement. |
+| :local_part | This is the "local-part" of the URL above. |
+| :domain | This is the domain of the URL above. |
+| :bad_characters | This true/false tells whether bad characters were found in the local-part. |
+| :wrong_dot_usage | This true/false tells whether dots were mis-used in the local-part. |
+| :ip | This is the IP from looking up the domain. |
+| :mxs | This is a list of one or more mail servers for this domain. |
+| :ips | This is a list of IPs obtained by looking up the MXs above. |
+
+### RCPT TO Values (a list)
+
+|Symbol |Description |
+|:--- |:--- |
+| :accepted | This true/false value indicates if the RCPT TO value appears to be acceptable. |
+| :value | This is the raw data presented on the RCPT TO line. |
+| :name | If a name preceeded the email address, it is put here. |
+| :url | This is the "pure" email address in the RCPT TO statement. |
+| :local_part | This is the "local-part" of the URL above. |
+| :domain | This is the domain of the URL above. |
+
+### DATA Values (including the email proper)
+
+|Symbol |Description |
+|:--- |:--- |
+| :accepted | This tru/false value indicates whether `Receiver` accepted the email from the sender. |
+| :value | This should be an empty string. |
+| :text | __This is the body of the email proper.__ It's organized as an array of lines with the CRLFs stripped off the ends. |
+
+**NOTE! If [:data][:accepted] is *true*, you have taken full responsibility for the email. You must either deliver it, forward it, or bounce it.**
+
+#### Headers (broken out to make access easier)
+
+Here's an example:
+
+|Key |Value |
+|:--- |:--- |
+| :date | "Fri, 23 Sep 2016 19:38:55 -0700", |
+| :to | "coco@example.com", |
+| :from | "admin@example.com", |
+| :subject | "test Fri, 23 Sep 2016 19:38:55 -0700", |
+| :x_mailer | "swaks v20130209.0 jetmore.org/john/code/swaks/" |
+
+
+The headers are put into a hash like this so that you may easily locate them, or test to see if they exist or not. Modifying this Hash *does not* modify the actual email.
+
+
+FIN

data/lib/net/extended_classes.rb

@@ -0,0 +1,182 @@
+# = net/extended_classes.rb
+#
+# Copyright (c) 2016 Michael J. Welch, Ph.D.
+#
+# Written and maintained by Michael J. Welch, Ph.D. <mjwelchphd@gmail.com>
+#
+# This work is not derived from any other author. It is original software.
+#
+# Documented by Michael J. Welch, Ph.D. <mjwelchphd@gmail.com>
+#
+# This program is free software. You can re-distribute and/or
+# modify this program under the same terms as Ruby itself.
+#
+# See the README.md for documentation.
+
+require 'resolv'
+require 'base64'
+require 'unix_crypt'
+
+LiveServerTestTimeout = 15
+
+class QueryError < StandardError; end
+
+class Object
+  def deepclone
+    case
+    when self.class==Hash
+      hash = {}
+      self.each { |k,v| hash[k] = v.deepclone }
+      hash
+    when self.class==Array
+      array = []
+      self.each { |v| array << v.deepclone }
+      array
+    else
+      if defined?(self.class.new)
+        self.class.new(self)
+      else
+        self
+      end
+    end
+  end
+end
+
+class NilClass
+  # these defs allow for the case where something wasn't found to
+  # give a nil response rather than crashing--for example:
+  #  mx = "example.com" # => nil (because example.com has no MX record)
+  #  ip = mx.dig_a # => nil, without crashing
+  # otherwise, it would be necessary to write:
+  #  mx = "example.com" # => nil (because example.com has no MX record)
+  #  ip = if mx then ip = mx.dig_a else ip = nil end
+  def dig_a; nil; end
+  def dig_aaaa; nil; end
+  def dig_mx; nil; end
+  def dig_dk; nil; end
+  def dig_ptr; nil; end
+  def mta_live?(port); nil; end
+  def validate_plain; return "", false; end
+  # the [] method allows for x[:a][:b]... type references where
+  # x[:a] --> nil to return another nil rather than crash
+  def [](what)
+    nil
+  end
+end
+
+class String
+  # returns list of IPV4 addresses, or nil
+  # (there should only be one IPV4 address)
+  def dig_a
+    Resolv::DNS.open do |dns|
+      txts = dns.getresources(self,Resolv::DNS::Resource::IN::A).collect { |r| r.address.to_s }
+      if txts.empty? then nil else txts[0] end
+    end
+  end
+
+  # returns list of IPV6 addresses, or nil
+  # (there should only be one IPV6 address)
+  def dig_aaaa
+    Resolv::DNS.open do |dns|
+      txts = dns.getresources(self,Resolv::DNS::Resource::IN::AAAA).collect { |r| r.address.to_s.downcase }
+      if txts.empty? then nil else txts[0] end
+    end
+  end
+
+  # returns list of MX names, or nil
+  # (there may be multiple MX names for a domain)
+  def dig_mx
+    Resolv::DNS.open do |dns|
+      txts = dns.getresources(self,Resolv::DNS::Resource::IN::MX).collect { |r| r.exchange.to_s }
+      if txts.empty? then nil else txts end
+    end
+  end
+
+  # returns a publibdomainkey, or nil
+  # (there should only be one DKIM public key)
+  def dig_dk
+    Resolv::DNS.open do |dns|
+      txts = dns.getresources(self,Resolv::DNS::Resource::IN::TXT).collect { |r| r.strings }
+      if txts.empty? then nil else txts[0][0] end
+    end
+  end
+
+  # returns a reverse DNS hostname or nil
+  def dig_ptr
+    begin
+      Resolv.new.getname(self.downcase)
+    rescue Resolv::ResolvError
+      nil
+    end
+  end
+
+  # returns true if the IP is blacklisted; otherwise false
+  # examples:
+  # barracuda = 'b.barracudacentral.org'.blacklisted?(ip)
+  # spamhaus = 'zen.spamhaus.org'.blacklisted?(ip)
+  def blacklisted?(dx)
+    domain = dx.split('.').reverse.join('.')+"."+self
+    a = []
+    Resolv::DNS.open do |dns|
+      begin
+        a = dns.getresources(domain, Resolv::DNS::Resource::IN::A)
+      rescue Resolv::NXDomainError
+        a=[]
+      end
+    end
+    if a.size>0 then true else false end
+  end
+
+  # returns a UTF-8 encoded string -- be carefule using this with email:
+  # email has to be received and transported with NO changes, except the
+  # addition of extra headers at the beginning (before any DKIM headers)
+  def utf8
+    self.encode('UTF-8', 'binary', :invalid => :replace, :undef => :replace, :replace => '?')
+  end
+
+  # opens a socket to the IP/port to see if there is an SMTP server
+  # there - returns "250 ..." if the server is there, or 
+  # times out in 5 seconds to prevent hanging the process
+  def mta_live?(port)
+    tcp_socket = nil
+    welcome = nil
+    begin
+      Timeout.timeout(LiveServerTestTimeout) do
+        begin
+          tcp_socket = TCPSocket.open(self,port)
+        rescue Errno::ECONNREFUSED => e
+          return "421 Service not available (port closed)"
+        end
+        begin
+          welcome = tcp_socket.gets
+          return welcome if welcome[1]!='2'
+          tcp_socket.write("QUIT\r\n")
+          line = tcp_socket.gets
+          return line if line[1]!='2'
+        ensure
+          tcp_socket.close if tcp_socket
+        end
+      end
+      return "250 #{welcome.chomp[4..-1]}"
+    rescue SocketError => e
+      return "421 Service not available (#{e.to_s})"
+    rescue Timeout::Error => e
+      return "421 Service not available (#{e.to_s})"
+    end
+  end
+
+  # this validates a password with the base64 plaintext in an AUTH command
+  # encoded -> AGNvY29AY3phcm1haWwuY29tAG15LXBhc3N3b3Jk => ["coco@example.com", "my-password"]
+  # "my-password" --> {CRYPT}IwYH/ZXeR8vUM
+  # "AGNvY29AY3phcm1haWwuY29tAG15LXBhc3N3b3Jk".validate_plain { "{CRYPT}IwYH/ZXeR8vUM" } => "coco@example.com", true
+  # "AGNvY29AY3phcm1haWwuY29tAHh4LXBhc3N3b3Jk".validate_plain { "{CRYPT}IwYH/ZXeR8vUM" } => "coco@example.com", false
+  def validate_plain
+    # decode and split up the username and password)
+    username, password = Base64::decode64(self).split("\x00")[1..-1]
+    return "", false if username.nil? || password.nil?
+    passwd_hash = yield(username) # get the hash
+    return "", false if passwd_hash.nil?
+    m = passwd_hash.match(/^{(.*)}(.*)$/)
+    return username, UnixCrypt.valid?(password, m[2])
+  end
+end

data/lib/net/item_of_mail.rb

@@ -0,0 +1,77 @@
+# = net/item_of_mail.rb
+#
+# Copyright (c) 2016 Michael J. Welch, Ph.D.
+#
+# Written and maintained by Michael J. Welch, Ph.D. <mjwelchphd@gmail.com>
+#
+# This work is not derived from any other author. It is original software.
+#
+# Documented by Michael J. Welch, Ph.D. <mjwelchphd@gmail.com>
+#
+# This program is free software. You can re-distribute and/or
+# modify this program under the same terms as Ruby itself.
+#
+# See the README.md for documentation.
+
+require 'sequel'
+
+module Net
+  class ItemOfMail < Hash
+    def initialize(local_port, local_hostname, remote_port, remote_hostname, remote_ip)
+      self[:local_port] = local_port
+      self[:local_hostname] = local_hostname
+      self[:remote_port] = remote_port
+      self[:remote_hostname] = remote_hostname
+      self[:remote_ip] = remote_ip
+
+      new_id = []
+      new_id[0] = Time.now.tv_sec.to_s(36).upcase
+      new_id[1] = ("000000"+(2176782336*rand).to_i.to_s(36))[-6..-1].upcase
+      new_id[2] = ("00"+(Time.now.usec/1000).to_i.to_s(36))[-2..-1].upcase
+      self[:id] = new_id.join("-")
+
+      self[:time] = Time.now.strftime("%Y-%m-%d %H:%M:%S %z")
+    end
+
+    def parse_headers
+      self[:data][:headers] = {}
+      header = ""
+      self[:data][:text].each do |line|
+        case
+        when line.nil?
+          break
+        when line =~ /^[ \t]/
+          header << String::new(line)
+        when line.empty?
+          break
+        when !header.empty?
+          keyword, value = header.split(":", 2)
+          self[:data][:headers][keyword.downcase.gsub("-","_").to_sym] = value.strip
+          header = String::new(line)
+        else
+          header = String::new(line)
+        end
+      end
+      if !header.empty?
+        keyword, value = header.split(":", 2)
+        self[:data][:headers][keyword.downcase.gsub("-","_").to_sym] = if !value.nil? then value.strip else "" end
+      end
+    end
+
+    def spf_check(scope,identity,ip,ehlo)
+      spf_server = SPF::Server.new
+      begin
+        request = SPF::Request.new(
+          versions:      [1, 2],
+          scope:         scope,
+          identity:      identity,
+          ip_address:    ip,
+          helo_identity: ehlo)
+        spf_server.process(request).code
+      rescue SPF::OptionRequiredError => e
+        @log.info("%06d"%Process::pid) {"SPF check failed: #{e.to_s}"}
+        :fail
+      end
+    end
+  end
+end

data/lib/net/receiver.rb

@@ -0,0 +1,522 @@
+# = net/receiver.rb
+#
+# Copyright (c) 2016 Michael J. Welch, Ph.D.
+#
+# Written and maintained by Michael J. Welch, Ph.D. <mjwelchphd@gmail.com>
+#
+# This work is not derived from any other author. It is original software.
+#
+# Documented by Michael J. Welch, Ph.D. <mjwelchphd@gmail.com>
+#
+# This program is free software. You can re-distribute and/or
+# modify this program under the same terms as Ruby itself.
+#
+# See the README.md for documentation.
+
+require 'net/server'
+require 'net/item_of_mail'
+require 'net/extended_classes'
+require 'pdkim'
+
+class Quit < Exception; end
+
+module Net
+
+  # == An Email receiver
+  class ReceiverCore
+    CRLF = "\r\n"
+    Patterns = [
+      [0, "[ /t]*QUIT[ /t]*", :do_quit],
+      [0, "[ /t]*SLAM[ /t]*", :do_slam],
+      [0, "[ /t]*TIMEOUT[ /t]*", :do_timeout],
+      [1, "[ /t]*AUTH[ /t]*(.+)", :do_auth],
+      [1, "[ /t]*EHLO(.*)", :do_ehlo],
+      [1, "[ /t]*EXPN[ /t]*", :do_expn],
+      [1, "[ /t]*HELO[ /t]+(.*)", :do_ehlo],
+      [1, "[ /t]*HELP[ /t]*", :do_help],
+      [1, "[ /t]*NOOP[ /t]*", :do_noop],
+      [1, "[ /t]*RSET[ /t]*", :do_rset],
+      [1, "[ /t]*TIMEOUT[ /t]*", :do_timeout],
+      [1, "[ /t]*VFRY[ /t]*", :do_vfry],
+      [2, "[ /t]*STARTTLS[ /t]*", :do_starttls],
+      [2, "[ /t]*MAIL FROM[ /t]*:[ \t]*(.+)", :do_mail_from],
+      [3, "[ /t]*RCPT TO[ /t]*:[ \t]*(.+)", :do_rcpt_to],
+      [4, "[ /t]*DATA[ /t]*", :do_data]
+    ]
+    Kind = {:mailfrom=>"MAIL FROM", :rcptto=>"RCPT TO"}
+    ReceiverTimeout = 30
+    LogConversation = true
+    Unexpectedly = "; probably caused by the client closing the connection unexpectedly"
+
+    include PDKIM
+
+    DkimOutcomes = {
+      PDKIM_VERIFY_NONE=>"PDKIM_VERIFY_NONE",
+      PDKIM_VERIFY_INVALID=>"PDKIM_VERIFY_INVALID",
+      PDKIM_VERIFY_FAIL=>"PDKIM_VERIFY_FAIL",
+      PDKIM_VERIFY_PASS=>"PDKIM_VERIFY_PASS",
+      PDKIM_FAIL=>"PDKIM_FAIL",
+      PDKIM_ERR_OOM=>"PDKIM_ERR_OOM",
+      PDKIM_ERR_RSA_PRIVKEY=>"PDKIM_ERR_RSA_PRIVKEY",
+      PDKIM_ERR_RSA_SIGNING=>"PDKIM_ERR_RSA_SIGNING",
+      PDKIM_ERR_LONG_LINE=>"PDKIM_ERR_LONG_LINE",
+      PDKIM_ERR_BUFFER_TOO_SMALL=>"PDKIM_ERR_BUFFER_TOO_SMALL"
+    }
+
+    def initialize(connection, options)
+      @connection = connection
+      @option_list = [[:ehlo_validation_check, false], [:sender_character_check, true],
+        [:recipient_character_check, false], [:sender_mx_check, true],
+        [:recipient_mx_check, false],[:max_failed_msgs_per_period,3]]
+      @options = options
+      @option_list.each do |key,value|
+        @options[key] = value if !options.has_key?(key)
+      end
+      @enc_ind = '-'
+    end
+
+#-------------------------------------------------------#
+#--- Send text to the client ---------------------------#
+#-------------------------------------------------------#
+    def log_msg_if_level_5(msg)
+      if msg[0]=='5'
+        m = msg.match(/^([0-9]{3} [0-9]\.[0-9]\.[0-9] )/)
+        start = if !m then 0 else m[1].size end
+        LOG.error("%06d"%Process::pid) {msg[start..-1]}
+      end
+    end
+
+    def write_text(text, echo)
+puts "<#{@enc_ind}  #{text.inspect}"
+      @connection.write(text)
+      @connection.write(CRLF)
+      @has_level_5_warnings = true if text[0]=='5'
+      LOG.info("%06d"%Process::pid) {"<#{@enc_ind}  #{text}"} if echo && LogConversation
+      log_msg_if_level_5(text)
+    end
+
+    def send_text(text,echo=true)
+      begin
+        case
+        when text.nil?
+          # do nothing
+        when text.class==Array
+          text.each { |line| write_text(line, echo) }
+        when text.class==String
+          write_text(text, echo)
+        end
+      rescue Errno::EPIPE => e
+        LOG.error("%06d"%Process::pid) {"#{e.to_s}#{Unexpectedly}"}
+        raise Quit
+      rescue Errno::EIO => e
+        LOG.error("%06d"%Process::pid) {"#{e.to_s}#{Unexpectedly}"}
+        raise Quit
+      end
+    end
+
+#-------------------------------------------------------#
+#--- Receive text from the client ----------------------#
+#-------------------------------------------------------#
+    def recv_text(echo=true)
+      begin
+        Timeout.timeout(ReceiverTimeout) do
+          begin
+            temp = @connection.gets
+            if temp.nil?
+              LOG.warn("%06d"%Process::pid) {"The client abruptly closed the connection"}
+              text = "QUIT"
+            else
+              text = temp.chomp
+            end
+          rescue Errno::ECONNRESET => e
+            LOG.warn("%06d"%Process::pid) {"The client slammed the connection shut"}
+            text = "SLAM"
+          end
+          LOG.info("%06d"%Process::pid) {" #{@enc_ind}> #{text}"} if echo && LogConversation
+puts " #{@enc_ind}> #{text.inspect}"
+          return text
+        end
+      rescue Errno::EIO => e
+        LOG.error("%06d"%Process::pid) {"#{e.to_s}"}
+        raise Quit
+      rescue Timeout::Error => e
+puts " #{@enc_ind}> \"TIMEOUT\""
+        return "TIMEOUT"
+      end
+puts " #{@enc_ind}> *669* Investigate why this got here"
+    end
+
+#-------------------------------------------------------#
+#--- Parse the email address and investigate it --------#
+#-------------------------------------------------------#
+    def psych_value(kind, part, value)
+      # the value gets set in both MAIL FROM and RCPT TO
+      part[:value] = value
+
+      # there MUST be a sender/recipient address
+      return "501 5.1.7 '#{part[:value]}' No proper address (<...>) on the #{Kind[kind]} line" \
+        if (m = value.match(/^(.*)<(.+@.+\..+)>$/)).nil?
+
+      # break up the address
+      part[:name] = m[1].strip
+      part[:url] = url = m[2].strip
+
+      # parse out the local-part and domain
+      local_part, domain = url.split("@")
+      part[:local_part] = local_part
+      part[:domain] = domain
+
+      if ((kind==:mailfrom) && (@options[:sender_character_check])) \
+        || ((kind==:rcptto) && (@options[:recipient_character_check]))
+        # check the local part:
+        # uppercase and lowercase English letters (a-z, A-Z)
+        # digits 0 to 9
+        # characters ! # $ % & ' * + - / = ? ^ _ ` { | } ~
+        part[:bad_characters] = local_part.match(/^[a-zA-Z0-9\!\#\$%&'*+-\/?^_`{|}~]+$/).nil?
+        # check character . must not be first or last character,
+        #   and must not appear two or more times consecutively
+        part[:wrong_dot_usage] = !(local_part[0]=='.' || local_part[-1]=='.' || local_part.index('..')).nil?
+      end
+
+      # skip this if not needed
+      if ((kind==:mailfrom) && (@options[:sender_mx_check])) \
+        || ((kind==:rcptto) && (@options[:recipient_mx_check]))
+        # get the ip for this domain
+        part[:ip] = ip = domain.dig_a
+
+        # get the mx record(s)
+        part[:mxs] = mxs = domain.dig_mx
+
+        # get the mx's ip records
+        if mxs
+          part[:ips] = ips = []
+          mxs.each { |mx| ips << mx.dig_a }
+        end
+      end
+
+      # email address investigation completed
+      return nil
+    end
+
+#-------------------------------------------------------#
+#--- Receive the connection ----------------------------#
+#-------------------------------------------------------#
+    def receive(local_port, local_hostname, remote_port, remote_hostname, remote_ip)
+      # Start a hash to collect the information gathered from the receive process
+      @mail = Net::ItemOfMail::new(local_port, local_hostname, remote_port, remote_hostname, remote_ip)
+      @mail[:accepted] = false
+      @mail[:prohibited] = false
+
+      # start the main receiving process here
+      @done = false
+      @mail[:encrypted] = false
+      @mail[:authenticated] = false
+      send_text(do_connect(remote_ip))
+      @level = 1
+      @has_level_5_warnings = false
+
+      begin
+        break if @done
+        text = recv_text
+        unrecognized = true
+        Patterns.each do |pattern|
+          break if pattern[0]>@level
+          m = text.match(/^#{pattern[1]}$/i)
+          if m
+            case
+            when pattern[2]==:do_quit
+              send_text(do_quit(m[1]))
+            when pattern[2]==:do_slam
+              send_text(do_slam(m[1]))
+            when @mail[:prohibited]
+              send_text("450 4.7.1 Sender IP #{@mail[:remote_ip]} is temporarily prohibited from sending")
+            when pattern[0]>@level
+              send_text("503 5.5.1 Command out of sequence")
+            else
+              send_text(send(pattern[2], m[1].to_s.strip))
+            end
+            unrecognized = false
+            break
+          end
+        end
+        if unrecognized
+          response = "500 5.5.1 Unrecognized command #{text.inspect}, incorrectly formatted command, or command out of sequence"
+          send_text(response)
+        end
+      rescue OpenSSL::SSL::SSLError => e
+        LOG.error("%06d"%Process::pid) {"SSL error: #{e.inspect}"}
+        e.backtrace.each { |line| LOG.error("%06d"%Process::pid) {line} }
+        @done = true
+      end until @done
+
+    rescue Quit => e
+      @mail[:accepted] = false
+      # nothing to do but exit
+
+    rescue => e
+      # this is the "rescue of last resort"... "for when sh*t happens"
+      LOG.fatal("%06d"%Process::pid) {e.inspect}
+      e.backtrace.each { |line| LOG.fatal("%06d"%Process::pid) {line} }
+      @mail[:accepted] = false
+
+    ensure
+      # the email is either "received" or not, then when the
+      # return is executed, the process terminates
+      status = if @mail[:accepted] then 'Received' else 'Rejected' end
+      LOG.info("%06d"%Process::pid) {"#{status} mail with id '#{@mail[:id]}'"}
+      received(@mail)
+      # This is the end, beautiful friend
+      # This is the end, my only friend
+      # The end -- Jim Morrison
+      return nil # terminates the process
+    end
+
+#=======================================================================
+# these methods provide all the basic processing
+
+    def ok?(msg)
+      msg[0]!='4' && msg[0]!='5'
+    end
+
+    def do_connect(value)
+      LOG.info("%06d"%Process::pid) {"New item of mail opened with id '#{@mail[:id]}'"}
+      @mail[:connect] = p = {}
+      p[:value] = value
+
+      # this doesn't work with IPv4 addresses 'mapped' into IPv6, ie, ::ffff...
+      p[:domain] = value.dig_ptr
+
+      @level = 1 if ok?(msg = connect(p))
+      return msg
+    end
+
+    def do_ehlo(value)
+      @mail[:ehlo] = p = {}
+      p[:value] = value
+      p[:fip] = p[:rip] = nil
+      p[:rip] = rip = value.dig_a # reverse IP
+      p[:domain] = domain = rip.dig_ptr if rip
+      p[:fip] = domain.dig_a if domain # forward IP
+
+      return ("550 5.5.0 The domain name in EHLO does not validate") \
+        if @options[:ehlo_validation_check] && (p[:rip].nil? || p[:fip].nil? || p[:rip]!=p[:fip])
+
+      @level = 2 if ok?(msg = ehlo(p))
+      return msg
+    end
+
+    def do_quit(value)
+      @done = true if ok?(msg = quit(value))
+      return msg
+    end
+
+    def do_slam(value)
+      LOG.info("%06d"%Process::pid) {"Sender slammed the connection shut IP=#{@mail[:remote_ip]}"}
+      @done = true
+      @mail[:accepted] = false
+      return nil
+    end
+
+    def do_timeout(value)
+      @done = true
+      @mail[:accepted] = false
+      return ("501 5.4.7 Closing connection due to inactivity--#{@mail[:id]} was NOT saved")
+    end
+
+    def do_auth(value)
+      auth_type, auth_encoded = value.split
+      # auth_encoded contains both username and password
+      case auth_type.upcase
+      when "PLAIN"
+        # get the password hash from the database
+        username, ok = auth_encoded.validate_plain do |username|
+          password(username)
+        end
+        if ok
+          @mail[:authenticated] = username
+          return "235 2.0.0 Authentication succeeded"
+        else
+          return "530 5.7.5 Authentication failed"
+        end
+      else
+        return "504 5.7.6 authentication mechanism not supported"
+      end
+    end
+
+    def do_expn(value)
+      @mail[:expn] = p = {}
+      p[:value] = value
+      return expn(p)
+    end
+
+    def do_help(value)
+      return help(value)
+    end
+
+    def do_noop(value)
+      return noop(value)
+    end
+
+    def do_rset(value)
+      @level = 0 if ok?(msg = rset(value))
+      return msg
+    end
+
+    def do_vfry(value)
+      @mail[:vfry] = p = {}
+      p[:value] = value
+      return vfry(p)
+    end
+
+    def do_starttls(value)
+      send_text("220 2.0.0 TLS go ahead")
+      @connection.accept
+      @mail[:encrypted] = true
+      @enc_ind = '~'
+      return nil
+    end
+
+    def do_mail_from(value)
+      @mail[:mailfrom] = p = {:accepted=>false}
+      @mail[:rcptto] = []
+# TODO! A special case is the NULL envelope sender address (i.e. MAIL FROM: <>)
+      msg = psych_value(:mailfrom, p, value)
+      return (msg) if msg
+
+      if ok?(msg = mail_from(p))
+        p[:accepted] = true
+        @level = 3
+      end
+      return msg
+    end
+
+    def do_rcpt_to(value)
+      @mail[:rcptto] ||= []
+      @mail[:rcptto] << p = {:accepted=>false}
+
+      msg = psych_value(:rcptto, p, value)
+      return (msg) if msg
+
+      if ok?(msg = rcpt_to(p))
+        p[:accepted] = true
+        @level = 4
+      end
+      return msg
+    end
+
+    def do_data(value)
+# http://www.tldp.org/HOWTO/Spam-Filtering-for-MX/datachecks.html
+      @mail[:data] = body = {}
+      body[:accepted] = false
+      # receive the body of the mail
+      body[:value] = value # this should be nil -- no argument on the DATA command
+      body[:text] = lines = []
+      send_text("354 3.0.0 Enter message, ending with \".\" on a line by itself", false)
+      LOG.info("%06d"%Process::pid) {" -> (email message)"} if LogConversation
+      while true
+        text = recv_text(false)
+        break if text.nil? # the  client closed the channel abruptly
+        lines << text
+        if text=="."
+          body[:accepted] = true
+          break
+        end
+      end
+      @mail.parse_headers
+# should contain:
+# To: ...
+# Date: ...
+# From: ...
+# Subject: ...
+# Message-ID: ...
+
+# DKIM
+# SPF
+
+      # check the DKIM headers, if any
+      ok, signatures = pdkim_verify_an_email(PDKIM_INPUT_NORMAL, @mail[:data][:text])
+      signatures.each do |signature|
+        @log.info("%06d"%Process::pid){"Signature for '#{signature[:domain]}': #{PdkimReturnCodes[signature[:verify_status]]}"}
+        @mail[:signatures] ||= []
+        @mail[:signatures] << [signature[:domain], signature[:verify_status], DkimOutcomes[signature[:verify_status]]]
+      end if ok==PDKIM_OK
+
+      # test all the RCPT TOs
+      all_rcptto_accepted = true
+      @mail[:rcptto].each { |p| all_rcptto_accepted = false if !p[:accepted] } if @mail.has_key?(:rcptto)
+      # passed thru the guantlet with no failures
+      @mail[:accepted] = true \
+        if @mail[:mailfrom][:accepted] &&
+          all_rcptto_accepted &&
+          @mail[:data][:accepted] &&
+          @has_level_5_warnings==false
+
+      msg = data(p)
+      @level = 1
+      return msg
+    end
+
+#=======================================================================
+# these are the defaults, in case the user doesn't override
+
+    def connect(remote_ip)
+      return "220 2.0.0 ESMTP RubyMTA 0.01 #{Time.new.strftime("%^a, %d %^b %Y %H:%M:%S %z")}"
+    end
+
+    def ehlo(p)
+      msg = ["250-2.0.0 #{p[:value]} Hello"]
+      msg << "250-STARTTLS" if !@mail[:encrypted]
+      msg << "250-AUTH PLAIN"
+      msg << "250 HELP"
+      return msg
+    end
+
+    def quit(value)
+      return "221 2.0.0 OK #{"example.com"} closing connection"
+    end
+
+    def auth(value)
+      return "235 2.0.0 Authentication succeeded"
+    end
+
+    def password(username)
+      return nil
+    end
+
+    def expn(value)
+      return "252 2.5.1 Administrative prohibition"
+    end
+
+    def help(value)
+      return "250 2.0.0 QUIT AUTH, EHLO, EXPN, HELO, HELP, NOOP, RSET, VFRY, STARTTLS, MAIL FROM, RCPT TO, DATA"
+    end
+
+    def noop(value)
+      return "250 2.0.0 OK"
+    end
+
+    def rset(value)
+      return "250 2.0.0 Reset OK"
+    end
+
+    def vfry(value)
+      return "252 2.5.1 Administrative prohibition"
+    end
+
+    def mail_from(value)
+      return "250 2.0.0 OK"
+    end
+
+    def rcpt_to(value)
+      return "250 2.0.0 OK"
+    end
+
+    def data(value)
+      return "250 2.0.0 OK id=#{@mail[:id]}"
+    end
+
+    def received(mail)
+      # nothing here--just a placeholder
+    end
+  end
+end

data/lib/net/version.rb

@@ -0,0 +1,4 @@
+module Net
+  BUILD_VERSION = "1.0.0"
+  BUILD_DATE = "2016-09-21"
+end

metadata

@@ -0,0 +1,50 @@
+--- !ruby/object:Gem::Specification
+name: net-receiver
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Michael J. Welch, Ph.D.
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2016-09-21 00:00:00.000000000 Z
+dependencies: []
+description: Ruby Net Receiver.
+email:
+- mjwelchphd@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- README.md
+- lib/net/extended_classes.rb
+- lib/net/item_of_mail.rb
+- lib/net/receiver.rb
+- lib/net/version.rb
+homepage: https://github.com/mjwelchphd/net-receiver
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.6
+signing_key: 
+specification_version: 4
+summary: Ruby Net Receiver.
+test_files: []