feed2email 0.5.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ac8af629627f99050a52d97c8c83ae87c03663c4
-  data.tar.gz: cfc63b86eb14679e57d8f445559b682e2edeea16
+  metadata.gz: f9a92661dec1838e02af880c1e7c0e4c2a37dced
+  data.tar.gz: 879f563ae943c253b987a1c5beeec95b41bb95c3
 SHA512:
-  metadata.gz: 8609da85c1b80a45cc69fa03ef66abbd3b5a4d0e588a720c3700fe887fe884afacbfa33f4b975037fed7d4480a2a6b219a9bebf01273cf28b5c1e25fd75fa23e
-  data.tar.gz: 3434bf8b41b8a0528f215a273c337198c16986e8ad04732e27417aa2b8af3b33d268110dbf94f4592679833b63fa10fb3ae77e0da06aefb6f33f07db3eaf6dfa
+  metadata.gz: ad5c3c4fe63c1e5e860a3ea7bfaa5af4b345806c0569853e4a365f34f53cd0c3672fac725e3e2067cca5f5b9a1348f0a1651e5113a2488ee4d8ed80ad82d14be
+  data.tar.gz: f90bb39169407674065bedd943645ec94aee5f2ab825070e0563cc51fd5053665efdc15e054bb51146f466715b06387db668b127bb3faf6e232d48377ecdbe29

data/CHANGELOG.md

@@ -1,3 +1,11 @@
+### 0.6.0
+
+* Render text/plain body as Markdown
+* Cache feed fetching with Last-Modified and ETag HTTP headers
+* Update feed URI on permanent redirect
+* Make sender a required config option
+* Maintain a separate history file per feed
+
 ### 0.5.0
 
 * Sanitize SMTP user in from address

data/LICENSE.txt

@@ -1,6 +1,6 @@
 The MIT License
 
-Copyright (c) 2013 Aggelos Orfanakos
+Copyright (c) 2013, 2014, 2015 Aggelos Orfanakos
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of
 this software and associated documentation files (the "Software"), to deal in

data/README.md

@@ -27,8 +27,8 @@ $ gem install feed2email
 
 Through a [YAML][] file at `~/.feed2email/config.yml`.
 
-It is possible to send email via SMTP or an [MTA][]. If `config.yml` contains
-options for both, feed2email will use SMTP.
+It is possible to send email via SMTP or an [MTA][] (default). If `config.yml`
+contains options for both, feed2email will use SMTP.
 
 [YAML]: http://en.wikipedia.org/wiki/YAML
 [MTA]: http://en.wikipedia.org/wiki/Message_transfer_agent
@@ -41,13 +41,11 @@ pair is separated with a colon: `foo: bar`
 ### Generic options
 
 * `recipient` (required) is the email address to send email to
-* `sender` (optional) is the email address to send email from (default is taken
-  from the feed entry author or, if missing, it is generated from the SMTP user
-  and host or, if missing, it is the same as `recipient`)
+* `sender` (required) is the email address to send email from (can be any)
 * `send_delay` (optional) is the number of seconds to wait between each email to
   avoid SMTP server throttling errors (default is `10`; use `0` to disable)
 * `log_path` (optional) is the _absolute_ path to the log file (default is
-  `true` which logs to standard output; use `false` to disable)
+  `true` which logs to standard output; use `false` to disable logging)
 * `log_level` (optional) is the logging verbosity level and can be `fatal`
   (least verbose), `error`, `warn`, `info` (default) and `debug` (most verbose)
 * `max_entries` (optional) is the maximum number of entries to process per feed
@@ -58,10 +56,11 @@ pair is separated with a colon: `foo: bar`
 For this method you need to have access to an SMTP service. [Mailgun][] has a
 free plan.
 
-* `smtp_host` is the SMTP service hostname to connect to
-* `smtp_port` is the SMTP service port to connect to
-* `smtp_user` is the username of your email account
-* `smtp_pass` is the password of your email account (see the warning below)
+* `smtp_host` (required) is the SMTP service hostname to connect to
+* `smtp_port` (required) is the SMTP service port to connect to
+* `smtp_user` (required) is the username of your email account
+* `smtp_pass` (required) is the password of your email account (see the warning
+   below)
 * `smtp_tls` (optional) controls TLS (default is `true`; can also be `false`)
 * `smtp_auth` (optional) controls the authentication method (default is `login`;
    can also be `plain` or `cram_md5`)
@@ -87,6 +86,8 @@ interface setup and working in your system like [msmtp][] or [Postfix][].
 
 ## Use
 
+### Managing feeds
+
 Create `~/.feed2email/feeds.yml` and add the address of each feed you want to
 subscribe to, prefixed with a dash and a space:
 
@@ -94,30 +95,57 @@ subscribe to, prefixed with a dash and a space:
 - https://github.com/agorf/feed2email/commits.atom
 ~~~
 
-To disable a feed temporarily, comment it:
+To disable a feed, comment it:
 
 ~~~ yaml
 #- https://github.com/agorf/feed2email/commits.atom
 ~~~
 
-You are now ready to run the program:
+### Running
+
+Simply:
 
 ~~~ sh
 $ feed2email
 ~~~
 
-When run for the first time, feed2email enters "dry run" mode and exits almost
-immediately. During dry run mode:
+When feed2email runs for the first time or after adding a new feed:
+
+* All feed entries are skipped (no email sent)
+* `~/.feed2email/history-<digest>.yml` is created for each feed containing these
+  (old) entries, where `<digest>` is the MD5 hex digest of the feed URL
+
+**Warning:** Versions prior to 0.6.0 used a single history file for all feeds.
+Before using version 0.6.0 for the first time, please make sure you run the
+provided migration script: `feed2email-migrate-history` If you don't, feed2email
+will think it's run for the first time and will treat all entries as old (thus
+no email will be sent and you may miss some entries).
+
+To receive existing entries from a new feed:
 
-* No feeds are fetched and, thus, no email is sent (existing feed entries are
-  considered already seen)
-* `~/.feed2email/history.yml` is created containing processed (seen) entries per
-  feed
+1. Add it to `feeds.yml` (see above)
+1. Run feed2email once so that the feed's history file is generated
+1. Remove the entries you want to receive from the feed's history (i.e. with
+   your text editor)
+1. Remove the feed's meta file (`meta-<digest>.yml`) to bust feed fetching
+   caching
 
-If you want to receive existing entries from a specific feed, you can manually
-delete them from `history.yml`. Next time feed2email runs, they will be
+Next time feed2email runs, these entries will be treated as new and will be
 processed (sent as email).
 
+### Permanent redirections
+
+Before processing each feed, feed2email issues a [HEAD request][] to check
+whether it has been permanently moved by looking for a _301 Moved Permanently_
+HTTP status and its respective _Location_ header. In such case, feed2email
+updates `feeds.yml` with the new location and all feed entries are skipped (no
+email sent). If you do want to have some of them sent as email, please refer to
+the _Running_ section.
+
+[HEAD request]: http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods
+
+### Automating
+
 You can use [cron][] to run feed2email automatically e.g. once every hour.
 
 [cron]: http://en.wikipedia.org/wiki/Cron

data/TODO.md

@@ -0,0 +1,13 @@
+# TODO
+
+* Specs
+* Do not mark entry as sent if email was not sent
+* Show entry metadata in email (e.g. pubdate, author)
+* Implement a command-line interface to manage feeds.yml
+* Detect entry URI changes (maybe by comparing body hashes?)
+* Filters (e.g. skip entries matching a pattern)
+* Support "dispatch interfaces" where email is one such interface (another could
+  be writing to the filesystem)
+* Profiles (support many feed lists and recipients)
+* Send email notifications to user (e.g. when a feed is not available anymore)
+* Plugin architecture

data/bin/feed2email

@@ -1,5 +1,6 @@
 #!/usr/bin/env ruby
 
 require 'feed2email'
+require 'feed2email/feed'
 
 Feed2Email::Feed.process_all

data/bin/feed2email-migrate-history

@@ -0,0 +1,29 @@
+#!/usr/bin/env ruby
+
+require 'digest/md5'
+require 'yaml'
+
+CONFIG_DIR = File.expand_path('~/.feed2email')
+HISTORY_FILE = File.join(CONFIG_DIR, 'history.yml')
+
+if !File.exist?(HISTORY_FILE)
+  $stderr.puts "Missing history file #{HISTORY_FILE}"
+  exit 1
+end
+
+history_data = YAML.load(open(HISTORY_FILE))
+
+if !history_data.is_a?(Hash)
+  $stderr.puts "Invalid data type for history file #{HISTORY_FILE}"
+  exit 2
+end
+
+history_data.each do |feed_uri, entries|
+  hist_filename = "history-#{Digest::MD5.hexdigest(feed_uri)}.yml"
+  hist_path = File.join(CONFIG_DIR, hist_filename)
+  open(hist_path, 'w') {|f| f.write(entries.to_yaml) }
+  puts "Extracted history of #{feed_uri} to #{hist_filename}"
+end
+
+File.rename(HISTORY_FILE, "#{HISTORY_FILE}.bak")
+puts "Renamed #{HISTORY_FILE} to #{HISTORY_FILE}.bak"

data/lib/feed2email/config.rb

@@ -1,31 +1,106 @@
-module Feed2Email
-  CONFIG_DIR = File.expand_path('~/.feed2email')
+require 'yaml'
 
+module Feed2Email
   class Config
-    include Singleton
+    class MissingConfigError < StandardError; end
+    class InvalidConfigPermissionsError < StandardError; end
+    class InvalidConfigSyntaxError < StandardError; end
+    class InvalidConfigDataTypeError < StandardError; end
+    class MissingConfigOptionError < StandardError; end
+
+    def initialize(path)
+      @path = File.expand_path(path)
+      check
+    end
 
-    CONFIG_FILE = File.join(CONFIG_DIR, 'config.yml')
+    def [](option)
+      merged_config[option] # delegate
+    end
 
-    attr_reader :config
+    private
 
-    def read!
-      FileUtils.mkdir_p(CONFIG_DIR)
+    def path
+      @path
+    end
 
-      @config = YAML.load(open(CONFIG_FILE)) rescue nil
+    def data
+      @data
+    end
 
-      if !@config.is_a? Hash
-        STDERR.puts "Error: missing or invalid config file #{CONFIG_FILE}"
-        exit 1
+    def check
+      check_existence
+      check_permissions
+      check_syntax
+      check_data_type
+      check_recipient_existence
+      check_sender_existence
+    end
+
+    def check_existence
+      if !File.exist?(path)
+        raise MissingConfigError, "Missing config file #{path}"
       end
+    end
 
-      if '%o' % (File.stat(CONFIG_FILE).mode & 0777) != '600'
-        STDERR.puts "Error: invalid permissions for config file #{CONFIG_FILE}"
-        exit 2
+    def check_permissions
+      if '%o' % (File.stat(path).mode & 0777) != '600'
+        raise InvalidConfigPermissionsError,
+          "Invalid permissions for config file #{path}"
       end
+    end
+
+    def check_syntax
+      begin
+        load_yaml
+      rescue Psych::SyntaxError
+        raise InvalidConfigSyntaxError,
+          "Invalid YAML syntax for config file #{path}"
+      end
+    end
+
+    def check_data_type
+      if !data.is_a?(Hash)
+        raise InvalidConfigDataTypeError,
+          "Invalid data type (not a Hash) for config file #{path}"
+      end
+    end
+
+    def check_recipient_existence
+      check_option_existence('recipient')
+    end
+
+    def check_sender_existence
+      check_option_existence('sender')
+    end
+
+    def load_yaml
+      @data = YAML.load(read_file)
+    end
+
+    def read_file
+      File.read(path)
+    end
+
+    def merged_config
+      @merged_config ||= defaults.merge(data)
+    end
+
+    def defaults
+      {
+        'log_level'     => 'info',
+        'log_path'      => true,
+        'max_entries'   => 20,
+        'send_delay'    => 10,
+        'sendmail_path' => '/usr/sbin/sendmail',
+        'smtp_auth'     => 'login',
+        'smtp_tls'      => true,
+      }
+    end
 
-      if @config['recipient'].nil?
-        STDERR.puts "Error: recipient missing from config file #{CONFIG_FILE}"
-        exit 3
+    def check_option_existence(option)
+      if data[option].nil?
+        raise MissingConfigOptionError,
+          "Option #{option} missing from config file #{path}"
       end
     end
   end

data/lib/feed2email/core_ext.rb

@@ -1,11 +1,23 @@
+require 'cgi'
+require 'reverse_markdown'
+require 'sanitize'
+
 class String
   def escape_html
     CGI.escapeHTML(self)
   end
 
+  def pluralize(count, plural = self + 's')
+    "#{count} #{count == 1 ? self : plural}"
+  end
+
   def strip_html
     CGI.unescapeHTML(Sanitize.clean(self))
   end
+
+  def to_markdown
+    ReverseMarkdown.convert(self, unknown_tags: :drop)
+  end
 end
 
 class Time

data/lib/feed2email/entry.rb

@@ -1,3 +1,5 @@
+require 'feed2email/mail'
+
 module Feed2Email
   class Entry
     def initialize(data, feed_uri, feed_title)
@@ -14,22 +16,25 @@ module Feed2Email
       @data.content || @data.summary
     end
 
-    def process
+    def send_mail
       Mail.new(self, @feed_title).send
     end
 
     def title
-      @data.title
+      @data.title.strip
     end
 
     def uri
-      @uri ||= begin
-        if @data.url[0] == '/' # invalid entry URL is a path
-          @feed_uri[%r{https?://[^/]+}] + @data.url # prepend feed URI
-        else
-          @data.url
-        end
+      return @uri if @uri
+
+      @uri = @data.url
+
+      # Make relative entry URL absolute by prepending feed URL
+      if @uri && @uri.start_with?('/')
+        @uri = @feed_uri[%r{https?://[^/]+}] + @uri
       end
+
+      @uri
     end
   end
 end