feed2email 0.7.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f18bbfd4e8a09cb263f70c7966af0fa9e6602a3c
-  data.tar.gz: fe00a0445e6010eca1d847861f33687beb7a8dc7
+  metadata.gz: 51069bce6917475aa8499384d330396eebde34c1
+  data.tar.gz: cde703782a959046c866aa459123d050e6df5fc2
 SHA512:
-  metadata.gz: 8fddbfd1416024208e32e65ba16a5eeb51c3e6537d6b39d0e49ba945ae8ab5c9bf9c8c14f3e976b173da9f06976e15b54ebc18ac73c62250727f2e3a58eafa1b
-  data.tar.gz: 0f1cbbdcba1768b9f75ad29c21b3c5c4d17cd55446dbaa1aed49fc72d9b63ec58a3c333dd1d8a0001e31f9cc6555ac106a8c40db764bcc127b004a2e0605f69c
+  metadata.gz: 702e5ab54cd401d097b572bfdea2acfc48af045382ddc988135a1f93d8b100790c950b1cf1c32a0405feab4ae8d746c289fffd3ed328ff6974e2418eef87f383
+  data.tar.gz: 271c7c5d97acbc461dcdf24ac481255d0ddd6a387535c8fb9dc1fef58fcb6783aed9235c87d7a089de6e625036e06ae6399ec2f3b861486af257c697bb526f0f

data/CHANGELOG.md

@@ -1,3 +1,17 @@
+### 0.8.0
+
+* Command-line interface for managing feeds (c20bf0c, e5c6dec)
+* Perform feed autodiscovery in `add` command (46523ee)
+* Store feed metadata in feed list, so no more feed files (cfbd725)
+* Add `f2e` symlink to `feed2email` binary for running convenience (6add6b9)
+* Improve send delay between entry processing (2118aef)
+* Fix feed fetching exception handling (8eb68cf)
+* Sync feed metadata only if all entries are processed (73f0947)
+* Record entry to history only if email was sent (db59e77)
+* Always fetch feed when permanently redirected (4fb147f)
+* Ignore redirections to the same location (26513d1e)
+* Major rewrite of README file with new instructions
+
 ### 0.7.0
 
 * Prevent simultaneous running instances
@@ -11,7 +25,7 @@
 * Render text/plain body as Markdown
 * Cache feed fetching with Last-Modified and ETag HTTP headers
 * Update feed URI on permanent redirect
-* Make sender a required config option
+* Make `sender` a required config option
 * Maintain a separate history file per feed
 
 ### 0.5.0

data/README.md

@@ -1,22 +1,25 @@
 # feed2email [![Gem Version](https://badge.fury.io/rb/feed2email.svg)](http://badge.fury.io/rb/feed2email)
 
-RSS/Atom feed updates in your email
-
-## Why
-
-I don't like having a separate application for feeds when I'm already checking
-my email. I also never read a thing when feeds are kept in a separate place.
-
 feed2email is a [headless][] RSS/Atom feed aggregator that sends feed entries
-via email. It was written primarily as a replacement of [rss2email][] and aims
+via email. It was initially written as a replacement of [rss2email][] and aims
 to be simple, fast and easy to use.
 
 [headless]: http://en.wikipedia.org/wiki/Headless_software
 [rss2email]: http://www.allthingsrss.com/rss2email/
 
+## Features
+
+* Easy command-line feed management (add, remove, enable/disable)
+* Feed fetching caching
+* Feed autodiscovery
+* Support for sending email with SMTP or a local MTA (e.g. sendmail)
+* _text/html_ and _text/plain_ (Markdown) multipart emails
+* Support for feed permanent redirections
+* Auto-fixing relative feed entry permalinks
+
 ## Installation
 
-Install as a [gem][] from [RubyGems][]:
+As a [gem][] from [RubyGems][]:
 
 ~~~ sh
 $ gem install feed2email
@@ -27,19 +30,24 @@ $ gem install feed2email
 
 ## Configuration
 
-Through a [YAML][] file at `~/.feed2email/config.yml`.
+Through a [YAML][] file that you create at `~/.feed2email/config.yml`.
 
 [YAML]: http://en.wikipedia.org/wiki/YAML
 
 Each line in the configuration file contains a key-value pair. Each key-value
-pair is separated with a colon: `foo: bar`
+pair is separated with a colon, e.g.: `foo: bar`
 
-### Generic options
+### General options
 
 * `recipient` (required) is the email address to send email to
 * `sender` (required) is the email address to send email from (can be any)
 * `send_delay` (optional) is the number of seconds to wait between each email to
   avoid SMTP server throttling errors (default is `10`; use `0` to disable)
+* `max_entries` (optional) is the maximum number of entries to process per feed
+  (default is `20`; use `0` for unlimited)
+
+### Logging options
+
 * `log_path` (optional) is the _absolute_ path to the log file (default is
   `true` which logs to standard output; use `false` to disable logging)
 * `log_level` (optional) is the logging verbosity level and can be `fatal`
@@ -50,15 +58,15 @@ pair is separated with a colon: `foo: bar`
 * `log_shift_size` (optional) is the maximum log file size in _megabytes_ and it
   only applies when `log_shift_age` is a number greater than zero (default is
   `1`)
-* `max_entries` (optional) is the maximum number of entries to process per feed
-  (default is `20`; use `0` for unlimited)
+
+### Delivery options
 
 It is possible to send email via SMTP or an [MTA][] (default). If `config.yml`
 contains options for both, feed2email will use SMTP.
 
 [MTA]: http://en.wikipedia.org/wiki/Message_transfer_agent
 
-### SMTP
+#### SMTP
 
 For this method you need to have access to an SMTP service. [Mailgun][] has a
 free plan.
@@ -76,11 +84,11 @@ free plan.
 **Warning:** Unless it has correct restricted permissions, anyone with access in
 your system will be able to read `config.yml` and your password. To prevent
 this, feed2email will not run and complain if it detects the wrong permissions.
-To set the correct permissions, issue `chmod 600 ~/.feed2email/config.yml`
+To set the correct permissions, issue `chmod 600 ~/.feed2email/config.yml`.
 
 [Mailgun]: http://www.mailgun.com/
 
-### MTA
+#### MTA
 
 For this method you need to have an [MTA][] with a [Sendmail][]-compatible
 interface set up and working in your system like [msmtp][] or [Postfix][].
@@ -96,68 +104,125 @@ interface set up and working in your system like [msmtp][] or [Postfix][].
 
 ### Managing feeds
 
-Create or edit `~/.feed2email/feeds.yml` and add the URL of the feed you want to
-subscribe to, prefixed with a dash and a space:
+First, add some feeds:
+
+~~~ sh
+$ feed2email add https://github.com/agorf.atom
+Added feed https://github.com/agorf.atom at index 0
+$ feed2email add https://github.com/agorf/feed2email/commits.atom
+Added feed https://github.com/agorf/feed2email/commits.atom at index 1
+~~~
+
+**Tip:** You only have to type a feed2email command until it is unambiguous e.g.
+instead of `feed2email list`, you can simply issue `feed2email l` as long as
+there is no other command beginning with an `l`.
 
-~~~ yaml
-- https://github.com/agorf/feed2email/commits.atom
+It is also possible to pass a website URL and let feed2email autodiscover any
+feeds:
+
+~~~ sh
+$ feed2email add http://www.rubyinside.com/
+0: http://www.rubyinside.com/feed/ "Ruby Inside" (application/rss+xml)
+Please enter a feed to subscribe to: 0
+Added feed http://www.rubyinside.com/feed/ at index 2
+$ feed2email add http://thechangelog.com/137/
+0: http://thechangelog.com/137/feed/ "The Changelog » #137: Better GitHub Issues with HuBoard and Ryan Rauh Comments Feed" (application/rss+xml)
+1: http://thechangelog.com/feed/ "RSS 2.0 Feed" (application/rss+xml)
+Please enter a feed to subscribe to: 1
+Added feed http://thechangelog.com/feed/ at index 3
+$ feed2email add http://thechangelog.com/137/
+0: http://thechangelog.com/137/feed/ "The Changelog » #137: Better GitHub Issues with HuBoard and Ryan Rauh Comments Feed" (application/rss+xml)
+Please enter a feed to subscribe to: ^C
+~~~
+
+Note that on the last command, feed2email autodiscovers the same two feeds as in
+the second command, but only lists the ones that haven't been already added.
+Autodiscovery is then cancelled by pressing `Ctrl-C`.
+
+The feed list so far:
+
+~~~ sh
+$ feed2email list
+0: https://github.com/agorf.atom
+1: https://github.com/agorf/feed2email/commits.atom
+2: http://www.rubyinside.com/feed/
+3: http://thechangelog.com/feed/
+~~~
+
+To disable a feed so that it is not processed with `feed2email process`, toggle
+it:
+
+~~~ sh
+$ feed2email toggle 1
+Disabled feed at index 1
+$ feed2email list
+0: https://github.com/agorf.atom
+1: DISABLED https://github.com/agorf/feed2email/commits.atom
+2: http://www.rubyinside.com/feed/
+3: http://thechangelog.com/feed/
 ~~~
 
-To disable a feed, comment its line by prefixing it with a hash symbol:
+It is also possible to remove it from the list:
 
-~~~ yaml
-#- https://github.com/agorf/feed2email/commits.atom
+~~~ sh
+$ feed2email remove 1
+Removed feed at index 1
+Warning: Feed list indices have changed!
 ~~~
 
-### Running for the first time
+It has been removed, but what is that weird warning?
 
-When feed2email runs for the first time or after adding a new feed:
+Since the feed that got removed was at index 1, every feed below it got
+reindexed. So feed2email warns you that the feed indices have changed: the feed
+at index 2 is now at index 1 and the feed at index 3 is now at index 2.
 
-* All feed entries are skipped (no email sent)
-* `~/.feed2email/history-<digest>.yml` is created for each feed containing these
-  (old) entries, where `<digest>` is the MD5 hex digest of the feed URL
+Indeed:
 
-**Warning:** Versions prior to 0.6.0 used a single history file for all feeds.
-Before using version 0.6.0 for the first time, please make sure you run the
-provided migration script: `feed2email-migrate-history` If you don't, feed2email
-will think it's run for the first time and will treat all entries as old (thus
-no email will be sent and you may miss some entries).
+~~~ sh
+$ feed2email list
+0: https://github.com/agorf.atom
+1: http://www.rubyinside.com/feed/
+2: http://thechangelog.com/feed/
+~~~
 
-### Receiving specific entries from a feed
+**Tip:** feed2email installs `f2e` as a symbolic link to the feed2email binary,
+so you can use that to avoid typing the whole name every time, e.g.: `f2e list`
+or even `f2e l`.
 
-1. Add the feed URL to `~/.feed2email/feeds.yml`
-1. Run feed2email once so that the feed's history file is generated
-1. Remove the entries you want to receive from the feed's history (i.e. with
-   your text editor)
-1. Remove the feed's meta file (`meta-<digest>.yml`, where `<digest>` is the MD5
-   hex digest of the feed URL) to bust feed fetching caching
+### Running
 
-Next time feed2email runs, these entries will be treated as new and will be
-processed (sent as email).
+~~~ sh
+$ feed2email process
+~~~
 
-### Permanent redirections
+When run, feed2email will go through your feed list, fetch each feed (if
+necessary) and send an email for each new entry. Any output is logged to the
+standard output, unless configured otherwise.
 
-Before processing each feed, feed2email issues a [HEAD request][] to check
-whether it has been permanently moved by looking for a _301 Moved Permanently_
-HTTP status and its respective _Location_ header. In such case, feed2email
-updates `~/.feed2email/feeds.yml` with the new location and all feed entries are
-skipped (no email sent). If you do want to have some of them sent as email, see
-[Receiving specific entries from a feed](#receiving-specific-entries-from-a-feed).
+**Warning:** Prior to version 0.8.0 where a command-line interface was
+introduced, the way to run feed2email was simply `feed2email`. Now this will
+just print helpful text on how to use it.
 
-[HEAD request]: http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods
+When a new feed is detected (which is the case when feed2email runs for the
+first time on your feed list), all of its entries are skipped and no email is
+sent. This is so that you don't get spammed when you add a feed for the first
+time.
 
-### Feed caching
+If you want to receive a specific entry from a newly added feed, edit the feed's
+history file with `feed2email history` and remove the entry. Then issue
+`feed2email fetch` to clear the feed's fetch cache. Next time
+`feed2email process` runs, the entry will be treated as new and will be
+processed.
 
-feed2email caches fetched feeds with the _Last-Modified_ and _Etag_ HTTP
-headers. If you want to force a feed to be fetched, remove the feed's meta file
-(`~/.feed2email/meta-<digest>.yml`, where `<digest>` is the MD5 hex digest of
-the feed URL). Next time feed2email runs, the feed will be fetched.
+### Getting help
 
-### Automating
+Issue `feed2email` or `feed2email help` at any point to get helpful text on how
+to use feed2email.
 
-You can use [cron][] to run feed2email automatically e.g. once every hour.
+## Contributing
 
-[cron]: http://en.wikipedia.org/wiki/Cron
+Using feed2email and want to help? Just [contact me](http://agorf.gr/) and
+describe how you use it and if you have any ideas on how it can be improved.
 
 ## License
 

data/bin/f2e

@@ -0,0 +1,9 @@
+#!/usr/bin/env ruby
+
+if File.new('/tmp/feed2email.lock', 'w').flock(File::LOCK_NB | File::LOCK_EX)
+  require 'feed2email/cli'
+  Feed2Email::Cli.start(ARGV)
+else
+  abort 'An instance of feed2email is already running. Exiting...'
+end
+

data/bin/feed2email

@@ -1,11 +1,9 @@
 #!/usr/bin/env ruby
 
-require 'feed2email'
-require 'feed2email/feed'
-
 if File.new('/tmp/feed2email.lock', 'w').flock(File::LOCK_NB | File::LOCK_EX)
-  Feed2Email::Feed.process_all
+  require 'feed2email/cli'
+  Feed2Email::Cli.start(ARGV)
 else
-  $stderr.puts 'An instance of feed2email is already running. Exiting...'
-  exit 1
+  abort 'An instance of feed2email is already running. Exiting...'
 end
+

data/bin/feed2email-migrate-feedlist

@@ -0,0 +1,36 @@
+#!/usr/bin/env ruby
+
+require 'yaml'
+
+CONFIG_DIR = File.expand_path('~/.feed2email')
+FEEDS_FILE = File.join(CONFIG_DIR, 'feeds.yml')
+
+if !File.exist?(FEEDS_FILE)
+  $stderr.puts "Missing feed list file #{FEEDS_FILE}"
+  exit 1
+end
+
+feeds_data = YAML.load(open(FEEDS_FILE))
+
+unless feeds_data.is_a?(Array) && feeds_data.all? {|data| data.is_a?(String) }
+  $stderr.puts "Invalid data type for feed list file #{FEEDS_FILE}"
+  exit 2
+end
+
+File.rename(FEEDS_FILE, "#{FEEDS_FILE}.bak")
+
+begin
+  open(FEEDS_FILE, 'w') do |f|
+    f.write(
+      feeds_data.map {|uri|
+        { uri: uri, enabled: true }
+      }.to_yaml
+    )
+  end
+
+  puts "Migrated feed list file #{FEEDS_FILE}"
+  puts "Renamed #{FEEDS_FILE} to #{FEEDS_FILE}.bak"
+rescue
+  File.rename("#{FEEDS_FILE}.bak", FEEDS_FILE)
+  raise
+end

data/lib/feed2email.rb

@@ -1,22 +1,27 @@
 require 'feed2email/config'
+require 'feed2email/feed_list'
 require 'feed2email/lazy_smtp_connection'
 require 'feed2email/logger'
 
 module Feed2Email
-  CONFIG_DIR = File.expand_path('~/.feed2email')
+  CONFIG_DIR = File.join(ENV['HOME'], '.feed2email')
 
-  def self.config; @config end
-
-  def self.logger
-    @logger.logger # delegate
+  def self.config
+    @config ||= Config.new(File.join(CONFIG_DIR, 'config.yml'))
   end
 
-  def self.smtp_connection; @smtp_connection end
-
-  @config = Config.new(File.join(CONFIG_DIR, 'config.yml'))
+  def self.feed_list
+    @feed_list ||= FeedList.new(File.join(CONFIG_DIR, 'feeds.yml'))
+  end
 
-  @logger = Logger.new(config['log_path'], config['log_level'],
-                       config['log_shift_age'], config['log_shift_size'])
+  def self.logger
+    @logger ||= Logger.new(
+      config['log_path'], config['log_level'], config['log_shift_age'],
+      config['log_shift_size']
+    ).logger
+  end
 
-  @smtp_connection = LazySMTPConnection.new
+  def self.smtp_connection
+    @smtp_connection ||= LazySMTPConnection.new
+  end
 end

data/lib/feed2email/cli.rb

@@ -0,0 +1,158 @@
+require 'thor'
+require 'feed2email'
+require 'feed2email/feed_autodiscoverer'
+require 'feed2email/redirection_checker'
+
+module Feed2Email
+  class Cli < Thor
+    desc 'add URL', 'subscribe to feed at URL'
+    def add(uri)
+      uri = handle_permanent_redirection(uri)
+      uri = perform_feed_autodiscovery(uri)
+
+      begin
+        feed_list << uri
+      rescue FeedList::DuplicateFeedError => e
+        abort e.message
+      end
+
+      if feed_list.sync
+        puts "Added feed #{uri} at index #{feed_list.size - 1}"
+      else
+        abort 'Failed to add feed'
+      end
+    end
+
+    desc 'fetch FEED', 'clear fetch cache for feed at index FEED'
+    def fetch(index)
+      index = check_feed_index(index, in: (0...feed_list.size))
+      feed_list.clear_fetch_cache(index)
+
+      if feed_list.sync
+        puts "Cleared fetch cache for feed at index #{index}"
+      else
+        abort "Failed to clear fetch cache for feed at index #{index}"
+      end
+    end
+
+    desc 'history FEED', 'edit history file of feed at index FEED with $EDITOR'
+    def history(index)
+      abort '$EDITOR not set' unless ENV['EDITOR']
+
+      index = check_feed_index(index, in: (0...feed_list.size))
+      require 'feed2email/feed_history'
+      history_path = FeedHistory.new(feed_list[index][:uri]).path
+      exec(ENV['EDITOR'], history_path)
+    end
+
+    desc 'remove FEED', 'unsubscribe from feed at index FEED'
+    def remove(index)
+      index = check_feed_index(index, in: (0...feed_list.size))
+      deleted = feed_list.delete_at(index)
+
+      if deleted && feed_list.sync
+        puts "Removed feed at index #{index}"
+
+        if feed_list.size != index # feed was not the last
+          puts 'Warning: Feed list indices have changed!'
+        end
+      else
+        abort "Failed to remove feed at index #{index}"
+      end
+    end
+
+    desc 'toggle FEED', 'enable/disable feed at index FEED'
+    def toggle(index)
+      index   = check_feed_index(index, in: (0...feed_list.size))
+      toggled = feed_list.toggle(index)
+      enabled = feed_list[index][:enabled]
+
+      if toggled && feed_list.sync
+        puts "#{enabled ? 'En' : 'Dis'}abled feed at index #{index}"
+      else
+        abort "Failed to #{enabled ? 'en' : 'dis'}able feed at index #{index}"
+      end
+    end
+
+    desc 'list', 'list feed subscriptions'
+    def list
+      puts feed_list
+    end
+
+    desc 'process', 'process feed subscriptions'
+    def process
+      feed_list.process
+    end
+
+    desc 'version', 'show feed2email version'
+    def version
+      require 'feed2email/version'
+      puts "feed2email #{Feed2Email::VERSION}"
+    end
+
+    no_commands do
+      def check_feed_index(index, options = {})
+        if index.to_i.to_s != index ||
+            (options[:in] && !options[:in].include?(index.to_i))
+          puts if index.nil? # Ctrl-D
+          abort 'Invalid index'
+        end
+
+        index.to_i
+      end
+
+      def feed_list
+        Feed2Email.feed_list # delegate
+      end
+
+      def handle_permanent_redirection(uri)
+        checker = RedirectionChecker.new(uri)
+
+        if checker.permanently_redirected?
+          puts "Got permanently redirected to #{checker.location}"
+          checker.location
+        else
+          uri
+        end
+      end
+
+      def perform_feed_autodiscovery(uri)
+        discoverer = FeedAutodiscoverer.new(uri)
+
+        discovered_feeds = discoverer.feeds.reject {|feed|
+          feed_list.include?(feed[:uri])
+        }
+
+        if discovered_feeds.empty?
+          if discoverer.content_type == 'text/html'
+            puts 'Could not find any new feeds'
+            exit
+          end
+
+          return uri
+        end
+
+        justify = discovered_feeds.size.to_s.size
+
+        discovered_feeds.each_with_index do |feed, i|
+          puts '%{index}: %{uri} %{title}(%{content_type})' % {
+            index:        i.to_s.rjust(justify),
+            uri:          feed[:uri],
+            title:        feed[:title] ? "\"#{feed[:title]}\" " : '',
+            content_type: feed[:content_type]
+          }
+        end
+
+        begin
+          response = ask('Please enter a feed to subscribe to:')
+        rescue Interrupt # Ctrl-C
+          puts
+          exit
+        end
+
+        index = check_feed_index(response, in: (0...discovered_feeds.size))
+        discovered_feeds[index][:uri]
+      end
+    end
+  end
+end