feedigest 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 401be82539f92da17495ec1bbdc38e9308ee2c1d2bf1d991dd2440f8c7ce9dfd
-  data.tar.gz: 9ed6fdf1eb2ddfef7862cf3de2cf0052ed38d02eeb3f7a2e054692163234dc07
+  metadata.gz: 5e5e0608a5ec2045aa61335d068fe5acb0a8b620f850ff5e44933e3ce1564638
+  data.tar.gz: 6c450ac10c9017169004d91390eccb94c9581cf793ec5503525f09d91db728e2
 SHA512:
-  metadata.gz: 52813c1c6d54b757a5173673d65d436ce5418222314ddac2ddeb7c9b811089e72efbf6a194bf55e0c761cb7ee8d45a7fd86db180c867c8fa98ed160ef3ebc483
-  data.tar.gz: b8bb424b7a4fd0b7fcede8ccea5709b4ec88f666de8faf5fe8e5916123ae82c70e4f66a3c8960a677e06e6662bf8f3521b408fdef41da02762fcfb4c53544815
+  metadata.gz: 59dea377dc71be1dcbe0f4c162313262fe56ab3817837ce55cf35c01e2c1305f030941e4c3bd75fca7f175e16e4cfb95a16a1152de1a42a84eb686d2433c39ce
+  data.tar.gz: 68fc1ce197775bae4175ff2c9e030d260044742a3924de089880ee48b19d286b4df3dfae804a81c5ae6458ccd62d134886700b11f15357fb171508c04bc8b527

data/CHANGELOG.md

@@ -7,6 +7,20 @@ The format is based on [Keep a Changelog][] and this project adheres to
 
 ## [Unreleased][]
 
+## [0.2.0][] - 2018-07-18
+
+### Added
+
+- Support filtering feeds through an external command.
+- Opening informational text in digest email.
+- Support for specifying path to YAML configuration file.
+- Support for command-line options.
+
+### Changed
+
+- Configuration from environment variables to a YAML file.
+- `entry_window` config option unit from seconds to hours.
+
 ## [0.1.0][] - 2018-07-04
 
 ### Changed
@@ -50,4 +64,5 @@ Initial release.
 [0.0.2]: https://github.com/agorf/feedigest/compare/0.0.1...0.0.2
 [0.0.3]: https://github.com/agorf/feedigest/compare/0.0.2...0.0.3
 [0.1.0]: https://github.com/agorf/feedigest/compare/0.0.3...0.1.0
-[Unreleased]: https://github.com/agorf/feedigest/compare/0.1.0...HEAD
+[0.2.0]: https://github.com/agorf/feedigest/compare/0.1.0...0.2.0
+[Unreleased]: https://github.com/agorf/feedigest/compare/0.2.0...HEAD

data/README.md

@@ -14,104 +14,136 @@ It was written as a simpler alternative to [feed2email][].
 
 As a [gem][] from [RubyGems][]:
 
-~~~ sh
+```sh
 gem install feedigest
-~~~
+```
 
 If the above command fails, make sure the following system packages are
 installed. For Debian/Ubuntu, issue as root:
 
-~~~ sh
+```sh
 apt-get install build-essential patch ruby-dev zlib1g-dev liblzma-dev
-~~~
+```
 
 [gem]: http://rubygems.org/gems/feedigest
 [RubyGems]: http://rubygems.org/
 
 ## Configuration
 
-feedigest is configured through shell environment variables. Instead of passing
-them one by one when you run it, you can store them in a separate file and
-source it to make them available:
+feedigest is configured through a [YAML][] configuration file located under
+`~/.feedigest/config.yaml`:
 
-~~~ sh
-$ mkdir -p ~/.feedigest
-$ touch ~/.feedigest/env
-$ chmod 600 ~/.feedigest/env
-$ cat > ~/.feedigest/env
-FEEDIGEST_EMAIL_RECIPIENT=me@mydomain.com
-...
-^D
-~~~
+[YAML]: https://en.wikipedia.org/wiki/YAML
 
-**Note:** In the example above, `^D` stands for pressing `Ctrl-D`.
+The following configuration options are supported:
 
-The following environment variables are supported:
-
-* `FEEDIGEST_ENTRY_WINDOW` (default: `86400`) the maximum age, in seconds, of
-  entries to include in the digest
-* `FEEDIGEST_EMAIL_SENDER` (default: `feedigest@hostname`) the "from" address in
+* `entry_window` (default: `24`) the maximum age, in hours, of entries to
+  include in the digest
+* `email_sender` (default: `feedigest@hostname`) the "from" address in
   the email
-* `FEEDIGEST_EMAIL_RECIPIENT` the email address to send the email to
+* `email_recipient` (required) the email address to send the email to
 
-feedigest uses SMTP to send emails. You can get a free plan from [Mailgun][].
-The relevant environment variables are:
+feedigest uses SMTP to send emails ([Mailgun][] has a free plan). The relevant
+configuration options are:
 
 [Mailgun]: http://www.mailgun.com/
 
-* `FEEDIGEST_SMTP_ADDRESS`
-* `FEEDIGEST_SMTP_PORT` (default: `587`)
-* `FEEDIGEST_SMTP_USERNAME`
-* `FEEDIGEST_SMTP_PASSWORD`
-* `FEEDIGEST_SMTP_AUTH` (default: `plain`)
-* `FEEDIGEST_SMTP_STARTTLS` (default: `true`)
+* `smtp_address` (required) SMTP service address to connect to
+* `smtp_port` (default: `587`) SMTP service port to connect to
+* `smtp_username` (required) username of your email account
+* `smtp_password` (required) password of your email account
+* `smtp_auth` (default: `plain`) controls authentication method (can also be
+  `login` or `cram_md5`)
+* `smtp_starttls` (default: `true`) controls use of STARTTLS
 
-Finally, you need to provide to the standard input (stdin) of feedigest, a
-line-separated list of feed URLs:
+Finally, you will need a line-separated list of feed URLs:
 
-~~~ sh
+```sh
 $ cat > ~/.feedigest/feeds.txt
 https://github.com/agorf/feed2email/commits.atom
 https://github.com/agorf.atom
 ...
 ^D
-~~~
+```
+
+**Note:** In the example above, `^D` stands for pressing `Ctrl-D`.
 
 ## Use
 
-~~~ sh
-export $(cat ~/.feedigest/env | xargs) && feedigest send < ~/.feedigest/feeds.txt
-~~~
+```sh
+feedigest --feeds ~/.feedigest/feeds.txt
+```
 
 You can run this with [cron][] e.g. once per day at 10 am:
 
 [cron]: https://en.wikipedia.org/wiki/Cron
 
-~~~
-0 10 * * * export $(cat ~/.feedigest/env | xargs) && feedigest send < ~/.feedigest/feeds.txt
-~~~
+```
+0 10 * * * feedigest --feeds ~/.feedigest/feeds.txt
+```
 
 Alternatively, you can have feedigest simply print the generated email so that
-you can send it yourself e.g. by piping it to sendmail. To do that, you simply
-replace `feedigest send` with `feedigest print`:
-
-~~~ sh
-feedigest print < ~/.feedigest/feeds.txt
-~~~
-
-## Contributing
-
-Using feedigest and want to help? Please [let me know][contact] how you use it
-and if you have any ideas on how to improve it.
-
-[contact]: https://agorf.gr/contact/
+you can send it yourself e.g. by piping it to sendmail:
+
+```sh
+feedigest --dry-run --feeds ~/.feedigest/feeds.txt
+```
+
+It is also possible to have each feed filtered by a custom command. For example,
+the following script fixes a feed's entry publication dates that use Greek month
+names and don't follow the required RFC822 format:
+
+```ruby
+require 'nokogiri'
+
+feed_data = $stdin.read
+doc = Nokogiri.XML(feed_data)
+
+case feed_data
+when /advendure\.com/
+  doc.css('pubDate').each do |pubdate|
+    %w[
+      Ιαν Φεβ Μαρ Απρ Μαι Ιουν Ιουλ Αυγ Σεπ Οκτ Νοβ Δεκ
+      Δε Τρ Τε Πε Πα Σα Κυ
+    ].zip(
+      %w[
+        Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
+        Mo Tu We Th Fr Sa Su
+      ]
+    ).each do |greek, latin|
+      pubdate.content = pubdate.content.sub(greek, latin)
+    end
+  end
+
+  print doc
+else
+  print feed_data # Do nothing
+end
+```
+
+The script reads the feed XML from its standard input (stdin) and writes the
+modified XML to its standard output (stdout). To use it as a filter, you simply
+pass as a command-line argument the necessary command to execute it:
+
+```sh
+feedigest --filter 'ruby /path/to/filter.rb' --feeds ~/.feedigest/feeds.txt
+```
+
+Finally, it is possible to specify the path to the YAML configuration file with
+`--config`:
+
+```sh
+feedigest --feeds ~/.feedigest/feeds.txt --config ~/.config/feedigest.yaml
+```
+
+You can issue `feedigest -h` to get some help text on the supported options.
 
 ## License
 
-Licensed under the MIT license (see [LICENSE.txt][license]).
+[MIT][]
 
-[license]: https://github.com/agorf/feedigest/blob/master/LICENSE.txt
+[MIT]: https://github.com/agorf/feedigest/blob/master/LICENSE.txt
 
 ## Author
 
-Angelos Orfanakos, <https://agorf.gr/>
+[Angelos Orfanakos](https://agorf.gr/contact/)

data/bin/feedigest

@@ -1,23 +1,21 @@
 #!/usr/bin/env ruby
 
-if !%w[send print].include?(ARGV[0])
-  puts "#$0 <send|print>"
-  exit 1
-end
-
 require 'feedigest'
 
-puts 'Please provide a list of feed URLs (one per line) and press Ctrl-D'
+options = Feedigest::OptionParser.new(ARGV).options
+
+Feedigest.config_path = File.expand_path(options[:config])
 
-feed_urls = $stdin.readlines.map(&:strip)
-feeds = Feedigest::FeedFetcher.new(feed_urls).feeds
+feed_urls = IO.readlines(options[:feeds]).map(&:strip)
+feeds = Feedigest::FeedFetcher.new(feed_urls, options[:filter]).feeds
 
 exit if feeds.empty?
 
 mail = Feedigest::MailComposer.new(feeds).mail
 mail_sender = Feedigest::MailSender.new(mail)
 
-case ARGV[0]
-when 'send' then mail_sender.deliver
-when 'print' then puts mail_sender
+if options[:dry_run]
+  puts mail_sender
+else
+  mail_sender.deliver
 end

data/lib/feedigest.rb

@@ -1,6 +1,18 @@
 module Feedigest; end
 
-require 'feedigest/options'
+require 'feedigest/config'
+
+module Feedigest
+  class << self
+    attr_accessor :config_path
+  end
+
+  def self.config
+    @config ||= Feedigest::Config.new(config_path)
+  end
+end
+
+require 'feedigest/option_parser'
 require 'feedigest/feed_fetcher'
 require 'feedigest/mail_composer'
 require 'feedigest/mail_sender'

data/lib/feedigest/config.rb

@@ -0,0 +1,51 @@
+require 'yaml'
+
+class Feedigest::Config
+  attr_reader :path
+
+  def initialize(path)
+    @path = path
+  end
+
+  def fetch(key)
+    options.fetch(key) # Delegate
+  end
+
+  def options
+    return @options if @options
+
+    @options = default_options.merge(user_options)
+    @options[:smtp_starttls] = @options[:smtp_starttls] == 'true'
+
+    @options
+  end
+
+  # Translate SMTP options of feedigest for Mail gem
+  def mail_gem_smtp_options
+    {
+      address: options.fetch(:smtp_address),
+      port: options.fetch(:smtp_port),
+      user_name: options.fetch(:smtp_username),
+      password: options.fetch(:smtp_password),
+      authentication: options.fetch(:smtp_auth),
+      enable_starttls: options.fetch(:smtp_starttls)
+    }
+  end
+
+  private
+
+  def user_options
+    YAML.safe_load(File.read(path)).
+      map { |k, v| [k.to_sym, v] }.to_h # Symbolize keys
+  end
+
+  def default_options
+    {
+      entry_window: 24,
+      email_sender: "feedigest@#{`hostname`.strip}",
+      smtp_port: 587,
+      smtp_auth: 'plain',
+      smtp_starttls: 'true' # Gets converted to boolean
+    }
+  end
+end

data/lib/feedigest/feed_fetcher.rb

@@ -1,13 +1,17 @@
-require 'feedigest/options'
+require 'feedigest/config'
 require 'feedjira'
+require 'open-uri'
 
 class Feedigest::FeedFetcher
   Feed = Struct.new(:url, :title, :entries, :error)
 
-  attr_reader :feed_urls
+  USER_AGENT = "feedigest/#{Feedigest::VERSION}".freeze
 
-  def initialize(feed_urls)
+  attr_reader :feed_urls, :filter_cmd
+
+  def initialize(feed_urls, filter_cmd = nil)
     @feed_urls = feed_urls
+    @filter_cmd = filter_cmd
   end
 
   def feeds
@@ -34,9 +38,18 @@ class Feedigest::FeedFetcher
     )
   end
 
+  def fetch_feed(url)
+    feed_fd = OpenURI.open_uri(url, 'User-Agent' => USER_AGENT)
+
+    if filter_cmd
+      filter_feed(feed_fd)
+    else
+      feed_fd.read
+    end
+  end
+
   def fetch_and_parse_feed(url)
-    feed = Feedjira::Feed.fetch_and_parse(url)
-    [feed, nil]
+    [Feedjira::Feed.parse(fetch_feed(url)), nil]
   rescue StandardError => e
     [nil, e.message]
   end
@@ -50,6 +63,14 @@ class Feedigest::FeedFetcher
   end
 
   def window_start
-    @window_start ||= Time.now - Feedigest.options[:entry_window]
+    @window_start ||= Time.now - Feedigest.config.fetch(:entry_window) * 3600
+  end
+
+  def filter_feed(feed_fd)
+    IO.popen(filter_cmd, 'r+') do |io|
+      IO.copy_stream(feed_fd, io)
+      io.close_write
+      io.read
+    end
   end
 end

data/lib/feedigest/mail_composer.rb

@@ -1,4 +1,4 @@
-require 'feedigest/options'
+require 'feedigest/config'
 require 'feedigest/version'
 require 'nokogiri'
 require 'reverse_markdown'
@@ -14,8 +14,8 @@ class Feedigest::MailComposer
 
   def mail
     @mail ||= Mail.new(
-      Feedigest.options[:email_sender],
-      Feedigest.options[:email_recipient],
+      Feedigest.config.fetch(:email_sender),
+      Feedigest.config.fetch(:email_recipient),
       subject,
       html_body,
       text_body
@@ -36,6 +36,8 @@ class Feedigest::MailComposer
     @html_body ||=
       Nokogiri::HTML::Builder.new(encoding: 'utf-8') { |builder|
         builder.div do
+          header_html(builder)
+
           feeds.each do |feed|
             feed_html(builder, feed)
           end
@@ -49,6 +51,18 @@ class Feedigest::MailComposer
     ReverseMarkdown.convert(html_body)
   end
 
+  def header_html(builder)
+    builder.p(
+      sprintf(
+        'In the last %{hours}, %{entries} %{were} published in %{feeds}:',
+        hours: pluralize(Feedigest.config.fetch(:entry_window), 'hour'),
+        entries: pluralize(entries_count, 'entry', 'entries'),
+        were: entries_count == 1 ? 'was' : 'were',
+        feeds: pluralize(feeds_without_error.size, 'feed')
+      )
+    )
+  end
+
   def feed_html(builder, feed)
     builder.div do
       if feed.error

data/lib/feedigest/mail_sender.rb

@@ -1,4 +1,4 @@
-require 'feedigest/options'
+require 'feedigest/config'
 require 'forwardable'
 require 'mail'
 
@@ -47,6 +47,6 @@ class Feedigest::MailSender
   end
 
   def setup_delivery_method!(mail)
-    mail.delivery_method(:smtp, Feedigest.smtp_options)
+    mail.delivery_method(:smtp, Feedigest.config.mail_gem_smtp_options)
   end
 end

data/lib/feedigest/option_parser.rb

@@ -0,0 +1,61 @@
+require 'feedigest/version'
+require 'slop'
+
+class Feedigest::OptionParser
+  attr_reader :args
+
+  def initialize(args)
+    @args = args
+  end
+
+  def options
+    options = build_options
+    parser = Slop::Parser.new(options)
+
+    begin
+      parser.parse(args).to_hash
+    rescue Slop::Error => e
+      puts "error: #{e.message}"
+      puts
+      puts options
+      exit
+    end
+  end
+
+  private
+
+  def build_options
+    Slop::Options.new do |o|
+      o.string(
+        '--feeds',
+        'path to file that contains a line-separated list of feed URLs',
+        required: true
+      )
+
+      o.string '--filter', 'command to filter each feed with'
+
+      o.string(
+        '--config',
+        'path to YAML configuration file',
+        default: '~/.feedigest/config.yaml'
+      )
+
+      o.bool(
+        '-n',
+        '--dry-run',
+        'print email instead of sending it',
+        default: false
+      )
+
+      o.on '--version', 'print version' do
+        puts Feedigest::VERSION
+        exit
+      end
+
+      o.on '-h', '--help', 'print help' do
+        puts o
+        exit
+      end
+    end
+  end
+end

data/lib/feedigest/version.rb

@@ -1,3 +1,3 @@
 module Feedigest
-  VERSION = '0.1.0'.freeze
+  VERSION = '0.2.0'.freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: feedigest
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Angelos Orfanakos
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-07-04 00:00:00.000000000 Z
+date: 2018-07-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: feedjira
@@ -66,6 +66,26 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1.1'
+- !ruby/object:Gem::Dependency
+  name: slop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.6'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 4.6.2
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.6'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 4.6.2
 - !ruby/object:Gem::Dependency
   name: bundler-audit
   requirement: !ruby/object:Gem::Requirement
@@ -92,10 +112,11 @@ files:
 - README.md
 - bin/feedigest
 - lib/feedigest.rb
+- lib/feedigest/config.rb
 - lib/feedigest/feed_fetcher.rb
 - lib/feedigest/mail_composer.rb
 - lib/feedigest/mail_sender.rb
-- lib/feedigest/options.rb
+- lib/feedigest/option_parser.rb
 - lib/feedigest/version.rb
 homepage: https://github.com/agorf/feedigest
 licenses:

data/lib/feedigest/options.rb

@@ -1,23 +0,0 @@
-module Feedigest
-  def self.options
-    @options ||= {
-      entry_window: ENV.fetch('FEEDIGEST_ENTRY_WINDOW', 60 * 60 * 24), # Seconds
-      email_sender: ENV.fetch(
-        'FEEDIGEST_EMAIL_SENDER',
-        "feedigest@#{`hostname`.strip}"
-      ),
-      email_recipient: ENV.fetch('FEEDIGEST_EMAIL_RECIPIENT')
-    }
-  end
-
-  def self.smtp_options
-    @smtp_options ||= {
-      address: ENV.fetch('FEEDIGEST_SMTP_ADDRESS'),
-      port: ENV.fetch('FEEDIGEST_SMTP_PORT', '587').to_i,
-      user_name: ENV.fetch('FEEDIGEST_SMTP_USERNAME'),
-      password: ENV.fetch('FEEDIGEST_SMTP_PASSWORD'),
-      authentication: ENV.fetch('FEEDIGEST_SMTP_AUTH', 'plain'),
-      enable_starttls: ENV.fetch('FEEDIGEST_SMTP_STARTTLS', 'true') == 'true'
-    }
-  end
-end