newsfetcher 0.84

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: bb18377fe7a1b55c9a1f9778b6824294ab8c65cad6f8260d388b61629247de48
+  data.tar.gz: a0648cec658c5299075ac08a3e5524a766d14aac20f0b8efd37603a607e4be93
+SHA512:
+  metadata.gz: 636aad38f91c50934a524f7862143bc0b31b7b544b5a2f82686dcf51ce6ac3a91d361e407fb55190b7c13465e396c241bfc550b1f660e13176644c53eb4f8f3f
+  data.tar.gz: 3b5710a57847e4653c822b57242437df7d21576bcedc39d5ba1eb7b91df605e9e6590ff23434fb4121dc1f904ba155085c24e78bf2142a20f6a54387260e5300

data/.gitignore

@@ -0,0 +1,11 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+
+test/tmp

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2025 John Labovitz
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,123 @@
+# NewsFetcher
+
+NewsFetcher monitors RSS and Atom feeds and delivers new items as individual HTML emails. It tracks which feed items have already been delivered, so you only receive new content.
+
+
+## Features
+
+- Monitors multiple RSS/Atom feeds organized into subscriptions
+- Delivers each new feed item as a separate HTML email with custom styling
+- Tracks item history to avoid re-delivering the same content
+- Supports multiple delivery methods (SMTP, Maildir, etc.)
+- Configurable update intervals and age limits per subscription
+- Multi-threaded updating for efficiency
+- Feed discovery from web pages
+- Per-subscription and global configuration
+
+
+## Installation
+
+```
+gem install newsfetcher
+```
+
+
+## Quick Start
+
+Initialize a profile with your email settings:
+
+```
+newsfetcher init --mail-from feeds@example.com --mail-to you@example.com
+```
+
+Discover a feed for a website:
+
+```
+newsfetcher discover https://example.com/
+```
+
+Add a feed subscription:
+
+```
+newsfetcher add https://example.com/feed.xml
+```
+
+Update all subscriptions and deliver new items:
+
+```
+newsfetcher update
+```
+
+
+## Usage
+
+NewsFetcher organizes feeds into subscriptions stored in `~/.newsfetcher` by default. Each subscription has its own configuration and tracks which items have been delivered.
+
+### Commands
+
+- `init` — Initialize a new profile with email settings
+- `add URI [PATH] [ID]` — Add a new feed subscription
+- `update [IDs...]` — Update subscriptions and deliver new items
+- `show [IDs...]` — Display subscription details
+- `enable ID` — Enable a disabled subscription
+- `disable ID` — Disable a subscription
+- `remove ID` — Remove a subscription
+- `reset ID` — Clear item history (next update will re-deliver all items)
+- `edit ID` — Edit subscription configuration
+- `dir` — Show the profile directory path
+- `discover URI` — Find feeds in a web page
+- `get URI` — Fetch and display a feed without saving
+- `uniq` — Remove duplicate items from history
+- `fix` — Fix subscription issues
+
+### Configuration
+
+Configuration lives in JSON files at the profile and subscription levels. Profile-level settings in `~/.newsfetcher/config.json` serve as defaults for all subscriptions.
+
+Key configuration options:
+
+- `mail_from`, `mail_to` — Email sender and recipient
+- `mail_subject` — Subject line template (supports ERB)
+- `delivery_method` — How to deliver mail (`:smtp`, `:maildir`, etc.)
+- `delivery_params` — Parameters for the delivery method
+- `update_interval` — Minimum time between updates (default: 1 hour)
+- `max_age` — How long to track items (default: 30 days)
+- `max_threads` — Number of concurrent subscription updates (default: 100)
+- `disabled` — Whether to skip this subscription during updates
+- `ignore_uris` — Array of regex patterns to ignore items
+- `root_folder` — Prefix for maildir folders or mail subject
+- `consolidate` — Use shorter folder names
+
+Subscription-specific settings override profile defaults.
+
+### Styling
+
+Email messages use HTML with embedded CSS compiled from SCSS. The default stylesheet is included, but you can specify additional stylesheets:
+
+```json
+{
+  "aux_stylesheets": ["~/.newsfetcher/custom.scss"]
+}
+```
+
+
+## Automation
+
+Run newsfetcher update periodically using cron, launchd, or your preferred scheduler to automatically check feeds and deliver new items.
+
+
+## How It Works
+
+NewsFetcher fetches each feed, parses it with Feedjira, and compares items against the subscription’s history. New items (not previously seen and within the configured age limit) are formatted as HTML emails and delivered using Ruby’s Mail gem.
+
+Each subscription maintains two history files: one tracking delivered items, one tracking HTTP responses. These histories are automatically pruned based on the `max_age` setting.
+
+
+## Requirements
+
+Ruby 2.7 or later.
+
+
+## License
+
+MIT

data/Rakefile

@@ -0,0 +1 @@
+require 'simple-rake-tasks'

data/TODO.md

@@ -0,0 +1,67 @@
+# TODO
+
+## BUGS
+
+
+## IMPROVEMENTS
+
+- add 'audit' command
+  - interactively modify subscriptions
+    - show activity, dormancy, errors
+    - enable/disable, delete, rename
+
+- keep original web address in config
+  - to solve problems when feed disappears
+
+- implement retry on error in fetching/parsing
+  - eg, to handle timeouts, connection/read errors, bad data, etc.
+  - only show error if > retry count
+
+- add new Feed class to encapsulate title/items
+
+- save last fetch response to file (Marshaled?)
+
+- rename 'path' to 'section'?
+
+- add per-subscription locks to avoid access by multiple processes/threads
+
+- convert HTML document builder to ERB, for easier customization
+
+
+## FEATURES
+
+- allow unit suffixes on durations (s, m, h, d, w)
+
+- allow 'add' to take 'id' option to customize ID
+
+- add 'remove' feature to remove HTML element
+  - specifies XPath expression to remove
+
+- auto-discover on 'add'
+  - or auto-add on discover?
+
+- allow update by section (eg, world)
+
+- add 'config' command
+  - '--root' specifies root config
+  - use readline to show/edit values
+
+- expand 'ignore' feature
+  - each rule can match on any fields
+    ignore:
+      uri: /foo
+      title: Bar
+
+- add check/validate command
+  - fetch HTML page for feed
+  - verify that feed matches <link> element
+
+
+## RELEASE
+
+- write README documentation
+- bump version to 1.0
+- reset git history
+- push to Github
+- release to Rubygems
+- release publicly

data/bin/newsfetcher

@@ -0,0 +1,10 @@
+#!/usr/bin/env ruby
+
+require 'newsfetcher'
+
+begin
+  Simple::CommandParser.run(ARGV)
+rescue Simple::CommandParser::UsageError => error
+  warn error
+  exit(1)
+end

data/lib/newsfetcher/command.rb

@@ -0,0 +1,20 @@
+module NewsFetcher
+
+  class Command < Simple::CommandParser::Command
+
+    attr_accessor :dir
+    attr_accessor :log_level
+    attr_accessor :max_threads
+
+    def run(args)
+      dir = Path.new(@dir || DefaultProfileDir)
+      config_file = dir / ConfigFileName
+      config = config_file.exist? ? BaseConfig.load(config_file) : BaseConfig.make
+      config.log_level = @log_level.downcase.to_sym if @log_level
+      config.max_threads = @max_threads&.to_i if @max_threads
+      @profile = Profile.new(dir: dir, config: config) or raise Error, "Profile not loaded"
+    end
+
+  end
+
+end

data/lib/newsfetcher/commands/add.rb

@@ -0,0 +1,19 @@
+module NewsFetcher
+
+  module Commands
+
+    class Add < Command
+
+      def run(args)
+        super
+        uri, path, id = *args
+        raise Error, "No URI specified" unless uri
+        dir = @profile.add_subscription(uri: uri, id: id, path: path)
+        warn "Added subscription: #{dir}"
+      end
+
+    end
+
+  end
+
+end

data/lib/newsfetcher/commands/dir.rb

@@ -0,0 +1,18 @@
+module NewsFetcher
+
+  module Commands
+
+    class Dir < Command
+
+      def run(args)
+        super
+        @profile.find_subscriptions(ids: args).each do |subscription|
+          puts subscription.dir
+        end
+      end
+
+    end
+
+  end
+
+end

data/lib/newsfetcher/commands/disable.rb

@@ -0,0 +1,18 @@
+module NewsFetcher
+
+  module Commands
+
+    class Disable < Command
+
+      def run(args)
+        super
+        @profile.find_subscriptions(ids: args).each do |subscription|
+          subscription.disable
+        end
+      end
+
+    end
+
+  end
+
+end

data/lib/newsfetcher/commands/discover.rb

@@ -0,0 +1,28 @@
+module NewsFetcher
+
+  module Commands
+
+    class Discover < Command
+
+      def run(args)
+        super
+        args.map { |a| Addressable::URI.parse(a) }.each do |uri|
+          fetcher = Fetcher.get(uri)
+          if fetcher.success?
+            fetcher.find_feeds.each do |feed|
+              feed.each do |key, value|
+                puts "%10s: %s" % [key, value]
+              end
+              puts
+            end
+          else
+            warn "#{uri}: HTTP error #{fetcher.response_status} (#{fetcher.response_reason})"
+          end
+        end
+      end
+
+    end
+
+  end
+
+end

data/lib/newsfetcher/commands/edit.rb

@@ -0,0 +1,19 @@
+module NewsFetcher
+
+  module Commands
+
+    class Edit < Command
+
+      def run(args)
+        super
+        editor = ENV['EDITOR'] or raise Error, "No editor defined in $EDITOR"
+        @profile.find_subscriptions(ids: args).each do |subscription|
+          system(editor, (subscription.dir / ConfigFileName).to_s)
+        end
+      end
+
+    end
+
+  end
+
+end

data/lib/newsfetcher/commands/enable.rb

@@ -0,0 +1,18 @@
+module NewsFetcher
+
+  module Commands
+
+    class Enable < Command
+
+      def run(args)
+        super
+        @profile.find_subscriptions(ids: args).each do |subscription|
+          subscription.enable
+        end
+      end
+
+    end
+
+  end
+
+end

data/lib/newsfetcher/commands/fix.rb

@@ -0,0 +1,18 @@
+module NewsFetcher
+
+  module Commands
+
+    class Fix < Command
+
+      def run(args)
+        super
+        @profile.find_subscriptions(ids: args).each do |subscription|
+          subscription.fix
+        end
+      end
+
+    end
+
+  end
+
+end

data/lib/newsfetcher/commands/get.rb

@@ -0,0 +1,31 @@
+module NewsFetcher
+
+  module Commands
+
+    class Get < Command
+
+      def run(args)
+        super
+        args.each do |uri|
+          fetcher = Fetcher.get(uri)
+          if fetcher.success?
+            feed = fetcher.parse_feed
+            puts
+            puts "URI: #{uri}"
+            puts "Title: #{feed[:title]}"
+            puts "Items:"
+            feed[:items].each do |item|
+              item.print
+              puts
+            end
+          else
+            warn "#{uri}: HTTP error #{fetcher.response_status} (#{fetcher.response_reason})"
+          end
+        end
+      end
+
+    end
+
+  end
+
+end

data/lib/newsfetcher/commands/init.rb

@@ -0,0 +1,28 @@
+module NewsFetcher
+
+  module Commands
+
+    class Init < Command
+
+      attr_accessor :mail_from
+      attr_accessor :mail_to
+
+      def run(args)
+        super
+        raise Error, "Must specify mail_from" unless @mail_from
+        raise Error, "Must specify mail_to" unless @mail_to
+        @profile = Profile.new(
+          dir: @dir,
+          config: BaseConfig.make(
+            mail_from: @mail_from,
+            mail_to: @mail_to,
+          ),
+        )
+        profile.save
+      end
+
+    end
+
+  end
+
+end

data/lib/newsfetcher/commands/remove.rb

@@ -0,0 +1,19 @@
+module NewsFetcher
+
+  module Commands
+
+    class Remove < Command
+
+      def run(args)
+        super
+        @profile.find_subscriptions(ids: args).each do |subscription|
+          dir = subscription.dir or raise Error, "dir not set"
+          dir.rmtree
+        end
+      end
+
+    end
+
+  end
+
+end

data/lib/newsfetcher/commands/reset.rb

@@ -0,0 +1,16 @@
+module NewsFetcher
+
+  module Commands
+
+    class Reset < Command
+
+      def run(args)
+        super
+        @profile.find_subscriptions(ids: args).each(&:reset)
+      end
+
+    end
+
+  end
+
+end

data/lib/newsfetcher/commands/show.rb

@@ -0,0 +1,26 @@
+module NewsFetcher
+
+  module Commands
+
+    class Show < Command
+
+      attr_accessor :status
+      attr_accessor :sort
+
+      def run(args)
+        super
+        @profile.find_subscriptions(
+          ids: args,
+          status: @status&.split(',')&.map(&:to_sym),
+          sort: @sort&.to_sym,
+        ).each do |subscription|
+          subscription.print
+          puts
+        end
+      end
+
+    end
+
+  end
+
+end

data/lib/newsfetcher/commands/update.rb

@@ -0,0 +1,39 @@
+module NewsFetcher
+
+  module Commands
+
+    class Update < Command
+
+      def run(args)
+        super
+        subscriptions = @profile.find_subscriptions(ids: args).reject(&:disabled?)
+        if @profile.config.max_threads == 0
+          subscriptions.each(&:update)
+        else
+          run_threads(subscriptions, &:update)
+        end
+      end
+
+      def run_threads(objects, &block)
+        threads = []
+        objects.each do |object|
+          if threads.length >= @profile.config.max_threads
+            $logger.debug { "Waiting for #{threads.length} threads to finish" }
+            threads.map(&:join)
+            threads = []
+          end
+          threads << Thread.new do
+            $logger.debug { "Started thread for #{object.id}" }
+            yield(object)
+          end
+        end
+        unless threads.empty?
+          $logger.debug { "Waiting for last #{threads.length} threads to finish" }
+          threads.map(&:join)
+        end
+      end
+    end
+
+  end
+
+end

data/lib/newsfetcher/error.rb

@@ -0,0 +1,6 @@
+module NewsFetcher
+
+  class Error < StandardError
+  end
+
+end

data/lib/newsfetcher/extensions/addressable-uri.rb

@@ -0,0 +1,24 @@
+module Addressable
+
+  class URI
+
+    def make_subscription_id
+      [
+        host.to_s \
+          .sub(/^(www|ssl|en|feeds|rss|blogs?|news).*?\./i, '') \
+          .sub(/\.(com|org|net|info|edu|co\.uk|wordpress\.com|blogspot\.com|feedburner\.com)$/i, ''),
+        path.to_s \
+          .gsub(/\b(\.?feeds?|index|atom|rss|rss2|xml|rdf|php|blog|posts|default)\b/i, ''),
+        query.to_s \
+          .gsub(/\b(format|feed|type|q)=(atom|rss\.xml|rss2?|xml)/i, ''),
+      ] \
+        .join(' ')
+        .downcase
+        .gsub(/[^a-z0-9]+/, ' ')  # non-alphanumeric
+        .strip
+        .gsub(/\s+/, '-')
+    end
+
+  end
+
+end

data/lib/newsfetcher/extensions/mail.rb

@@ -0,0 +1,22 @@
+require 'maildir'
+
+module Mail
+
+  class Maildir
+
+    attr_accessor :settings
+
+    def initialize(values)
+      self.settings = values
+    end
+
+    def deliver!(mail)
+      dir = settings[:dir] or raise Error, "'dir' not found in settings"
+      maildir = ::Maildir.new(dir)
+      maildir.serializer = ::Maildir::Serializer::Mail.new
+      maildir.add(mail)
+    end
+
+  end
+
+end

data/lib/newsfetcher/extensions/string.rb

@@ -0,0 +1,15 @@
+class String
+
+  def html?
+    self =~ /(<[a-z]+)|(\&\S+;)/i
+  end
+
+  def to_html
+    if html?
+      self
+    else
+      RubyPants.new(self).to_html
+    end
+  end
+
+end