automatic 14.12.2 → 26.09

data/doc/VERSIONS

@@ -0,0 +1,266 @@
+automaticruby Repository Version History
+========================================
+
+v26.09 (2026-09-23)
+-------------------
+- Add FilterKimi for the Kimi API, using its OpenAI-compatible Chat Completions
+  interface.
+- Add FilterLimit, FilterBatch, and FilterPresent for pipeline-wide limiting,
+  batching, and required-field filtering.
+- Add per-fetch interval handling to FilterFullFeed, FilterImageSource, and
+  FilterDescriptionLink.
+- Refresh the bundled LDRFullFeed siteinfo database from a newer upstream
+  snapshot.
+- Make doc/PLUGINS.md section 6 the plugin catalogue's source of truth.
+- Harden the framework's execution boundary: fix CLI contract drift, distinguish
+  dependency from load failures, and preflight-validate Recipe plugins.
+- Preserve standard item metadata when FeedMaker rebuilds pipelines so filters
+  keep source, enclosure, and full content.
+- Preserve nil-link pipeline items and remove SubscriptionText placeholder
+  title and link values.
+- Preserve SubscriptionText TSV column positions and ignore empty input rows.
+- Preserve local file paths when StoreFile hands file URIs to PublishAmazonS3.
+- Stop StoreFullText from passing items downstream when database writes fail.
+
+v26.08 (2026-08-22)
+-------------------
+- Support Ruby 3.3 through 4.0 and modernize the codebase for current Ruby and
+  maintained library APIs, validating representative versions in CI.
+- Harden Recipe loading with safe YAML parsing, structural validation and
+  framework-specific errors.
+- Restructure the CLI with help and version options, predictable error reporting
+  and documented exit statuses.
+- Report a clear CLI error when feed inspection discovers no feed instead of
+  raising an internal exception.
+- Reject HTTP and HTTPS URLs without a host before attempting a network request.
+- Verify TLS certificates when publishing to Instapaper instead of accepting an
+  unverified connection.
+- Modernize gem packaging: separate core from optional plugin dependencies so an
+  install can be minimal or extended one plugin at a time.
+- Classify every shipped plugin by its current support status rather than
+  simulating obsolete services in tests.
+- Modernize the bundled plugins for current Ruby, libraries and services, and
+  remove the integrations that are no longer viable.
+- Remove the plugins for services that no longer exist: Twitter, Pocket, HipChat,
+  Google Calendar, livedoor Weather, So-net G-Guide and Chan-Toru.
+- Add the plugin CustomFeedWeb, building one feed per HTML index page from its
+  article links, selected by CSS selector and filtered by host, pattern and count.
+- Add the plugin StoreDigest, which drops items by content rather than by URL, on
+  a SHA-256 digest of the item fields a Recipe names.
+- Add the plugin FilterJoin, which joins the whole pipeline into one item carrying
+  each item's title, link and body, and invents no permalink for it.
+- Add one AI filter plugin per service (FilterOpenAI, FilterClaude, FilterGemini,
+  FilterSakuraAI), replacing an item's description with that service's answer.
+- Rebuild the test and CI strategy for current RSpec and Ruby, with deterministic
+  isolation from user data and external services.
+- Add Markdown as the primary service-independent publication format, with a
+  documented and tested first-run workflow.
+- Rebuild the maintained documentation around current usage, architecture, policy,
+  plugins and deployment, and remove superseded historical documents.
+- Offer the project under either GPLv3 or LGPLv3, retaining third-party
+  attribution and shipping both licence texts.
+
+v14.12.2 (2015-01-04)
+---------------------
+- Pin every gem in the Gemfile to an exact minor version.
+
+v14.12.1 (2015-01-03)
+---------------------
+- Depend on activerecord 4.1.
+
+v14.12.0 (2014-12-27)
+---------------------
+- Rewrite obsolete methods.
+- Update Gemfile.
+
+v14.10.1 (2014-10-23)
+---------------------
+- Version-only release. No change beyond the version number is recorded for it.
+
+v14.10.0 (2014-10-23)
+---------------------
+- Use Logger for log output.
+- Store::FullText ignores an entry whose title is already stored.
+- Add the plugins Filter::Clear, Filter::GoogleNews and Filter::DescriptionLink.
+
+v14.5.0 (2014-05-31)
+--------------------
+- Update to activesupport and activerecord 4.1.
+
+v14.3.0 (2014-03-23)
+--------------------
+- Downgrade the nokogiri version for OS X.
+
+v14.2.0 (2014-02-26)
+--------------------
+- Add the plugin Publish::AmazonS3.
+- Rename Store::TargetLink to Store::File, and have it rewrite the link to a file
+  URI scheme.
+- Enable the file URI scheme in an RSS 2.0 feed, and let Store::File fetch a file
+  from Amazon S3.
+- Read text from TSV formatted files in Subscription::Text, and add the
+  description, author and comments fields to the RSS 2.0 feed.
+- Give the log messages more detail.
+- Use 'double' instead of 'mock' and 'stub' for rspec-mocks.
+- Improve CI with a network connection.
+- Divide Feed::Parser into Feed::Maker.
+
+v14.1.0 (2014-01-24)
+--------------------
+- Fix a dependency conflict.
+- Refactor the error retry handling.
+
+v13.12.0 (2013-12-17)
+---------------------
+- Fix the twitter gem version.
+
+v13.7.0 (2013-07-28)
+--------------------
+- Add the plugins Subscription::Xml and Provide::Fluentd.
+- Fix some problems.
+
+v13.6.0 (2013-06-28)
+--------------------
+- Add eleven plugins spanning Subscription, Filter and Publish, including Pocket,
+  GGuide, ChanToru, Accept, Sanitize, ConsoleLink, Hipchat, Memcached and Fluentd.
+- Correspond to activesupport and activerecord 4.0, temporarily removing the mail
+  plugins for actionmailer 4.0.
+- Ruby 1.8 support was outdated.
+- And other improvements and bug fixes.
+
+v13.5.0 (2013-05-18)
+--------------------
+- Add the plugins Subscription::Text, Subscription::Weather, Filter::One,
+  Filter::Rand, Publish::Twitter and Publish::Pocket.
+- Move the repository from id774/automaticruby to automaticruby/automaticruby.
+
+v13.4.1 (2013-04-10)
+--------------------
+- Merge the command automatic-config into automatic.
+- Improve the retry functions.
+- Run the tests on Ruby 2.0.
+
+v13.2.0 (2013-02-13)
+--------------------
+- Implement the retry function in all subscription and download plugins.
+- Plugin Filter::Ignore has become an item-based filter.
+- Rename the plugin Filter::Reverse to Filter::Sort.
+- Add the plugins Subscription::GoogleReaderStar, Filter::FullFeed and
+  Publish::Instapaper.
+- Improve the tests.
+- And other minor bug fixes.
+
+v13.1.0 (2013-01-08)
+--------------------
+- Fix the interval bugs.
+- Improve the timing of logging.
+- Return only a new feed.
+- Refactoring.
+
+v12.10.0 (2012-10-16)
+---------------------
+- Add the plugin Subscription::Tumblr.
+
+v12.9.1 (2012-09-18)
+--------------------
+- Consider the case of a feed link including nil.
+
+v12.9.0 (2012-09-18)
+--------------------
+- Unify the format of the pipeline to a feed object, an array of feeds; some
+  plugins lost backward compatibility.
+- Add the plugins Subscription::Link, Subscription::Twitter, Filter::Image and
+  Filter::AbsoluteURI.
+- Remove the obsolete plugins Subscription::URI, Extract::Link, Filter::ImageLink,
+  Store::Link and Store::Target.
+
+v12.6.0 (2012-06-18)
+--------------------
+- Add the plugins Subscription::URI, Extract::Link, Filter::ImageLink, Store::Link
+  and Store::Target.
+- Make scaffold create the config and db directories, and add the unscaffold
+  subcommand to automatic-config.
+- Search the user directory's config when given -c with a simple filename, and use
+  the user's db directory when it exists.
+- Show the recipe and db name in the log.
+- Refactor the plug-in names and the header fields.
+- Add tests, taking RSpec code coverage over 99%, and add the all option to
+  script/build.
+
+v12.4.0 (2012-04-30)
+--------------------
+- Add the inspect subcommand to automatic-config.
+- Add the plugins Filter::Reverse, Publish::SMTP and Publish::Mail.
+- Run all integration tests with CI.
+
+v12.3.1 (2012-03-16)
+--------------------
+- Write the English documents doc/README and doc/PLUGINS.
+- Let a user who installed the gem package create their own plugins, under
+  ~/.automatic/plugins.
+- Merge the utilities into the automatic-config command, adding the subcommands
+  scaffold, autodiscovery, opmlparser, feedparser and log.
+- Add the integration test script script/build for CI.
+
+v12.03 (2012-03-07)
+-------------------
+- Distribute as a gem library.
+- Add the plugins CustomFeed::SVNLog, Notify::Ikachan, Filter::TumblrResize,
+  Filter::Image and Store::TargetLink.
+- Add the CI and build script script/bootstrap.
+- Significant refactoring.
+
+v12.02.1 (2012-02-25)
+---------------------
+- Point release: a large refactoring of the core, a reorganization of the
+  namespace, and a few plugins added.
+
+v12.02 (2012-02-18)
+-------------------
+- Initial release.
+
+
+Notes on this record
+====================
+- Entries from v12.02 to v14.12.0 were moved here from doc/ChangeLog, unchanged in
+  date or content, one bullet per change.
+- v14.10.1, v14.12.1 and v14.12.2 were released and recorded in the commit log but
+  never in doc/ChangeLog; their entries above are reconstructed from it.
+- No Git tag exists for any release before v26.08; the version numbers above are
+  those recorded in the VERSION file at the time.
+- Between v14.12.2 and v26.08 the only committed change was adding
+  test/integration/test_pdf2local.yml (April 2015), folded into v26.08.
+
+
+Version History Guidelines
+==========================
+- Record externally meaningful or architecturally significant outcomes, not a
+  diary of implementation work.
+- Omit development-time corrections and intermediate states that were reverted
+  or superseded before release; Git history retains that detail.
+- Summarize related low-level changes under their substantive outcome.
+- Write one coherent change per bullet, at most two physical lines.
+- A single line at or under 80 columns is preferred whenever practical.
+  Necessary file names, module names, plugin names, option names, and
+  configuration names may pass 80 columns on the bullet's one or two lines
+  when they cannot be shortened without losing meaning.
+- When a version history already exists, balance a new entry against the length
+  of the existing lines, so that the whole file stays consistent.
+- When an entry becomes substantially longer, omit or abstract implementation
+  details, examples, reasons, and secondary effects where possible.
+- Preserve the changed target, externally observable behavior, compatibility
+  or security impact, and important option or configuration names.
+- Combine related changes within the same version, especially changes to the
+  same file or feature.
+- Place closely related entries near each other.
+- Append an independent new change to the end of the current version entry.
+- Never create more than one repository version for the same calendar date.
+- Independent changes may use separate bullets, but they stay in that day's
+  single version entry.
+- Past entries and their dates are a record. Do not rewrite, renumber or merge
+  them. Past releases that share a date stay recorded as they were made; they
+  do not conform to the rule above and are not precedent for new releases.
+- This file records the history of the repository. A change that is not
+  observable from outside does not need an entry here.
+- The first entry, at the lowest version this file reaches, reads only
+  "Initial release." and nothing else.

data/lib/automatic/cli.rb

@@ -0,0 +1,270 @@
+# -*- coding: utf-8 -*-
+# Name::        Automatic::CLI
+# Author:       id774 (More info: https://id774.net)
+# Source Code:: https://github.com/id774/automaticruby
+# License::     The GPL version 3, or LGPL version 3 (Dual License).
+# Contact::     idnanashi@gmail.com
+# Created::     Aug 14, 2026
+# Updated::     Sep  9, 2026
+# Copyright::   Copyright (c) 2012-2026 Automatic Ruby Developers.
+#
+# Everything that belongs to being a command: option parsing, the subcommands,
+# and turning a failure into a message and an exit status.
+#
+# CLI.run returns an Integer and never calls exit. Deciding the process's fate
+# is bin/automatic's job, and returning the status is what makes the command
+# line testable without spawning a process. See doc/BASIC_DESIGN.md section 4.2.
+
+require 'automatic'
+require 'fileutils'
+require 'optparse'
+
+module Automatic
+  class CLI
+    EXIT_SUCCESS = 0
+    EXIT_FAILURE = 1
+    EXIT_USAGE   = 2
+
+    # The installation root: the directory holding plugins/, config/ and db/.
+    DEFAULT_ROOT_DIR = File.expand_path('../..', __dir__)
+
+    def self.run(argv, **options)
+      new(**options).run(argv)
+    end
+
+    def initialize(root_dir: DEFAULT_ROOT_DIR, stdout: $stdout, stderr: $stderr)
+      @root_dir = root_dir
+      @stdout   = stdout
+      @stderr   = stderr
+    end
+
+    # Returns the process exit status. --help and --version, and a subcommand
+    # invoked without its argument, finish early by throwing :automatic_exit.
+    def run(argv)
+      catch(:automatic_exit) do
+        argv = argv.dup
+        recipe_path = nil
+
+        parser = build_parser { |path| recipe_path = path }
+
+        begin
+          parser.order!(argv)
+        rescue OptionParser::ParseError => e
+          @stderr.puts e.message
+          @stderr.puts parser.help
+          throw :automatic_exit, EXIT_USAGE
+        end
+
+        next run_recipe(recipe_path) unless recipe_path.to_s.empty?
+        next usage(parser) if argv.empty?
+
+        run_subcommand(argv)
+      end
+    end
+
+    private
+
+    SUBCOMMAND_USAGE = {
+      'scaffold'      => '',
+      'unscaffold'    => '',
+      'autodiscovery' => '<url>',
+      'feedparser'    => '<url>',
+      'inspect'       => '<url>',
+      'opmlparser'    => '<opml path>',
+      'log'           => '<level> <message>'
+    }.freeze
+
+    def build_parser
+      OptionParser.new do |parser|
+        parser.version = Automatic::VERSION
+        parser.banner = "Automatic Ruby #{Automatic::VERSION}\n" \
+                        "    Usage: automatic [options] [subcommand]"
+        parser.separator 'SubCommands:'
+        SUBCOMMAND_USAGE.each do |name, arguments|
+          parser.separator "    #{name} #{arguments}".rstrip
+        end
+        parser.separator 'Options:'
+        parser.on('-c', '--config FILE', String, 'recipe YAML file') do |path|
+          yield path
+        end
+        parser.on('-h', '--help', 'show this message') do
+          @stdout.puts parser.help
+          throw :automatic_exit, EXIT_SUCCESS
+        end
+        parser.on('-v', '--version', 'show the version') do
+          @stdout.puts Automatic::VERSION
+          throw :automatic_exit, EXIT_SUCCESS
+        end
+      end
+    end
+
+    def usage(parser)
+      @stdout.puts parser.help
+      EXIT_FAILURE
+    end
+
+    # Automatic::OptionalDependencyError is one of the failures reported as a
+    # message here: a Recipe naming a plugin whose optional gem is not
+    # installed is an operator's mistake with an answer, and the answer is in
+    # the message rather than in a backtrace. A LoadError raised from inside a
+    # plugin's own load -- for a different, unrelated missing file -- is not
+    # this, and is left alone along with everything else the plugin itself
+    # raises; see doc/PLUGINS.md section 2.7.
+    def run_recipe(path)
+      unless File.exist?(resolve_recipe(path))
+        @stderr.puts "automatic: no such recipe: #{path}"
+        return EXIT_FAILURE
+      end
+
+      Automatic.run(recipe: Automatic::Recipe.new(path), root_dir: @root_dir)
+      EXIT_SUCCESS
+    rescue Automatic::Error, Psych::Exception, SystemCallError, IOError,
+           Automatic::OptionalDependencyError => e
+      @stderr.puts "automatic: #{e.message}"
+      EXIT_FAILURE
+    end
+
+    # Mirrors Automatic::Recipe's own resolution, so that a missing file is
+    # reported as a message rather than as a backtrace.
+    def resolve_recipe(path)
+      in_user_dir = File.join(Automatic.user_config_dir, path.to_s)
+      File.exist?(in_user_dir) ? in_user_dir : path.to_s
+    end
+
+    def run_subcommand(argv)
+      name = argv.shift
+      handler = subcommands[name]
+
+      unless handler
+        @stderr.puts "automatic: no such subcommand: #{name}"
+        return EXIT_FAILURE
+      end
+
+      handler.call(argv)
+      EXIT_SUCCESS
+    rescue Automatic::Error, SystemCallError, IOError, Automatic::OptionalDependencyError => e
+      @stderr.puts "automatic: #{e.message}"
+      EXIT_FAILURE
+    end
+
+    def missing_argument(name)
+      @stderr.puts "Usage: automatic #{name} #{SUBCOMMAND_USAGE[name]}"
+      throw :automatic_exit, EXIT_FAILURE
+    end
+
+    # Each subcommand requires what it needs, so that `automatic --version`
+    # loads neither a feed parser nor an OPML parser.
+    #
+    # `feedbag` is an optional dependency: two subcommands out of seven use it,
+    # and neither running a Recipe nor any other subcommand does. It is
+    # required through Automatic.require_optional, so that not having it is a
+    # sentence rather than a backtrace. See doc/POLICY.md section 9.1.
+    def subcommands
+      {
+        'scaffold'      => method(:scaffold),
+        'unscaffold'    => method(:unscaffold),
+        'autodiscovery' => method(:autodiscovery),
+        'feedparser'    => method(:feedparser),
+        'inspect'       => method(:inspect_url),
+        'opmlparser'    => method(:opmlparser),
+        'log'           => method(:log)
+      }
+    end
+
+    def scaffold(_argv)
+      Dir.children(File.join(@root_dir, 'plugins')).sort.each do |category|
+        next unless File.directory?(File.join(@root_dir, 'plugins', category))
+
+        create_dir(File.join(Automatic.user_dir, 'plugins', category))
+      end
+
+      create_dir(File.join(Automatic.user_dir, 'db'))
+
+      assets = File.join(Automatic.user_dir, 'assets')
+      create_dir(assets)
+      copy_bundled(File.join(@root_dir, 'assets', 'siteinfo'), File.join(assets, 'siteinfo'))
+
+      config = Automatic.user_config_dir
+      create_dir(config)
+      copy_bundled(File.join(@root_dir, 'config'), File.join(config, 'example'))
+    end
+
+    def unscaffold(_argv)
+      dir = Automatic.user_dir
+      return unless File.directory?(dir)
+
+      @stdout.puts "Removing #{dir}"
+      FileUtils.rm_r(dir)
+    end
+
+    def autodiscovery(argv)
+      Automatic.require_optional('feedbag', needed_by: 'the autodiscovery subcommand')
+      require 'pp'
+      url = argv.shift || missing_argument('autodiscovery')
+      @stdout.puts Feedbag.find(url).pretty_inspect
+    end
+
+    def feedparser(argv)
+      require 'automatic/feed_parser'
+      require 'pp'
+      url = argv.shift || missing_argument('feedparser')
+      validate_fetch_url(url)
+      @stdout.puts Automatic::FeedParser.get_url(url).pretty_inspect
+    end
+
+    # Converts only the URL validation failure `Automatic::Http.uri` already
+    # applies into the CLI's ordinary failure path; an unexpected error from
+    # inside FeedParser itself is left to propagate.
+    def validate_fetch_url(url)
+      Automatic::Http.uri(url)
+    rescue ArgumentError, URI::InvalidURIError => e
+      raise Automatic::Error, e.message
+    end
+
+    def inspect_url(argv)
+      require 'automatic/feed_parser'
+      Automatic.require_optional('feedbag', needed_by: 'the inspect subcommand')
+      require 'pp'
+      url = argv.shift || missing_argument('inspect')
+      feeds = Feedbag.find(url)
+      raise Automatic::Error, "no feed found at #{url}" if feeds.empty?
+
+      @stdout.puts feeds.pretty_inspect
+      @stdout.puts Automatic::FeedParser.get_url(feeds.first).pretty_inspect
+    end
+
+    def opmlparser(argv)
+      require 'automatic/opml'
+      path = argv.shift || missing_argument('opmlparser')
+      parser = Automatic::OPML::Parser.new(File.read(path))
+      parser.each_outline { |_opml, outline| @stdout.puts outline.xmlUrl }
+    end
+
+    def log(argv)
+      level = argv.shift || missing_argument('log')
+      message = argv.shift || missing_argument('log')
+      Automatic::Log.puts(level, message)
+    end
+
+    def create_dir(path)
+      return false if File.exist?(path)
+
+      FileUtils.mkdir_p(path)
+      @stdout.puts "Creating #{path}"
+      true
+    end
+
+    # Bundled initial data (siteinfo, example config) is copied whenever its
+    # own destination is missing, independent of whether the parent user
+    # directory already existed. That is what lets a re-run of scaffold heal a
+    # partially-scaffolded user directory instead of only ever populating a
+    # brand new one. See doc/DEPLOYMENT.md and doc/POLICY.md on scaffold only
+    # adding what is missing.
+    def copy_bundled(src, dest)
+      return if File.exist?(dest)
+
+      FileUtils.cp_r(src, dest)
+      @stdout.puts "Creating #{dest}"
+    end
+  end
+end

data/lib/automatic/environment.rb

@@ -1,8 +1,34 @@
 # -*- coding: utf-8 -*-
-require 'rubygems'
+# Name::        Automatic::Environment
+# Author:       id774 (More info: https://id774.net)
+# Source Code:: https://github.com/id774/automaticruby
+# License::     The GPL version 3, or LGPL version 3 (Dual License).
+# Contact::     idnanashi@gmail.com
+# Created::     Feb 18, 2012
+# Updated::     Aug 14, 2026
+# Copyright::   Copyright (c) 2012-2026 Automatic Ruby Developers.
+#
+# Bundler setup for a source checkout, and nothing more.
+#
+# When this repository is used from a checkout, bin/automatic should resolve
+# the gems the checkout's Gemfile locks. When the gem is installed normally
+# there is no Gemfile beside lib/, this file does nothing, and that is correct:
+# an installed library must not impose a bundle on the program requiring it.
+#
+# This deliberately does not call Bundler.require. Loading every gem in the
+# bundle is how a plugin's dependency used to become everyone's; see
+# doc/POLICY.md section 2.5.
 
-# Set up gems listed in the Gemfile.
-ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../Gemfile', __FILE__)
+gemfile = File.expand_path('../../Gemfile', __dir__)
 
-require 'bundler/setup' if File.exists?(ENV['BUNDLE_GEMFILE'])
-Bundler.require :default if defined?(Bundler)
+if File.exist?(gemfile)
+  ENV['BUNDLE_GEMFILE'] ||= gemfile
+  begin
+    require 'bundler/setup'
+  rescue LoadError, StandardError
+    # Bundler is absent, or the bundle is not installed. Fall back to whatever
+    # RubyGems can activate; a genuinely missing dependency will surface as a
+    # LoadError naming it, which is the more useful message anyway.
+    nil
+  end
+end