source_monitor 0.3.3 → 0.4.0

data/docs/setup.md

@@ -49,7 +49,12 @@ This ensures Bundler can load SourceMonitor so the commands below are available.
    ```bash
    bin/rails solid_queue:start
    ```
-   Recurring jobs (fetch scheduling, scraping, cleanup) are automatically configured in `config/recurring.yml` by the install generator. They'll run automatically with `bin/dev` or `bin/jobs`.
+   The install generator automatically handles all worker configuration:
+   - **Recurring jobs** are configured in `config/recurring.yml` (fetch scheduling, scraping, cleanup).
+   - **Procfile.dev** is patched with a `jobs:` entry so `bin/dev` starts Solid Queue alongside the web server.
+   - **Queue dispatcher** is patched with `recurring_schedule: config/recurring.yml` in `config/queue.yml` so recurring jobs load on startup.
+
+   All three steps are idempotent. If any configuration is missing, re-run: `bin/rails generate source_monitor:install`
 
 4. **Visit the dashboard** at the chosen mount path, create a source, and trigger “Fetch Now” to validate realtime updates and Solid Queue processing.
 
@@ -87,6 +92,8 @@ Prefer to script each step or plug SourceMonitor into an existing deployment che
 | 4 | `bin/rails railties:install:migrations FROM=source_monitor` | Copy engine migrations (idempotent) |
 | 5 | `bin/rails db:migrate` | Apply schema updates, including Solid Queue tables |
 | 6 | `bin/rails solid_queue:start` | Ensure jobs process via Solid Queue |
+| 6a | Handled by generator (patches `Procfile.dev`) | Ensure `bin/dev` starts Solid Queue workers |
+| 6b | Handled by generator (patches `config/queue.yml`) | Wire recurring jobs into Solid Queue dispatcher |
 | 7 | `bin/jobs --recurring_schedule_file=config/recurring.yml` | Start recurring scheduler (optional but recommended) |
 | 8 | `bin/source_monitor verify` | Confirm Solid Queue/Action Cable readiness and emit telemetry |
 
@@ -100,6 +107,8 @@ Prefer to script each step or plug SourceMonitor into an existing deployment che
 4. **Apply database changes** using `bin/rails db:migrate`. If your host already installed Solid Queue migrations manually, delete duplicate files before migrating.
 5. **Wire Action Cable** if necessary. SourceMonitor defaults to Solid Cable; confirm `ApplicationCable::Connection`/`Channel` exist and that `config/initializers/source_monitor.rb` uses the adapter you expect. To switch to Redis, set `config.realtime.adapter = :redis` and `config.realtime.redis_url`.
 6. **Start workers** with `bin/rails solid_queue:start` (or your process manager). The install generator automatically configures recurring jobs in `config/recurring.yml` for fetch scheduling, scraping, and cleanup. They'll run with `bin/dev` or `bin/jobs`.
+   - **Procfile.dev:** The generator automatically patches `Procfile.dev` with a `jobs:` entry for Solid Queue. Verify the file contains `jobs: bundle exec rake solid_queue:start` after running the generator.
+   - **Recurring schedule:** The generator automatically patches `config/queue.yml` dispatchers with `recurring_schedule: config/recurring.yml`. Verify the key is present after running the generator.
 7. **Review the initializer** and tune queue names, HTTP timeouts, scraping adapters, retention limits, authentication hooks, and Mission Control integration. The [configuration reference](configuration.md) details every option.
 8. **Verify the install**: run `bin/source_monitor verify` to ensure Solid Queue workers and Action Cable are healthy, then visit the mount path to trigger a fetch manually. Enable telemetry if you want JSON logs recorded for support.
 

data/docs/troubleshooting.md

@@ -20,45 +20,76 @@ This guide lists common issues you might encounter while installing, upgrading,
 - Ensure at least one Solid Queue worker is running; the dashboard reads visibility data via `SourceMonitor::Jobs::Visibility`.
 - When using mission control integration, keep `config.mission_control_dashboard_path` pointing at a valid route helper; otherwise the dashboard hides the link.
 
-## 4. Realtime Updates Do Not Stream
+## 4. Recurring Jobs Not Running
+
+- **Symptoms:** Fetch scheduling, scrape scheduling, and cleanup jobs never fire. Sources never auto-fetch on their configured intervals.
+- **Primary fix:** Re-run the install generator, which automatically patches the dispatcher config:
+  ```bash
+  bin/rails generate source_monitor:install
+  ```
+- **Diagnostics:** Run `bin/source_monitor verify` to check recurring task registration. The RecurringScheduleVerifier will report whether SourceMonitor recurring tasks are loaded into Solid Queue.
+- **Manual check:** Verify `config/queue.yml` includes `recurring_schedule: config/recurring.yml` under the `dispatchers:` section. Without this key, Solid Queue's dispatcher will not load the recurring schedule even though `config/recurring.yml` exists.
+- **Manual fix (if generator cannot patch):**
+  ```yaml
+  dispatchers:
+    - polling_interval: 1
+      batch_size: 500
+      recurring_schedule: config/recurring.yml
+  ```
+
+## 5. Jobs Not Processing with bin/dev
+
+- **Symptoms:** `bin/dev` starts the web server but jobs never run. Running `bin/rails solid_queue:start` manually works fine.
+- **Primary fix:** Re-run the install generator, which automatically patches `Procfile.dev`:
+  ```bash
+  bin/rails generate source_monitor:install
+  ```
+- **Diagnostics:** Run `bin/source_monitor verify` to check Solid Queue worker status. The SolidQueueVerifier will suggest Procfile.dev if no workers are detected.
+- **Manual check:** Verify `Procfile.dev` includes a `jobs:` line:
+  ```
+  jobs: bundle exec rake solid_queue:start
+  ```
+- Most Rails 8 apps use foreman or overmind via `bin/dev`. Without a `jobs:` entry, the process manager only starts the web server and asset watchers -- Solid Queue workers are not launched.
+
+## 6. Realtime Updates Do Not Stream
 
 - Confirm Action Cable is mounted and `ApplicationCable` classes exist (see installation guide).
 - In production, verify WebSocket proxy settings allow the `/cable` endpoint.
 - When switching to Redis, add `config.realtime.adapter = :redis` and `config.realtime.redis_url` in the initializer, then restart web and worker processes.
 - For Solid Cable, check that the `solid_cable_messages` table exists and that no other process clears it unexpectedly.
 
-## 5. Fetch Jobs Keep Failing
+## 7. Fetch Jobs Keep Failing
 
 - Review the most recent fetch log entry for the source; it stores the HTTP status, error class, and error message.
 - Increase `config.http.timeout` or `config.http.retry_max` if the feed is slow or prone to transient errors.
 - Supply custom headers or basic auth credentials via the source form when feeds require authentication.
 - Check for TLS issues on self-signed feeds; you may need to configure Faraday with custom SSL options.
 
-## 6. Scraping Returns "Failed"
+## 8. Scraping Returns "Failed"
 
 - Confirm the source has scraping enabled and the configured adapter exists.
 - Override selectors in the source's scrape settings if the default Readability extraction misses key elements.
 - Inspect the scrape log to see the adapter status and content length. Logs store the HTTP status and any exception raised by the adapter.
 - Retry manually from the item detail page after fixing selectors.
 
-## 7. Cleanup Rake Tasks Fail
+## 9. Cleanup Rake Tasks Fail
 
 - Pass numeric values for `FETCH_LOG_DAYS` or `SCRAPE_LOG_DAYS` environment variables (e.g., `FETCH_LOG_DAYS=30`).
 - Ensure workers or the console environment have permission to soft delete (`SOFT_DELETE=true`) if you expect tombstones.
 - If job classes cannot load, verify `SourceMonitor.configure` ran before calling `rake source_monitor:cleanup:*`.
 
-## 8. Test Suite Cannot Launch a Browser
+## 10. Test Suite Cannot Launch a Browser
 
 - System tests rely on Selenium + Chrome. Install Chrome/Chromium and set `SELENIUM_CHROME_BINARY` if the binary lives in a non-standard path.
 - You can run `rbenv exec bin/test-coverage --verbose` to inspect failures with additional logging.
 
-## 9. Mission Control Jobs Link Returns 404
+## 11. Mission Control Jobs Link Returns 404
 
 - Mount `MissionControl::Jobs::Engine` in your host routes (for example, `mount MissionControl::Jobs::Engine, at: "/mission_control"`).
 - Keep `config.mission_control_enabled = true` **and** `config.mission_control_dashboard_path` pointing at that mounted route helper. Call `SourceMonitor.mission_control_dashboard_path` in the Rails console to confirm it resolves.
 - When hosting Mission Control in a separate app, provide a full URL instead of a route helper and ensure CORS/WebSocket settings allow the dashboard iframe.
 
-## 10. Tailwind Build Fails or Admin UI Loads Without Styles
+## 12. Tailwind Build Fails or Admin UI Loads Without Styles
 
 - Running `test/dummy/bin/dev` before configuring the bundling pipeline will serve the admin UI without Tailwind styles or Stimulus behaviours. This happens because the engine no longer ships precompiled assets; see `.ai/engine-asset-configuration.md:11-44` for the required npm setup.
 - Fix by running `npm install` followed by `npm run build` inside the engine root so that `app/assets/builds/source_monitor/application.css` and `application.js` exist. The Rake task `app:source_monitor:assets:build` wraps the same scripts for CI usage.

data/lib/generators/source_monitor/install/install_generator.rb

@@ -49,7 +49,48 @@ module SourceMonitor
         end
       end
 
+      def patch_procfile_dev
+        procfile_path = File.join(destination_root, "Procfile.dev")
+
+        if File.exist?(procfile_path)
+          content = File.read(procfile_path)
+          if content.match?(/^jobs:/)
+            say_status :skip, "Procfile.dev (jobs entry already present)", :yellow
+            return
+          end
+
+          File.open(procfile_path, "a") { |f| f.puts("", PROCFILE_JOBS_ENTRY) }
+          say_status :append, "Procfile.dev", :green
+        else
+          File.write(procfile_path, "web: bin/rails server -p 3000\n#{PROCFILE_JOBS_ENTRY}\n")
+          say_status :create, "Procfile.dev", :green
+        end
+      end
+
+      def configure_queue_dispatcher
+        queue_path = File.join(destination_root, "config/queue.yml")
+
+        unless File.exist?(queue_path)
+          say_status :skip, "config/queue.yml (file not found — create it or run rails app:update to generate)", :yellow
+          return
+        end
+
+        parsed = YAML.safe_load(File.read(queue_path), aliases: true) || {}
+
+        if queue_config_has_recurring_schedule?(parsed)
+          say_status :skip, "config/queue.yml (recurring_schedule already configured)", :yellow
+          return
+        end
+
+        add_recurring_schedule_to_dispatchers!(parsed)
+        File.write(queue_path, YAML.dump(parsed))
+        say_status :append, "config/queue.yml (added recurring_schedule to dispatchers)", :green
+      end
+
       def print_next_steps
+        say_status :info,
+          "Procfile.dev configured — run bin/dev to start both web server and Solid Queue workers.",
+          :green
         say_status :info,
           "Recurring jobs configured in config/recurring.yml — they'll run automatically with bin/dev or bin/jobs.",
           :green
@@ -60,6 +101,8 @@ module SourceMonitor
 
       private
 
+      PROCFILE_JOBS_ENTRY = "jobs: bundle exec rake solid_queue:start"
+
       RECURRING_ENTRIES = {
         "source_monitor_schedule_fetches" => {
           "class" => "SourceMonitor::ScheduleFetchesJob",
@@ -154,6 +197,64 @@ module SourceMonitor
         path = (raw_path && !raw_path.strip.empty?) ? raw_path.strip : "/source_monitor"
         path.start_with?("/") ? path : "/#{path}"
       end
+
+      RECURRING_SCHEDULE_VALUE = "config/recurring.yml"
+
+      DEFAULT_DISPATCHER = {
+        "polling_interval" => 1,
+        "batch_size" => 500,
+        "recurring_schedule" => RECURRING_SCHEDULE_VALUE
+      }.freeze
+
+      def queue_config_has_recurring_schedule?(parsed)
+        parsed.each_value do |value|
+          next unless value.is_a?(Hash)
+
+          dispatchers = value["dispatchers"] || value[:dispatchers]
+          if dispatchers.is_a?(Array)
+            return true if dispatchers.any? { |d| d.is_a?(Hash) && d.key?("recurring_schedule") }
+          end
+
+          return true if queue_config_has_recurring_schedule?(value)
+        end
+
+        # Check top-level dispatchers (flat config)
+        if parsed.key?("dispatchers") && parsed["dispatchers"].is_a?(Array)
+          return true if parsed["dispatchers"].any? { |d| d.is_a?(Hash) && d.key?("recurring_schedule") }
+        end
+
+        false
+      end
+
+      def add_recurring_schedule_to_dispatchers!(parsed)
+        found_dispatchers = false
+
+        parsed.each_value do |value|
+          next unless value.is_a?(Hash)
+
+          if value.key?("dispatchers") && value["dispatchers"].is_a?(Array)
+            value["dispatchers"].each do |dispatcher|
+              next unless dispatcher.is_a?(Hash)
+              dispatcher["recurring_schedule"] ||= RECURRING_SCHEDULE_VALUE
+            end
+            found_dispatchers = true
+          end
+        end
+
+        # Check top-level dispatchers (flat config)
+        if parsed.key?("dispatchers") && parsed["dispatchers"].is_a?(Array)
+          parsed["dispatchers"].each do |dispatcher|
+            next unless dispatcher.is_a?(Hash)
+            dispatcher["recurring_schedule"] ||= RECURRING_SCHEDULE_VALUE
+          end
+          found_dispatchers = true
+        end
+
+        # No dispatchers found at all — add a default section
+        unless found_dispatchers
+          parsed["dispatchers"] = [ DEFAULT_DISPATCHER.dup ]
+        end
+      end
     end
   end
 end

data/lib/source_monitor/configuration/http_settings.rb

@@ -13,7 +13,10 @@ module SourceMonitor
         :retry_interval,
         :retry_interval_randomness,
         :retry_backoff_factor,
-        :retry_statuses
+        :retry_statuses,
+        :ssl_ca_file,
+        :ssl_ca_path,
+        :ssl_verify
 
       def initialize
         reset!
@@ -31,6 +34,9 @@ module SourceMonitor
         @retry_interval_randomness = 0.5
         @retry_backoff_factor = 2
         @retry_statuses = nil
+        @ssl_ca_file = nil
+        @ssl_ca_path = nil
+        @ssl_verify = true
       end
 
       private

data/lib/source_monitor/configuration/images_settings.rb

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+module SourceMonitor
+  class Configuration
+    class ImagesSettings
+      attr_accessor :download_to_active_storage,
+        :max_download_size,
+        :download_timeout,
+        :allowed_content_types
+
+      DEFAULT_MAX_DOWNLOAD_SIZE = 10 * 1024 * 1024 # 10 MB
+      DEFAULT_DOWNLOAD_TIMEOUT = 30 # seconds
+      DEFAULT_ALLOWED_CONTENT_TYPES = %w[
+        image/jpeg
+        image/png
+        image/gif
+        image/webp
+        image/svg+xml
+      ].freeze
+
+      def initialize
+        reset!
+      end
+
+      def reset!
+        @download_to_active_storage = false
+        @max_download_size = DEFAULT_MAX_DOWNLOAD_SIZE
+        @download_timeout = DEFAULT_DOWNLOAD_TIMEOUT
+        @allowed_content_types = DEFAULT_ALLOWED_CONTENT_TYPES.dup
+      end
+
+      def download_enabled?
+        !!download_to_active_storage
+      end
+    end
+  end
+end

data/lib/source_monitor/configuration.rb

@@ -8,6 +8,7 @@ require "source_monitor/configuration/scraping_settings"
 require "source_monitor/configuration/realtime_settings"
 require "source_monitor/configuration/retention_settings"
 require "source_monitor/configuration/authentication_settings"
+require "source_monitor/configuration/images_settings"
 require "source_monitor/configuration/scraper_registry"
 require "source_monitor/configuration/events"
 require "source_monitor/configuration/validation_definition"
@@ -26,7 +27,7 @@ module SourceMonitor
       :mission_control_enabled,
       :mission_control_dashboard_path
 
-    attr_reader :http, :scrapers, :retention, :events, :models, :realtime, :fetching, :health, :authentication, :scraping
+    attr_reader :http, :scrapers, :retention, :events, :models, :realtime, :fetching, :health, :authentication, :scraping, :images
 
     DEFAULT_QUEUE_NAMESPACE = "source_monitor"
 
@@ -50,6 +51,7 @@ module SourceMonitor
       @health = HealthSettings.new
       @authentication = AuthenticationSettings.new
       @scraping = ScrapingSettings.new
+      @images = ImagesSettings.new
     end
 
     def queue_name_for(role)

data/lib/source_monitor/dashboard/queries/recent_activity_query.rb

@@ -37,7 +37,8 @@ module SourceMonitor
             item_title: row["item_title"],
             item_url: row["item_url"],
             source_name: row["source_name"],
-            source_id: row["source_id"]
+            source_id: row["source_id"],
+            source_feed_url: row["source_feed_url"]
           )
         end
 
@@ -57,7 +58,8 @@ module SourceMonitor
                    item_title,
                    item_url,
                    source_name,
-                   source_id
+                   source_id,
+                   source_feed_url
             FROM (
               #{fetch_log_sql}
               UNION ALL
@@ -83,9 +85,12 @@ module SourceMonitor
               NULL AS scraper_adapter,
               NULL AS item_title,
               NULL AS item_url,
-              NULL AS source_name,
-              #{SourceMonitor::FetchLog.quoted_table_name}.source_id AS source_id
+              #{SourceMonitor::Source.quoted_table_name}.#{quoted_source_name} AS source_name,
+              #{SourceMonitor::FetchLog.quoted_table_name}.source_id AS source_id,
+              #{SourceMonitor::Source.quoted_table_name}.feed_url AS source_feed_url
             FROM #{SourceMonitor::FetchLog.quoted_table_name}
+            LEFT JOIN #{SourceMonitor::Source.quoted_table_name}
+              ON #{SourceMonitor::Source.quoted_table_name}.id = #{SourceMonitor::FetchLog.quoted_table_name}.source_id
           SQL
         end
 
@@ -100,12 +105,15 @@ module SourceMonitor
               NULL AS items_updated,
               #{SourceMonitor::ScrapeLog.quoted_table_name}.scraper_adapter AS scraper_adapter,
               NULL AS item_title,
-              NULL AS item_url,
+              #{SourceMonitor::Item.quoted_table_name}.url AS item_url,
               #{SourceMonitor::Source.quoted_table_name}.#{quoted_source_name} AS source_name,
-              #{SourceMonitor::ScrapeLog.quoted_table_name}.source_id AS source_id
+              #{SourceMonitor::ScrapeLog.quoted_table_name}.source_id AS source_id,
+              NULL AS source_feed_url
             FROM #{SourceMonitor::ScrapeLog.quoted_table_name}
             LEFT JOIN #{SourceMonitor::Source.quoted_table_name}
               ON #{SourceMonitor::Source.quoted_table_name}.id = #{SourceMonitor::ScrapeLog.quoted_table_name}.source_id
+            LEFT JOIN #{SourceMonitor::Item.quoted_table_name}
+              ON #{SourceMonitor::Item.quoted_table_name}.id = #{SourceMonitor::ScrapeLog.quoted_table_name}.item_id
           SQL
         end
 
@@ -122,7 +130,8 @@ module SourceMonitor
               #{SourceMonitor::Item.quoted_table_name}.title AS item_title,
               #{SourceMonitor::Item.quoted_table_name}.url AS item_url,
               #{SourceMonitor::Source.quoted_table_name}.#{quoted_source_name} AS source_name,
-              #{SourceMonitor::Item.quoted_table_name}.source_id AS source_id
+              #{SourceMonitor::Item.quoted_table_name}.source_id AS source_id,
+              NULL AS source_feed_url
             FROM #{SourceMonitor::Item.quoted_table_name}
             LEFT JOIN #{SourceMonitor::Source.quoted_table_name}
               ON #{SourceMonitor::Source.quoted_table_name}.id = #{SourceMonitor::Item.quoted_table_name}.source_id

data/lib/source_monitor/dashboard/recent_activity.rb

@@ -15,6 +15,7 @@ module SourceMonitor
         :item_url,
         :source_name,
         :source_id,
+        :source_feed_url,
         keyword_init: true
       ) do
         def type

data/lib/source_monitor/dashboard/recent_activity_presenter.rb

@@ -30,13 +30,16 @@ module SourceMonitor
       end
 
       def fetch_event(event)
+        domain = source_domain(event.source_feed_url)
         {
           label: "Fetch ##{event.id}",
           description: "#{event.items_created.to_i} created / #{event.items_updated.to_i} updated",
           status: event.success? ? :success : :failure,
           type: :fetch,
           time: event.occurred_at,
-          path: url_helpers.fetch_log_path(event.id)
+          path: url_helpers.fetch_log_path(event.id),
+          url_display: domain,
+          url_href: event.source_feed_url
         }
       end
 
@@ -47,10 +50,20 @@ module SourceMonitor
           status: event.success? ? :success : :failure,
           type: :scrape,
           time: event.occurred_at,
-          path: url_helpers.scrape_log_path(event.id)
+          path: url_helpers.scrape_log_path(event.id),
+          url_display: event.item_url,
+          url_href: event.item_url
         }
       end
 
+      def source_domain(feed_url)
+        return nil if feed_url.blank?
+
+        URI.parse(feed_url.to_s).host
+      rescue URI::InvalidURIError
+        nil
+      end
+
       def item_event(event)
         {
           label: event.item_title.presence || "New Item",

data/lib/source_monitor/fetching/feed_fetcher/entry_processor.rb

@@ -38,6 +38,7 @@ module SourceMonitor
                 created += 1
                 created_items << result.item
                 SourceMonitor::Events.after_item_created(item: result.item, source:, entry:, result: result)
+                enqueue_image_download(result.item)
               else
                 updated += 1
                 updated_items << result.item
@@ -61,6 +62,18 @@ module SourceMonitor
 
         private
 
+        def enqueue_image_download(item)
+          return unless SourceMonitor.config.images.download_enabled?
+          return if item.content.blank?
+
+          SourceMonitor::DownloadContentImagesJob.perform_later(item.id)
+        rescue StandardError => error
+          # Image download enqueue failure must never break feed processing
+          if defined?(Rails) && Rails.respond_to?(:logger) && Rails.logger
+            Rails.logger.error("[SourceMonitor] Failed to enqueue image download for item #{item.id}: #{error.message}")
+          end
+        end
+
         def normalize_item_error(entry, error)
           {
             guid: safe_entry_guid(entry),

data/lib/source_monitor/http.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require "openssl"
 require "faraday"
 require "faraday/retry"
 require "faraday/follow_redirects"
@@ -57,9 +58,31 @@ module SourceMonitor
           connection.headers[key] = value
         end
 
+        configure_ssl(connection, settings)
+
         connection.adapter Faraday.default_adapter
       end
 
+      # Configure SSL to use a proper cert store. Without this, some systems
+      # fail to verify certificate chains that depend on intermediate CAs
+      # (e.g., Medium/Netflix on AWS). OpenSSL::X509::Store#set_default_paths
+      # loads all system-trusted CAs including intermediates.
+      def configure_ssl(connection, settings)
+        connection.ssl.verify = settings.ssl_verify != false
+
+        if settings.ssl_ca_file
+          connection.ssl.ca_file = settings.ssl_ca_file
+        elsif settings.ssl_ca_path
+          connection.ssl.ca_path = settings.ssl_ca_path
+        else
+          connection.ssl.cert_store = default_cert_store
+        end
+      end
+
+      def default_cert_store
+        OpenSSL::X509::Store.new.tap(&:set_default_paths)
+      end
+
       def default_headers(settings)
         base_headers = {
           "User-Agent" => resolve_callable(settings.user_agent).presence || DEFAULT_USER_AGENT,

data/lib/source_monitor/images/content_rewriter.rb

@@ -0,0 +1,81 @@
+# frozen_string_literal: true
+
+require "nokolexbor"
+require "uri"
+
+module SourceMonitor
+  module Images
+    class ContentRewriter
+      attr_reader :html, :base_url
+
+      def initialize(html, base_url: nil)
+        @html = html.to_s
+        @base_url = base_url
+      end
+
+      # Returns an array of absolute image URLs found in <img> tags.
+      # Skips data: URIs, blank src, and invalid URLs.
+      def image_urls
+        return [] if html.blank?
+
+        doc = parse_fragment
+        urls = []
+
+        doc.css("img[src]").each do |img|
+          url = resolve_url(img["src"])
+          urls << url if url && downloadable_url?(url)
+        end
+
+        urls.uniq
+      end
+
+      # Rewrites <img src="..."> attributes by yielding each original URL
+      # to the block and replacing with the block's return value.
+      # Returns the rewritten HTML string.
+      # If the block returns nil, the original URL is preserved (graceful fallback).
+      def rewrite
+        return html if html.blank?
+
+        doc = parse_fragment
+
+        doc.css("img[src]").each do |img|
+          original_url = resolve_url(img["src"])
+          next unless original_url && downloadable_url?(original_url)
+
+          new_url = yield(original_url)
+          img["src"] = new_url if new_url.present?
+        end
+
+        doc.to_html
+      end
+
+      private
+
+      def parse_fragment
+        Nokolexbor::DocumentFragment.parse(html)
+      end
+
+      def resolve_url(src)
+        src = src.to_s.strip
+        return nil if src.blank?
+        return nil if src.start_with?("data:")
+
+        uri = URI.parse(src)
+        if uri.relative? && base_url.present?
+          URI.join(base_url, src).to_s
+        elsif uri.absolute?
+          src
+        end
+      rescue URI::InvalidURIError
+        nil
+      end
+
+      def downloadable_url?(url)
+        uri = URI.parse(url)
+        uri.is_a?(URI::HTTP) || uri.is_a?(URI::HTTPS)
+      rescue URI::InvalidURIError
+        false
+      end
+    end
+  end
+end

data/lib/source_monitor/images/downloader.rb

@@ -0,0 +1,82 @@
+# frozen_string_literal: true
+
+require "faraday"
+require "securerandom"
+
+module SourceMonitor
+  module Images
+    class Downloader
+      Result = Struct.new(:io, :filename, :content_type, :byte_size, keyword_init: true)
+
+      attr_reader :url, :settings
+
+      def initialize(url, settings: nil)
+        @url = url
+        @settings = settings || SourceMonitor.config.images
+      end
+
+      # Downloads the image and returns a Result, or nil if download fails
+      # or the image does not meet validation criteria.
+      def call
+        response = fetch_image
+        return unless response
+
+        content_type = response.headers["content-type"]&.split(";")&.first&.strip&.downcase
+        return unless allowed_content_type?(content_type)
+
+        body = response.body
+        return unless body && body.bytesize > 0
+        return if body.bytesize > settings.max_download_size
+
+        filename = derive_filename(url, content_type)
+
+        Result.new(
+          io: StringIO.new(body),
+          filename: filename,
+          content_type: content_type,
+          byte_size: body.bytesize
+        )
+      rescue Faraday::Error, URI::InvalidURIError, Timeout::Error
+        nil
+      end
+
+      private
+
+      def fetch_image
+        connection = Faraday.new do |f|
+          f.options.timeout = settings.download_timeout
+          f.options.open_timeout = [ settings.download_timeout / 2, 5 ].min
+          f.headers["User-Agent"] = SourceMonitor.config.http.user_agent || "SourceMonitor/#{SourceMonitor::VERSION}"
+          f.headers["Accept"] = "image/*"
+          f.adapter Faraday.default_adapter
+        end
+
+        response = connection.get(url)
+        return response if response.status == 200
+
+        nil
+      end
+
+      def allowed_content_type?(content_type)
+        return false if content_type.blank?
+
+        settings.allowed_content_types.include?(content_type)
+      end
+
+      def derive_filename(image_url, content_type)
+        uri = URI.parse(image_url)
+        basename = File.basename(uri.path) if uri.path.present?
+
+        if basename.present? && basename.include?(".")
+          basename
+        else
+          ext = Rack::Mime::MIME_TYPES.invert[content_type] || ".bin"
+          "image-#{SecureRandom.hex(8)}#{ext}"
+        end
+      rescue URI::InvalidURIError
+        ext = Rack::Mime::MIME_TYPES.invert[content_type] || ".bin"
+        "image-#{SecureRandom.hex(8)}#{ext}"
+      end
+    end
+  end
+end