source_monitor 0.3.2 → 0.4.0

data/docs/setup.md

@@ -48,8 +48,13 @@ This ensures Bundler can load SourceMonitor so the commands below are available.
 3. **Start background workers:**
    ```bash
    bin/rails solid_queue:start
-   bin/jobs --recurring_schedule_file=config/recurring.yml # optional recurring scheduler
    ```
+   The install generator automatically handles all worker configuration:
+   - **Recurring jobs** are configured in `config/recurring.yml` (fetch scheduling, scraping, cleanup).
+   - **Procfile.dev** is patched with a `jobs:` entry so `bin/dev` starts Solid Queue alongside the web server.
+   - **Queue dispatcher** is patched with `recurring_schedule: config/recurring.yml` in `config/queue.yml` so recurring jobs load on startup.
+
+   All three steps are idempotent. If any configuration is missing, re-run: `bin/rails generate source_monitor:install`
 
 4. **Visit the dashboard** at the chosen mount path, create a source, and trigger “Fetch Now” to validate realtime updates and Solid Queue processing.
 
@@ -83,10 +88,12 @@ Prefer to script each step or plug SourceMonitor into an existing deployment che
 | --- | --- | --- |
 | 1 | `gem "source_monitor", github: "dchuk/source_monitor"` | Add the engine to your Gemfile (skip if already present) |
 | 2 | `bundle install` | Install Ruby dependencies |
-| 3 | `bin/rails generate source_monitor:install --mount-path=/source_monitor` | Mount the engine and create the initializer |
+| 3 | `bin/rails generate source_monitor:install --mount-path=/source_monitor` | Mount the engine, create the initializer, and configure recurring jobs |
 | 4 | `bin/rails railties:install:migrations FROM=source_monitor` | Copy engine migrations (idempotent) |
 | 5 | `bin/rails db:migrate` | Apply schema updates, including Solid Queue tables |
 | 6 | `bin/rails solid_queue:start` | Ensure jobs process via Solid Queue |
+| 6a | Handled by generator (patches `Procfile.dev`) | Ensure `bin/dev` starts Solid Queue workers |
+| 6b | Handled by generator (patches `config/queue.yml`) | Wire recurring jobs into Solid Queue dispatcher |
 | 7 | `bin/jobs --recurring_schedule_file=config/recurring.yml` | Start recurring scheduler (optional but recommended) |
 | 8 | `bin/source_monitor verify` | Confirm Solid Queue/Action Cable readiness and emit telemetry |
 
@@ -95,11 +102,13 @@ Prefer to script each step or plug SourceMonitor into an existing deployment che
 ### Step-by-step Details
 
 1. **Add the gem** to the host `Gemfile` (GitHub edge or released version) and run `bundle install`. If your host manages node tooling, run `npm install` also.
-2. **Install the engine** via `bin/rails generate source_monitor:install --mount-path=/source_monitor`. The generator mounts the engine, creates `config/initializers/source_monitor.rb`, and prints follow-up instructions. Re-running the generator is safe; it detects existing mounts/initializers.
+2. **Install the engine** via `bin/rails generate source_monitor:install --mount-path=/source_monitor`. The generator mounts the engine, creates `config/initializers/source_monitor.rb`, and configures recurring Solid Queue jobs in `config/recurring.yml`. Re-running the generator is safe; it detects existing mounts/initializers and skips entries that are already present.
 3. **Copy migrations** with `bin/rails railties:install:migrations FROM=source_monitor`. This brings in the SourceMonitor tables plus Solid Cable/Queue schema when needed. The command is idempotent—run it again after upgrading the gem.
 4. **Apply database changes** using `bin/rails db:migrate`. If your host already installed Solid Queue migrations manually, delete duplicate files before migrating.
 5. **Wire Action Cable** if necessary. SourceMonitor defaults to Solid Cable; confirm `ApplicationCable::Connection`/`Channel` exist and that `config/initializers/source_monitor.rb` uses the adapter you expect. To switch to Redis, set `config.realtime.adapter = :redis` and `config.realtime.redis_url`.
-6. **Start workers** with `bin/rails solid_queue:start` (or your process manager). Add a recurring process via `bin/jobs --recurring_schedule_file=config/recurring.yml` when you need fetch/scrape schedules.
+6. **Start workers** with `bin/rails solid_queue:start` (or your process manager). The install generator automatically configures recurring jobs in `config/recurring.yml` for fetch scheduling, scraping, and cleanup. They'll run with `bin/dev` or `bin/jobs`.
+   - **Procfile.dev:** The generator automatically patches `Procfile.dev` with a `jobs:` entry for Solid Queue. Verify the file contains `jobs: bundle exec rake solid_queue:start` after running the generator.
+   - **Recurring schedule:** The generator automatically patches `config/queue.yml` dispatchers with `recurring_schedule: config/recurring.yml`. Verify the key is present after running the generator.
 7. **Review the initializer** and tune queue names, HTTP timeouts, scraping adapters, retention limits, authentication hooks, and Mission Control integration. The [configuration reference](configuration.md) details every option.
 8. **Verify the install**: run `bin/source_monitor verify` to ensure Solid Queue workers and Action Cable are healthy, then visit the mount path to trigger a fetch manually. Enable telemetry if you want JSON logs recorded for support.
 

data/docs/troubleshooting.md

@@ -20,45 +20,76 @@ This guide lists common issues you might encounter while installing, upgrading,
 - Ensure at least one Solid Queue worker is running; the dashboard reads visibility data via `SourceMonitor::Jobs::Visibility`.
 - When using mission control integration, keep `config.mission_control_dashboard_path` pointing at a valid route helper; otherwise the dashboard hides the link.
 
-## 4. Realtime Updates Do Not Stream
+## 4. Recurring Jobs Not Running
+
+- **Symptoms:** Fetch scheduling, scrape scheduling, and cleanup jobs never fire. Sources never auto-fetch on their configured intervals.
+- **Primary fix:** Re-run the install generator, which automatically patches the dispatcher config:
+  ```bash
+  bin/rails generate source_monitor:install
+  ```
+- **Diagnostics:** Run `bin/source_monitor verify` to check recurring task registration. The RecurringScheduleVerifier will report whether SourceMonitor recurring tasks are loaded into Solid Queue.
+- **Manual check:** Verify `config/queue.yml` includes `recurring_schedule: config/recurring.yml` under the `dispatchers:` section. Without this key, Solid Queue's dispatcher will not load the recurring schedule even though `config/recurring.yml` exists.
+- **Manual fix (if generator cannot patch):**
+  ```yaml
+  dispatchers:
+    - polling_interval: 1
+      batch_size: 500
+      recurring_schedule: config/recurring.yml
+  ```
+
+## 5. Jobs Not Processing with bin/dev
+
+- **Symptoms:** `bin/dev` starts the web server but jobs never run. Running `bin/rails solid_queue:start` manually works fine.
+- **Primary fix:** Re-run the install generator, which automatically patches `Procfile.dev`:
+  ```bash
+  bin/rails generate source_monitor:install
+  ```
+- **Diagnostics:** Run `bin/source_monitor verify` to check Solid Queue worker status. The SolidQueueVerifier will suggest Procfile.dev if no workers are detected.
+- **Manual check:** Verify `Procfile.dev` includes a `jobs:` line:
+  ```
+  jobs: bundle exec rake solid_queue:start
+  ```
+- Most Rails 8 apps use foreman or overmind via `bin/dev`. Without a `jobs:` entry, the process manager only starts the web server and asset watchers -- Solid Queue workers are not launched.
+
+## 6. Realtime Updates Do Not Stream
 
 - Confirm Action Cable is mounted and `ApplicationCable` classes exist (see installation guide).
 - In production, verify WebSocket proxy settings allow the `/cable` endpoint.
 - When switching to Redis, add `config.realtime.adapter = :redis` and `config.realtime.redis_url` in the initializer, then restart web and worker processes.
 - For Solid Cable, check that the `solid_cable_messages` table exists and that no other process clears it unexpectedly.
 
-## 5. Fetch Jobs Keep Failing
+## 7. Fetch Jobs Keep Failing
 
 - Review the most recent fetch log entry for the source; it stores the HTTP status, error class, and error message.
 - Increase `config.http.timeout` or `config.http.retry_max` if the feed is slow or prone to transient errors.
 - Supply custom headers or basic auth credentials via the source form when feeds require authentication.
 - Check for TLS issues on self-signed feeds; you may need to configure Faraday with custom SSL options.
 
-## 6. Scraping Returns "Failed"
+## 8. Scraping Returns "Failed"
 
 - Confirm the source has scraping enabled and the configured adapter exists.
 - Override selectors in the source's scrape settings if the default Readability extraction misses key elements.
 - Inspect the scrape log to see the adapter status and content length. Logs store the HTTP status and any exception raised by the adapter.
 - Retry manually from the item detail page after fixing selectors.
 
-## 7. Cleanup Rake Tasks Fail
+## 9. Cleanup Rake Tasks Fail
 
 - Pass numeric values for `FETCH_LOG_DAYS` or `SCRAPE_LOG_DAYS` environment variables (e.g., `FETCH_LOG_DAYS=30`).
 - Ensure workers or the console environment have permission to soft delete (`SOFT_DELETE=true`) if you expect tombstones.
 - If job classes cannot load, verify `SourceMonitor.configure` ran before calling `rake source_monitor:cleanup:*`.
 
-## 8. Test Suite Cannot Launch a Browser
+## 10. Test Suite Cannot Launch a Browser
 
 - System tests rely on Selenium + Chrome. Install Chrome/Chromium and set `SELENIUM_CHROME_BINARY` if the binary lives in a non-standard path.
 - You can run `rbenv exec bin/test-coverage --verbose` to inspect failures with additional logging.
 
-## 9. Mission Control Jobs Link Returns 404
+## 11. Mission Control Jobs Link Returns 404
 
 - Mount `MissionControl::Jobs::Engine` in your host routes (for example, `mount MissionControl::Jobs::Engine, at: "/mission_control"`).
 - Keep `config.mission_control_enabled = true` **and** `config.mission_control_dashboard_path` pointing at that mounted route helper. Call `SourceMonitor.mission_control_dashboard_path` in the Rails console to confirm it resolves.
 - When hosting Mission Control in a separate app, provide a full URL instead of a route helper and ensure CORS/WebSocket settings allow the dashboard iframe.
 
-## 10. Tailwind Build Fails or Admin UI Loads Without Styles
+## 12. Tailwind Build Fails or Admin UI Loads Without Styles
 
 - Running `test/dummy/bin/dev` before configuring the bundling pipeline will serve the admin UI without Tailwind styles or Stimulus behaviours. This happens because the engine no longer ships precompiled assets; see `.ai/engine-asset-configuration.md:11-44` for the required npm setup.
 - Fix by running `npm install` followed by `npm run build` inside the engine root so that `app/assets/builds/source_monitor/application.css` and `application.js` exist. The Rake task `app:source_monitor:assets:build` wraps the same scripts for CI usage.

data/lib/generators/source_monitor/install/install_generator.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require "yaml"
 require "rails/generators"
 require "rails/generators/base"
 
@@ -32,7 +33,67 @@ module SourceMonitor
         template "source_monitor.rb.tt", initializer_path
       end
 
+      def configure_recurring_jobs
+        recurring_path = "config/recurring.yml"
+        destination = File.join(destination_root, recurring_path)
+
+        if recurring_file_has_source_monitor_entries?(destination)
+          say_status :skip, "#{recurring_path} (SourceMonitor entries already present)", :yellow
+          return
+        end
+
+        if File.exist?(destination)
+          merge_into_existing_recurring(destination, recurring_path)
+        else
+          create_recurring_file(destination, recurring_path)
+        end
+      end
+
+      def patch_procfile_dev
+        procfile_path = File.join(destination_root, "Procfile.dev")
+
+        if File.exist?(procfile_path)
+          content = File.read(procfile_path)
+          if content.match?(/^jobs:/)
+            say_status :skip, "Procfile.dev (jobs entry already present)", :yellow
+            return
+          end
+
+          File.open(procfile_path, "a") { |f| f.puts("", PROCFILE_JOBS_ENTRY) }
+          say_status :append, "Procfile.dev", :green
+        else
+          File.write(procfile_path, "web: bin/rails server -p 3000\n#{PROCFILE_JOBS_ENTRY}\n")
+          say_status :create, "Procfile.dev", :green
+        end
+      end
+
+      def configure_queue_dispatcher
+        queue_path = File.join(destination_root, "config/queue.yml")
+
+        unless File.exist?(queue_path)
+          say_status :skip, "config/queue.yml (file not found — create it or run rails app:update to generate)", :yellow
+          return
+        end
+
+        parsed = YAML.safe_load(File.read(queue_path), aliases: true) || {}
+
+        if queue_config_has_recurring_schedule?(parsed)
+          say_status :skip, "config/queue.yml (recurring_schedule already configured)", :yellow
+          return
+        end
+
+        add_recurring_schedule_to_dispatchers!(parsed)
+        File.write(queue_path, YAML.dump(parsed))
+        say_status :append, "config/queue.yml (added recurring_schedule to dispatchers)", :green
+      end
+
       def print_next_steps
+        say_status :info,
+          "Procfile.dev configured — run bin/dev to start both web server and Solid Queue workers.",
+          :green
+        say_status :info,
+          "Recurring jobs configured in config/recurring.yml — they'll run automatically with bin/dev or bin/jobs.",
+          :green
         say_status :info,
           "Next steps: review docs/setup.md for the guided + manual install walkthrough and docs/troubleshooting.md for common fixes.",
           :green
@@ -40,6 +101,88 @@ module SourceMonitor
 
       private
 
+      PROCFILE_JOBS_ENTRY = "jobs: bundle exec rake solid_queue:start"
+
+      RECURRING_ENTRIES = {
+        "source_monitor_schedule_fetches" => {
+          "class" => "SourceMonitor::ScheduleFetchesJob",
+          "args" => [ { "limit" => 100 } ],
+          "schedule" => "every minute"
+        },
+        "source_monitor_schedule_scrapes" => {
+          "command" => "SourceMonitor::Scraping::Scheduler.run(limit: 100)",
+          "schedule" => "every 2 minutes"
+        },
+        "source_monitor_item_cleanup" => {
+          "class" => "SourceMonitor::ItemCleanupJob",
+          "schedule" => "at 2am every day"
+        },
+        "source_monitor_log_cleanup" => {
+          "class" => "SourceMonitor::LogCleanupJob",
+          "args" => [ { "fetch_logs_older_than_days" => 90, "scrape_logs_older_than_days" => 60 } ],
+          "schedule" => "at 3am every day"
+        }
+      }.freeze
+
+      def recurring_file_has_source_monitor_entries?(path)
+        return false unless File.exist?(path)
+
+        content = File.read(path)
+        content.include?("source_monitor_schedule_fetches")
+      end
+
+      def merge_into_existing_recurring(destination, recurring_path)
+        parsed = YAML.safe_load(File.read(destination), aliases: true) || {}
+        default_key = parsed.key?("default") ? "default" : nil
+
+        if default_key
+          parsed["default"] = (parsed["default"] || {}).merge(RECURRING_ENTRIES)
+        else
+          parsed.merge!(RECURRING_ENTRIES)
+        end
+
+        write_recurring_yaml(destination, parsed, has_environments: parsed.key?("development"))
+        say_status :append, recurring_path, :green
+      end
+
+      def create_recurring_file(destination, recurring_path)
+        FileUtils.mkdir_p(File.dirname(destination))
+        yaml_content = build_fresh_recurring_yaml
+        File.write(destination, yaml_content)
+        say_status :create, recurring_path, :green
+      end
+
+      def build_fresh_recurring_yaml
+        entries_yaml = format_entries_yaml(RECURRING_ENTRIES)
+
+        "default: &default\n#{entries_yaml}\n" \
+          "development:\n  <<: *default\n\n" \
+          "test:\n  <<: *default\n\n" \
+          "production:\n  <<: *default\n"
+      end
+
+      def write_recurring_yaml(destination, parsed, has_environments: false)
+        if has_environments
+          default_entries = parsed["default"] || {}
+          entries_yaml = format_entries_yaml(default_entries)
+          envs = %w[development test production].select { |e| parsed.key?(e) }
+          env_sections = envs.map { |e| "#{e}:\n  <<: *default" }.join("\n\n")
+
+          content = "default: &default\n#{entries_yaml}"
+          content += "\n#{env_sections}\n" unless envs.empty?
+          File.write(destination, content)
+        else
+          File.write(destination, YAML.dump(parsed))
+        end
+      end
+
+      def format_entries_yaml(entries)
+        entries.map { |key, value|
+          entry = YAML.dump({ key => value }).delete_prefix("---\n")
+          entry.gsub(/^/, "  ")
+        }.join("\n")
+      end
+
       def engine_already_mounted?(mount_path)
         routes_path = File.join(destination_root, "config/routes.rb")
         return false unless File.exist?(routes_path)
@@ -54,6 +197,64 @@ module SourceMonitor
         path = (raw_path && !raw_path.strip.empty?) ? raw_path.strip : "/source_monitor"
         path.start_with?("/") ? path : "/#{path}"
       end
+
+      RECURRING_SCHEDULE_VALUE = "config/recurring.yml"
+
+      DEFAULT_DISPATCHER = {
+        "polling_interval" => 1,
+        "batch_size" => 500,
+        "recurring_schedule" => RECURRING_SCHEDULE_VALUE
+      }.freeze
+
+      def queue_config_has_recurring_schedule?(parsed)
+        parsed.each_value do |value|
+          next unless value.is_a?(Hash)
+
+          dispatchers = value["dispatchers"] || value[:dispatchers]
+          if dispatchers.is_a?(Array)
+            return true if dispatchers.any? { |d| d.is_a?(Hash) && d.key?("recurring_schedule") }
+          end
+
+          return true if queue_config_has_recurring_schedule?(value)
+        end
+
+        # Check top-level dispatchers (flat config)
+        if parsed.key?("dispatchers") && parsed["dispatchers"].is_a?(Array)
+          return true if parsed["dispatchers"].any? { |d| d.is_a?(Hash) && d.key?("recurring_schedule") }
+        end
+
+        false
+      end
+
+      def add_recurring_schedule_to_dispatchers!(parsed)
+        found_dispatchers = false
+
+        parsed.each_value do |value|
+          next unless value.is_a?(Hash)
+
+          if value.key?("dispatchers") && value["dispatchers"].is_a?(Array)
+            value["dispatchers"].each do |dispatcher|
+              next unless dispatcher.is_a?(Hash)
+              dispatcher["recurring_schedule"] ||= RECURRING_SCHEDULE_VALUE
+            end
+            found_dispatchers = true
+          end
+        end
+
+        # Check top-level dispatchers (flat config)
+        if parsed.key?("dispatchers") && parsed["dispatchers"].is_a?(Array)
+          parsed["dispatchers"].each do |dispatcher|
+            next unless dispatcher.is_a?(Hash)
+            dispatcher["recurring_schedule"] ||= RECURRING_SCHEDULE_VALUE
+          end
+          found_dispatchers = true
+        end
+
+        # No dispatchers found at all — add a default section
+        unless found_dispatchers
+          parsed["dispatchers"] = [ DEFAULT_DISPATCHER.dup ]
+        end
+      end
     end
   end
 end

data/lib/source_monitor/configuration/http_settings.rb

@@ -13,7 +13,10 @@ module SourceMonitor
         :retry_interval,
         :retry_interval_randomness,
         :retry_backoff_factor,
-        :retry_statuses
+        :retry_statuses,
+        :ssl_ca_file,
+        :ssl_ca_path,
+        :ssl_verify
 
       def initialize
         reset!
@@ -31,6 +34,9 @@ module SourceMonitor
         @retry_interval_randomness = 0.5
         @retry_backoff_factor = 2
         @retry_statuses = nil
+        @ssl_ca_file = nil
+        @ssl_ca_path = nil
+        @ssl_verify = true
       end
 
       private

data/lib/source_monitor/configuration/images_settings.rb

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+module SourceMonitor
+  class Configuration
+    class ImagesSettings
+      attr_accessor :download_to_active_storage,
+        :max_download_size,
+        :download_timeout,
+        :allowed_content_types
+
+      DEFAULT_MAX_DOWNLOAD_SIZE = 10 * 1024 * 1024 # 10 MB
+      DEFAULT_DOWNLOAD_TIMEOUT = 30 # seconds
+      DEFAULT_ALLOWED_CONTENT_TYPES = %w[
+        image/jpeg
+        image/png
+        image/gif
+        image/webp
+        image/svg+xml
+      ].freeze
+
+      def initialize
+        reset!
+      end
+
+      def reset!
+        @download_to_active_storage = false
+        @max_download_size = DEFAULT_MAX_DOWNLOAD_SIZE
+        @download_timeout = DEFAULT_DOWNLOAD_TIMEOUT
+        @allowed_content_types = DEFAULT_ALLOWED_CONTENT_TYPES.dup
+      end
+
+      def download_enabled?
+        !!download_to_active_storage
+      end
+    end
+  end
+end

data/lib/source_monitor/configuration.rb

@@ -8,6 +8,7 @@ require "source_monitor/configuration/scraping_settings"
 require "source_monitor/configuration/realtime_settings"
 require "source_monitor/configuration/retention_settings"
 require "source_monitor/configuration/authentication_settings"
+require "source_monitor/configuration/images_settings"
 require "source_monitor/configuration/scraper_registry"
 require "source_monitor/configuration/events"
 require "source_monitor/configuration/validation_definition"
@@ -26,7 +27,7 @@ module SourceMonitor
       :mission_control_enabled,
       :mission_control_dashboard_path
 
-    attr_reader :http, :scrapers, :retention, :events, :models, :realtime, :fetching, :health, :authentication, :scraping
+    attr_reader :http, :scrapers, :retention, :events, :models, :realtime, :fetching, :health, :authentication, :scraping, :images
 
     DEFAULT_QUEUE_NAMESPACE = "source_monitor"
 
@@ -50,6 +51,7 @@ module SourceMonitor
       @health = HealthSettings.new
       @authentication = AuthenticationSettings.new
       @scraping = ScrapingSettings.new
+      @images = ImagesSettings.new
     end
 
     def queue_name_for(role)

data/lib/source_monitor/dashboard/queries/recent_activity_query.rb

@@ -37,7 +37,8 @@ module SourceMonitor
             item_title: row["item_title"],
             item_url: row["item_url"],
             source_name: row["source_name"],
-            source_id: row["source_id"]
+            source_id: row["source_id"],
+            source_feed_url: row["source_feed_url"]
           )
         end
 
@@ -57,7 +58,8 @@ module SourceMonitor
                    item_title,
                    item_url,
                    source_name,
-                   source_id
+                   source_id,
+                   source_feed_url
             FROM (
               #{fetch_log_sql}
               UNION ALL
@@ -83,9 +85,12 @@ module SourceMonitor
               NULL AS scraper_adapter,
               NULL AS item_title,
               NULL AS item_url,
-              NULL AS source_name,
-              #{SourceMonitor::FetchLog.quoted_table_name}.source_id AS source_id
+              #{SourceMonitor::Source.quoted_table_name}.#{quoted_source_name} AS source_name,
+              #{SourceMonitor::FetchLog.quoted_table_name}.source_id AS source_id,
+              #{SourceMonitor::Source.quoted_table_name}.feed_url AS source_feed_url
             FROM #{SourceMonitor::FetchLog.quoted_table_name}
+            LEFT JOIN #{SourceMonitor::Source.quoted_table_name}
+              ON #{SourceMonitor::Source.quoted_table_name}.id = #{SourceMonitor::FetchLog.quoted_table_name}.source_id
           SQL
         end
 
@@ -100,12 +105,15 @@ module SourceMonitor
               NULL AS items_updated,
               #{SourceMonitor::ScrapeLog.quoted_table_name}.scraper_adapter AS scraper_adapter,
               NULL AS item_title,
-              NULL AS item_url,
+              #{SourceMonitor::Item.quoted_table_name}.url AS item_url,
               #{SourceMonitor::Source.quoted_table_name}.#{quoted_source_name} AS source_name,
-              #{SourceMonitor::ScrapeLog.quoted_table_name}.source_id AS source_id
+              #{SourceMonitor::ScrapeLog.quoted_table_name}.source_id AS source_id,
+              NULL AS source_feed_url
             FROM #{SourceMonitor::ScrapeLog.quoted_table_name}
             LEFT JOIN #{SourceMonitor::Source.quoted_table_name}
               ON #{SourceMonitor::Source.quoted_table_name}.id = #{SourceMonitor::ScrapeLog.quoted_table_name}.source_id
+            LEFT JOIN #{SourceMonitor::Item.quoted_table_name}
+              ON #{SourceMonitor::Item.quoted_table_name}.id = #{SourceMonitor::ScrapeLog.quoted_table_name}.item_id
           SQL
         end
 
@@ -122,7 +130,8 @@ module SourceMonitor
               #{SourceMonitor::Item.quoted_table_name}.title AS item_title,
               #{SourceMonitor::Item.quoted_table_name}.url AS item_url,
               #{SourceMonitor::Source.quoted_table_name}.#{quoted_source_name} AS source_name,
-              #{SourceMonitor::Item.quoted_table_name}.source_id AS source_id
+              #{SourceMonitor::Item.quoted_table_name}.source_id AS source_id,
+              NULL AS source_feed_url
             FROM #{SourceMonitor::Item.quoted_table_name}
             LEFT JOIN #{SourceMonitor::Source.quoted_table_name}
               ON #{SourceMonitor::Source.quoted_table_name}.id = #{SourceMonitor::Item.quoted_table_name}.source_id

data/lib/source_monitor/dashboard/recent_activity.rb

@@ -15,6 +15,7 @@ module SourceMonitor
         :item_url,
         :source_name,
         :source_id,
+        :source_feed_url,
         keyword_init: true
       ) do
         def type

data/lib/source_monitor/dashboard/recent_activity_presenter.rb

@@ -30,13 +30,16 @@ module SourceMonitor
       end
 
       def fetch_event(event)
+        domain = source_domain(event.source_feed_url)
         {
           label: "Fetch ##{event.id}",
           description: "#{event.items_created.to_i} created / #{event.items_updated.to_i} updated",
           status: event.success? ? :success : :failure,
           type: :fetch,
           time: event.occurred_at,
-          path: url_helpers.fetch_log_path(event.id)
+          path: url_helpers.fetch_log_path(event.id),
+          url_display: domain,
+          url_href: event.source_feed_url
         }
       end
 
@@ -47,10 +50,20 @@ module SourceMonitor
           status: event.success? ? :success : :failure,
           type: :scrape,
           time: event.occurred_at,
-          path: url_helpers.scrape_log_path(event.id)
+          path: url_helpers.scrape_log_path(event.id),
+          url_display: event.item_url,
+          url_href: event.item_url
         }
       end
 
+      def source_domain(feed_url)
+        return nil if feed_url.blank?
+
+        URI.parse(feed_url.to_s).host
+      rescue URI::InvalidURIError
+        nil
+      end
+
       def item_event(event)
         {
           label: event.item_title.presence || "New Item",

data/lib/source_monitor/fetching/feed_fetcher/entry_processor.rb

@@ -38,6 +38,7 @@ module SourceMonitor
                 created += 1
                 created_items << result.item
                 SourceMonitor::Events.after_item_created(item: result.item, source:, entry:, result: result)
+                enqueue_image_download(result.item)
               else
                 updated += 1
                 updated_items << result.item
@@ -61,6 +62,18 @@ module SourceMonitor
 
         private
 
+        def enqueue_image_download(item)
+          return unless SourceMonitor.config.images.download_enabled?
+          return if item.content.blank?
+
+          SourceMonitor::DownloadContentImagesJob.perform_later(item.id)
+        rescue StandardError => error
+          # Image download enqueue failure must never break feed processing
+          if defined?(Rails) && Rails.respond_to?(:logger) && Rails.logger
+            Rails.logger.error("[SourceMonitor] Failed to enqueue image download for item #{item.id}: #{error.message}")
+          end
+        end
+
         def normalize_item_error(entry, error)
           {
             guid: safe_entry_guid(entry),

data/lib/source_monitor/http.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require "openssl"
 require "faraday"
 require "faraday/retry"
 require "faraday/follow_redirects"
@@ -57,9 +58,31 @@ module SourceMonitor
           connection.headers[key] = value
         end
 
+        configure_ssl(connection, settings)
+
         connection.adapter Faraday.default_adapter
       end
 
+      # Configure SSL to use a proper cert store. Without this, some systems
+      # fail to verify certificate chains that depend on intermediate CAs
+      # (e.g., Medium/Netflix on AWS). OpenSSL::X509::Store#set_default_paths
+      # loads all system-trusted CAs including intermediates.
+      def configure_ssl(connection, settings)
+        connection.ssl.verify = settings.ssl_verify != false
+
+        if settings.ssl_ca_file
+          connection.ssl.ca_file = settings.ssl_ca_file
+        elsif settings.ssl_ca_path
+          connection.ssl.ca_path = settings.ssl_ca_path
+        else
+          connection.ssl.cert_store = default_cert_store
+        end
+      end
+
+      def default_cert_store
+        OpenSSL::X509::Store.new.tap(&:set_default_paths)
+      end
+
       def default_headers(settings)
         base_headers = {
           "User-Agent" => resolve_callable(settings.user_agent).presence || DEFAULT_USER_AGENT,