source_monitor 0.8.1 → 0.9.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 16d5152ca0edc61e1d7f67878516e93769dd43c34daa757c554d36355b998fec
-  data.tar.gz: f93013eedf10c9af8b758b302993f701531b51170a399ea1e09c0f53efb4498d
+  metadata.gz: c3dd7577c86e15ec9926a631998d423b1d2fd1bc18cbdfc83e8d7dc57b6be365
+  data.tar.gz: 65fc2870418c04d3741a98404558ffdd3e8f5a901294681446f337b662dd50f2
 SHA512:
-  metadata.gz: d06eedf5dd22a3cf5c96a252883870b7f416c53f41da5121100b3a66564e7c3021a963512c6355f3aa52be6e30aba23d74b135e47bc1e6fbc17401fc929d5865
-  data.tar.gz: 70655afb16134fecaa25f6547ed8ef05e2fcb7c61fbf3d2b46c7c9b1199eb8f14503db99eb8e165b71927bc9ba4301c3e035e6973eb3fbec85fc3f5be17b1eeb
+  metadata.gz: 115775737ef8f40ea9323932d58e941fccb9b6903371cbdab80d62bcdfbf31d85f44c5eacbcc7b536489483972a2b02e7e547454715dae9af8b19083dce62a62
+  data.tar.gz: 2a285b946a069420c28a1588f580f721f0ea23da6b5c002a0dfb8d149c338c120291e4ad8e77609b65f59ae55bfadec7db6b85c49f92d2097616588d9425c363

data/.claude/skills/sm-domain-model/SKILL.md

@@ -155,7 +155,7 @@ Items use soft delete via `deleted_at` column (NOT default_scope):
 | `url` | presence |
 
 ### Content Delegation
-`scraped_html` and `scraped_content` delegate to `ItemContent`. Setting these values auto-creates/destroys the ItemContent association.
+`scraped_html` and `scraped_content` delegate to `ItemContent`. Setting these values auto-creates the ItemContent association. `ensure_feed_content_record` also creates ItemContent for feed word count tracking. ItemContent is only auto-destroyed when both scraped fields are blank AND the item has no feed content.
 
 ## LogEntry Delegated Type
 

data/.claude/skills/sm-domain-model/reference/model-graph.md

@@ -30,8 +30,8 @@
      v         v
   ItemContent  ScrapeLog
   ===========  (also belongs_to :source)
-  Scraped
-  content
+  Feed + scraped
+  word counts
 ```
 
 ## Association Details
@@ -43,10 +43,11 @@
 - Counter cache: `items_count` on Source (tracks active items)
 
 ### Item -> ItemContent
-- `has_one :item_content` -- Lazy-created when scraped content is assigned
+- `has_one :item_content` -- Created when item has feed content (via `ensure_feed_content_record`) or when scraped content is assigned
 - `dependent: :destroy, autosave: true`
 - `touch: true` on the belongs_to side
-- Content is auto-destroyed when both `scraped_html` and `scraped_content` become blank
+- Stores `feed_word_count` (from `item.content`) and `scraped_word_count` (from `scraped_content`)
+- Content is auto-destroyed only when both scraped fields are blank AND item has no feed content
 
 ### Source -> FetchLog
 - `has_many :fetch_logs, dependent: :destroy`

data/CHANGELOG.md

@@ -15,6 +15,34 @@ All notable changes to this project are documented below. The format follows [Ke
 
 - No unreleased changes yet.
 
+## [0.9.1] - 2026-02-22
+
+### Fixed
+
+- **Feed word counts now always computed.** Items fetched from feeds but never scraped now get an `ItemContent` record with `feed_word_count` automatically. Previously, only scraped items had `ItemContent`, so the `backfill_word_counts` rake task found nothing for feed-only items.
+- **Backfill task creates missing ItemContent records.** `source_monitor:backfill_word_counts` now has a two-phase approach: first creates `ItemContent` for items with feed content but no record, then recomputes all word counts.
+- **ItemContent preserved when item has feed content.** Clearing scraped fields no longer destroys the `ItemContent` record if the item still has feed content (which provides `feed_word_count`).
+
+## [0.9.0] - 2026-02-22
+
+### Added
+
+- **Sources pagination and filtering.** Sources index now paginates (25 per page, configurable) with Previous/Next controls. Dropdown filters for Status, Health, Format, and Scraper Adapter auto-submit on change. Active filters shown as dismissible badges. Text search and dropdown filters compose as intersection and persist across pagination.
+- **Per-source scrape rate limiting.** New `min_scrape_interval` column on sources allows time-based throttling between scrapes. Global default (1.0s) configurable via `config.scraping.min_scrape_interval`. Per-source overrides via the column value. ScrapeItemJob and Enqueuer check last scrape time from `scrape_logs` and re-enqueue with delay when rate-limited.
+- **Word count metrics.** New `feed_word_count` and `scraped_word_count` columns on `item_contents`. Feed content is HTML-stripped before counting; scraped content counted as-is (readability-cleaned). Separate "Avg Feed Words" and "Avg Scraped Words" columns on sources index. Separate "Feed Words" and "Scraped Words" columns on items index and source detail items table. Backfill rake task: `source_monitor:backfill_word_counts`.
+
+### Fixed
+
+- Show `created_at` fallback when `published_at` is nil in items table.
+- Handle source destroy failures with proper error responses instead of silent failures.
+- UI fixes: navigation warning indicator positioning, toast container placement, dashboard table alignment.
+- N+1 query fix: source detail items table now uses `includes(:item_content)`.
+
+### Testing
+
+- 1,175 tests, 3,683 assertions, 0 failures.
+- RuboCop: 0 offenses (423 files).
+
 ## [0.8.1] - 2026-02-21
 
 ### Fixed

data/CLAUDE.md

@@ -4,10 +4,10 @@
 
 ## Active Context
 
-**Milestone:** (none active)
-**Last shipped:** polish-and-reliability (2026-02-21) -- v0.8.0, 3 phases, 1134 tests
-**Previous:** aia-ssl-fix (2026-02-20), upgrade-assurance (2026-02-13)
-**Next action:** /vbw:vibe to start a new milestone
+**Milestone:** polish-and-reliability (extended)
+**Phase:** 4 of 5 -- Bug Fixes & Polish (pending planning)
+**Previous phases:** Backend Fixes, Favicon Support, Toast Stacking (all complete)
+**Next action:** /vbw:vibe to plan and execute Phase 4
 
 ## Key Decisions
 

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    source_monitor (0.8.1)
+    source_monitor (0.9.1)
       cssbundling-rails (~> 1.4)
       faraday (~> 2.9)
       faraday-follow_redirects (~> 0.4)

data/VERSION

@@ -1 +1 @@
-0.8.1
+0.9.1

data/app/assets/builds/source_monitor/application.css

@@ -671,8 +671,8 @@ video {
   top: 50%;
 }
 
-.fm-admin .top-4 {
-  top: 1rem;
+.fm-admin .top-16 {
+  top: 4rem;
 }
 
 .fm-admin .z-10 {
@@ -697,6 +697,14 @@ video {
   margin-bottom: 0.25rem;
 }
 
+.fm-admin .mb-1 {
+  margin-bottom: 0.25rem;
+}
+
+.fm-admin .ml-0\.5 {
+  margin-left: 0.125rem;
+}
+
 .fm-admin .ml-2 {
   margin-left: 0.5rem;
 }
@@ -849,10 +857,30 @@ video {
   width: 20rem;
 }
 
+.fm-admin .w-\[15\%\] {
+  width: 15%;
+}
+
+.fm-admin .w-\[18\%\] {
+  width: 18%;
+}
+
+.fm-admin .w-\[22\%\] {
+  width: 22%;
+}
+
+.fm-admin .w-\[45\%\] {
+  width: 45%;
+}
+
 .fm-admin .w-full {
   width: 100%;
 }
 
+.fm-admin .min-w-\[12rem\] {
+  min-width: 12rem;
+}
+
 .fm-admin .min-w-full {
   min-width: 100%;
 }
@@ -1567,6 +1595,10 @@ video {
   line-height: 1rem;
 }
 
+.fm-admin .font-bold {
+  font-weight: 700;
+}
+
 .fm-admin .font-medium {
   font-weight: 500;
 }
@@ -1902,6 +1934,11 @@ video {
   color: rgb(37 99 235 / var(--tw-text-opacity, 1));
 }
 
+.fm-admin .hover\:text-blue-700:hover {
+  --tw-text-opacity: 1;
+  color: rgb(29 78 216 / var(--tw-text-opacity, 1));
+}
+
 .fm-admin .hover\:text-rose-500:hover {
   --tw-text-opacity: 1;
   color: rgb(244 63 94 / var(--tw-text-opacity, 1));
@@ -2030,10 +2067,6 @@ video {
     width: auto;
   }
 
-  .fm-admin .sm\:min-w-\[16rem\] {
-    min-width: 16rem;
-  }
-
   .fm-admin .sm\:grid-cols-2 {
     grid-template-columns: repeat(2, minmax(0, 1fr));
   }

data/app/controllers/source_monitor/items_controller.rb

@@ -5,7 +5,7 @@ module SourceMonitor
     include ActionView::RecordIdentifier
     include SourceMonitor::SanitizesSearchParams
 
-    searchable_with scope: -> { Item.active.includes(:source) }, default_sorts: [ "published_at desc", "created_at desc" ]
+    searchable_with scope: -> { Item.active.includes(:source, :item_content) }, default_sorts: [ "published_at desc", "created_at desc" ]
 
     PER_PAGE = 25
     SEARCH_FIELD = :title_or_summary_or_url_or_source_name_cont

data/app/controllers/source_monitor/sources_controller.rb

@@ -12,6 +12,7 @@ module SourceMonitor
     searchable_with scope: -> { Source.all }, default_sorts: [ "created_at desc" ]
 
     ITEMS_PREVIEW_LIMIT = SourceMonitor::Scraping::BulkSourceScraper::DEFAULT_PREVIEW_LIMIT
+    PER_PAGE = 25
 
     before_action :set_source, only: %i[show edit update destroy]
 
@@ -21,14 +22,23 @@ module SourceMonitor
       @search_params = sanitized_search_params
       @q = build_search_query
 
-      @sources = @q.result
+      paginator = SourceMonitor::Pagination::Paginator.new(
+        scope: @q.result,
+        page: params[:page],
+        per_page: params[:per_page] || PER_PAGE
+      ).paginate
+
+      @sources = paginator.records
+      @page = paginator.page
+      @has_next_page = paginator.has_next_page
+      @has_previous_page = paginator.has_previous_page
 
       @search_term = @search_params[SEARCH_FIELD.to_s].to_s.strip
       @search_field = SEARCH_FIELD
 
       metrics = SourceMonitor::Analytics::SourcesIndexMetrics.new(
         base_scope: Source.all,
-        result_scope: @sources,
+        result_scope: paginator.records,
         search_params: @search_params
       )
 
@@ -38,12 +48,26 @@ module SourceMonitor
       @fetch_interval_filter = metrics.fetch_interval_filter
       @selected_fetch_interval_bucket = metrics.selected_fetch_interval_bucket
       @item_activity_rates = metrics.item_activity_rates
+
+      source_ids = @sources.map(&:id)
+      if source_ids.any?
+        base = ItemContent.joins(:item).where(sourcemon_items: { source_id: source_ids })
+        @avg_feed_word_counts = base.where.not(feed_word_count: nil)
+                                    .group("sourcemon_items.source_id")
+                                    .average(:feed_word_count)
+        @avg_scraped_word_counts = base.where.not(scraped_word_count: nil)
+                                      .group("sourcemon_items.source_id")
+                                      .average(:scraped_word_count)
+      else
+        @avg_feed_word_counts = {}
+        @avg_scraped_word_counts = {}
+      end
     end
 
     def show
       @recent_fetch_logs = @source.fetch_logs.order(started_at: :desc).limit(5)
       @recent_scrape_logs = @source.scrape_logs.order(started_at: :desc).limit(5)
-      @items = @source.items.recent.limit(ITEMS_PREVIEW_LIMIT)
+      @items = @source.items.recent.includes(:item_content).limit(ITEMS_PREVIEW_LIMIT)
       @bulk_scrape_selection = :current
     end
 
@@ -75,7 +99,17 @@ module SourceMonitor
 
     def destroy
       search_params = sanitized_search_params
-      @source.destroy
+
+      begin
+        unless @source.destroy
+          handle_destroy_failure(search_params, "Could not delete source: #{@source.errors.full_messages.join(', ')}")
+          return
+        end
+      rescue ActiveRecord::InvalidForeignKey
+        handle_destroy_failure(search_params, "Cannot delete source: other records still reference it. Remove dependent records first.")
+        return
+      end
+
       message = "Source deleted"
 
       respond_to do |format|
@@ -132,6 +166,20 @@ module SourceMonitor
       sanitized.start_with?("/") ? sanitized : nil
     end
 
+    def handle_destroy_failure(search_params, error_message)
+      respond_to do |format|
+        format.turbo_stream do
+          responder = SourceMonitor::TurboStreams::StreamResponder.new
+          responder.toast(message: error_message, level: :error)
+          render turbo_stream: responder.render(view_context), status: :unprocessable_entity
+        end
+
+        format.html do
+          redirect_to source_monitor.sources_path(q: search_params), alert: error_message
+        end
+      end
+    end
+
     def enqueue_favicon_fetch(source)
       return unless defined?(ActiveStorage)
       return unless SourceMonitor.config.favicons.enabled?

data/app/jobs/source_monitor/scrape_item_job.rb

@@ -18,6 +18,14 @@ module SourceMonitor
         return
       end
 
+      remaining = time_until_scrape_allowed(source)
+      if remaining&.positive?
+        SourceMonitor::Scraping::State.clear_inflight!(item)
+        self.class.set(wait: remaining.seconds).perform_later(item_id)
+        log("job:deferred", item: item, wait_seconds: remaining)
+        return
+      end
+
       SourceMonitor::Scraping::State.mark_processing!(item)
       SourceMonitor::Scraping::ItemScraper.new(item:, source:).call
       log("job:completed", item: item, status: item.scrape_status)
@@ -31,6 +39,18 @@ module SourceMonitor
 
     private
 
+    def time_until_scrape_allowed(source)
+      interval = source.min_scrape_interval || SourceMonitor.config.scraping.min_scrape_interval
+      return nil if interval.nil? || interval <= 0
+
+      last_scrape_at = source.scrape_logs.maximum(:started_at)
+      return nil unless last_scrape_at
+
+      elapsed = Time.current - last_scrape_at
+      remaining = interval - elapsed
+      remaining.positive? ? remaining.ceil : nil
+    end
+
     def log(stage, item: nil, item_id: nil, **extra)
       return unless defined?(Rails) && Rails.respond_to?(:logger) && Rails.logger
 

data/app/models/source_monitor/item.rb

@@ -33,6 +33,15 @@ module SourceMonitor
 
     SourceMonitor::ModelExtensions.register(self, :item)
 
+    # Creates an ItemContent record for feed word count computation when one doesn't exist.
+    # The before_save callback on ItemContent will compute feed_word_count from item.content.
+    def ensure_feed_content_record
+      return if item_content.present?
+      return if content.blank?
+
+      create_item_content!
+    end
+
     class << self
       def ransackable_attributes(_auth_object = nil)
         %w[title summary url published_at created_at scrape_status]
@@ -91,6 +100,7 @@ module SourceMonitor
     def cleanup_item_content_if_blank
       return unless item_content
       return if item_content.scraped_html.present? || item_content.scraped_content.present?
+      return if content.present?
 
       if item_content.persisted?
         item_content.mark_for_destruction

data/app/models/source_monitor/item_content.rb

@@ -8,6 +8,35 @@ module SourceMonitor
 
     has_many_attached :images if defined?(ActiveStorage)
 
+    before_save :compute_word_counts
+
     SourceMonitor::ModelExtensions.register(self, :item_content)
+
+    def total_word_count
+      [ scraped_word_count, feed_word_count ].compact.max || 0
+    end
+
+    private
+
+    def compute_word_counts
+      compute_scraped_word_count
+      compute_feed_word_count
+    end
+
+    def compute_scraped_word_count
+      return unless scraped_content_changed? || new_record? || (scraped_word_count.nil? && scraped_content.present?)
+
+      self.scraped_word_count = scraped_content.present? ? scraped_content.split.size : nil
+    end
+
+    def compute_feed_word_count
+      content = item&.content
+      if content.blank?
+        self.feed_word_count = nil
+      else
+        stripped = ActionView::Base.full_sanitizer.sanitize(content)
+        self.feed_word_count = stripped.present? ? stripped.split.size : nil
+      end
+    end
   end
 end

data/app/models/source_monitor/source.rb

@@ -62,7 +62,8 @@ module SourceMonitor
       end
 
       def ransackable_attributes(_auth_object = nil)
-        %w[name feed_url website_url created_at fetch_interval_minutes items_count last_fetched_at]
+        %w[name feed_url website_url created_at fetch_interval_minutes items_count last_fetched_at
+           active health_status feed_format scraper_adapter]
       end
 
       def ransackable_associations(_auth_object = nil)
@@ -103,6 +104,13 @@ module SourceMonitor
       update_columns(items_count: actual_count)
     end
 
+    def avg_word_count
+      items.joins(:item_content)
+           .where.not(sourcemon_item_contents: { scraped_word_count: nil })
+           .average("sourcemon_item_contents.scraped_word_count")
+           &.round
+    end
+
     private
 
     def health_auto_pause_threshold_within_bounds

data/app/views/layouts/source_monitor/application.html.erb

@@ -13,7 +13,7 @@
 </head>
 <body class="fm-admin">
   <%= turbo_stream_from "source_monitor_notifications" %>
-  <div class="pointer-events-none fixed inset-x-0 top-4 z-50 flex justify-end px-6"
+  <div class="pointer-events-none fixed inset-x-0 top-16 z-50 flex justify-end px-6"
        data-controller="notification-container">
     <div class="flex w-full max-w-sm flex-col items-end gap-3">
       <div id="source_monitor_notifications"

data/app/views/source_monitor/dashboard/_fetch_schedule.html.erb

@@ -25,13 +25,13 @@
 
         <% if group.sources.any? %>
           <div class="overflow-x-auto px-5 py-4">
-            <table class="min-w-full divide-y divide-slate-200 text-left text-sm">
+            <table class="min-w-full table-fixed divide-y divide-slate-200 text-left text-sm">
               <thead class="text-xs font-semibold uppercase tracking-wide text-slate-500">
                 <tr>
-                  <th scope="col" class="px-4 py-2">Source</th>
-                  <th scope="col" class="px-4 py-2">Status</th>
-                  <th scope="col" class="px-4 py-2">Next Fetch</th>
-                  <th scope="col" class="px-4 py-2">Interval</th>
+                  <th scope="col" class="w-[45%] px-4 py-2">Source</th>
+                  <th scope="col" class="w-[15%] px-4 py-2">Status</th>
+                  <th scope="col" class="w-[22%] px-4 py-2">Next Fetch</th>
+                  <th scope="col" class="w-[18%] px-4 py-2">Interval</th>
                 </tr>
               </thead>
               <tbody class="divide-y divide-slate-100 text-slate-700">

data/app/views/source_monitor/import_sessions/steps/_confirm.html.erb

@@ -112,7 +112,8 @@
 
       <%= form_with model: import_session,
             url: source_monitor.step_import_session_path(import_session, step: "confirm"),
-            method: :patch do |form| %>
+            method: :patch,
+            data: { action: "submit->confirm-navigation#disable" } do |form| %>
         <%= hidden_field_tag :next_step, "confirm", name: "import_session[next_step]" %>
         <%= form.submit "Start import", class: "inline-flex items-center rounded-md bg-blue-600 px-4 py-2 text-sm font-semibold text-white shadow hover:bg-blue-500" %>
       <% end %>

data/app/views/source_monitor/items/_details.html.erb

@@ -36,7 +36,11 @@
           <span class="text-slate-400">No source</span>
         <% end %>
         &middot;
-        <%= item.published_at ? item.published_at.strftime("%b %d, %Y %H:%M %Z") : "Unpublished" %>
+        <% if item.published_at %>
+          <%= item.published_at.strftime("%b %d, %Y %H:%M %Z") %>
+        <% else %>
+          <span class="text-slate-400"><%= item.created_at.strftime("%b %d, %Y %H:%M %Z") %></span>
+        <% end %>
       </p>
     </div>
   </div>
@@ -56,7 +60,7 @@
                "URL" => (item.url.present? ? external_link_to(item.url, item.url, class: "text-slate-900 hover:text-blue-500") : "\u2014"),
                "Canonical URL" => (item.canonical_url.present? ? external_link_to(item.canonical_url, item.canonical_url, class: "text-slate-900 hover:text-blue-500") : "\u2014"),
                "Author" => item.author || "—",
-               "Published At" => (item.published_at&.strftime("%b %d, %Y %H:%M %Z") || "—"),
+               "Published At" => (item.published_at&.strftime("%b %d, %Y %H:%M %Z") || item.created_at.strftime("%b %d, %Y %H:%M %Z")),
                "Updated At (Source)" => (item.updated_at_source&.strftime("%b %d, %Y %H:%M %Z") || "—"),
                "Language" => item.language || "—",
                "Categories" => categories_list.present? ? categories_list.join(", ") : "—",
@@ -126,7 +130,9 @@
         </div>
         <div class="space-y-2 px-5 py-4 text-sm text-slate-700">
           <p><span class="font-medium text-slate-600">Comments:</span> <%= item.comments_count || 0 %></p>
-          <p><span class="font-medium text-slate-600">Feed Items in Source:</span> <%= source&.items_count || "—" %></p>
+          <p><span class="font-medium text-slate-600">Feed Items in Source:</span> <%= source&.items_count || "\u2014" %></p>
+          <p><span class="font-medium text-slate-600">Feed Word Count:</span> <%= item.item_content&.feed_word_count || "\u2014" %></p>
+          <p><span class="font-medium text-slate-600">Scraped Word Count:</span> <%= item.item_content&.scraped_word_count || "\u2014" %></p>
         </div>
       </div>
 

data/app/views/source_monitor/items/index.html.erb

@@ -64,6 +64,8 @@
               </span>
             </th>
             <th scope="col" class="px-6 py-3">Scrape Status</th>
+            <th scope="col" class="px-6 py-3">Feed Words</th>
+            <th scope="col" class="px-6 py-3">Scraped Words</th>
           </tr>
         </thead>
         <tbody class="divide-y divide-slate-100 text-slate-700">
@@ -91,7 +93,11 @@
                 <% end %>
               </td>
               <td class="px-6 py-4 text-xs text-slate-500">
-                <%= item.published_at ? item.published_at.strftime("%b %d, %Y %H:%M") : "Unpublished" %>
+                <% if item.published_at %>
+                  <%= item.published_at.strftime("%b %d, %Y %H:%M") %>
+                <% else %>
+                  <span class="text-slate-400"><%= item.created_at.strftime("%b %d, %Y %H:%M") %></span>
+                <% end %>
               </td>
               <td class="px-6 py-4 text-xs">
                 <% status_label, status_classes =
@@ -107,12 +113,18 @@
                      end %>
                 <span class="inline-flex items-center rounded-full px-3 py-1 font-semibold <%= status_classes %>"><%= status_label %></span>
               </td>
+              <td class="px-6 py-4 text-xs text-slate-500">
+                <%= item.item_content&.feed_word_count || "\u2014" %>
+              </td>
+              <td class="px-6 py-4 text-xs text-slate-500">
+                <%= item.item_content&.scraped_word_count || "\u2014" %>
+              </td>
             </tr>
           <% end %>
 
           <% if @items.blank? %>
             <tr>
-              <td colspan="4" class="px-6 py-6 text-center text-sm text-slate-500">
+              <td colspan="6" class="px-6 py-6 text-center text-sm text-slate-500">
                 No items found.
               </td>
             </tr>

data/app/views/source_monitor/sources/_details.html.erb

@@ -2,7 +2,7 @@
 <% recent_fetch_logs = source.fetch_logs.order(started_at: :desc).limit(5) %>
 <% recent_scrape_logs = source.scrape_logs.order(started_at: :desc).limit(5) %>
 <% preview_limit = SourceMonitor::Scraping::BulkSourceScraper::DEFAULT_PREVIEW_LIMIT %>
-<% items = source.items.recent.limit(preview_limit) %>
+<% items = source.items.recent.includes(:item_content).limit(preview_limit) %>
 <% fetch_status = async_status_badge(source.fetch_status) %>
 <% health_status_override = local_assigns[:health_status_override] %>
 
@@ -281,6 +281,8 @@
                   <th scope="col" class="px-5 py-3">Tags</th>
                   <th scope="col" class="px-5 py-3">Published</th>
                   <th scope="col" class="px-5 py-3">Scrape Status</th>
+                  <th scope="col" class="px-5 py-3">Feed Words</th>
+                  <th scope="col" class="px-5 py-3">Scraped Words</th>
                 </tr>
               </thead>
               <tbody class="divide-y divide-slate-100 text-slate-700">
@@ -299,7 +301,11 @@
                     <td class="px-5 py-4 text-xs text-slate-500"><%= categories.present? ? categories.join(", ") : "—" %></td>
                     <td class="px-5 py-4 text-xs text-slate-500"><%= tags.present? ? tags.join(", ") : "—" %></td>
                     <td class="px-5 py-4 text-xs text-slate-500">
-                      <%= item.published_at ? item.published_at.strftime("%b %d, %Y %H:%M") : "Unpublished" %>
+                      <% if item.published_at %>
+                        <%= item.published_at.strftime("%b %d, %Y %H:%M") %>
+                      <% else %>
+                        <span class="text-slate-400"><%= item.created_at.strftime("%b %d, %Y %H:%M") %></span>
+                      <% end %>
                     </td>
                     <td class="px-5 py-4 text-xs">
                       <% scrape_badge = item_scrape_status_badge(item: item, source: source) %>
@@ -312,6 +318,12 @@
                         <%= scrape_badge[:label] %>
                       </span>
                     </td>
+                    <td class="px-5 py-4 text-xs text-slate-500">
+                      <%= item.item_content&.feed_word_count || "\u2014" %>
+                    </td>
+                    <td class="px-5 py-4 text-xs text-slate-500">
+                      <%= item.item_content&.scraped_word_count || "\u2014" %>
+                    </td>
                   </tr>
                 <% end %>
               </tbody>

data/app/views/source_monitor/sources/_row.html.erb

@@ -1,4 +1,6 @@
 <% rate_map = local_assigns[:item_activity_rates] || {} %>
+<% avg_feed_words_map = local_assigns[:avg_feed_word_counts] || {} %>
+<% avg_scraped_words_map = local_assigns[:avg_scraped_word_counts] || {} %>
 <% activity_rate = rate_map.fetch(source.id, 0.0) %>
 <% health_status_override = local_assigns[:health_status_override] %>
 <% health_status = if !source.active?
@@ -66,6 +68,12 @@
     <%= number_with_precision(activity_rate, precision: 2) %>
     <span class="text-xs text-slate-500">/ day</span>
   </td>
+  <td class="px-6 py-4 text-sm text-slate-500">
+    <%= avg_feed_words_map[source.id]&.round || "\u2014" %>
+  </td>
+  <td class="px-6 py-4 text-sm text-slate-500">
+    <%= avg_scraped_words_map[source.id]&.round || "\u2014" %>
+  </td>
   <td class="px-6 py-4 text-xs text-slate-500">
     <%= source.last_fetched_at ? source.last_fetched_at.strftime("%b %d, %H:%M") : "Never" %>
   </td>

data/app/views/source_monitor/sources/index.html.erb

@@ -5,22 +5,42 @@
       <h1 class="text-3xl font-semibold text-slate-900">Sources</h1>
       <p class="mt-1 text-sm text-slate-500">Manage feed endpoints and scraping settings.</p>
     </div>
-    <div class="flex w-full flex-col gap-3 sm:w-auto sm:flex-row sm:items-center">
-      <%= search_form_for @q, url: source_monitor.sources_path, method: :get, html: { class: "w-full sm:w-auto", data: { turbo_frame: "source_monitor_sources_table" } } do |form| %>
-        <%= form.label @search_field, "Search sources", class: "sr-only" %>
-        <div class="flex rounded-md shadow-sm">
-          <%= form.search_field @search_field, placeholder: "Search name or URL…", class: "w-full rounded-l-md border border-slate-200 bg-white px-3 py-2 text-sm text-slate-700 focus:border-blue-500 focus:outline-none focus:ring-1 focus:ring-blue-500 sm:min-w-[16rem]" %>
-          <%= form.submit "Search", class: "rounded-r-md bg-blue-600 px-4 py-2 text-sm font-semibold text-white hover:bg-blue-500" %>
-        </div>
-      <% end %>
-      <div class="flex flex-col gap-2 sm:flex-row sm:items-center sm:gap-3">
-        <%= link_to "New Source", source_monitor.new_source_path, class: "inline-flex items-center justify-center rounded-md bg-blue-600 px-4 py-2 text-sm font-semibold text-white shadow hover:bg-blue-500" %>
-        <%= link_to "Import OPML", source_monitor.new_import_session_path,
-              class: "inline-flex items-center justify-center rounded-md border border-slate-200 px-4 py-2 text-sm font-semibold text-slate-700 shadow-sm hover:bg-slate-50" %>
-      </div>
+    <div class="flex flex-col gap-2 sm:flex-row sm:items-center sm:gap-3">
+      <%= link_to "New Source", source_monitor.new_source_path, class: "inline-flex items-center justify-center rounded-md bg-blue-600 px-4 py-2 text-sm font-semibold text-white shadow hover:bg-blue-500" %>
+      <%= link_to "Import OPML", source_monitor.new_import_session_path,
+            class: "inline-flex items-center justify-center rounded-md border border-slate-200 px-4 py-2 text-sm font-semibold text-slate-700 shadow-sm hover:bg-slate-50" %>
     </div>
   </div>
 
+  <%= search_form_for @q, url: source_monitor.sources_path, method: :get, html: { class: "flex flex-wrap items-end gap-3", data: { turbo_frame: "source_monitor_sources_table" } } do |form| %>
+    <div class="flex-1 min-w-[12rem]">
+      <%= form.label @search_field, "Search sources", class: "sr-only" %>
+      <div class="flex rounded-md shadow-sm">
+        <%= form.search_field @search_field, placeholder: "Search name or URL…", class: "w-full rounded-l-md border border-slate-200 bg-white px-3 py-2 text-sm text-slate-700 focus:border-blue-500 focus:outline-none focus:ring-1 focus:ring-blue-500" %>
+        <%= form.submit "Search", class: "rounded-r-md bg-blue-600 px-4 py-2 text-sm font-semibold text-white hover:bg-blue-500" %>
+      </div>
+    </div>
+    <div class="flex flex-wrap items-end gap-2">
+      <div>
+        <%= form.label :active_eq, "Status", class: "block text-xs font-medium text-slate-500 mb-1" %>
+        <%= form.select :active_eq, options_for_select([["All Statuses", ""], ["Active", "true"], ["Paused", "false"]], @search_params["active_eq"].to_s), {}, class: "rounded-md border border-slate-200 bg-white px-2 py-2 text-sm text-slate-700 focus:border-blue-500 focus:outline-none focus:ring-1 focus:ring-blue-500", onchange: "this.form.requestSubmit()" %>
+      </div>
+      <div>
+        <%= form.label :health_status_eq, "Health", class: "block text-xs font-medium text-slate-500 mb-1" %>
+        <%= form.select :health_status_eq, options_for_select([["All Health", ""], ["Healthy", "healthy"], ["Warning", "warning"], ["Declining", "declining"], ["Critical", "critical"]], @search_params["health_status_eq"].to_s), {}, class: "rounded-md border border-slate-200 bg-white px-2 py-2 text-sm text-slate-700 focus:border-blue-500 focus:outline-none focus:ring-1 focus:ring-blue-500", onchange: "this.form.requestSubmit()" %>
+      </div>
+      <div>
+        <%= form.label :feed_format_eq, "Format", class: "block text-xs font-medium text-slate-500 mb-1" %>
+        <%= form.select :feed_format_eq, options_for_select([["All Formats", ""], ["RSS", "rss"], ["Atom", "atom"], ["JSON", "json"]], @search_params["feed_format_eq"].to_s), {}, class: "rounded-md border border-slate-200 bg-white px-2 py-2 text-sm text-slate-700 focus:border-blue-500 focus:outline-none focus:ring-1 focus:ring-blue-500", onchange: "this.form.requestSubmit()" %>
+      </div>
+      <div>
+        <% adapter_options = SourceMonitor::Source.distinct.where.not(scraper_adapter: [nil, ""]).order(:scraper_adapter).pluck(:scraper_adapter) %>
+        <%= form.label :scraper_adapter_eq, "Adapter", class: "block text-xs font-medium text-slate-500 mb-1" %>
+        <%= form.select :scraper_adapter_eq, options_for_select([["All Adapters", ""]] + adapter_options.map { |a| [a.titleize, a] }, @search_params["scraper_adapter_eq"].to_s), {}, class: "rounded-md border border-slate-200 bg-white px-2 py-2 text-sm text-slate-700 focus:border-blue-500 focus:outline-none focus:ring-1 focus:ring-blue-500", onchange: "this.form.requestSubmit()" %>
+      </div>
+    </div>
+  <% end %>
+
   <%= render "source_monitor/sources/import_history_panel", import_histories: @recent_import_histories %>
 
   <%= render "source_monitor/sources/fetch_interval_heatmap",
@@ -30,7 +50,10 @@
 
   <div class="overflow-hidden rounded-lg border border-slate-200 bg-white shadow-sm">
     <%= turbo_frame_tag "source_monitor_sources_table" do %>
-      <% if @search_term.present? || @fetch_interval_filter.present? %>
+      <% dropdown_filter_keys = %w[active_eq health_status_eq feed_format_eq scraper_adapter_eq] %>
+      <% active_dropdown_filters = dropdown_filter_keys.select { |k| @search_params[k].present? } %>
+      <% has_any_filter = @search_term.present? || @fetch_interval_filter.present? || active_dropdown_filters.any? %>
+      <% if has_any_filter %>
         <div class="rounded-t-lg border-b border-blue-100 bg-blue-50 px-4 py-3 text-xs text-blue-700">
           <% if @search_term.present? %>
             <% clear_search_query = @search_params.dup %>
@@ -56,6 +79,32 @@
                     class: "font-medium text-blue-600 hover:text-blue-500" %>
             </div>
           <% end %>
+
+          <% if active_dropdown_filters.any? %>
+            <div class="mt-1 flex flex-wrap items-center gap-2">
+              <span>Filtered by</span>
+              <% filter_labels = {
+                "active_eq" => @search_params["active_eq"] == "true" ? "Status: Active" : "Status: Paused",
+                "health_status_eq" => "Health: #{@search_params['health_status_eq']&.titleize}",
+                "feed_format_eq" => "Format: #{@search_params['feed_format_eq']&.upcase}",
+                "scraper_adapter_eq" => "Adapter: #{@search_params['scraper_adapter_eq']&.titleize}"
+              } %>
+              <% active_dropdown_filters.each do |filter_key| %>
+                <% clear_query = @search_params.dup %>
+                <% clear_query.delete(filter_key) %>
+                <% clear_query = if clear_query.respond_to?(:compact_blank)
+                  clear_query.compact_blank
+                else
+                  clear_query.reject { |_k, v| v.respond_to?(:blank?) ? v.blank? : v.nil? }
+                end %>
+                <% clear_path = clear_query.empty? ? source_monitor.sources_path : source_monitor.sources_path(q: clear_query) %>
+                <span class="inline-flex items-center gap-1 rounded-full bg-blue-100 px-2 py-0.5 text-xs font-medium text-blue-700">
+                  <%= filter_labels[filter_key] %>
+                  <%= link_to "×", clear_path, class: "ml-0.5 font-bold text-blue-500 hover:text-blue-700", data: { turbo_frame: "source_monitor_sources_table" } %>
+                </span>
+              <% end %>
+            </div>
+          <% end %>
         </div>
       <% end %>
       <table class="min-w-full divide-y divide-slate-200 text-left text-sm">
@@ -120,6 +169,8 @@
               </span>
             </th>
             <th scope="col" class="px-6 py-3">New Items / Day</th>
+            <th scope="col" class="px-6 py-3">Avg Feed Words</th>
+            <th scope="col" class="px-6 py-3">Avg Scraped Words</th>
             <th scope="col"
                 class="px-6 py-3"
                 data-sort-column="last_fetched_at"
@@ -148,12 +199,39 @@
                 as: :source,
                 locals: {
                   item_activity_rates: @item_activity_rates,
+                  avg_feed_word_counts: @avg_feed_word_counts,
+                  avg_scraped_word_counts: @avg_scraped_word_counts,
                   search_params: @search_params
                 } %>
 
           <%= render("source_monitor/sources/empty_state_row") if @sources.blank? %>
         </tbody>
       </table>
+      <div class="flex flex-col items-center gap-3 border-t border-slate-200 px-6 py-4 sm:flex-row sm:justify-between">
+        <div class="text-xs text-slate-500">
+          Page <%= @page %>
+        </div>
+        <div class="flex gap-2">
+          <% prev_params = { page: @page - 1 } %>
+          <% prev_params[:q] = @search_params if @search_params.present? %>
+          <% prev_params[:per_page] = params[:per_page] if params[:per_page].present? %>
+          <% next_params = { page: @page + 1 } %>
+          <% next_params[:q] = @search_params if @search_params.present? %>
+          <% next_params[:per_page] = params[:per_page] if params[:per_page].present? %>
+
+          <% if @has_previous_page %>
+            <%= link_to "Previous", source_monitor.sources_path(prev_params), class: "inline-flex items-center rounded-md border border-slate-300 px-3 py-2 text-sm font-medium text-slate-700 hover:bg-slate-50", data: { turbo_frame: "source_monitor_sources_table" } %>
+          <% else %>
+            <span class="inline-flex items-center rounded-md border border-slate-200 px-3 py-2 text-sm font-medium text-slate-300">Previous</span>
+          <% end %>
+
+          <% if @has_next_page %>
+            <%= link_to "Next", source_monitor.sources_path(next_params), class: "inline-flex items-center rounded-md border border-slate-300 px-3 py-2 text-sm font-medium text-slate-700 hover:bg-slate-50", data: { turbo_frame: "source_monitor_sources_table" } %>
+          <% else %>
+            <span class="inline-flex items-center rounded-md border border-slate-200 px-3 py-2 text-sm font-medium text-slate-300">Next</span>
+          <% end %>
+        </div>
+      </div>
     <% end %>
   </div>
 </div>

data/config/coverage_baseline.json

@@ -240,6 +240,13 @@
     59,
     63
   ],
+  "lib/source_monitor/configuration/scraping_settings.rb": [
+    31,
+    45,
+    46,
+    48,
+    49
+  ],
   "lib/source_monitor/configuration/validation_definition.rb": [
     15,
     17,

data/db/migrate/20260222120000_add_min_scrape_interval_to_sources.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+class AddMinScrapeIntervalToSources < ActiveRecord::Migration[8.1]
+  def change
+    add_column :sourcemon_sources, :min_scrape_interval, :decimal, precision: 10, scale: 2, null: true, default: nil
+  end
+end

data/db/migrate/20260222194201_add_word_counts_to_item_contents.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+class AddWordCountsToItemContents < ActiveRecord::Migration[8.0]
+  def change
+    add_column :sourcemon_item_contents, :scraped_word_count, :integer
+    add_column :sourcemon_item_contents, :feed_word_count, :integer
+  end
+end

data/lib/source_monitor/configuration/scraping_settings.rb

@@ -3,10 +3,11 @@
 module SourceMonitor
   class Configuration
     class ScrapingSettings
-      attr_accessor :max_in_flight_per_source, :max_bulk_batch_size
+      attr_accessor :max_in_flight_per_source, :max_bulk_batch_size, :min_scrape_interval
 
       DEFAULT_MAX_IN_FLIGHT = nil
       DEFAULT_MAX_BULK_BATCH_SIZE = 100
+      DEFAULT_MIN_SCRAPE_INTERVAL = 1.0
 
       def initialize
         reset!
@@ -15,6 +16,7 @@ module SourceMonitor
       def reset!
         @max_in_flight_per_source = DEFAULT_MAX_IN_FLIGHT
         @max_bulk_batch_size = DEFAULT_MAX_BULK_BATCH_SIZE
+        @min_scrape_interval = DEFAULT_MIN_SCRAPE_INTERVAL
       end
 
       def max_in_flight_per_source=(value)
@@ -25,6 +27,10 @@ module SourceMonitor
         @max_bulk_batch_size = normalize_numeric(value)
       end
 
+      def min_scrape_interval=(value)
+        @min_scrape_interval = normalize_numeric_float(value)
+      end
+
       private
 
       def normalize_numeric(value)
@@ -34,6 +40,14 @@ module SourceMonitor
         integer = value.respond_to?(:to_i) ? value.to_i : value
         integer.positive? ? integer : nil
       end
+
+      def normalize_numeric_float(value)
+        return nil if value.nil?
+        return nil if value == ""
+
+        float = value.respond_to?(:to_f) ? value.to_f : value
+        float.positive? ? float : nil
+      end
     end
   end
 end

data/lib/source_monitor/items/item_creator.rb

@@ -120,6 +120,7 @@ module SourceMonitor
         new_item = source.items.new
         apply_attributes(new_item, attributes)
         new_item.save!
+        new_item.ensure_feed_content_record
         Result.new(item: new_item, status: :created)
       rescue ActiveRecord::RecordNotUnique
         handle_concurrent_duplicate(attributes, raw_guid_present:)

data/lib/source_monitor/scraping/enqueuer.rb

@@ -14,8 +14,12 @@ module SourceMonitor
           status == :already_enqueued
         end
 
+        def deferred?
+          status == :deferred
+        end
+
         def failure?
-          !enqueued? && !already_enqueued?
+          !enqueued? && !already_enqueued? && !deferred?
         end
       end
 
@@ -44,6 +48,8 @@ module SourceMonitor
         already_queued = false
         rate_limited = false
         rate_limit_info = nil
+        time_limited = false
+        time_limit_info = nil
 
         item.with_lock do
           item.reload
@@ -61,6 +67,13 @@ module SourceMonitor
             next
           end
 
+          limited, t_info = time_rate_limited?
+          if limited
+            time_limited = true
+            time_limit_info = t_info
+            next
+          end
+
           SourceMonitor::Scraping::State.mark_pending!(item, broadcast: false, lock: false)
         end
 
@@ -75,6 +88,14 @@ module SourceMonitor
           return Result.new(status: :rate_limited, message:, item: item)
         end
 
+        if time_limited
+          wait_seconds = time_limit_info[:wait_seconds]
+          job_class.set(wait: wait_seconds.seconds).perform_later(item.id)
+          message = "Scrape deferred: source was scraped #{time_limit_info[:interval]}s ago, re-enqueued with #{wait_seconds}s delay."
+          log("enqueue:deferred", item:, wait_seconds:, interval: time_limit_info[:interval])
+          return Result.new(status: :deferred, message:, item: item)
+        end
+
         job_class.perform_later(item.id)
         log("enqueue:job_enqueued", item:, job_class: job_class.name)
         Result.new(status: :enqueued, message: "Scrape has been enqueued for processing.", item: item)
@@ -105,6 +126,22 @@ module SourceMonitor
         nil
       end
 
+      def time_rate_limited?
+        interval = source.min_scrape_interval || SourceMonitor.config.scraping.min_scrape_interval
+        return [ false, nil ] if interval.nil? || interval <= 0
+
+        last_scrape_at = source.scrape_logs.maximum(:started_at)
+        return [ false, nil ] unless last_scrape_at
+
+        elapsed = Time.current - last_scrape_at
+        if elapsed < interval
+          wait_seconds = (interval - elapsed).ceil
+          [ true, { wait_seconds:, interval:, last_scrape_at: } ]
+        else
+          [ false, nil ]
+        end
+      end
+
       def rate_limit_exhausted?
         limit = SourceMonitor.config.scraping.max_in_flight_per_source
         return [ false, nil ] unless limit

data/lib/source_monitor/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SourceMonitor
-  VERSION = "0.8.1"
+  VERSION = "0.9.1"
 end

data/lib/tasks/source_monitor_tasks.rake

@@ -1,6 +1,34 @@
 # frozen_string_literal: true
 
 namespace :source_monitor do
+  desc "Backfill word counts for existing item_content records."
+  task backfill_word_counts: :environment do
+    # Phase 1: Create ItemContent for items with feed content but no ItemContent
+    items_needing_content = SourceMonitor::Item
+      .where.not(content: [ nil, "" ])
+      .where.missing(:item_content)
+
+    created = 0
+    items_needing_content.find_each do |item|
+      item.ensure_feed_content_record
+      created += 1
+      puts "Created #{created} missing ItemContent records..." if (created % 100).zero?
+    end
+    puts "Created #{created} ItemContent records for feed-only items." if created > 0
+
+    # Phase 2: Recompute word counts for all existing ItemContent
+    total = SourceMonitor::ItemContent.count
+    processed = 0
+
+    SourceMonitor::ItemContent.find_each do |content|
+      content.save!
+      processed += 1
+      puts "Processed #{processed}/#{total} records..." if (processed % 100).zero?
+    end
+
+    puts "Done. Backfilled word counts for #{processed} records (#{created} newly created)."
+  end
+
   namespace :cleanup do
     desc "Run retention pruning across sources. Accepts SOURCE_IDS and SOFT_DELETE env vars."
     task items: :environment do

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: source_monitor
 version: !ruby/object:Gem::Version
-  version: 0.8.1
+  version: 0.9.1
 platform: ruby
 authors:
 - dchuk
@@ -490,6 +490,8 @@ files:
 - db/migrate/20251124153000_add_health_fields_to_import_sessions.rb
 - db/migrate/20251125094500_create_import_histories.rb
 - db/migrate/20260210204022_add_composite_index_to_log_entries.rb
+- db/migrate/20260222120000_add_min_scrape_interval_to_sources.rb
+- db/migrate/20260222194201_add_word_counts_to_item_contents.rb
 - docs/configuration.md
 - docs/deployment.md
 - docs/gh-cli-workflow.md