source_monitor 0.10.2 → 0.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 65fe10d6cb1f109a3f95af0c20ee5abbcbfc5c9e0bd188dc65d6c118662db667
-  data.tar.gz: 5ba3db65f470b9569f543a137df5dab30a8e730ab2a4d396d6160b8d88011c34
+  metadata.gz: 522d0a31c2068abc659c59ab47fb1b96cd4855729257a823aa08877c3593552c
+  data.tar.gz: 0e4278b1f628e7439befb9433f67618eb587f8f24860eef2e5f7db92207affd7
 SHA512:
-  metadata.gz: d6b374cac60f5f5d057212033ec101a859a35e3c1d09a3ba7dc7f64e15bfb2554e24c8648b96c06855cd176ab297d5e395198e97ff4fb4f02e33b69e29182a19
-  data.tar.gz: '00954ab911a13c252358a44fd5c822d928c8ab2e2c6f7b9b2bc00df21af0035486edbc45903bd805ac2904539f49e5baa92508ce6e5ce1c788c5f1516f53f3a2'
+  metadata.gz: 65e8e108bb043d0065a0fed294f02d5b3d232ceab7b9bb153ba1d3130cdad8ae5dfd1b0aa6a3d9c7bd172dfbd593bfc015a2d9b6b5c665435452586ed2209ba0
+  data.tar.gz: e691d60468521daccab51c9ae8a71ddfbb695edbc62df88401a93252250de68cfd9a36d33e3792ce3026d4e9a666fb2be8b8707b3910e83f2c21efa2cd32c2da

data/.claude/agent-memory/vbw-vbw-debugger/MEMORY.md

@@ -0,0 +1,15 @@
+# Debugger Memory
+
+## Rails Association Cache Pollution Pattern
+- `source.items.new` AND `Item.new(source: source)` both add to the loaded association cache via inverse_of
+- Only `Item.new(source_id: source.id)` truly bypasses inverse_of and avoids cache pollution
+- When unsaved/invalid records are in a loaded has_many cache, `parent.update!` triggers auto-save and fails with `RecordInvalid: Items is invalid`
+- `update_columns` bypasses all callbacks and auto-save, safe to use with polluted caches
+- After `update_columns`, call `reload` so the in-memory object reflects DB state
+
+## Test Patterns
+- Use `clean_source_monitor_tables!` in setup for blank-slate DB
+- `create_source!` is the factory helper (in test_helper.rb)
+- WebMock stubs + VCR cassettes for HTTP; `stub_request(:get, url)`
+- Stub class methods with `singleton_class.define_method` pattern
+- Always restore stubs in `ensure` block

data/.claude/skills/sm-configuration-setting/reference/settings-catalog.md

@@ -75,8 +75,7 @@ Has `reset!` method. All attributes are plain `attr_accessor`.
 | Attribute | Type | Default | Description |
 |-----------|------|---------|-------------|
 | `window_size` | Integer | `20` | Rolling window of fetches for health calc |
-| `healthy_threshold` | Float | `0.8` | Success rate above = healthy |
-| `warning_threshold` | Float | `0.5` | Success rate above = warning |
+| `healthy_threshold` | Float | `0.8` | Success rate above = working |
 | `auto_pause_threshold` | Float | `0.2` | Success rate below = auto-pause |
 | `auto_resume_threshold` | Float | `0.6` | Success rate above = auto-resume |
 | `auto_pause_cooldown_minutes` | Integer | `60` | Cooldown before auto-resume check |
@@ -91,8 +90,9 @@ Has `reset!` method. All attributes are plain `attr_accessor`.
 
 | Attribute | Type | Default | Description |
 |-----------|------|---------|-------------|
-| `max_in_flight_per_source` | Integer/nil | `25` | Max concurrent scrape jobs per source |
+| `max_in_flight_per_source` | Integer/nil | `nil` | Max concurrent scrape jobs per source |
 | `max_bulk_batch_size` | Integer/nil | `100` | Max items in a bulk scrape batch |
+| `scrape_recommendation_threshold` | Integer/nil | `200` | Avg feed word count threshold for scrape recommendations |
 
 Has `reset!` method. Custom setters normalize values:
 - `nil` -> `nil`

data/.claude/skills/sm-configure/reference/configuration-reference.md

@@ -95,8 +95,7 @@ Tunes automatic pause/resume heuristics per source.
 | Setting | Type | Default | Description |
 |---|---|---|---|
 | `window_size` | Integer | `20` | Number of fetch attempts to evaluate |
-| `healthy_threshold` | Float | `0.8` | Success ratio for "healthy" badge |
-| `warning_threshold` | Float | `0.5` | Success ratio for "warning" badge |
+| `healthy_threshold` | Float | `0.8` | Success ratio for "working" status |
 | `auto_pause_threshold` | Float | `0.2` | Auto-pause source below this ratio |
 | `auto_resume_threshold` | Float | `0.6` | Auto-resume source above this ratio |
 | `auto_pause_cooldown_minutes` | Integer | `60` | Grace period before re-enabling |
@@ -162,8 +161,9 @@ Class: `SourceMonitor::Configuration::ScrapingSettings`
 
 | Setting | Type | Default | Description |
 |---|---|---|---|
-| `max_in_flight_per_source` | Integer/nil | `25` | Max concurrent scrapes per source |
+| `max_in_flight_per_source` | Integer/nil | `nil` | Max concurrent scrapes per source |
 | `max_bulk_batch_size` | Integer/nil | `100` | Max items per bulk scrape enqueue |
+| `scrape_recommendation_threshold` | Integer/nil | `200` | Minimum average feed word count below which a source is recommended for scraping |
 
 Values are normalized to positive integers. Set to `nil` to disable limits.
 

data/.claude/skills/sm-domain-model/SKILL.md

@@ -92,7 +92,7 @@ URL normalization and format validation. Used by Source and Item.
 | Field | Values | Notes |
 |-------|--------|-------|
 | `fetch_status` | `idle`, `queued`, `fetching`, `failed`, `invalid` | DB CHECK constraint |
-| `health_status` | `healthy` (default) | String, extensible |
+| `health_status` | `working` (default) | Values: working, declining, improving, failing |
 | `active` | `true`/`false` | Boolean toggle |
 
 ### Key Scopes
@@ -101,7 +101,7 @@ URL normalization and format validation. Used by Source and Item.
 |-------|---------|
 | `active` | `WHERE active = true` |
 | `failed` | `failure_count > 0 OR last_error IS NOT NULL OR last_error_at IS NOT NULL` |
-| `healthy` | `active AND failure_count = 0 AND last_error IS NULL AND last_error_at IS NULL` |
+| `healthy` | `active AND failure_count = 0 AND last_error IS NULL AND last_error_at IS NULL` (Note: scope name preserved for backward compat; health_status uses "working") |
 | `due_for_fetch(reference_time:)` | Class method. Active sources where `next_fetch_at IS NULL OR <= reference_time` |
 
 ### Validations

data/.claude/skills/sm-domain-model/reference/table-structure.md

@@ -44,10 +44,11 @@ Core feed source configuration and state.
 | adaptive_fetching_enabled | boolean | NO | true | Adaptive interval toggle |
 | feed_content_readability_enabled | boolean | NO | false | Process feed content through readability |
 | rolling_success_rate | decimal(5,4) | YES | | Rolling success rate (0.0-1.0) |
-| health_status | string | NO | "healthy" | Health status string |
+| health_status | string | NO | "working" | Health status (working/declining/improving/failing) |
 | health_status_changed_at | datetime | YES | | Last health status change |
 | auto_paused_at | datetime | YES | | When source was auto-paused |
 | auto_paused_until | datetime | YES | | Auto-pause expiry |
+| consecutive_fetch_failures | integer | NO | 0 | Consecutive fetch failure streak count |
 | health_auto_pause_threshold | decimal(5,4) | YES | | Custom pause threshold (0.0-1.0) |
 | created_at | datetime | NO | | |
 | updated_at | datetime | NO | | |
@@ -170,6 +171,7 @@ Records of feed fetch attempts.
 | feed_size_bytes | integer | YES | | Feed body size |
 | items_in_feed | integer | YES | | Total entries in feed |
 | job_id | string | YES | | ActiveJob ID |
+| error_category | string | YES | | Failure classification (e.g., timeout, dns, blocked) |
 | metadata | jsonb | NO | {} | Extra metadata (parser, errors) |
 | created_at | datetime | NO | | |
 | updated_at | datetime | NO | | |

data/.claude/skills/sm-engine-migration/SKILL.md

@@ -338,7 +338,7 @@ class AddHealthFieldsToSources < ActiveRecord::Migration[8.0]
   def change
     change_table :sourcemon_sources, bulk: true do |t|
       t.decimal :rolling_success_rate, precision: 5, scale: 4
-      t.string :health_status, null: false, default: "healthy"
+      t.string :health_status, null: false, default: "working"
       t.datetime :health_status_changed_at
       t.datetime :auto_paused_at
       t.datetime :auto_paused_until

data/.claude/skills/sm-engine-migration/reference/migration-conventions.md

@@ -41,7 +41,7 @@ Created in: `20241008120000_create_source_monitor_sources.rb`
 | `fetch_circuit_opened_at` | datetime | - | - |
 | `fetch_circuit_until` | datetime | - | - |
 | `rolling_success_rate` | decimal(5,4) | - | - |
-| `health_status` | string | NOT NULL, indexed | `"healthy"` |
+| `health_status` | string | NOT NULL, indexed | `"working"` |
 | `health_status_changed_at` | datetime | - | - |
 | `auto_paused_at` | datetime | - | - |
 | `auto_paused_until` | datetime | indexed | - |

data/.claude/skills/sm-health-rule/SKILL.md

@@ -35,7 +35,7 @@ Health Module (setup!)
   |
   +-- SourceHealthReset (manual reset)
         +-- Clears all health state
-        +-- Resets to "healthy" status
+        +-- Resets to "working" status
 ```
 
 ## Key Files
@@ -55,39 +55,37 @@ Health Module (setup!)
 
 | Status | Meaning | Trigger |
 |--------|---------|---------|
-| `healthy` | Source is reliable | success_rate >= healthy_threshold (0.8) |
-| `warning` | Some failures occurring | success_rate >= warning_threshold (0.5) but < healthy |
-| `critical` | High failure rate | success_rate < warning_threshold |
+| `working` | Source is reliable | success_rate >= healthy_threshold (0.8) |
 | `declining` | Consecutive failures | >= 3 consecutive failures in recent logs |
 | `improving` | Recovery in progress | >= 2 consecutive successes after a failure |
-| `auto_paused` | Automatically paused | success_rate < auto_pause_threshold (0.2) |
+| `failing` | High failure rate | success_rate < auto_pause_threshold (0.2) |
 
-### Status Priority (highest to lowest)
+### Status Decision Tree
 
 ```
-auto_paused > declining > improving > healthy > warning > critical
+failing > declining > improving > working
 ```
 
 Determination logic:
 
 ```ruby
-def determine_status(rate, auto_paused_until, logs)
-  if auto_paused_active?(auto_paused_until)
-    "auto_paused"
+def determine_status(rate, logs)
+  if rate < auto_pause_threshold
+    "failing"
   elsif consecutive_failures(logs) >= 3
     "declining"
   elsif improving_streak?(logs)
     "improving"
   elsif rate >= healthy_threshold
-    "healthy"
-  elsif rate >= warning_threshold
-    "warning"
+    "working"
   else
-    "critical"
+    "declining"  # fallback for intermediate rates
   end
 end
 ```
 
+Note: Auto-pause is tracked separately as operational state (`auto_paused_at`/`auto_paused_until`), not as a health status value. A source can be both "failing" and auto-paused.
+
 ## Health Configuration
 
 ### Default Settings
@@ -95,8 +93,7 @@ end
 | Setting | Default | Purpose |
 |---------|---------|---------|
 | `window_size` | 20 | Number of recent fetch logs to evaluate |
-| `healthy_threshold` | 0.8 | Success rate for "healthy" status |
-| `warning_threshold` | 0.5 | Success rate for "warning" status |
+| `healthy_threshold` | 0.8 | Success rate for "working" status |
 | `auto_pause_threshold` | 0.2 | Below this, source is auto-paused |
 | `auto_resume_threshold` | 0.6 | Above this, auto-pause is lifted |
 | `auto_pause_cooldown_minutes` | 60 | Minimum pause duration |
@@ -211,7 +208,7 @@ SourceMonitor::Health::SourceHealthReset.call(source: source)
 ```
 
 Resets:
-- `health_status` -> "healthy"
+- `health_status` -> "working"
 - `auto_paused_at`, `auto_paused_until` -> nil
 - `rolling_success_rate` -> nil
 - `failure_count` -> 0
@@ -230,7 +227,7 @@ Lightweight health check for import candidates (no Source record needed):
 
 ```ruby
 result = Health::ImportSourceHealthCheck.new(feed_url: url).call
-result.status        # => "healthy" or "unhealthy"
+result.status        # => "working" or "failing"
 result.error_message # => nil or error description
 result.http_status   # => HTTP status code
 ```
@@ -255,9 +252,9 @@ end
 Add the condition to `determine_status`:
 
 ```ruby
-def determine_status(rate, auto_paused_until, logs)
-  if auto_paused_active?(auto_paused_until)
-    "auto_paused"
+def determine_status(rate, logs)
+  if rate < auto_pause_threshold
+    "failing"
   elsif my_custom_condition?(logs)  # Add here
     "my_custom_status"
   elsif consecutive_failures(logs) >= 3

data/.claude/skills/sm-health-rule/reference/health-system.md

@@ -206,7 +206,7 @@ From migration `20251012090000_add_health_fields_to_sources`:
 
 ```ruby
 # Health status tracking
-:health_status            # string, default: "healthy"
+:health_status            # string, default: "working" (values: working, declining, improving, failing)
 :health_status_changed_at # datetime
 :rolling_success_rate     # float
 :health_auto_pause_threshold # float (per-source override)

data/.claude/skills/sm-host-setup/reference/initializer-template.md

@@ -108,8 +108,7 @@ SourceMonitor.configure do |config|
   # ===========================================================================
 
   config.health.window_size = 20                   # Fetch attempts to evaluate
-  config.health.healthy_threshold = 0.8            # Ratio for "healthy" badge
-  config.health.warning_threshold = 0.5            # Ratio for "warning" badge
+  config.health.healthy_threshold = 0.8            # Ratio for "working" status
   config.health.auto_pause_threshold = 0.2         # Auto-pause below this
   config.health.auto_resume_threshold = 0.6        # Auto-resume above this
   config.health.auto_pause_cooldown_minutes = 60   # Grace period before re-enable
@@ -137,6 +136,7 @@ SourceMonitor.configure do |config|
 
   # config.scraping.max_in_flight_per_source = 25  # Concurrent scrapes per source
   # config.scraping.max_bulk_batch_size = 100       # Max bulk enqueue size
+  # config.scraping.scrape_recommendation_threshold = 200
 
   # ===========================================================================
   # Event Callbacks

data/.claude/skills/sm-upgrade/reference/version-history.md

@@ -2,26 +2,31 @@
 
 Version-specific migration notes for each major/minor version transition. Agents should reference this file when guiding users through multi-version upgrades.
 
-## 0.9.x to next release
+## 0.10.2 to 0.11.0
 
 **Key changes:**
-- New third queue: `source_monitor_maintenance` for non-fetch jobs (health checks, cleanup, favicon, images, OPML import). Keeps the fetch queue dedicated to FetchFeedJob and ScheduleFetchesJob.
-- `config.maintenance_queue_name` (default `"source_monitor_maintenance"`) and `config.maintenance_queue_concurrency` (default `1`) for tuning the maintenance queue.
-- `config.fetching.scheduler_batch_size` (default `25`, was hardcoded `100`) limits sources per scheduler run. Optimized for 1-CPU/2GB servers.
-- `config.fetching.stale_timeout_minutes` (default `5`, was hardcoded `10`) controls stalled fetch recovery speed.
-- Fixed-interval sources now get ±10% jitter on `next_fetch_at` (previously exact intervals).
-- Fetch pipeline error handling hardened: DB errors in `update_source_state!` propagate instead of being silently swallowed, `ensure` block guarantees status reset from "fetching", `FollowUpHandler` rescues per-item enqueue failures.
-- New rake task: `source_monitor:maintenance:stagger_fetch_times` distributes overdue sources across a configurable window (`WINDOW_MINUTES` env var, default 10).
+- Health status simplified from 7 values to 4 (`working`, `declining`, `improving`, `failing`). Auto-pause tracked as operational state via `auto_paused_at`/`auto_paused_until` columns.
+- New `consecutive_fetch_failures` column (integer, NOT NULL, default 0) on `sourcemon_sources` for streak-based health detection.
+- New `error_category` column (string, nullable) on `sourcemon_fetch_logs` for classifying failure types.
+- New `config.scraping.scrape_recommendation_threshold` (default 200) controls the word-count threshold for dashboard scrape recommendations.
+- Dashboard pagination for sources and items lists.
+- Automatic Cloudflare bypass via cookie replay and UA rotation (no configuration needed).
+- Smart scrape recommendations widget on the dashboard highlights sources that may benefit from scraping.
+- New third queue: `source_monitor_maintenance` for non-fetch jobs (health checks, cleanup, favicon, images, OPML import).
+- `config.fetching.scheduler_batch_size` (default `25`, was hardcoded `100`) and `config.fetching.stale_timeout_minutes` (default `5`, was `10`).
+- Fixed-interval sources now get +/-10% jitter on `next_fetch_at`.
+- Fetch pipeline error handling hardened: DB errors propagate, `ensure` block guarantees status reset.
 
 **Action items:**
-1. **Action required:** Add the maintenance queue to your `solid_queue.yml`:
+1. **Action required:** If your initializer sets `config.health.warning_threshold`, remove that line. The setting no longer exists.
+2. **Action required:** Add the maintenance queue to your `solid_queue.yml`:
    ```yaml
    source_monitor_maintenance:
      concurrency: <%= ENV.fetch("SOURCE_MONITOR_MAINTENANCE_CONCURRENCY", 1) %>
    ```
-2. If you have many overdue sources after upgrading, run `bin/rails source_monitor:maintenance:stagger_fetch_times` to break the thundering herd.
-3. For larger servers (4+ CPUs, 8GB+), increase batch size: `config.fetching.scheduler_batch_size = 50` (or higher).
-4. All existing configuration remains valid. No breaking changes.
+3. If your host app queries `health_status` directly (e.g., `Source.where(health_status: "healthy")`), update to use the new values (`working`, `declining`, `improving`, `failing`).
+4. If you have many overdue sources after upgrading, run `bin/rails source_monitor:maintenance:stagger_fetch_times` to break the thundering herd.
+5. All existing configuration (except `warning_threshold`) remains valid.
 
 ## 0.7.x to 0.8.0
 

data/CHANGELOG.md

@@ -15,6 +15,48 @@ All notable changes to this project are documented below. The format follows [Ke
 
 - No unreleased changes yet.
 
+## [0.11.0] - 2026-03-13
+
+### Breaking Changes
+
+- **Health status model simplified from 7 values to 4.** `healthy`/`auto_paused`/`unknown` → `working`, `warning`/`critical` → `failing`. `declining` and `improving` unchanged. Requires migration: `bin/rails source_monitor:upgrade && bin/rails db:migrate`.
+- **`config.health.warning_threshold` removed.** Delete this line from your initializer if present.
+
+### Added
+
+- **Smart scrape recommendations.** Dashboard widget and sources index badge identify sources with low feed word counts that would benefit from scraping. Configurable threshold via `config.scraping.scrape_recommendation_threshold`.
+- **Bulk scrape enablement.** Select multiple recommended sources and enable scraping in one action with confirmation modal.
+- **Test scrape with modal results.** "Test Scrape" button on source detail page runs a synchronous scrape of the latest item and shows feed vs scraped word count comparison in a modal with "Enable Auto-Scraping" action.
+- **Cloudflare bypass techniques.** Cookie replay and UA rotation for Cloudflare-challenged feeds with clear "Blocked" badge on affected sources.
+- **Consecutive failure auto-pause.** Sources that fail consecutively are automatically paused with a `consecutive_fetch_failures` counter and configurable threshold.
+- **Error categorization on fetch logs.** New `error_category` column classifies fetch failures (blocked, auth, timeout, parse, etc.) for structured diagnostics.
+- **Dashboard pagination.** Turbo Frame pagination for sources list, fetch schedule buckets, and health distribution badges. Handles 100+ sources efficiently.
+- **Health distribution badges.** Dashboard shows working/declining/improving/failing source counts below stats cards.
+- **Sortable computed columns.** New Items/Day, Avg Feed Words, and Avg Scraped Words columns sortable on sources index.
+- **Dismissible OPML import banner.** Import history banner includes a dismiss button.
+- **SVG favicon conversion.** SVG favicons auto-converted to PNG via MiniMagick.
+
+### Changed
+
+- Health badge colors: working (green), declining (yellow), improving (sky), failing (rose).
+- Sources index splits Health and Fetch Status into separate columns.
+- Fetch interval shown as rate "(X.Xx / day)" under fetch status badge instead of separate column.
+- Scrape recommended badge positioned below source URL for better wrapping.
+- Enabling auto-scrape now sets both `scraping_enabled` and `auto_scrape`, and queues existing unscraped items.
+- Bulk scrape enablement also sets `auto_scrape` alongside `scraping_enabled`.
+
+### Fixed
+
+- Dashboard fetch schedule source links no longer show "Content missing" (turbo_frame _top added).
+- Force-fetch lock contention handled gracefully (no permanent failure after retries).
+- Blocked response detection via HTML body sniffing.
+
+### Migration Required
+
+- Run `bin/rails source_monitor:upgrade` then `bin/rails db:migrate`.
+- New columns: `consecutive_fetch_failures` on sources, `error_category` on fetch logs.
+- Health status value migration automatically remaps existing records.
+
 ## [0.10.2] - 2026-02-26
 
 ### Fixed

data/CLAUDE.md

@@ -4,8 +4,8 @@
 
 ## Active Context
 
-**Last shipped:** polish-and-reliability (6 phases, 17 plans, 35 commits)
-**Next action:** /vbw:vibe to start new work
+**Last shipped:** ui-fixes-and-smart-scraping (5 phases, 22 plans)
+**Next action:** /vbw:vibe to plan Phase 01 (Ultimate Turbo Modal Integration)
 
 ## Key Decisions
 

data/Gemfile

@@ -27,6 +27,7 @@ end
 # gem "debug", ">= 1.0.0"
 
 group :test do
+  gem "mini_magick"
   gem "simplecov", require: false
   gem "test-prof", require: false
   gem "stackprof", require: false

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    source_monitor (0.10.2)
+    source_monitor (0.11.0)
       cssbundling-rails (~> 1.4)
       faraday (~> 2.9)
       faraday-follow_redirects (~> 0.4)
@@ -177,6 +177,8 @@ GEM
       net-smtp
     marcel (1.1.0)
     matrix (0.4.3)
+    mini_magick (5.3.1)
+      logger
     mini_mime (1.1.5)
     mini_portile2 (2.8.9)
     minitest (6.0.1)
@@ -404,6 +406,7 @@ DEPENDENCIES
   cgi
   digest
   json
+  mini_magick
   minitest-mock
   ostruct
   pg

data/README.md

@@ -9,8 +9,8 @@ SourceMonitor is a production-ready Rails 8 mountable engine for ingesting, norm
 In your host Rails app:
 
 ```bash
-bundle add source_monitor --version "~> 0.10.0"
-# or add `gem "source_monitor", "~> 0.10.0"` manually, then run:
+bundle add source_monitor --version "~> 0.11.0"
+# or add `gem "source_monitor", "~> 0.11.0"` manually, then run:
 bundle install
 ```
 
@@ -43,7 +43,7 @@ This exposes `bin/source_monitor` (via Bundler binstubs) so you can run the guid
 Before running any SourceMonitor commands inside your host app, add the gem and install dependencies:
 
 ```bash
-bundle add source_monitor --version "~> 0.10.0"
+bundle add source_monitor --version "~> 0.11.0"
 # or edit your Gemfile, then run
 bundle install
 ```

data/VERSION

@@ -1 +1 @@
-0.10.2
+0.11.0