source_monitor 0.3.2 → 0.4.0

data/.vbw-planning/ROADMAP.md

@@ -0,0 +1,125 @@
+<!-- VBW ROADMAP -- Phase decomposition with requirement mapping -->
+<!-- Created during /vbw scope -->
+
+# SourceMonitor Generator Enhancements Roadmap
+
+**Milestone:** generator-enhancements
+**Goal:** Make the install generator and verification suite catch the two most common host-app setup failures: missing Procfile.dev jobs entry and missing recurring_schedule dispatcher wiring.
+
+## Phases
+
+1. [x] ~~Phase 0: Documentation Gaps~~ (shipped via quick fix ea788ea)
+2. [x] Phase 1: Install Generator Steps (Procfile.dev + Queue Config)
+3. [x] Phase 2: Recurring Schedule Verifier
+4. [x] Phase 3: Skills & Documentation Alignment
+5. [x] Phase 4: Dashboard UX Improvements
+6. [x] Phase 5: Active Storage Image Downloads
+7. [x] Phase 6: Netflix Feed Investigation
+
+## Phase Details
+
+### Phase 1: Install Generator Steps
+
+**Goal:** Add two new idempotent steps to the install generator: (a) patch `Procfile.dev` with a `jobs:` entry for Solid Queue, and (b) patch the queue config dispatcher with `recurring_schedule: config/recurring.yml`.
+
+**Requirements:** REQ-16, REQ-17, REQ-18
+
+**Success Criteria:**
+- `bin/rails generate source_monitor:install` patches Procfile.dev when present (idempotent, skip if entry exists)
+- `bin/rails generate source_monitor:install` patches queue.yml dispatcher with recurring_schedule (idempotent)
+- Both steps wired into `Setup::Workflow` for the guided installer
+- Generator tests cover: fresh file, existing file with entry, existing file without entry, missing file
+- `bin/rails test` passes, RuboCop clean
+
+### Phase 2: Recurring Schedule Verifier
+
+**Goal:** Add a `RecurringScheduleVerifier` to the verification suite that checks whether recurring tasks are actually registered with Solid Queue dispatchers, and enhance the existing `SolidQueueVerifier` to suggest Procfile.dev when workers aren't detected.
+
+**Requirements:** REQ-19, REQ-20
+
+**Success Criteria:**
+- `bin/source_monitor verify` checks that recurring tasks are registered (not just that workers heartbeat)
+- Warning when no recurring tasks found with actionable remediation message
+- SolidQueueVerifier remediation mentions Procfile.dev for `bin/dev` users
+- Verifier tests with mocked Solid Queue state
+- `bin/rails test` passes, RuboCop clean
+
+### Phase 3: Skills & Documentation Alignment
+
+**Goal:** Update all `sm-*` skills and docs to reflect that the generator now automatically handles Procfile.dev and recurring_schedule wiring. Remove manual steps that are now automated.
+
+**Requirements:** REQ-21
+
+**Success Criteria:**
+- sm-host-setup skill reflects new generator capabilities (auto-patching, not manual steps)
+- sm-configure skill references the automatic recurring_schedule wiring
+- docs/setup.md updated to note the generator handles both automatically
+- docs/troubleshooting.md updated with improved diagnostics
+- setup-checklist.md reflects automation (checked by default, not manual)
+
+### Phase 4: Dashboard UX Improvements
+
+**Goal:** Show source URLs in fetch log entries for both successes and failures on the dashboard, and make links to sources and items clickable (opening in a new tab).
+
+**Requirements:** REQ-22, REQ-23
+
+**Success Criteria:**
+- Fetch log entries on the dashboard display the source URL alongside the existing summary
+- Both success and failure fetch logs show the URL
+- Source names and item titles are clickable links that open in a new tab
+- Existing dashboard layout is preserved
+- `bin/rails test` passes, RuboCop clean
+
+### Phase 5: Active Storage Image Downloads
+
+**Goal:** Add a configurable option to download inline images from feed items to Active Storage instead of loading them directly from the source URL. This prevents broken images when sources go offline and improves page load performance.
+
+**Requirements:** REQ-24
+
+**Success Criteria:**
+- New configuration option (`config.images.download_to_active_storage` or similar) defaults to `false`
+- When enabled, inline images in item content are detected and downloaded to Active Storage
+- Original image URLs are replaced with Active Storage URLs in the stored content
+- Images that fail to download gracefully fall back to original URLs
+- Configuration is documented in sm-configure skill
+- `bin/rails test` passes, RuboCop clean
+
+### Phase 6: Netflix Feed Investigation
+
+**Goal:** Investigate and fix the failing fetch for `https://netflixtechblog.com/feed`. Determine whether the issue is in the feed parser, HTTP client configuration, or content format, and apply appropriate fixes.
+
+**Requirements:** REQ-25
+
+**Success Criteria:**
+- Root cause identified and documented
+- Fix applied (parser, HTTP client, or configuration change)
+- Netflix Tech Blog feed fetches successfully
+- No regressions in other feed types
+- `bin/rails test` passes, RuboCop clean
+
+## Progress
+
+| Phase | Status | Plans |
+|-------|--------|-------|
+| 0 | Complete | - |
+| 1 | Complete | PLAN-01 (5 tasks, 4 commits) |
+| 2 | Complete | PLAN-01 (5 tasks, 1 commit) |
+| 3 | Complete | PLAN-01 (5 tasks, 1 commit) |
+| 4 | Complete | PLAN-01 (5 tasks, 5 commits) |
+| 5 | Complete | PLAN-01 (4 tasks, 5 commits) + PLAN-02 (4 tasks, 4 commits) |
+| 6 | Complete | PLAN-01 (5 tasks, 5 commits) |
+
+## Requirement Mapping
+
+| REQ | Phase | Description |
+|-----|-------|-------------|
+| REQ-16 | 1 | Generator patches Procfile.dev with jobs: entry |
+| REQ-17 | 1 | Generator patches queue config with recurring_schedule |
+| REQ-18 | 1 | Guided workflow integrates both new steps |
+| REQ-19 | 2 | RecurringScheduleVerifier checks recurring task registration |
+| REQ-20 | 2 | SolidQueueVerifier remediation mentions Procfile.dev |
+| REQ-21 | 3 | Skills and docs reflect automated setup |
+| REQ-22 | 4 | Fetch logs show source URL on dashboard |
+| REQ-23 | 4 | Dashboard links clickable in new tab |
+| REQ-24 | 5 | Download inline images to Active Storage |
+| REQ-25 | 6 | Fix Netflix Tech Blog feed fetch |

data/.vbw-planning/STATE.md

@@ -0,0 +1,43 @@
+<!-- VBW STATE -- Current milestone progress -->
+
+# State
+
+**Milestone:** generator-enhancements
+**Current Phase:** All phases complete
+**Status:** Ready for archive
+**Date:** 2026-02-12
+
+## Progress
+
+- Phase 0: Complete (quick fix ea788ea)
+- Phase 1: Complete (1 plan, 5 tasks, 4 commits)
+- Phase 2: Complete (1 plan, 5 tasks, 1 commit)
+- Phase 3: Complete (1 plan, 5 tasks, 1 commit) -- QA PASS
+- Phase 4: Complete (1 plan, 5 tasks, 5 commits) -- QA PASS 23/23
+- Phase 5: Complete (2 plans, 9 tasks, 9 commits) -- QA PASS 34/34
+- Phase 6: Complete (1 plan, 5 tasks, 5 commits) -- QA PASS 18/18
+
+## Decisions
+
+| Decision | Date | Rationale |
+|----------|------|-----------|
+| Docs-first approach | 2026-02-11 | Shipped doc fixes before code changes to unblock users immediately |
+| 3 phases for code changes | 2026-02-11 | Generator steps, verification, then docs alignment -- each independently testable |
+| Always create/patch Procfile.dev | 2026-02-11 | Maximum hand-holding for host app setup |
+| Target queue.yml only | 2026-02-11 | Rails 8 default, no legacy naming support |
+| External links open new tab with icon | 2026-02-12 | Consistent UX for all external URLs across dashboard, logs, sources, items |
+| Fetch log URL display: domain for RSS, item URL for scrapes | 2026-02-12 | User requested contextual URL display based on event type |
+| Image downloads via background job to ItemContent | 2026-02-12 | Content images only, opt-in via config.images.download_to_active_storage |
+| SSL fix: general cert store config, not Netflix-specific | 2026-02-12 | OpenSSL::X509::Store#set_default_paths on every Faraday connection |
+
+## Metrics
+
+| Metric | Value |
+|--------|-------|
+| Phases | 7 (Phase 0-6) |
+| Plans completed | 7 (across all phases) |
+| Tasks completed | 34 |
+| Commits | ~26 |
+| Tests | 973 (up from 841) |
+| RuboCop | 389 files, 0 offenses |
+| Brakeman | 0 warnings |

data/.vbw-planning/config.json

@@ -8,5 +8,7 @@
   "visual_format": "unicode",
   "compaction_trigger": 130000,
   "max_tasks_per_plan": 5,
-  "agent_teams": true
+  "agent_teams": true,
+  "model_profile": "quality",
+  "model_overrides": {}
 }

data/.vbw-planning/discovery.json

@@ -19,6 +19,8 @@
     "User wants comprehensive cleanup, not surface-level",
     "Public API changes are acceptable for convention alignment",
     "Tests should be updated to match any changes, not removed",
-    "All layers: models, controllers, services, dead code"
+    "All layers: models, controllers, services, dead code",
+    "Generator should be maximally helpful -- create files rather than just warn",
+    "Target Rails 8 defaults (queue.yml) rather than supporting legacy naming"
   ]
 }

data/.vbw-planning/phases/01-generator-steps/01-CONTEXT.md

@@ -0,0 +1,33 @@
+# Phase 1 Context: Install Generator Steps
+
+## User Vision
+Enhance the install generator to automatically handle the two most common Solid Queue setup failures in host apps: missing Procfile.dev jobs entry and missing recurring_schedule dispatcher wiring.
+
+## Essential Features
+- Generator creates Procfile.dev if missing (with web: and jobs: entries)
+- Generator patches existing Procfile.dev to add jobs: entry (idempotent)
+- Generator patches config/queue.yml dispatcher with recurring_schedule: config/recurring.yml
+- Both steps integrated into the guided Setup::Workflow
+
+## Technical Preferences
+- **Procfile.dev:** Always create/patch -- maximum hand-holding. Create if missing, patch if exists.
+- **Queue config:** Target `config/queue.yml` (Rails 8 default). Don't worry about legacy `solid_queue.yml` naming.
+- Follow existing generator patterns (idempotent, skip-if-present, say_status output)
+
+## Boundaries
+- Don't modify any other config files (cable.yml, database.yml, etc.)
+- Don't modify the initializer template
+- Don't change existing generator steps (routes, initializer, recurring.yml)
+- Don't add verification logic here (that's Phase 2)
+
+## Acceptance Criteria
+- `bin/rails generate source_monitor:install` creates Procfile.dev with web: + jobs: when none exists
+- `bin/rails generate source_monitor:install` adds jobs: line to existing Procfile.dev without duplicating
+- `bin/rails generate source_monitor:install` adds recurring_schedule to queue.yml dispatcher
+- `bin/source_monitor install` (guided) runs both new steps
+- All existing generator tests still pass
+- New tests cover: fresh Procfile.dev, existing with entry, existing without entry, missing queue.yml
+
+## Decisions Made
+- Create Procfile.dev if missing (not just warn)
+- Target config/queue.yml only (Rails 8 default), not solid_queue.yml

data/.vbw-planning/phases/01-generator-steps/01-VERIFICATION.md

@@ -0,0 +1,86 @@
+# PLAN-01 Verification Report
+
+## Verdict: PASS
+
+**Verified by:** QA agent (deep tier, 30 checks)
+**Date:** 2026-02-11
+
+---
+
+## Functional Verification
+
+| # | Check | Result | Notes |
+|---|-------|--------|-------|
+| 1 | Generator tests pass | PASS | 20 runs, 109 assertions, 0 failures, 0 errors |
+| 2 | Workflow tests pass | PASS | 8 runs, 22 assertions, 0 failures, 0 errors |
+| 3 | Full test suite passes | PASS | 867 runs, 2898 assertions, 0 failures, 0 errors |
+| 4 | RuboCop clean | PASS | 376 files inspected, no offenses detected |
+| 5 | Brakeman clean | PASS | 0 warnings, 0 errors |
+
+## Code Review Checks
+
+| # | Check | Result | Notes |
+|---|-------|--------|-------|
+| 6 | Public method order correct | PASS | Order is: `add_routes_mount` (L17), `create_initializer` (L24), `configure_recurring_jobs` (L36), `patch_procfile_dev` (L52), `configure_queue_dispatcher` (L70), `print_next_steps` (L90) |
+| 7 | `patch_procfile_dev` handles 3 cases | PASS | Create (L64-66), append (L62-63), skip (L57-60) |
+| 8 | `configure_queue_dispatcher` handles 4 cases | PASS | Missing file (L73-76), already configured (L80-83), needs patching (L85-87), no dispatchers (L254-256 via `add_recurring_schedule_to_dispatchers!`) |
+| 9 | Private methods/constants in private section | PASS | `PROCFILE_JOBS_ENTRY` (L104), `RECURRING_SCHEDULE_VALUE` (L201), `DEFAULT_DISPATCHER` (L203), `queue_config_has_recurring_schedule?` (L209), `add_recurring_schedule_to_dispatchers!` (L229) -- all after `private` on L102 |
+| 10 | `frozen_string_literal: true` on all new files | PASS | `procfile_patcher.rb` (L1), `queue_config_patcher.rb` (L1) both have it |
+| 11 | Idempotency: both steps safe to run multiple times | PASS | `patch_procfile_dev`: checks `/^jobs:/` before acting; `configure_queue_dispatcher`: checks `has_recurring_schedule?` before acting. Test `test_does_not_duplicate_jobs_entry_when_rerun` explicitly verifies. |
+| 12 | `YAML.safe_load` uses `aliases: true` | PASS | Generator L78: `YAML.safe_load(File.read(queue_path), aliases: true)`. QueueConfigPatcher L24: `YAML.safe_load(path.read, aliases: true)` |
+| 13 | ProcfilePatcher matches `/^jobs:/` regex | PASS | Both generator (L57) and ProcfilePatcher (L17) use `/^jobs:/` regex |
+| 14 | QueueConfigPatcher recursive dispatcher search | PASS | `has_recurring_schedule?` (L36-53) recursively iterates `parsed.each_value`, checks nested hashes, and also checks top-level `dispatchers` key for flat configs |
+| 15 | Workflow has both new kwargs with defaults | PASS | `procfile_patcher: ProcfilePatcher.new` (L40), `queue_config_patcher: QueueConfigPatcher.new` (L41) |
+| 16 | Workflow calls patchers in correct position | PASS | Called at L73-74, after `initializer_patcher.ensure_navigation_hint` (L72) and before devise check (L76) |
+| 17 | Autoload entries in `lib/source_monitor.rb` | PASS | `ProcfilePatcher` (L164), `QueueConfigPatcher` (L165) both present in `Setup` module autoloads |
+| 18 | No hardcoded paths or env-specific assumptions | PASS | Paths are relative to `destination_root` (generator) or injected via constructor kwargs (patcher classes). No absolute paths. |
+| 19 | Test isolation (no cross-test contamination) | PASS | Generator tests use `prepare_destination` in setup. Workflow tests use Spy/Mock objects for all collaborators. No shared state. WORKER_SUFFIX handles parallel workers. |
+| 20 | `say_status` calls use correct symbols | PASS | `:create` (L66), `:append` (L63), `:skip` (L58, L74, L81), `:info` (L91-99) -- all correct |
+
+## Edge Case Verification
+
+| # | Check | Result | Notes |
+|---|-------|--------|-------|
+| 21 | Procfile.dev has `jobs:` with different content | PASS | Regex `/^jobs:/` matches any line starting with `jobs:` regardless of the command after it. Correctly skips without overwriting. |
+| 22 | queue.yml nested environments with different dispatchers | PASS | `queue_config_has_recurring_schedule?` recursively walks all hash values. `add_recurring_schedule_to_dispatchers!` processes all nested dispatcher arrays under any env key. Both handle flat and nested configs. |
+| 23 | queue.yml empty/nil after parsing | PASS | L78: `|| {}` ensures nil YAML parse result becomes empty hash. `queue_config_has_recurring_schedule?({})` returns false. `add_recurring_schedule_to_dispatchers!({})` adds default dispatcher section. |
+| 24 | Procfile.dev has trailing newlines | PASS | When appending (L62), uses `puts("", PROCFILE_JOBS_ENTRY)` which adds a blank line before the jobs entry. This handles trailing newlines gracefully -- the blank line acts as separator. |
+| 25 | queue.yml dispatchers is empty array | PASS | If `dispatchers: []`, the `any?` check in `has_recurring_schedule?` returns false (no items to check). `add_recurring_schedule_to_dispatchers!` iterates the empty array (no-op) but sets `found_dispatchers = true` since the key exists. This means an empty dispatchers array stays empty with no recurring_schedule added. **Minor note**: This is a debatable edge case -- an empty dispatchers array means the user intentionally has no dispatchers, so not adding one is reasonable behavior. |
+
+## Requirements Verification
+
+| # | Check | Result | Notes |
+|---|-------|--------|-------|
+| 26 | REQ-16: Generator patches Procfile.dev | PASS | `patch_procfile_dev` creates/appends/skips as needed. 4 tests cover all scenarios. |
+| 27 | REQ-17: Generator patches queue.yml | PASS | `configure_queue_dispatcher` patches/skips/handles-missing as needed. 4 tests cover all scenarios. |
+| 28 | REQ-18: Workflow integrates both | PASS | `workflow.rb` L73-74 calls both patchers. Workflow test L96-97 verifies both are called. |
+| 29 | Test count increased | PASS | Was 841, now 867 (+26 tests: 8 new generator tests + 18 other additions from this phase) |
+| 30 | No regressions in existing tests | PASS | Full suite: 867 runs, 2898 assertions, 0 failures, 0 errors |
+
+---
+
+## Issues Found
+
+### Minor (non-blocking)
+
+1. **Empty dispatchers array edge case (Check #25)**: If `queue.yml` has `dispatchers: []`, the code sees `found_dispatchers = true` (the key exists) but doesn't add any recurring_schedule since there are no dispatcher hashes to iterate. This means the empty array is left as-is. This is arguably correct behavior (respecting user intent), but worth documenting.
+
+2. **Generator test count**: The plan expected "11 existing + 8 new = 19" but the actual count is 20 (12 existing + 8 new). The plan summary correctly notes 20. The 12th existing test was likely the `test_outputs_next_steps_with_doc_links` test that was already present. No issue -- just a plan estimate mismatch.
+
+### None (blocking)
+
+No blocking issues found.
+
+---
+
+## Risk Assessment
+
+**Risk level: LOW**
+
+- All 867 tests pass with 0 failures
+- RuboCop and Brakeman clean
+- Both new generator steps follow established idempotent patterns
+- Workflow integration is minimal (2 unconditional patcher calls)
+- New files are self-contained with no side effects
+- Test coverage includes all happy paths, skip paths, and edge cases
+- No security concerns (file operations are scoped to destination_root)

data/.vbw-planning/phases/01-generator-steps/PLAN-01-SUMMARY.md

@@ -0,0 +1,61 @@
+# PLAN-01 Summary: procfile-queue-generator-steps
+
+## Status: COMPLETE
+
+## What was done
+
+### Task 1: Add Procfile.dev generator step
+- Added `patch_procfile_dev` public method to `InstallGenerator` (between `configure_recurring_jobs` and `print_next_steps`)
+- Handles 3 cases: create new file, append to existing, skip if already present
+- Added `PROCFILE_JOBS_ENTRY` private constant
+- Updated `print_next_steps` with Procfile.dev info message
+
+### Task 2: Add queue.yml dispatcher step
+- Added `configure_queue_dispatcher` public method to `InstallGenerator` (between `patch_procfile_dev` and `print_next_steps`)
+- Handles 4 cases: missing file, already configured, needs patching, no dispatchers key
+- Added private helpers: `queue_config_has_recurring_schedule?`, `add_recurring_schedule_to_dispatchers!`
+- Added `RECURRING_SCHEDULE_VALUE` and `DEFAULT_DISPATCHER` private constants
+
+### Task 3: Add 8 generator tests
+- 4 Procfile.dev tests: create, append, skip, no-duplicate
+- 4 queue.yml tests: patch dispatchers, skip when configured, skip when missing, add default dispatcher
+- All 20 generator tests pass (12 existing + 8 new)
+
+### Task 4: Add workflow helpers and integration
+- Created `lib/source_monitor/setup/procfile_patcher.rb` (lightweight Pathname-based helper)
+- Created `lib/source_monitor/setup/queue_config_patcher.rb` (YAML parsing with recursive dispatcher search)
+- Modified `lib/source_monitor/setup/workflow.rb`: added 2 new kwargs, wired into `run` after `initializer_patcher` and before devise check
+- Added autoload entries in `lib/source_monitor.rb`
+- Updated all 8 workflow tests with new collaborator spies
+
+### Task 5: Full suite verification
+- 867 test runs, 2898 assertions, 0 failures, 0 errors
+- RuboCop: 376 files, 0 offenses
+- Brakeman: 0 warnings
+
+## Files modified
+- `lib/generators/source_monitor/install/install_generator.rb` -- 2 new public methods, 4 private helpers/constants
+- `test/lib/generators/install_generator_test.rb` -- 8 new tests
+- `lib/source_monitor/setup/procfile_patcher.rb` -- NEW
+- `lib/source_monitor/setup/queue_config_patcher.rb` -- NEW
+- `lib/source_monitor/setup/workflow.rb` -- 2 new kwargs, 2 new patcher calls
+- `test/lib/source_monitor/setup/workflow_test.rb` -- added patcher spies to all tests
+- `lib/source_monitor.rb` -- 2 new autoload entries
+
+## Commits
+1. `af59468` feat(generator): add Procfile.dev patching step to install generator
+2. `29250af` feat(generator): add queue.yml dispatcher step to install generator
+3. `96365b9` feat(generator): add 8 tests for Procfile.dev and queue.yml steps
+4. `4393d17` feat(generator): add workflow helpers and wire patchers into guided install
+
+## Acceptance criteria met
+- Generator creates Procfile.dev with web: + jobs: entries when none exists (REQ-16)
+- Generator appends jobs: entry to existing Procfile.dev without duplicating (REQ-16)
+- Generator skips Procfile.dev when jobs: entry already present (REQ-16 idempotency)
+- Generator patches queue.yml dispatchers with recurring_schedule (REQ-17)
+- Generator skips queue.yml when recurring_schedule already configured (REQ-17 idempotency)
+- Generator handles missing queue.yml gracefully (REQ-17 edge case)
+- Guided workflow runs both patchers after generator step (REQ-18)
+- All existing tests continue to pass (no regressions)
+- 8 new generator tests cover all scenarios
+- RuboCop clean, Brakeman clean