source_monitor 0.7.1 → 0.8.0

data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/PLAN-02.md

@@ -1,195 +0,0 @@
----
-phase: 4
-plan: 2
-title: item-creator-extraction
-wave: 1
-depends_on: []
-skills_used: []
-cross_phase_deps:
-  - "Phase 3 Plan 01 -- FeedFetcher extraction pattern (sub-module directory with require from main file)"
-  - "Phase 2 Plan 02 -- ItemCreator tests exist at test/lib/source_monitor/items/item_creator_test.rb"
-must_haves:
-  truths:
-    - "Running `wc -l lib/source_monitor/items/item_creator.rb` shows fewer than 300 lines"
-    - "Running `bin/rails test test/lib/source_monitor/items/item_creator_test.rb` exits 0 with zero failures"
-    - "Running `bin/rails test` exits 0 with 760+ runs and 0 failures"
-    - "Running `ruby -c lib/source_monitor/items/item_creator.rb` exits 0 (valid syntax)"
-    - "Running `ruby -c lib/source_monitor/items/item_creator/content_extractor.rb` exits 0"
-    - "Running `ruby -c lib/source_monitor/items/item_creator/entry_parser.rb` exits 0"
-    - "Running `bin/rubocop lib/source_monitor/items/item_creator.rb lib/source_monitor/items/item_creator/` exits 0"
-  artifacts:
-    - "lib/source_monitor/items/item_creator/entry_parser.rb -- extracted entry field parsing (guid, url, authors, enclosures, media, metadata, etc.)"
-    - "lib/source_monitor/items/item_creator/content_extractor.rb -- extracted feed content processing and readability"
-    - "lib/source_monitor/items/item_creator.rb -- slimmed to orchestrator under 300 lines"
-  key_links:
-    - "Phase 4 success criterion #1 -- all service objects follow established conventions"
-    - "No single file exceeds 300 lines (extends Phase 3 criterion)"
-    - "Public API unchanged -- ItemCreator.call(source:, entry:) returns Result struct"
----
-
-# Plan 02: item-creator-extraction
-
-## Objective
-
-Extract `lib/source_monitor/items/item_creator.rb` (601 lines, 50+ methods) into focused sub-modules following the exact same extraction pattern used by `FeedFetcher` in Phase 3 (sub-module directory with require from main file). The public API (`ItemCreator.call(source:, entry:)` returning a `Result` struct) must remain unchanged. All existing ItemCreator tests must continue to pass without modification.
-
-## Context
-
-<context>
-@lib/source_monitor/items/item_creator.rb -- 601 lines with 50+ methods. The largest file in the codebase after Phase 3 refactoring. Contains three clearly separable responsibility clusters:
-
-**Cluster 1: Core attribute building (build_attributes, ~90 lines)**
-The `build_attributes` method (lines 233-271) assembles all item attributes by calling field extraction methods. This is the main orchestration method and should stay in the main file.
-
-**Cluster 2: Field extraction from feed entries (~300 lines)**
-Methods that extract specific fields from Feedjira entry objects:
-- `extract_guid` (lines 273-287)
-- `extract_url` (lines 288-311)
-- `extract_summary` (lines 312-317)
-- `extract_content` (lines 318-327)
-- `extract_timestamp` (lines 328-337)
-- `extract_updated_timestamp` (lines 338-343)
-- `extract_author` (lines 344-347)
-- `extract_authors` (lines 348-384)
-- `extract_categories` (lines 385-394)
-- `extract_tags` (lines 395-408)
-- `extract_keywords` (lines 409-415)
-- `extract_enclosures` (lines 416-467)
-- `extract_media_thumbnail_url` (lines 468-476)
-- `extract_media_content` (lines 477-500)
-- `extract_language` (lines 501-512)
-- `extract_copyright` (lines 513-524)
-- `extract_comments_url` (lines 525-528)
-- `extract_comments_count` (lines 529-535)
-- `extract_metadata` (lines 536-544)
-Plus utility methods: `generate_fingerprint`, `string_or_nil`, `sanitize_string_array`, `split_keywords`, `safe_integer`, `json_entry?`, `atom_entry?`, `normalize_metadata` (lines 545-601)
-
-**Cluster 3: Feed content processing (~75 lines)**
-Methods for processing raw feed content through readability:
-- `process_feed_content` (lines 137-158)
-- `should_process_feed_content?` (lines 160-165)
-- `feed_content_parser_class` (lines 167-170)
-- `wrap_content_for_readability` (lines 171-186)
-- `default_feed_readability_options` (lines 187-193)
-- `build_feed_content_metadata` (lines 194-209)
-- `html_fragment?` (lines 210-213)
-- `deep_copy` (lines 214-231)
-
-**What stays in the main file (~200 lines):**
-- Result struct definition
-- Constants (FINGERPRINT_SEPARATOR, CONTENT_METHODS, etc.)
-- Constructor, `self.call`, `call` method
-- `existing_item_for`, `find_item_by_guid`, `find_item_by_fingerprint`
-- `instrument_duplicate`, `update_existing_item`, `create_new_item`
-- `handle_concurrent_duplicate`, `find_conflicting_item`, `apply_attributes`
-- `build_attributes` (calls into extracted modules)
-- Lazy accessor methods for sub-modules
-
-@lib/source_monitor/fetching/feed_fetcher.rb -- 285 lines. The extraction pattern to follow: main file requires sub-modules, uses lazy accessors (e.g., `def source_updater; @source_updater ||= SourceUpdater.new(...); end`), delegates method calls.
-@lib/source_monitor/fetching/feed_fetcher/source_updater.rb -- Example sub-module: namespaced under FeedFetcher, constructor receives dependencies.
-@lib/source_monitor/fetching/feed_fetcher/entry_processor.rb -- Another example sub-module.
-@test/lib/source_monitor/items/item_creator_test.rb -- Existing tests. Must pass without modification.
-</context>
-
-## Tasks
-
-### Task 1: Extract EntryParser module
-
-- **name:** extract-entry-parser
-- **files:**
-  - `lib/source_monitor/items/item_creator/entry_parser.rb` (new)
-  - `lib/source_monitor/items/item_creator.rb`
-- **action:** Create `lib/source_monitor/items/item_creator/entry_parser.rb` containing a `SourceMonitor::Items::ItemCreator::EntryParser` class. Move these methods from item_creator.rb into the new class:
-  - `extract_guid` -- entry GUID extraction with JSON/Atom fallbacks
-  - `extract_url` -- URL extraction with canonical/alternate link resolution
-  - `extract_summary` -- summary text extraction
-  - `extract_content` -- content extraction from multiple methods
-  - `extract_timestamp` -- published_at extraction
-  - `extract_updated_timestamp` -- updated_at extraction
-  - `extract_author` -- single author extraction
-  - `extract_authors` -- multi-author extraction with JSON parsing
-  - `extract_categories` -- category extraction
-  - `extract_tags` -- tag extraction
-  - `extract_keywords` -- keyword extraction with separator splitting
-  - `extract_enclosures` -- enclosure/attachment extraction
-  - `extract_media_thumbnail_url` -- media thumbnail extraction
-  - `extract_media_content` -- media content metadata extraction
-  - `extract_language` -- language detection
-  - `extract_copyright` -- copyright extraction
-  - `extract_comments_url` -- comments link extraction
-  - `extract_comments_count` -- comments count extraction
-  - `extract_metadata` -- raw metadata extraction
-  - `generate_fingerprint` -- content fingerprint generation
-  - Utility methods: `string_or_nil`, `sanitize_string_array`, `split_keywords`, `safe_integer`, `json_entry?`, `atom_entry?`, `normalize_metadata`
-
-  The EntryParser constructor takes `source:` and `entry:` (same as ItemCreator). It exposes a single public method `parse` that returns a hash of all extracted attributes (what `build_attributes` currently assembles). Add `require_relative "item_creator/entry_parser"` at the top of item_creator.rb. In ItemCreator, create an `entry_parser` lazy accessor and delegate the field extraction to it.
-- **verify:** `ruby -c lib/source_monitor/items/item_creator/entry_parser.rb` exits 0 AND `bin/rails test test/lib/source_monitor/items/item_creator_test.rb` exits 0 with zero failures
-- **done:** EntryParser extracted with all field extraction methods. Tests pass unchanged.
-
-### Task 2: Extract ContentExtractor module
-
-- **name:** extract-content-extractor
-- **files:**
-  - `lib/source_monitor/items/item_creator/content_extractor.rb` (new)
-  - `lib/source_monitor/items/item_creator.rb`
-- **action:** Create `lib/source_monitor/items/item_creator/content_extractor.rb` containing a `SourceMonitor::Items::ItemCreator::ContentExtractor` class. Move these methods:
-  - `process_feed_content` -- orchestrates content processing through readability
-  - `should_process_feed_content?` -- determines if content should be processed
-  - `feed_content_parser_class` -- resolves the parser class
-  - `wrap_content_for_readability` -- wraps raw content with HTML structure for parsing
-  - `default_feed_readability_options` -- default options for readability
-  - `build_feed_content_metadata` -- builds metadata about processing results
-  - `html_fragment?` -- checks if content is HTML
-  - `deep_copy` -- deep copies complex values
-
-  The ContentExtractor constructor takes `source:`. It exposes `process_feed_content(raw_content, title:)` as the primary public method. Add `require_relative "item_creator/content_extractor"` at the top of item_creator.rb. In ItemCreator, create a `content_extractor` lazy accessor. The EntryParser from Task 1 should call `content_extractor.process_feed_content(...)` instead of the local method -- wire this through the constructor or pass as a dependency.
-- **verify:** `ruby -c lib/source_monitor/items/item_creator/content_extractor.rb` exits 0 AND `bin/rails test test/lib/source_monitor/items/item_creator_test.rb` exits 0
-- **done:** ContentExtractor extracted. Feed content processing isolated. Tests pass unchanged.
-
-### Task 3: Slim main ItemCreator and wire modules
-
-- **name:** slim-item-creator-and-wire
-- **files:**
-  - `lib/source_monitor/items/item_creator.rb`
-- **action:** After Tasks 1-2, the main item_creator.rb should contain:
-  - Require statements for 2 sub-modules
-  - Existing requires (digest, json, cgi, etc.)
-  - Result struct definition
-  - Constants (FINGERPRINT_SEPARATOR, CONTENT_METHODS, TIMESTAMP_METHODS, etc.)
-  - Constructor and `self.call`
-  - `call` method (find or create)
-  - `existing_item_for`, `find_item_by_guid`, `find_item_by_fingerprint`
-  - `instrument_duplicate`, `update_existing_item`, `create_new_item`
-  - `handle_concurrent_duplicate`, `find_conflicting_item`, `apply_attributes`
-  - `build_attributes` (now delegates to entry_parser.parse)
-  - Lazy accessor methods for entry_parser and content_extractor
-
-  Clean up any dead code, orphaned requires, or duplicated constants. Ensure the main file is under 300 lines. Run RuboCop on all modified/new files.
-- **verify:** `wc -l lib/source_monitor/items/item_creator.rb` shows fewer than 300 lines AND `bin/rubocop lib/source_monitor/items/item_creator.rb lib/source_monitor/items/item_creator/` exits 0 AND `bin/rails test test/lib/source_monitor/items/item_creator_test.rb` exits 0
-- **done:** ItemCreator main file under 300 lines. All sub-modules wired. RuboCop clean.
-
-### Task 4: Full test suite regression check
-
-- **name:** full-regression-check
-- **files:** (no new modifications -- verification only)
-- **action:** Run the complete test suite to verify no regressions from the extraction. Check that: (a) all 760+ tests pass, (b) no new RuboCop violations, (c) ItemCreator public API (`ItemCreator.call(source:, entry:)` returning `Result` struct) works identically to before the extraction. Verify by inspecting any tests that use ItemCreator in other test files (e.g., feed_fetcher_test.rb, import_opml_job tests) to confirm they still pass.
-- **verify:** `bin/rails test` exits 0 with 760+ runs and 0 failures AND `bin/rubocop -f simple` shows `no offenses detected`
-- **done:** Full suite passes. Zero RuboCop violations. No regressions from extraction.
-
-## Verification
-
-1. `wc -l lib/source_monitor/items/item_creator.rb` shows fewer than 300 lines
-2. `wc -l lib/source_monitor/items/item_creator/entry_parser.rb lib/source_monitor/items/item_creator/content_extractor.rb` shows both exist
-3. `bin/rails test test/lib/source_monitor/items/item_creator_test.rb` exits 0 with zero failures
-4. `bin/rails test` exits 0 with 760+ runs and 0 failures
-5. `bin/rubocop lib/source_monitor/items/` exits 0
-
-## Success Criteria
-
-- [ ] ItemCreator main file under 300 lines
-- [ ] Two sub-modules created: entry_parser.rb, content_extractor.rb
-- [ ] Public API unchanged -- ItemCreator.call(source:, entry:) returns Result struct
-- [ ] All existing tests pass without modification
-- [ ] Full test suite passes (760+ runs, 0 failures)
-- [ ] RuboCop passes on all modified/new files
-- [ ] No file in app/ or lib/ exceeds 300 lines (extends Phase 3 success criterion)

data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/PLAN-03-SUMMARY.md

@@ -1,79 +0,0 @@
----
-phase: 4
-plan: 3
-title: final-verification
-status: complete
----
-
-# Plan 03 Summary: final-verification
-
-## What Was Done
-
-1. **Regenerated coverage baseline** -- Coverage baseline reduced from 2117 to 510 uncovered lines (75.9% reduction, far exceeding the 60% target of 847).
-
-2. **Fixed test isolation** -- Scoped test queries to specific source/item to prevent cross-test contamination from parallel test state leakage. Affected files: log_cleanup_job_test.rb, paginator_test.rb, item_test.rb, scrape_log_test.rb.
-
-3. **Fixed coverage test infrastructure** -- Updated test_helper.rb to use threads with 1 worker for coverage runs (prevents SimpleCov data loss). Removed `refuse_coverage_drop :line` that was blocking coverage regeneration.
-
-4. **Fixed remaining RuboCop violations** -- Autocorrected 22 `Layout/SpaceInsideArrayLiteralBrackets` offenses in Phase 2 configuration test files plus 1 `Layout/TrailingEmptyLines` in a generated temp file.
-
-5. **Extracted modules to bring all files under 300 lines:**
-   - EntryParser (308->294): MediaExtraction module extracted
-   - Queries (356->163): StatsQuery and RecentActivityQuery extracted
-   - ApplicationHelper (346->236): TableSortHelper and HealthBadgeHelper extracted
-   - Added test/lib/tmp/ to .rubocop.yml exclusions
-
-6. **CI-equivalent verification passed:**
-   - `bin/rubocop -f simple`: 372 files inspected, no offenses detected
-   - `bin/brakeman --no-pager -q`: 0 warnings
-   - `bin/rails test`: 841 runs, 2776 assertions, 0 failures, 0 errors
-   - No file in app/ or lib/ exceeds 300 lines (max: 294)
-   - All models and controllers have frozen_string_literal: true
-
-7. **Conventions spot-check** -- All core models use ModelExtensions.register (ImportHistory/ImportSession intentionally excluded -- not in MODEL_KEYS). Concerns use ActiveSupport::Concern, jobs inherit from ApplicationJob, no commented-out code. One documented TODO in items_controller.rb. Struct keyword_init not needed (Ruby 4.0 default).
-
-## Files Modified
-
-- `config/coverage_baseline.json` -- Regenerated (510 uncovered lines)
-- `test/test_helper.rb` -- Fixed parallel/coverage interaction
-- `lib/source_monitor.rb` -- Added missing Scrapers::Fetchers autoload
-- `test/jobs/source_monitor/log_cleanup_job_test.rb` -- Test isolation fix
-- `test/lib/source_monitor/pagination/paginator_test.rb` -- Test isolation fix
-- `test/models/source_monitor/item_test.rb` -- Test isolation fix
-- `test/models/source_monitor/scrape_log_test.rb` -- Test isolation fix
-- `test/lib/source_monitor/configuration/*.rb` (6 files) -- RuboCop fixes
-- `.rubocop.yml` -- Added test/lib/tmp/ exclusion
-- `lib/source_monitor/items/item_creator/entry_parser.rb` -- Extracted MediaExtraction
-- `lib/source_monitor/items/item_creator/entry_parser/media_extraction.rb` -- New file
-- `lib/source_monitor/dashboard/queries.rb` -- Extracted StatsQuery/RecentActivityQuery
-- `lib/source_monitor/dashboard/queries/stats_query.rb` -- New file
-- `lib/source_monitor/dashboard/queries/recent_activity_query.rb` -- New file
-- `app/helpers/source_monitor/application_helper.rb` -- Extracted TableSort/HealthBadge
-- `app/helpers/source_monitor/table_sort_helper.rb` -- New file
-- `app/helpers/source_monitor/health_badge_helper.rb` -- New file
-
-## Test Results
-
-- 841 runs, 2776 assertions, 0 failures, 0 errors
-- 372 files inspected, 0 RuboCop offenses
-- 0 Brakeman warnings
-- Coverage: 86.97% line, 58.84% branch
-- Uncovered lines: 510 (75.9% reduction from 2117)
-- Max file size: 294 lines (entry_parser.rb)
-
-## Success Criteria
-
-- [x] Coverage baseline regenerated: 510 lines (75.9% reduction, target was 60%)
-- [x] Zero RuboCop violations
-- [x] Zero Brakeman warnings
-- [x] All 841 tests pass with 0 failures
-- [x] No file in app/ or lib/ exceeds 300 lines
-- [x] All conventions verified in final spot-check
-- [x] Phase 4 complete -- all ROADMAP success criteria met
-
-## Notes
-
-- ImportHistory and ImportSession intentionally excluded from ModelExtensions.register (not in MODEL_KEYS -- they're import workflow models, not core domain models).
-- Ruby 4.0.1 Struct accepts keyword args by default; keyword_init: true is redundant.
-- One documented TODO in items_controller.rb:39 for future CRUD extraction.
-- Transient PG deadlocks in Solid Queue test teardown occur intermittently -- pre-existing, unrelated to Phase 4 changes.

data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/PLAN-03.md

@@ -1,130 +0,0 @@
----
-phase: 4
-plan: 3
-title: final-verification
-wave: 2
-depends_on:
-  - "plan-01 (conventions-audit)"
-  - "plan-02 (item-creator-extraction)"
-skills_used: []
-cross_phase_deps:
-  - "Phase 1 -- coverage baseline established at 2117 uncovered lines across 105 files"
-  - "Phase 2 -- critical path test coverage added (500+ uncovered lines expected to be covered)"
-  - "Phase 3 -- large file refactoring (new files created, some lines shifted between files)"
-must_haves:
-  truths:
-    - "Running `bin/rails test` exits 0 with 760+ runs and 0 failures"
-    - "Running `bin/rubocop -f simple` shows `no offenses detected`"
-    - "Running `bin/brakeman --no-pager -q` exits 0 with zero warnings"
-    - "The regenerated `config/coverage_baseline.json` has at most 847 uncovered lines (60% reduction from 2117)"
-    - "No file in app/ or lib/ exceeds 300 lines"
-  artifacts:
-    - "config/coverage_baseline.json -- regenerated with current coverage data"
-  key_links:
-    - "Phase 4 success criterion #1 -- all models, controllers, service objects follow conventions"
-    - "Phase 4 success criterion #2 -- zero RuboCop violations"
-    - "Phase 4 success criterion #3 -- coverage baseline at least 60% smaller than original"
-    - "Phase 4 success criterion #4 -- CI pipeline fully green"
----
-
-# Plan 03: final-verification
-
-## Objective
-
-Regenerate the coverage baseline to reflect all test improvements from Phases 2-4, verify the 60% reduction target is met, run full CI-equivalent checks (tests, RuboCop, Brakeman), and confirm no file exceeds 300 lines. This plan is the final gate before Phase 4 (and the entire VBW roadmap) can be marked complete.
-
-## Context
-
-<context>
-@config/coverage_baseline.json -- Currently shows 2117 uncovered lines across 105 files. This baseline has NOT been regenerated since Phase 1. Phases 2 and 3 added significant test coverage (Phase 2 targeted ~630 lines directly plus indirect coverage, Phase 3 refactored files which shifted coverage around). The actual current uncovered count should be significantly lower.
-
-@bin/update-coverage-baseline -- Script that regenerates the baseline from SimpleCov results. Requires running the test suite with coverage first (`COVERAGE=1 bin/rails test` or `bin/test-coverage`).
-
-@bin/check-diff-coverage -- CI script that checks diff coverage against the baseline.
-
-@AGENTS.md -- Documents the workflow: "refresh config/coverage_baseline.json by running bin/test-coverage followed by bin/update-coverage-baseline"
-
-@test/test_helper.rb -- Coverage is enabled when `CI` or `COVERAGE` env var is set. Uses SimpleCov with branch coverage.
-
-**60% reduction target:** The original baseline has 2117 uncovered lines. A 60% reduction means the new baseline must have at most 847 uncovered lines (2117 * 0.4 = 847). Phase 2 directly targeted ~630 lines in top files, and indirect coverage should bring more. If the target is not met, this task must identify the gap and either add targeted tests or document which files still need coverage.
-
-**CI-equivalent checks:**
-- `bin/rubocop -f github` (lint job)
-- `bin/brakeman --no-pager` (security job)
-- `bin/rails test` (test job)
-- diff coverage check (test job)
-</context>
-
-## Tasks
-
-### Task 1: Regenerate coverage baseline
-
-- **name:** regenerate-coverage-baseline
-- **files:**
-  - `config/coverage_baseline.json`
-- **action:** Run the full test suite with coverage enabled: `COVERAGE=1 bin/rails test`. Then regenerate the baseline: `bin/update-coverage-baseline`. Compare the new uncovered line count to the original 2117. The target is at most 847 uncovered lines (60% reduction). If the target is met, commit the regenerated baseline. If not, document the gap and identify which files still have the most uncovered lines for targeted fix in Task 2.
-- **verify:** `ruby -rjson -e 'data = JSON.parse(File.read("config/coverage_baseline.json")); total = data.values.map(&:size).sum; puts "Uncovered: #{total}"; exit(total <= 847 ? 0 : 1)'` exits 0
-- **done:** Coverage baseline regenerated. Uncovered line count documented.
-
-### Task 2: Address coverage gap if target not met
-
-- **name:** address-coverage-gap
-- **files:**
-  - Test files as needed (determined by Task 1 gap analysis)
-  - `config/coverage_baseline.json` (re-regenerate after adding tests)
-- **action:** If Task 1 shows the 60% reduction target is NOT met, analyze the regenerated baseline to find the largest remaining gaps. Add targeted tests for the top uncovered files until the 847-line target is met. Focus on files with the most uncovered lines that are NOT in the `:nocov:` exclusion zones. After adding tests, re-run `COVERAGE=1 bin/rails test` and `bin/update-coverage-baseline` to verify. If the target IS already met from Task 1, this task is a no-op -- simply verify and move on.
-- **verify:** `ruby -rjson -e 'data = JSON.parse(File.read("config/coverage_baseline.json")); total = data.values.map(&:size).sum; puts "Uncovered: #{total}"; exit(total <= 847 ? 0 : 1)'` exits 0
-- **done:** Coverage baseline meets 60% reduction target (at most 847 uncovered lines).
-
-### Task 3: Run full CI-equivalent verification
-
-- **name:** full-ci-verification
-- **files:** (no modifications -- verification only)
-- **action:** Run all CI-equivalent checks in sequence:
-  1. `bin/rubocop -f simple` -- must show `no offenses detected`
-  2. `bin/brakeman --no-pager -q` -- must exit 0 with zero warnings
-  3. `bin/rails test` -- must exit 0 with 760+ runs and 0 failures
-  4. Verify no file in app/ or lib/ exceeds 300 lines: `find app lib -name '*.rb' -exec wc -l {} + | sort -rn | awk '$1 > 300 && $2 != "total" {print; found=1} END {exit found ? 1 : 0}'`
-  5. Verify all models have `frozen_string_literal: true`: `grep -rL 'frozen_string_literal: true' app/models/source_monitor/*.rb` returns empty
-  6. Verify all controllers have `frozen_string_literal: true`: `grep -rL 'frozen_string_literal: true' app/controllers/source_monitor/*.rb` returns empty
-
-  Document any failures and fix them before marking this task done.
-- **verify:** All 6 checks above pass
-- **done:** All CI-equivalent checks pass. Codebase fully clean.
-
-### Task 4: Final conventions spot-check
-
-- **name:** final-conventions-spot-check
-- **files:** (read-only audit, fix only if issues found)
-- **action:** Do a final walkthrough of all models, controllers, and service objects checking:
-  - All models use `ModelExtensions.register(self, :key)` (except ApplicationRecord)
-  - All models have appropriate validations for their associations
-  - All service objects follow the `initialize`/`call` pattern or `self.call` class method
-  - All jobs inherit from ApplicationJob and use `source_monitor_queue`
-  - All concerns use `extend ActiveSupport::Concern` and `included do...end`
-  - No commented-out code blocks remain
-  - No TODO/FIXME/HACK comments without associated tracking
-  - All Struct definitions use `keyword_init: true`
-
-  Fix any issues found. This should be a light pass since most conventions were already followed.
-- **verify:** `bin/rails test` exits 0 AND `bin/rubocop -f simple` shows `no offenses detected`
-- **done:** All conventions verified. Codebase passes final quality gate.
-
-## Verification
-
-1. `bin/rails test` exits 0 with 760+ runs and 0 failures
-2. `bin/rubocop -f simple` shows `no offenses detected`
-3. `bin/brakeman --no-pager -q` exits 0
-4. Coverage baseline has at most 847 uncovered lines
-5. No Ruby file in app/ or lib/ exceeds 300 lines
-6. All frozen_string_literal pragmas present
-
-## Success Criteria
-
-- [ ] Coverage baseline regenerated and at most 847 uncovered lines (60% reduction from 2117)
-- [ ] Zero RuboCop violations
-- [ ] Zero Brakeman warnings
-- [ ] All 760+ tests pass with 0 failures
-- [ ] No file in app/ or lib/ exceeds 300 lines
-- [ ] All conventions verified in final spot-check
-- [ ] Phase 4 complete -- all ROADMAP success criteria met

data/.vbw-planning/milestones/generator-enhancements/REQUIREMENTS.md

@@ -1,72 +0,0 @@
-<!-- VBW REQUIREMENTS TEMPLATE (ARTF-06) -- Structured requirements with traceability -->
-<!-- Created by Architect agent during /vbw scope -->
-
-# SourceMonitor Requirements
-
-Defined: 2026-02-09
-Core value: Drop-in Rails engine for feed monitoring, content scraping, and operational dashboards.
-
-## v1 Requirements
-
-### Test Coverage
-
-- [ ] **REQ-01**: Close coverage gaps in `FeedFetcher` -- add tests for uncovered branches in the fetch pipeline
-- [ ] **REQ-02**: Close coverage gaps in `ItemCreator` -- add tests for item creation edge cases
-- [ ] **REQ-03**: Close coverage gaps in `Configuration` -- test nested settings classes and edge cases
-- [ ] **REQ-04**: Close coverage gaps in `Dashboard::Queries` -- test dashboard query logic
-- [ ] **REQ-05**: Close coverage gaps in `Broadcaster` -- test realtime broadcasting logic
-- [ ] **REQ-06**: Close coverage gaps in `BulkSourceScraper` -- test bulk scraping workflows
-- [ ] **REQ-07**: Close coverage gaps in `SourcesIndexMetrics` -- test analytics calculations
-
-### Refactoring
-
-- [ ] **REQ-08**: Extract `FeedFetcher` (627 lines) into focused single-responsibility classes
-- [ ] **REQ-09**: Extract `Configuration` (655 lines) nested settings classes into separate files
-- [ ] **REQ-10**: Extract `ImportSessionsController` (792 lines) wizard steps into step-specific concerns or service objects
-- [ ] **REQ-11**: Fix `LogEntry` hard-coded table name to use configurable prefix system
-- [ ] **REQ-12**: Replace eager 102+ require statements in `lib/source_monitor.rb` with autoloading
-
-### Code Quality
-
-- [ ] **REQ-13**: Ensure frozen_string_literal is consistent across all Ruby files
-- [ ] **REQ-14**: Audit and fix any RuboCop violations against omakase ruleset
-- [ ] **REQ-15**: Ensure all models, controllers, and service objects follow Rails conventions
-
-### Generator Enhancements
-
-- [ ] **REQ-16**: Install generator patches `Procfile.dev` with a `jobs:` entry for Solid Queue
-- [ ] **REQ-17**: Install generator patches queue config dispatcher with `recurring_schedule: config/recurring.yml`
-- [ ] **REQ-18**: Guided workflow (`Setup::Workflow`) integrates both new generator steps
-- [ ] **REQ-19**: `RecurringScheduleVerifier` checks that recurring tasks are registered with Solid Queue
-- [ ] **REQ-20**: `SolidQueueVerifier` remediation suggests `Procfile.dev` when workers not detected
-- [ ] **REQ-21**: Skills and documentation updated to reflect automated Procfile.dev and recurring_schedule setup
-
-### Dashboard UX
-
-- [ ] **REQ-22**: Fetch logs show source URL for both success and failure entries on the dashboard
-- [ ] **REQ-23**: Dashboard links to sources and items are clickable and open in a new tab
-
-### Active Storage Image Downloads
-
-- [ ] **REQ-24**: Configurable option to download inline images from items to Active Storage instead of loading from source
-
-### Feed Compatibility
-
-- [ ] **REQ-25**: Investigate and fix failing fetch for Netflix Tech Blog feed (https://netflixtechblog.com/feed)
-
-## v2 Requirements
-
-- [ ] **REQ-XX**: Improve optional dependency loading with clear error messages
-- [ ] **REQ-XX**: Add database index verification tooling
-- [ ] **REQ-XX**: Document health check endpoint response format
-
-## Out of Scope
-
-| Item | Reason |
-|------|--------|
-| Multi-database support (MySQL/SQLite) | PostgreSQL-only simplifies development |
-| Built-in authentication | Host app responsibility |
-
-## Traceability
-
-Requirement-to-phase mapping is tracked in ROADMAP.md.

data/.vbw-planning/milestones/generator-enhancements/ROADMAP.md

@@ -1,125 +0,0 @@
-<!-- VBW ROADMAP -- Phase decomposition with requirement mapping -->
-<!-- Created during /vbw scope -->
-
-# SourceMonitor Generator Enhancements Roadmap
-
-**Milestone:** generator-enhancements
-**Goal:** Make the install generator and verification suite catch the two most common host-app setup failures: missing Procfile.dev jobs entry and missing recurring_schedule dispatcher wiring.
-
-## Phases
-
-1. [x] ~~Phase 0: Documentation Gaps~~ (shipped via quick fix ea788ea)
-2. [x] Phase 1: Install Generator Steps (Procfile.dev + Queue Config)
-3. [x] Phase 2: Recurring Schedule Verifier
-4. [x] Phase 3: Skills & Documentation Alignment
-5. [x] Phase 4: Dashboard UX Improvements
-6. [x] Phase 5: Active Storage Image Downloads
-7. [x] Phase 6: Netflix Feed Investigation
-
-## Phase Details
-
-### Phase 1: Install Generator Steps
-
-**Goal:** Add two new idempotent steps to the install generator: (a) patch `Procfile.dev` with a `jobs:` entry for Solid Queue, and (b) patch the queue config dispatcher with `recurring_schedule: config/recurring.yml`.
-
-**Requirements:** REQ-16, REQ-17, REQ-18
-
-**Success Criteria:**
-- `bin/rails generate source_monitor:install` patches Procfile.dev when present (idempotent, skip if entry exists)
-- `bin/rails generate source_monitor:install` patches queue.yml dispatcher with recurring_schedule (idempotent)
-- Both steps wired into `Setup::Workflow` for the guided installer
-- Generator tests cover: fresh file, existing file with entry, existing file without entry, missing file
-- `bin/rails test` passes, RuboCop clean
-
-### Phase 2: Recurring Schedule Verifier
-
-**Goal:** Add a `RecurringScheduleVerifier` to the verification suite that checks whether recurring tasks are actually registered with Solid Queue dispatchers, and enhance the existing `SolidQueueVerifier` to suggest Procfile.dev when workers aren't detected.
-
-**Requirements:** REQ-19, REQ-20
-
-**Success Criteria:**
-- `bin/source_monitor verify` checks that recurring tasks are registered (not just that workers heartbeat)
-- Warning when no recurring tasks found with actionable remediation message
-- SolidQueueVerifier remediation mentions Procfile.dev for `bin/dev` users
-- Verifier tests with mocked Solid Queue state
-- `bin/rails test` passes, RuboCop clean
-
-### Phase 3: Skills & Documentation Alignment
-
-**Goal:** Update all `sm-*` skills and docs to reflect that the generator now automatically handles Procfile.dev and recurring_schedule wiring. Remove manual steps that are now automated.
-
-**Requirements:** REQ-21
-
-**Success Criteria:**
-- sm-host-setup skill reflects new generator capabilities (auto-patching, not manual steps)
-- sm-configure skill references the automatic recurring_schedule wiring
-- docs/setup.md updated to note the generator handles both automatically
-- docs/troubleshooting.md updated with improved diagnostics
-- setup-checklist.md reflects automation (checked by default, not manual)
-
-### Phase 4: Dashboard UX Improvements
-
-**Goal:** Show source URLs in fetch log entries for both successes and failures on the dashboard, and make links to sources and items clickable (opening in a new tab).
-
-**Requirements:** REQ-22, REQ-23
-
-**Success Criteria:**
-- Fetch log entries on the dashboard display the source URL alongside the existing summary
-- Both success and failure fetch logs show the URL
-- Source names and item titles are clickable links that open in a new tab
-- Existing dashboard layout is preserved
-- `bin/rails test` passes, RuboCop clean
-
-### Phase 5: Active Storage Image Downloads
-
-**Goal:** Add a configurable option to download inline images from feed items to Active Storage instead of loading them directly from the source URL. This prevents broken images when sources go offline and improves page load performance.
-
-**Requirements:** REQ-24
-
-**Success Criteria:**
-- New configuration option (`config.images.download_to_active_storage` or similar) defaults to `false`
-- When enabled, inline images in item content are detected and downloaded to Active Storage
-- Original image URLs are replaced with Active Storage URLs in the stored content
-- Images that fail to download gracefully fall back to original URLs
-- Configuration is documented in sm-configure skill
-- `bin/rails test` passes, RuboCop clean
-
-### Phase 6: Netflix Feed Investigation
-
-**Goal:** Investigate and fix the failing fetch for `https://netflixtechblog.com/feed`. Determine whether the issue is in the feed parser, HTTP client configuration, or content format, and apply appropriate fixes.
-
-**Requirements:** REQ-25
-
-**Success Criteria:**
-- Root cause identified and documented
-- Fix applied (parser, HTTP client, or configuration change)
-- Netflix Tech Blog feed fetches successfully
-- No regressions in other feed types
-- `bin/rails test` passes, RuboCop clean
-
-## Progress
-
-| Phase | Status | Plans |
-|-------|--------|-------|
-| 0 | Complete | - |
-| 1 | Complete | PLAN-01 (5 tasks, 4 commits) |
-| 2 | Complete | PLAN-01 (5 tasks, 1 commit) |
-| 3 | Complete | PLAN-01 (5 tasks, 1 commit) |
-| 4 | Complete | PLAN-01 (5 tasks, 5 commits) |
-| 5 | Complete | PLAN-01 (4 tasks, 5 commits) + PLAN-02 (4 tasks, 4 commits) |
-| 6 | Complete | PLAN-01 (5 tasks, 5 commits) |
-
-## Requirement Mapping
-
-| REQ | Phase | Description |
-|-----|-------|-------------|
-| REQ-16 | 1 | Generator patches Procfile.dev with jobs: entry |
-| REQ-17 | 1 | Generator patches queue config with recurring_schedule |
-| REQ-18 | 1 | Guided workflow integrates both new steps |
-| REQ-19 | 2 | RecurringScheduleVerifier checks recurring task registration |
-| REQ-20 | 2 | SolidQueueVerifier remediation mentions Procfile.dev |
-| REQ-21 | 3 | Skills and docs reflect automated setup |
-| REQ-22 | 4 | Fetch logs show source URL on dashboard |
-| REQ-23 | 4 | Dashboard links clickable in new tab |
-| REQ-24 | 5 | Download inline images to Active Storage |
-| REQ-25 | 6 | Fix Netflix Tech Blog feed fetch |

data/.vbw-planning/milestones/generator-enhancements/SHIPPED.md

@@ -1,40 +0,0 @@
-# Shipped: generator-enhancements
-
-**Shipped:** 2026-02-12
-**Tag:** milestone/generator-enhancements
-**Release:** v0.4.0
-
-## Summary
-
-Made the install generator and verification suite catch the two most common host-app setup failures: missing Procfile.dev jobs entry and missing recurring_schedule dispatcher wiring. Also added dashboard UX improvements, Active Storage image downloads, and SSL certificate store configuration.
-
-## Metrics
-
-| Metric | Value |
-|--------|-------|
-| Phases | 7 (Phase 0-6) |
-| Plans completed | 7 |
-| Tasks completed | 34 |
-| Commits | ~26 |
-| Tests | 973 (up from 841) |
-| New requirements satisfied | 10 (REQ-16 through REQ-25) |
-
-## Phases
-
-1. Phase 0: Documentation Gaps (quick fix)
-2. Phase 1: Install Generator Steps (Procfile.dev + Queue Config)
-3. Phase 2: Recurring Schedule Verifier
-4. Phase 3: Skills & Documentation Alignment
-5. Phase 4: Dashboard UX Improvements
-6. Phase 5: Active Storage Image Downloads
-7. Phase 6: Netflix Feed Investigation (SSL cert store fix)
-
-## Key Decisions
-
-- Docs-first approach: shipped doc fixes before code changes
-- Always create/patch Procfile.dev for maximum hand-holding
-- Target queue.yml only (Rails 8 default)
-- External links open new tab with visual indicator icon
-- Fetch log URL display: domain for RSS, item URL for scrapes
-- Image downloads via background job to ItemContent (opt-in)
-- SSL fix: general cert store config, not Netflix-specific