source_monitor 0.2.0 → 0.3.0

data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-01.md

@@ -0,0 +1,142 @@
+---
+phase: 2
+plan: 1
+title: feed-fetcher-tests
+wave: 1
+depends_on: []
+skills_used: []
+must_haves:
+  truths:
+    - "Running `bin/rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb` exits 0 with zero failures"
+    - "Coverage report shows lib/source_monitor/fetching/feed_fetcher.rb has fewer than 50 uncovered lines (down from 245)"
+    - "All new tests use WebMock stubs or VCR cassettes -- no real HTTP requests"
+    - "Running `bin/rails test` exits 0 with no regressions"
+  artifacts:
+    - "test/lib/source_monitor/fetching/feed_fetcher_test.rb -- extended with new test methods covering retry strategy, error handling, header management, entry processing, and helper methods"
+  key_links:
+    - "REQ-01 substantially satisfied -- FeedFetcher branch coverage above 80%"
+---
+
+# Plan 01: feed-fetcher-tests
+
+## Objective
+
+Close the coverage gap in `lib/source_monitor/fetching/feed_fetcher.rb` (currently 245 uncovered lines out of 627). The existing test file covers basic RSS/Atom/JSON fetching, 304 handling, timeout/HTTP/parsing failures, and adaptive interval mechanics. This plan targets the remaining uncovered branches: retry strategy application, circuit breaker state transitions, connection error wrapping, last_modified header handling, entry processing edge cases, jitter computation, and private helper methods.
+
+## Context
+
+<context>
+@lib/source_monitor/fetching/feed_fetcher.rb -- 627 lines, the core fetch pipeline
+@lib/source_monitor/fetching/retry_policy.rb -- RetryPolicy with Decision struct
+@lib/source_monitor/fetching/fetch_error.rb -- error hierarchy (TimeoutError, ConnectionError, HTTPError, ParsingError, UnexpectedResponseError)
+@test/lib/source_monitor/fetching/feed_fetcher_test.rb -- existing test file with 12 tests covering success, 304, timeout, HTTP 404, parsing failure, adaptive intervals
+@config/coverage_baseline.json -- lists 245 uncovered lines for feed_fetcher.rb
+@test/test_helper.rb -- test infrastructure (WebMock, VCR, create_source!, with_queue_adapter)
+
+**Decomposition rationale:** FeedFetcher is the single largest coverage gap (245 lines). It is tested in isolation from ItemCreator (which has its own plan). The uncovered code falls into distinct categories that can each be addressed as a focused task: (1) retry/circuit breaker logic, (2) Faraday error wrapping and connection failures, (3) header and metadata management, (4) entry processing edge cases, (5) jitter and interval helpers. Each task adds test methods to the existing test file.
+
+**Trade-offs considered:**
+- Could split FeedFetcher tests across multiple test files (e.g., feed_fetcher_retry_test.rb), but keeping them in one file matches the existing codebase convention and avoids confusion.
+- Could use mocks for RetryPolicy, but testing the integration between FeedFetcher and RetryPolicy provides higher confidence.
+
+**What constrains the structure:**
+- Tests must use WebMock stubs (no real HTTP)
+- All tests go in the existing test file to avoid file proliferation
+- The `jitter: ->(_) { 0 }` pattern from existing tests should be reused to make interval assertions deterministic
+- Tests need `travel_to` for time-sensitive assertions
+</context>
+
+## Tasks
+
+### Task 1: Test retry strategy and circuit breaker transitions
+
+- **name:** test-retry-and-circuit-breaker
+- **files:**
+  - `test/lib/source_monitor/fetching/feed_fetcher_test.rb`
+- **action:** Add tests covering lines 262-298 (reset_retry_state!, apply_retry_strategy!) and lines 267-290 (retry/circuit decisions). Specifically:
+  1. Test that a first timeout failure sets fetch_retry_attempt to 1 (retry decision with wait)
+  2. Test that exhausting retry attempts opens the circuit (sets fetch_circuit_opened_at, fetch_circuit_until, next_fetch_at)
+  3. Test that a successful fetch after retries resets retry state (fetch_retry_attempt=0, circuit fields nil)
+  4. Test that apply_retry_strategy! handles StandardError by logging and setting defaults (lines 291-298) -- simulate by stubbing RetryPolicy to raise
+  5. Test that retry decision adjusts next_fetch_at to the minimum of current and retry time (line 283)
+  Use WebMock stubs: first stub raises Faraday::TimeoutError to trigger retries, then stub success for recovery tests. Set source.fetch_retry_attempt manually to simulate multiple failures.
+- **verify:** `bin/rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb -n /retry|circuit/` exits 0
+- **done:** Lines 262-298 covered. Source retry fields verified in assertions.
+
+### Task 2: Test Faraday error wrapping and connection failures
+
+- **name:** test-faraday-error-wrapping
+- **files:**
+  - `test/lib/source_monitor/fetching/feed_fetcher_test.rb`
+- **action:** Add tests covering lines 77-86 (perform_fetch error wrapping) and lines 405-417 (build_http_error_from_faraday). Specifically:
+  1. Test that Faraday::ConnectionFailed raises SourceMonitor::Fetching::ConnectionError with original_error preserved
+  2. Test that Faraday::SSLError raises ConnectionError
+  3. Test that Faraday::ClientError with response hash builds HTTPError via build_http_error_from_faraday (lines 405-417) with correct status, message, and ResponseWrapper
+  4. Test that generic Faraday::Error raises FetchError
+  5. Test that a non-Faraday StandardError is wrapped in UnexpectedResponseError (lines 52-54)
+  Use WebMock's `to_raise` for each error type. Assert the result.error class, message, and original_error.
+- **verify:** `bin/rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb -n /connection|ssl|client_error|unexpected|faraday/i` exits 0
+- **done:** Lines 77-86 and 405-417 covered. All Faraday error types correctly wrapped.
+
+### Task 3: Test Last-Modified header handling and request headers
+
+- **name:** test-last-modified-and-headers
+- **files:**
+  - `test/lib/source_monitor/fetching/feed_fetcher_test.rb`
+- **action:** Add tests covering lines 97-104 (request_headers with custom_headers, etag, last_modified), lines 203-215 and 228-240 (response Last-Modified parsing and storage), and lines 349-355 (parse_http_time). Specifically:
+  1. Test that If-Modified-Since header is sent when source.last_modified is set
+  2. Test that Last-Modified response header is parsed and stored on source
+  3. Test that malformed Last-Modified headers are silently ignored (parse_http_time returns nil for invalid dates, line 353)
+  4. Test that custom_headers from source are passed through to the request (lines 98)
+  5. Test that both ETag and Last-Modified are preserved on 304 not_modified responses (lines 228-234)
+  Use WebMock to verify request headers via `.with(headers: {...})` and return specific response headers.
+- **verify:** `bin/rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb -n /last_modified|custom_header|if_modified/i` exits 0
+- **done:** Lines 97-104, 203-215, 228-240, 349-355 covered.
+
+### Task 4: Test entry processing edge cases and error normalization
+
+- **name:** test-entry-processing-edges
+- **files:**
+  - `test/lib/source_monitor/fetching/feed_fetcher_test.rb`
+- **action:** Add tests covering lines 520-567 (process_feed_entries) and lines 603-624 (normalize_item_error, safe_entry_guid, safe_entry_title). Specifically:
+  1. Test that a feed without entries (feed that doesn't respond_to :entries) returns zero counts (line 529)
+  2. Test that Events.run_item_processors is called for each entry (line 542)
+  3. Test that Events.after_item_created is called only for created items (line 547), not updated ones
+  4. Test that normalize_item_error extracts guid via entry_id (line 615-616), falls back to id (line 617-618), and handles entries without either
+  5. Test that safe_entry_title returns nil when entry doesn't respond_to :title (line 623)
+  Use a simple XML feed fixture with known entries. Mock ItemCreator to control created vs updated results. Use Notifications subscriptions to verify event dispatch.
+- **verify:** `bin/rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb -n /entry_processing|item_processor|normalize_error|safe_entry/i` exits 0
+- **done:** Lines 520-567, 603-624 covered.
+
+### Task 5: Test jitter, interval helpers, and metadata management
+
+- **name:** test-jitter-and-interval-helpers
+- **files:**
+  - `test/lib/source_monitor/fetching/feed_fetcher_test.rb`
+- **action:** Add tests covering lines 490-518 (jitter_offset, adjusted_interval_with_jitter, body_digest, updated_metadata) and lines 569-600 (configured_seconds, configured_positive, configured_non_negative, extract_numeric, fetching_config). Specifically:
+  1. Test jitter_offset returns 0 when interval_seconds <= 0 (line 505)
+  2. Test jitter_offset uses jitter_proc when provided (line 506)
+  3. Test jitter_offset computes random jitter within expected range when no proc given
+  4. Test body_digest returns nil for blank body (line 515), returns SHA256 for non-blank
+  5. Test updated_metadata preserves existing metadata, removes dynamic_fetch_interval_seconds key, adds last_feed_signature
+  6. Test configured_seconds returns default when minutes_value is nil or non-positive (lines 570-571)
+  7. Test extract_numeric handles Numeric, responds_to :to_f, and non-numeric values (lines 590-596)
+  These are private methods -- test them through the public `call` method by configuring specific fetching settings and verifying the resulting source state. Alternatively, use `send` for the pure-function helpers.
+- **verify:** `bin/rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb -n /jitter|body_digest|metadata|configured_|extract_numeric/i` exits 0
+- **done:** Lines 490-518 and 569-600 covered.
+
+## Verification
+
+1. `bin/rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb` exits 0
+2. `COVERAGE=1 bin/rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb` shows feed_fetcher.rb with >80% branch coverage
+3. `bin/rails test` exits 0 (no regressions)
+
+## Success Criteria
+
+- [ ] FeedFetcher coverage drops from 245 uncovered lines to fewer than 50
+- [ ] All retry/circuit breaker branches tested
+- [ ] All Faraday error wrapping branches tested
+- [ ] All header handling branches tested
+- [ ] Entry processing and error normalization branches tested
+- [ ] Jitter, interval helpers, and metadata management tested
+- [ ] REQ-01 substantially satisfied

data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-02-SUMMARY.md

@@ -0,0 +1,64 @@
+# PLAN-02 Summary: item-creator-tests
+
+## Status: COMPLETE
+
+## Commit
+
+- **Hash:** `ce8ede4`
+- **Message:** `test(item-creator): close coverage gaps for URL/content, dupes, authors, errors, utils`
+- **Files changed:** 1 file, 941 insertions
+
+## Tasks Completed
+
+### Task 1: Test URL extraction fallbacks and content extraction chain
+- Tested extract_url with link_nodes alternate link, links array fallback, nil handling
+- Tested extract_content priority: content > content_encoded > summary
+- Tested extract_updated_timestamp returns entry.updated when present
+
+### Task 2: Test concurrent duplicate handling (RecordNotUnique)
+- Tested RecordNotUnique for guid conflict finds and updates existing item
+- Tested RecordNotUnique for fingerprint conflict finds by fingerprint
+- Tested handle_concurrent_duplicate returns Result with status: :updated
+
+### Task 3: Test multi-format author, enclosure, and media extraction
+- Tested extract_authors from dc_creators, author_nodes (email/uri), deduplication
+- Tested extract_enclosures from RSS enclosure_nodes, Atom link_nodes, JSON attachments
+- Tested extract_media_content with url, type, medium, dimensions
+- Tested extract_media_thumbnail_url fallback from nodes to entry.image
+- Tested blank URL filtering in enclosures
+
+### Task 4: Test feed content processing error paths and readability edge cases
+- Tested parser error produces item with raw content and error metadata
+- Tested should_process_feed_content? returns false for blank/non-HTML content
+- Tested deep_copy handles Hash, Array, deep_dup, TypeError rescue
+- Tested build_feed_content_metadata includes readability_text_length
+- Tested html_fragment? detection
+
+### Task 5: Test utility methods
+- Tested safe_integer: nil, Integer, string "42", non-numeric string
+- Tested split_keywords: commas, semicolons, whitespace stripping
+- Tested extract_guid dedup when entry_id equals URL
+- Tested extract_language from json_entry and entry.language paths
+- Tested normalize_metadata JSON roundtrip and unparseable values
+- Tested extract_comments_count slash_comments_raw/comments_count fallback
+
+## Deviations
+
+None -- plan executed as specified.
+
+## Verification Results
+
+| Check | Result |
+|-------|--------|
+| `bin/rails test test/lib/source_monitor/items/item_creator_test.rb` | All tests pass |
+| `bin/rails test` | 760 runs, 2626 assertions, 0 failures, 0 errors, 0 skips |
+
+## Success Criteria
+
+- [x] 56 new tests added (78 total, 941 lines)
+- [x] URL extraction fallback branches tested
+- [x] Concurrent duplicate handling tested
+- [x] Multi-format author/enclosure/media extraction tested
+- [x] Feed content processing error paths tested
+- [x] All utility methods tested
+- [x] REQ-02 substantially satisfied

data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-02.md

@@ -0,0 +1,138 @@
+---
+phase: 2
+plan: 2
+title: item-creator-tests
+wave: 1
+depends_on: []
+skills_used: []
+must_haves:
+  truths:
+    - "Running `bin/rails test test/lib/source_monitor/items/item_creator_test.rb` exits 0 with zero failures"
+    - "Coverage report shows lib/source_monitor/items/item_creator.rb has fewer than 40 uncovered lines (down from 228)"
+    - "Running `bin/rails test` exits 0 with no regressions"
+  artifacts:
+    - "test/lib/source_monitor/items/item_creator_test.rb -- extended with new test methods covering URL extraction, content extraction, concurrent duplicate handling, author/enclosure/media extraction, feed content processing, and utility methods"
+  key_links:
+    - "REQ-02 substantially satisfied -- ItemCreator branch coverage above 80%"
+---
+
+# Plan 02: item-creator-tests
+
+## Objective
+
+Close the coverage gap in `lib/source_monitor/items/item_creator.rb` (currently 228 uncovered lines out of 601). The existing test file covers basic RSS/Atom/JSON creation, fingerprint generation, guid fallback, readability processing, metadata extraction, and guid/fingerprint deduplication. This plan targets the remaining uncovered branches: URL extraction from link_nodes and links arrays, content extraction fallback chain, concurrent duplicate handling (RecordNotUnique), author extraction from multiple sources, enclosure extraction from Atom link_nodes and JSON attachments, media content extraction, language/copyright/comments extraction, keyword splitting, safe_integer edge cases, and the deep_copy utility.
+
+## Context
+
+<context>
+@lib/source_monitor/items/item_creator.rb -- 601 lines, item creation with dedup logic
+@test/lib/source_monitor/items/item_creator_test.rb -- existing test file with 8 tests
+@config/coverage_baseline.json -- lists 228 uncovered lines for item_creator.rb
+@test/fixtures/feeds/ -- RSS, Atom, JSON feed fixtures
+@test/test_helper.rb -- test infrastructure
+
+**Decomposition rationale:** ItemCreator is the second largest coverage gap. Its uncovered lines cluster into: (1) URL/content/timestamp extraction branches, (2) concurrent duplicate handling, (3) multi-format author/enclosure/media extraction, (4) feed content processing error paths, (5) utility methods. Each task targets a distinct cluster.
+
+**Trade-offs considered:**
+- Many branches are triggered by specific Feedjira entry types (AtomEntry, JSONFeedItem). Tests need mock entries or real parsed fixtures to exercise these.
+- The concurrent duplicate test (RecordNotUnique) requires careful setup to simulate a race condition without actually racing.
+- Testing private methods through the public `call` interface keeps tests realistic, but some deep utility methods (deep_copy, safe_integer, split_keywords) are easier to verify directly.
+
+**What constrains the structure:**
+- Must use Feedjira-parsed entries (not plain hashes) to exercise respond_to? checks
+- Atom and JSON-specific branches need entries of the correct type
+- Tests extend the existing test file
+</context>
+
+## Tasks
+
+### Task 1: Test URL extraction fallbacks and content extraction chain
+
+- **name:** test-url-and-content-extraction
+- **files:**
+  - `lib/source_monitor/items/item_creator.rb` -- (read-only reference)
+  - `test/lib/source_monitor/items/item_creator_test.rb`
+- **action:** Add tests covering lines 288-310 (extract_url with link_nodes and links arrays), lines 318-336 (extract_content with CONTENT_METHODS fallback chain), and lines 328-342 (extract_timestamp, extract_updated_timestamp). Specifically:
+  1. Test extract_url when entry.url is blank but entry.link_nodes has an alternate link with href -- use an Atom entry fixture
+  2. Test extract_url when entry.url is blank and link_nodes are empty but entry.links has a URL string
+  3. Test extract_url returns nil when no URL source is available (mock entry with no url/link_nodes/links)
+  4. Test extract_content tries :content, then :content_encoded, then :summary in order -- create mock entries that respond to different method subsets
+  5. Test extract_updated_timestamp returns entry.updated when present, nil otherwise
+  Use OpenStruct or Minitest::Mock to create entries with specific respond_to? patterns for edge cases not covered by real feed fixtures.
+- **verify:** `bin/rails test test/lib/source_monitor/items/item_creator_test.rb -n /url_extraction|content_extraction|updated_timestamp/i` exits 0
+- **done:** Lines 288-310, 318-342 covered.
+
+### Task 2: Test concurrent duplicate handling (RecordNotUnique)
+
+- **name:** test-concurrent-duplicate-handling
+- **files:**
+  - `test/lib/source_monitor/items/item_creator_test.rb`
+- **action:** Add tests covering lines 104-128 (create_new_item rescue RecordNotUnique, handle_concurrent_duplicate, find_conflicting_item). Specifically:
+  1. Test that when create_new_item raises ActiveRecord::RecordNotUnique for a guid conflict, the item is found and updated instead
+  2. Test that when RecordNotUnique fires for a fingerprint conflict (no raw guid), the item is found by fingerprint and updated
+  3. Test that handle_concurrent_duplicate returns a Result with status: :updated and the correct matched_by
+  Simulate RecordNotUnique by: (a) creating an item first, (b) stubbing source.items.new to return an item that raises RecordNotUnique on save!, then verifying the fallback path finds and updates the existing item.
+- **verify:** `bin/rails test test/lib/source_monitor/items/item_creator_test.rb -n /concurrent_duplicate|record_not_unique/i` exits 0
+- **done:** Lines 104-128 covered.
+
+### Task 3: Test multi-format author, enclosure, and media extraction
+
+- **name:** test-author-enclosure-media-extraction
+- **files:**
+  - `test/lib/source_monitor/items/item_creator_test.rb`
+- **action:** Add tests covering lines 348-383 (extract_authors with rss_authors, dc_creators, author_nodes, json authors), lines 416-466 (extract_enclosures from rss enclosure_nodes, atom link_nodes with rel=enclosure, json attachments), and lines 477-499 (extract_media_content). Specifically:
+  1. Test extract_authors collects from rss_authors, dc_creators, dc_creator, and author_nodes (name/email/uri) -- deduplicates and compacts
+  2. Test extract_enclosures from Atom link_nodes with rel="enclosure" produces entries with source: "atom_link"
+  3. Test extract_media_content builds array from media_content_nodes with url, type, medium, height, width, file_size, duration, expression -- compacted
+  4. Test extract_media_thumbnail_url falls back from media_thumbnail_nodes to entry.image
+  5. Test extract_enclosures skips entries with blank URLs
+  Build mock entries using Struct or OpenStruct to simulate the various node types. The JSON-specific paths are already partially tested; focus on Atom and RSS edge cases.
+- **verify:** `bin/rails test test/lib/source_monitor/items/item_creator_test.rb -n /authors|enclosure|media_content|media_thumbnail/i` exits 0
+- **done:** Lines 348-383, 416-466, 477-499 covered.
+
+### Task 4: Test feed content processing error path and readability edge cases
+
+- **name:** test-feed-content-processing-errors
+- **files:**
+  - `test/lib/source_monitor/items/item_creator_test.rb`
+- **action:** Add tests covering lines 137-158 (process_feed_content error rescue), lines 160-165 (should_process_feed_content?), lines 187-208 (default_feed_readability_options, build_feed_content_metadata), and lines 210-231 (html_fragment?, deep_copy). Specifically:
+  1. Test that when readability parser raises an error, the item is created with raw content and error metadata (lines 148-157): metadata has "status"=>"failed", "error_class", "error_message"
+  2. Test should_process_feed_content? returns false when content is blank or when content has no HTML tags (html_fragment? returns false)
+  3. Test deep_copy handles Hash, Array, and objects that support deep_dup, plus TypeError rescue
+  4. Test build_feed_content_metadata includes readability_text_length and title when present
+  5. Test html_fragment? returns true for `<p>text</p>` and false for plain text
+  Stub the parser class to raise for the error path test. Use source with feed_content_readability_enabled: true.
+- **verify:** `bin/rails test test/lib/source_monitor/items/item_creator_test.rb -n /processing_error|html_fragment|deep_copy|readability_metadata/i` exits 0
+- **done:** Lines 137-165, 187-231 covered.
+
+### Task 5: Test utility methods: safe_integer, split_keywords, string_or_nil, normalize_metadata, extract_guid edge cases
+
+- **name:** test-utility-methods
+- **files:**
+  - `test/lib/source_monitor/items/item_creator_test.rb`
+- **action:** Add tests covering lines 501-534 (extract_language, extract_copyright, extract_comments_url, extract_comments_count), lines 536-543 (extract_metadata with normalize_metadata), lines 555-597 (string_or_nil, sanitize_string_array, split_keywords, safe_integer, json_entry?, atom_entry?, normalize_metadata). Specifically:
+  1. Test safe_integer returns nil for nil, returns Integer for Integer, parses string "42", returns nil for non-numeric string (lines 574-584)
+  2. Test split_keywords splits on commas and semicolons, strips whitespace, removes blank entries (lines 565-572)
+  3. Test extract_guid returns nil when entry_id equals URL (dedup logic at line 283-285)
+  4. Test extract_language from json_entry? path (line 506-507) and from entry.language (line 502-503)
+  5. Test normalize_metadata returns empty hash for unparseable values (JSON roundtrip at lines 594-597)
+  6. Test extract_comments_count tries slash_comments_raw then comments_count (lines 529-533)
+  Use mock entries with targeted respond_to? patterns. For json_entry? and atom_entry? tests, use actual Feedjira-parsed entries from fixtures.
+- **verify:** `bin/rails test test/lib/source_monitor/items/item_creator_test.rb -n /safe_integer|split_keywords|extract_guid_edge|language|copyright|normalize_metadata|comments_count/i` exits 0
+- **done:** Lines 501-597 covered.
+
+## Verification
+
+1. `bin/rails test test/lib/source_monitor/items/item_creator_test.rb` exits 0
+2. `COVERAGE=1 bin/rails test test/lib/source_monitor/items/item_creator_test.rb` shows item_creator.rb with >80% branch coverage
+3. `bin/rails test` exits 0 (no regressions)
+
+## Success Criteria
+
+- [ ] ItemCreator coverage drops from 228 uncovered lines to fewer than 40
+- [ ] URL extraction fallback branches tested
+- [ ] Concurrent duplicate handling tested
+- [ ] Multi-format author/enclosure/media extraction tested
+- [ ] Feed content processing error paths tested
+- [ ] All utility methods tested
+- [ ] REQ-02 substantially satisfied

data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-03-SUMMARY.md

@@ -0,0 +1,85 @@
+# PLAN-03 Summary: configuration-tests
+
+## Status: COMPLETE
+
+## Commit
+
+- **Hash:** `66b8df2`
+- **Message:** `test(dev-plan05): close coverage gaps for bulk scraper and broadcaster`
+- **Files changed:** 1 file (configuration_test.rb), 771 insertions
+- **Note:** Commit was mislabeled as "dev-plan05" but the code changes are configuration tests matching Plan 03 tasks exactly.
+
+## Tasks Completed
+
+### Task 1: Test AuthenticationSettings handlers (lines 75-130)
+- Tested authenticate_with :symbol handler dispatches via public_send
+- Tested authenticate_with string handler converts to symbol
+- Tested authenticate_with callable with zero arity uses instance_exec
+- Tested authenticate_with callable with arity passes controller
+- Tested authenticate_with block handler
+- Tested authorize_with symbol and callable handlers
+- Tested Handler.call returns nil when callable is nil
+- Tested invalid handler raises ArgumentError
+- Tested reset! clears all handlers and methods
+- Tested authenticate_with nil returns nil handler
+
+### Task 2: Test ScrapingSettings and RetentionSettings edge cases (lines 132-164, 398-436)
+- Tested ScrapingSettings defaults (max_in_flight_per_source=25, max_bulk_batch_size=100)
+- Tested normalize_numeric: string, nil, empty string, zero, negative all handled correctly
+- Tested ScrapingSettings reset restores defaults
+- Tested RetentionSettings defaults (nil for days/max_items, :destroy strategy)
+- Tested strategy accepts :soft_delete, string "destroy", rejects :archive and non-symbolizable
+- Tested strategy normalizes nil to :destroy
+
+### Task 3: Test RealtimeSettings adapter validation and action_cable_config (lines 166-253)
+- Tested adapter= accepts :solid_cable, :redis, :async; rejects :websocket and nil
+- Tested action_cable_config for solid_cable returns merged SolidCableOptions
+- Tested action_cable_config for redis with/without url
+- Tested action_cable_config for async returns { adapter: "async" }
+- Tested SolidCableOptions.assign with hash, unknown keys, non-enumerable input
+- Tested SolidCableOptions.to_h compacts nil values
+- Tested realtime reset restores defaults
+
+### Task 4: Test Events callbacks and item_processors (lines 438-491)
+- Tested after_item_created with lambda and block registration
+- Tested after_item_scraped and after_fetch_completed registration
+- Tested multiple callbacks per event, callbacks_for unknown key returns []
+- Tested callbacks_for returns dup preventing mutation
+- Tested non-callable handler rejection (ArgumentError)
+- Tested register_item_processor with lambda, block, multiple, dup protection
+- Tested events reset! clears callbacks and item_processors
+
+### Task 5: Test Models, ModelDefinition, ConcernDefinition, ValidationDefinition (lines 493-652)
+- Tested Models.table_name_prefix default "sourcemon_"
+- Tested Models exposes all model keys, for(:source) returns definition
+- Tested Models.for(:unknown) raises ArgumentError
+- Tested include_concern with Module, block (anonymous module), string constant
+- Tested include_concern deduplication by signature (module, string, blocks differ)
+- Tested include_concern with invalid string raises on resolve
+- Tested validate with symbol, string, lambda, block; raises on invalid handler
+- Tested ValidationDefinition.signature for symbol, string, callable handlers
+- Tested each_concern returns Enumerator without block
+
+## Deviations
+
+| ID | Description | Impact |
+|----|-------------|--------|
+| DEVN-01 | Commit mislabeled as "dev-plan05" | None -- code changes are correct Plan 03 configuration tests |
+| DEVN-02 | Parallel test runner segfaults on PG fork when running single file | Tests pass with PARALLEL_WORKERS=1 and in full suite; environment issue, not code defect |
+
+## Verification Results
+
+| Check | Result |
+|-------|--------|
+| `PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/configuration_test.rb` | 81 runs, 178 assertions, 0 failures, 0 errors, 0 skips |
+| `bin/rails test` (full suite) | 760 runs, 2626 assertions, 0 failures, 0 errors, 0 skips |
+
+## Success Criteria
+
+- [x] 81 tests total (76 new, 5 existing), 771 lines added
+- [x] AuthenticationSettings handlers fully tested
+- [x] ScrapingSettings and RetentionSettings edge cases tested
+- [x] RealtimeSettings adapter validation and action_cable_config tested
+- [x] Events callbacks and item_processors tested
+- [x] Models and definition classes tested
+- [x] REQ-03 substantially satisfied

data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-03.md

@@ -0,0 +1,147 @@
+---
+phase: 2
+plan: 3
+title: configuration-tests
+wave: 1
+depends_on: []
+skills_used: []
+must_haves:
+  truths:
+    - "Running `bin/rails test test/lib/source_monitor/configuration_test.rb` exits 0 with zero failures"
+    - "Coverage report shows lib/source_monitor/configuration.rb has fewer than 20 uncovered lines (down from 94)"
+    - "Running `bin/rails test` exits 0 with no regressions"
+  artifacts:
+    - "test/lib/source_monitor/configuration_test.rb -- extended with new test methods covering AuthenticationSettings, ScrapingSettings, RealtimeSettings, RetentionSettings, Events, Models, and ModelDefinition"
+  key_links:
+    - "REQ-03 substantially satisfied -- Configuration branch coverage above 80%"
+---
+
+# Plan 03: configuration-tests
+
+## Objective
+
+Close the coverage gap in `lib/source_monitor/configuration.rb` (currently 94 uncovered lines out of 655). The existing test file has only 5 tests covering mission_control_dashboard_path, scraper registry, retention strategy default, and fetching settings. This plan targets the remaining uncovered branches across the 12 nested settings classes: AuthenticationSettings handlers, ScrapingSettings normalization, RealtimeSettings adapter validation and action_cable_config, RetentionSettings strategy validation, Events callbacks and item_processors, Models table_name_prefix and `for` method, ModelDefinition concerns and validations, ConcernDefinition resolution, and ValidationDefinition signatures.
+
+## Context
+
+<context>
+@lib/source_monitor/configuration.rb -- 655 lines with ~12 nested classes
+@test/lib/source_monitor/configuration_test.rb -- existing test file with 5 tests
+@config/coverage_baseline.json -- lists 94 uncovered lines for configuration.rb
+@test/test_helper.rb -- resets configuration each test
+
+**Decomposition rationale:** Configuration has many small nested classes, each with a few uncovered branches. Rather than one mega-task, we group by logical subsystem: (1) authentication handlers, (2) scraping/retention settings with edge cases, (3) realtime adapter validation, (4) events system, (5) models and concern/validation definitions.
+
+**Trade-offs considered:**
+- Could create separate test files per settings class, but the existing pattern is one configuration_test.rb file.
+- Some branches (like ConcernDefinition with constant string resolution) require careful setup to test constantize calls.
+- RealtimeSettings action_cable_config branches test all three adapter paths.
+
+**What constrains the structure:**
+- Each test must call SourceMonitor.reset_configuration! in setup/teardown (already present)
+- Tests should not leak state between settings classes
+- ModelDefinition tests need care around Module.new blocks
+</context>
+
+## Tasks
+
+### Task 1: Test AuthenticationSettings handlers
+
+- **name:** test-authentication-settings
+- **files:**
+  - `test/lib/source_monitor/configuration_test.rb`
+- **action:** Add tests covering lines 75-130 (AuthenticationSettings and Handler). Specifically:
+  1. Test authenticate_with(:some_method) creates a Handler with type :symbol that calls controller.public_send (lines 80-82)
+  2. Test authenticate_with { |c| c.redirect_to "/" } creates a Handler with type :callable that calls the block with controller (lines 84-88) -- test both zero-arity and one-arity blocks
+  3. Test authorize_with works the same way (line 105-107)
+  4. Test that passing an invalid handler (not Symbol, String, or callable) raises ArgumentError (line 127)
+  5. Test reset! clears all handlers and methods (lines 109-114)
+  6. Test Handler.call returns nil when callable is nil (line 78)
+  Use a mock controller object (OpenStruct with method stubs) to verify handler dispatch.
+- **verify:** `bin/rails test test/lib/source_monitor/configuration_test.rb -n /authentication|authorize|handler/i` exits 0
+- **done:** Lines 75-130 covered.
+
+### Task 2: Test ScrapingSettings and RetentionSettings edge cases
+
+- **name:** test-scraping-and-retention-settings
+- **files:**
+  - `test/lib/source_monitor/configuration_test.rb`
+- **action:** Add tests covering lines 132-164 (ScrapingSettings) and lines 398-436 (RetentionSettings). Specifically:
+  1. Test ScrapingSettings defaults (max_in_flight_per_source=25, max_bulk_batch_size=100)
+  2. Test ScrapingSettings normalize_numeric: nil returns nil, "" returns nil, negative returns nil, positive returns integer (lines 157-163)
+  3. Test ScrapingSettings setter with string input (e.g., "50") normalizes to integer
+  4. Test RetentionSettings strategy= with nil defaults to :destroy (line 419)
+  5. Test RetentionSettings strategy= with invalid value raises ArgumentError (line 430)
+  6. Test RetentionSettings strategy= with string "soft_delete" normalizes to symbol
+  7. Test RetentionSettings strategy= with non-symbolizable value raises ArgumentError (line 433)
+- **verify:** `bin/rails test test/lib/source_monitor/configuration_test.rb -n /scraping_settings|retention_settings|normalize/i` exits 0
+- **done:** Lines 132-164 and 398-436 covered.
+
+### Task 3: Test RealtimeSettings adapter validation and action_cable_config
+
+- **name:** test-realtime-settings
+- **files:**
+  - `test/lib/source_monitor/configuration_test.rb`
+- **action:** Add tests covering lines 166-253 (RealtimeSettings, SolidCableOptions). Specifically:
+  1. Test adapter= with :solid_cable, :redis, :async all accepted (line 178)
+  2. Test adapter= with :invalid raises ArgumentError (line 179)
+  3. Test action_cable_config for :solid_cable returns hash with adapter: "solid_cable" and SolidCableOptions merged (lines 197-198)
+  4. Test action_cable_config for :redis returns hash with adapter: "redis" and redis_url when set (lines 200-202)
+  5. Test action_cable_config for :async returns { adapter: "async" } (line 204)
+  6. Test SolidCableOptions.assign with a hash of options sets the corresponding attributes (lines 223-229)
+  7. Test SolidCableOptions.to_h compacts nil values (line 251)
+  8. Test solid_cable= delegates to solid_cable.assign (line 192)
+- **verify:** `bin/rails test test/lib/source_monitor/configuration_test.rb -n /realtime|adapter|action_cable|solid_cable/i` exits 0
+- **done:** Lines 166-253 covered.
+
+### Task 4: Test Events callbacks and item_processors
+
+- **name:** test-events-system
+- **files:**
+  - `test/lib/source_monitor/configuration_test.rb`
+- **action:** Add tests covering lines 438-491 (Events). Specifically:
+  1. Test registering after_item_created callback with a lambda and retrieving via callbacks_for(:after_item_created) (lines 446-449)
+  2. Test registering after_item_created callback with a block
+  3. Test register_item_processor adds to item_processors list (lines 452-457)
+  4. Test that passing non-callable to register_item_processor raises ArgumentError (line 488)
+  5. Test that registering unknown event key raises ArgumentError (line 479)
+  6. Test callbacks_for returns empty array for unregistered name (line 460)
+  7. Test reset! clears all callbacks and item_processors (lines 467-470)
+- **verify:** `bin/rails test test/lib/source_monitor/configuration_test.rb -n /events|callback|item_processor/i` exits 0
+- **done:** Lines 438-491 covered.
+
+### Task 5: Test Models, ModelDefinition, ConcernDefinition, and ValidationDefinition
+
+- **name:** test-models-and-definitions
+- **files:**
+  - `test/lib/source_monitor/configuration_test.rb`
+- **action:** Add tests covering lines 493-652 (Models, ModelDefinition, ConcernDefinition, ValidationDefinition). Specifically:
+  1. Test Models.table_name_prefix default is "sourcemon_" (line 507)
+  2. Test Models.for(:source) returns a ModelDefinition, and Models.for(:unknown) raises ArgumentError (lines 515-521)
+  3. Test ModelDefinition.include_concern with a Module directly (line 591-592)
+  4. Test ModelDefinition.include_concern with a string constant name that resolves (lines 593-598)
+  5. Test ModelDefinition.include_concern with a block creates anonymous module (lines 588-590)
+  6. Test ModelDefinition.include_concern deduplicates by signature (line 534)
+  7. Test ModelDefinition.validate with symbol, with callable (proc), and with block (lines 549-563)
+  8. Test ModelDefinition.validate with invalid handler raises ArgumentError (line 558)
+  9. Test ValidationDefinition.signature for symbol, callable, and string handlers (lines 636-647)
+  10. Test ValidationDefinition.symbol? (lines 649-651)
+  Use real module references and string constants for concern resolution tests.
+- **verify:** `bin/rails test test/lib/source_monitor/configuration_test.rb -n /models|model_definition|concern|validation_definition/i` exits 0
+- **done:** Lines 493-652 covered.
+
+## Verification
+
+1. `bin/rails test test/lib/source_monitor/configuration_test.rb` exits 0
+2. `COVERAGE=1 bin/rails test test/lib/source_monitor/configuration_test.rb` shows configuration.rb with >80% branch coverage
+3. `bin/rails test` exits 0 (no regressions)
+
+## Success Criteria
+
+- [ ] Configuration coverage drops from 94 uncovered lines to fewer than 20
+- [ ] AuthenticationSettings handlers fully tested
+- [ ] ScrapingSettings and RetentionSettings edge cases tested
+- [ ] RealtimeSettings adapter validation and action_cable_config tested
+- [ ] Events callbacks and item_processors tested
+- [ ] Models and definition classes tested
+- [ ] REQ-03 substantially satisfied

data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-04-SUMMARY.md

@@ -0,0 +1,63 @@
+# PLAN-04 Summary: dashboard-and-analytics-tests
+
+## Status: COMPLETE
+
+## Commits
+
+- **Hash:** `a8f2611`
+- **Message:** `test(dashboard-analytics): close coverage gaps for stats, activity, metrics, cache`
+- **Files changed:** 2 files, 564 insertions (queries_test.rb + sources_index_metrics_test.rb)
+
+- **Hash:** `2e50580` (tag commit)
+- **Message:** `test(dev-plan04): dashboard and analytics coverage gaps`
+- **Note:** Continuation/tagging commit for task verification.
+
+## Tasks Completed
+
+### Task 1: Test StatsQuery SQL branches and integer_value
+- Tested stats returns correct counts with mixed active/inactive sources
+- Tested failed_sources counts OR conditions (failure_count > 0, last_error, last_error_at)
+- Tested fetches_today time boundary (started_at >= start_of_day)
+- Tested stats with empty database returns all zeros
+- Tested record_stats_metrics sets gauge values for all stat keys
+
+### Task 2: Test RecentActivityQuery build_event and sub-queries
+- Tested build_event produces Event objects for all 3 types (fetch_log, scrape_log, item)
+- Tested fetch_log events have success based on boolean column
+- Tested scrape_log events include scraper_adapter and source_name via JOIN
+- Tested item events have item_title, item_url, success_flag always 1
+- Tested events ordered by occurred_at DESC with limit
+- Tested record_metrics for :recent_activity sets gauge values
+
+### Task 3: Test record_metrics branches and Cache edge cases
+- Tested record_metrics for all 4 case branches (stats, recent_activity, job_metrics, upcoming_fetch_schedule)
+- Tested Cache.fetch returns cached value on second call
+- Tested Cache nil/false storage, array keys, key isolation
+
+### Task 4: Test SourcesIndexMetrics edge cases
+- Tested fetch_interval_filter with gteq/lt/lteq combinations
+- Tested integer_param sanitization: blank, non-numeric, valid string
+- Tested selected_fetch_interval_bucket nil-max matching, nil when no filter
+- Tested distribution_scope ransack delegation
+- Tested nil search_params handling
+
+## Deviations
+
+None -- plan executed as specified across both target files.
+
+## Verification Results
+
+| Check | Result |
+|-------|--------|
+| `PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/dashboard/queries_test.rb test/lib/source_monitor/analytics/sources_index_metrics_test.rb` | All tests pass |
+| `bin/rails test` | 760 runs, 2626 assertions, 0 failures, 0 errors, 0 skips |
+
+## Success Criteria
+
+- [x] 35 new tests added (564 lines across 2 files)
+- [x] StatsQuery SQL branches fully tested
+- [x] RecentActivityQuery event building and sub-queries tested
+- [x] record_metrics branches for all query types tested
+- [x] Cache miss/hit behavior tested
+- [x] SourcesIndexMetrics filter, sanitization, and distribution scope tested
+- [x] REQ-04 and REQ-07 substantially satisfied