source_monitor 0.7.1 → 0.8.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (155) hide show
  1. checksums.yaml +4 -4
  2. data/.claude/commands/release.md +18 -6
  3. data/.claude/skills/sm-configure/SKILL.md +10 -1
  4. data/.claude/skills/sm-configure/reference/configuration-reference.md +44 -0
  5. data/.claude/skills/sm-host-setup/reference/initializer-template.md +17 -0
  6. data/.claude/skills/sm-host-setup/reference/setup-checklist.md +2 -0
  7. data/.claude/skills/sm-job/reference/job-conventions.md +26 -0
  8. data/.claude/skills/sm-upgrade/reference/version-history.md +22 -0
  9. data/.gitignore +10 -0
  10. data/AGENTS.md +1 -1
  11. data/CHANGELOG.md +45 -0
  12. data/CLAUDE.md +24 -3
  13. data/Gemfile.lock +1 -1
  14. data/README.md +6 -4
  15. data/Rakefile +0 -2
  16. data/VERSION +1 -1
  17. data/app/assets/builds/source_monitor/application.css +43 -0
  18. data/app/assets/builds/source_monitor/application.js +127 -0
  19. data/app/assets/builds/source_monitor/application.js.map +3 -3
  20. data/app/assets/javascripts/source_monitor/application.js +2 -0
  21. data/app/assets/javascripts/source_monitor/controllers/notification_container_controller.js +138 -0
  22. data/app/assets/javascripts/source_monitor/controllers/notification_controller.js +11 -0
  23. data/app/controllers/source_monitor/source_favicon_fetches_controller.rb +38 -0
  24. data/app/controllers/source_monitor/sources_controller.rb +11 -0
  25. data/app/helpers/source_monitor/application_helper.rb +51 -0
  26. data/app/jobs/source_monitor/favicon_fetch_job.rb +71 -0
  27. data/app/jobs/source_monitor/import_opml_job.rb +9 -0
  28. data/app/jobs/source_monitor/source_health_check_job.rb +10 -0
  29. data/app/models/source_monitor/source.rb +2 -0
  30. data/app/views/layouts/source_monitor/application.html.erb +23 -2
  31. data/app/views/source_monitor/import_sessions/steps/_preview.html.erb +7 -2
  32. data/app/views/source_monitor/shared/_toast.html.erb +1 -0
  33. data/app/views/source_monitor/sources/_details.html.erb +34 -5
  34. data/app/views/source_monitor/sources/_row.html.erb +11 -6
  35. data/config/routes.rb +1 -0
  36. data/docs/configuration.md +1 -1
  37. data/docs/upgrade.md +22 -0
  38. data/lib/generators/source_monitor/install/templates/source_monitor.rb.tt +15 -1
  39. data/lib/source_monitor/configuration/favicons_settings.rb +42 -0
  40. data/lib/source_monitor/configuration/http_settings.rb +1 -1
  41. data/lib/source_monitor/configuration/scraping_settings.rb +1 -1
  42. data/lib/source_monitor/configuration.rb +3 -1
  43. data/lib/source_monitor/favicons/discoverer.rb +196 -0
  44. data/lib/source_monitor/fetching/feed_fetcher/source_updater.rb +21 -0
  45. data/lib/source_monitor/fetching/feed_fetcher.rb +1 -0
  46. data/lib/source_monitor/http.rb +5 -3
  47. data/lib/source_monitor/version.rb +1 -1
  48. data/lib/source_monitor.rb +4 -0
  49. data/source_monitor.gemspec +1 -1
  50. metadata +6 -106
  51. data/.vbw-planning/PROJECT.md +0 -51
  52. data/.vbw-planning/ROADMAP.md +0 -53
  53. data/.vbw-planning/SHIPPED.md +0 -63
  54. data/.vbw-planning/STATE.md +0 -27
  55. data/.vbw-planning/codebase/ARCHITECTURE.md +0 -147
  56. data/.vbw-planning/codebase/CONCERNS.md +0 -99
  57. data/.vbw-planning/codebase/CONVENTIONS.md +0 -97
  58. data/.vbw-planning/codebase/DEPENDENCIES.md +0 -100
  59. data/.vbw-planning/codebase/INDEX.md +0 -86
  60. data/.vbw-planning/codebase/META.md +0 -42
  61. data/.vbw-planning/codebase/PATTERNS.md +0 -262
  62. data/.vbw-planning/codebase/STACK.md +0 -101
  63. data/.vbw-planning/codebase/STRUCTURE.md +0 -324
  64. data/.vbw-planning/codebase/TESTING.md +0 -154
  65. data/.vbw-planning/config.json +0 -53
  66. data/.vbw-planning/discovery.json +0 -26
  67. data/.vbw-planning/milestones/default/ROADMAP.md +0 -115
  68. data/.vbw-planning/milestones/default/STATE.md +0 -82
  69. data/.vbw-planning/milestones/default/phases/01-coverage-analysis-quick-wins/PLAN-01-SUMMARY.md +0 -56
  70. data/.vbw-planning/milestones/default/phases/01-coverage-analysis-quick-wins/PLAN-01.md +0 -187
  71. data/.vbw-planning/milestones/default/phases/01-coverage-analysis-quick-wins/PLAN-02-SUMMARY.md +0 -64
  72. data/.vbw-planning/milestones/default/phases/01-coverage-analysis-quick-wins/PLAN-02.md +0 -137
  73. data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-01-SUMMARY.md +0 -67
  74. data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-01.md +0 -142
  75. data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-02-SUMMARY.md +0 -64
  76. data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-02.md +0 -138
  77. data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-03-SUMMARY.md +0 -85
  78. data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-03.md +0 -147
  79. data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-04-SUMMARY.md +0 -63
  80. data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-04.md +0 -129
  81. data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-05-SUMMARY.md +0 -74
  82. data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-05.md +0 -154
  83. data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/03-VERIFICATION-wave1.md +0 -303
  84. data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/03-VERIFICATION.md +0 -510
  85. data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-01-SUMMARY.md +0 -61
  86. data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-01.md +0 -161
  87. data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-02-SUMMARY.md +0 -66
  88. data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-02.md +0 -132
  89. data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-03-SUMMARY.md +0 -59
  90. data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-03.md +0 -171
  91. data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-04-SUMMARY.md +0 -56
  92. data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-04.md +0 -152
  93. data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/04-CONTEXT.md +0 -33
  94. data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/PLAN-01-SUMMARY.md +0 -42
  95. data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/PLAN-01.md +0 -119
  96. data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/PLAN-02-SUMMARY.md +0 -52
  97. data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/PLAN-02.md +0 -195
  98. data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/PLAN-03-SUMMARY.md +0 -79
  99. data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/PLAN-03.md +0 -130
  100. data/.vbw-planning/milestones/generator-enhancements/REQUIREMENTS.md +0 -72
  101. data/.vbw-planning/milestones/generator-enhancements/ROADMAP.md +0 -125
  102. data/.vbw-planning/milestones/generator-enhancements/SHIPPED.md +0 -40
  103. data/.vbw-planning/milestones/generator-enhancements/STATE.md +0 -43
  104. data/.vbw-planning/milestones/generator-enhancements/phases/01-generator-steps/01-CONTEXT.md +0 -33
  105. data/.vbw-planning/milestones/generator-enhancements/phases/01-generator-steps/01-VERIFICATION.md +0 -86
  106. data/.vbw-planning/milestones/generator-enhancements/phases/01-generator-steps/PLAN-01-SUMMARY.md +0 -61
  107. data/.vbw-planning/milestones/generator-enhancements/phases/01-generator-steps/PLAN-01.md +0 -380
  108. data/.vbw-planning/milestones/generator-enhancements/phases/02-verification/02-VERIFICATION.md +0 -78
  109. data/.vbw-planning/milestones/generator-enhancements/phases/02-verification/PLAN-01-SUMMARY.md +0 -46
  110. data/.vbw-planning/milestones/generator-enhancements/phases/02-verification/PLAN-01.md +0 -500
  111. data/.vbw-planning/milestones/generator-enhancements/phases/03-docs-alignment/03-VERIFICATION.md +0 -89
  112. data/.vbw-planning/milestones/generator-enhancements/phases/03-docs-alignment/PLAN-01-SUMMARY.md +0 -48
  113. data/.vbw-planning/milestones/generator-enhancements/phases/03-docs-alignment/PLAN-01.md +0 -456
  114. data/.vbw-planning/milestones/generator-enhancements/phases/04-dashboard-ux/04-VERIFICATION.md +0 -129
  115. data/.vbw-planning/milestones/generator-enhancements/phases/04-dashboard-ux/PLAN-01-SUMMARY.md +0 -70
  116. data/.vbw-planning/milestones/generator-enhancements/phases/04-dashboard-ux/PLAN-01.md +0 -747
  117. data/.vbw-planning/milestones/generator-enhancements/phases/05-active-storage-images/05-VERIFICATION.md +0 -156
  118. data/.vbw-planning/milestones/generator-enhancements/phases/05-active-storage-images/PLAN-01-SUMMARY.md +0 -69
  119. data/.vbw-planning/milestones/generator-enhancements/phases/05-active-storage-images/PLAN-01.md +0 -455
  120. data/.vbw-planning/milestones/generator-enhancements/phases/05-active-storage-images/PLAN-02-SUMMARY.md +0 -39
  121. data/.vbw-planning/milestones/generator-enhancements/phases/05-active-storage-images/PLAN-02.md +0 -488
  122. data/.vbw-planning/milestones/generator-enhancements/phases/06-netflix-feed-fix/06-VERIFICATION.md +0 -100
  123. data/.vbw-planning/milestones/generator-enhancements/phases/06-netflix-feed-fix/PLAN-01-SUMMARY.md +0 -37
  124. data/.vbw-planning/milestones/generator-enhancements/phases/06-netflix-feed-fix/PLAN-01.md +0 -345
  125. data/.vbw-planning/milestones/upgrade-assurance/REQUIREMENTS.md +0 -80
  126. data/.vbw-planning/milestones/upgrade-assurance/ROADMAP.md +0 -75
  127. data/.vbw-planning/milestones/upgrade-assurance/STATE.md +0 -29
  128. data/.vbw-planning/milestones/upgrade-assurance/phases/01-upgrade-command/01-VERIFICATION.md +0 -144
  129. data/.vbw-planning/milestones/upgrade-assurance/phases/01-upgrade-command/PLAN-01-SUMMARY.md +0 -43
  130. data/.vbw-planning/milestones/upgrade-assurance/phases/01-upgrade-command/PLAN-01.md +0 -405
  131. data/.vbw-planning/milestones/upgrade-assurance/phases/02-config-deprecation/PLAN-01-SUMMARY.md +0 -27
  132. data/.vbw-planning/milestones/upgrade-assurance/phases/02-config-deprecation/PLAN-01.md +0 -303
  133. data/.vbw-planning/milestones/upgrade-assurance/phases/03-upgrade-skill-docs/03-VERIFICATION.md +0 -380
  134. data/.vbw-planning/milestones/upgrade-assurance/phases/03-upgrade-skill-docs/PLAN-01-SUMMARY.md +0 -36
  135. data/.vbw-planning/milestones/upgrade-assurance/phases/03-upgrade-skill-docs/PLAN-01.md +0 -652
  136. data/.vbw-planning/phases/01-aia-certificate-resolution/.context-dev.md +0 -17
  137. data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-01-SUMMARY.md +0 -26
  138. data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-01.md +0 -71
  139. data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-02-SUMMARY.md +0 -16
  140. data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-02.md +0 -56
  141. data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-03-SUMMARY.md +0 -17
  142. data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-03.md +0 -98
  143. data/.vbw-planning/phases/02-test-performance/.context-dev.md +0 -75
  144. data/.vbw-planning/phases/02-test-performance/.context-lead.md +0 -89
  145. data/.vbw-planning/phases/02-test-performance/.context-qa.md +0 -23
  146. data/.vbw-planning/phases/02-test-performance/02-RESEARCH.md +0 -56
  147. data/.vbw-planning/phases/02-test-performance/02-VERIFICATION.md +0 -51
  148. data/.vbw-planning/phases/02-test-performance/PLAN-01-SUMMARY.md +0 -37
  149. data/.vbw-planning/phases/02-test-performance/PLAN-01.md +0 -156
  150. data/.vbw-planning/phases/02-test-performance/PLAN-02-SUMMARY.md +0 -33
  151. data/.vbw-planning/phases/02-test-performance/PLAN-02.md +0 -120
  152. data/.vbw-planning/phases/02-test-performance/PLAN-03-SUMMARY.md +0 -30
  153. data/.vbw-planning/phases/02-test-performance/PLAN-03.md +0 -154
  154. data/.vbw-planning/phases/02-test-performance/PLAN-04-SUMMARY.md +0 -28
  155. data/.vbw-planning/phases/02-test-performance/PLAN-04.md +0 -133
@@ -1,26 +0,0 @@
1
- ---
2
- phase: 1
3
- plan: 1
4
- status: complete
5
- ---
6
- # Plan 01 Summary: AIA Resolver Module
7
-
8
- ## Tasks Completed
9
- - [x] Task 1: Created lib/source_monitor/http/aia_resolver.rb
10
- - [x] Task 2: Created test/lib/source_monitor/http/aia_resolver_test.rb
11
-
12
- ## Commits
13
- - 4c9568a: feat(1-1): add AIA intermediate certificate resolver
14
-
15
- ## Files Modified
16
- - lib/source_monitor/http/aia_resolver.rb (created)
17
- - test/lib/source_monitor/http/aia_resolver_test.rb (created)
18
-
19
- ## What Was Built
20
- - `SourceMonitor::HTTP::AIAResolver` module with thread-safe cached resolution of missing intermediate SSL certificates via AIA (Authority Information Access) X.509 extension
21
- - Public API: `resolve(hostname)`, `enhanced_cert_store(certs)`, `clear_cache!`, `cache_size`
22
- - Private methods: `fetch_leaf_certificate` (VERIFY_NONE + SNI), `extract_aia_url` (uses `cert.ca_issuer_uris`), `download_certificate` (DER-first, PEM fallback)
23
- - 11 unit tests covering all public/private methods, caching, TTL expiration, and error handling
24
-
25
- ## Deviations
26
- - None
@@ -1,71 +0,0 @@
1
- ---
2
- phase: 1
3
- plan: 1
4
- title: "AIA Resolver Module"
5
- wave: 1
6
- depends_on: []
7
- must_haves:
8
- - AIAResolver module with resolve, enhanced_cert_store, clear_cache!, cache_size
9
- - Thread-safe Mutex + Hash cache with 1-hour TTL per hostname
10
- - fetch_leaf_certificate with VERIFY_NONE and SNI support
11
- - extract_aia_url using cert.ca_issuer_uris (not regex)
12
- - download_certificate with DER-first, PEM-fallback parsing
13
- - All methods rescue StandardError and return nil
14
- - Unit tests covering all public and private methods
15
- ---
16
-
17
- # Plan 01: AIA Resolver Module
18
-
19
- ## Goal
20
-
21
- Create `SourceMonitor::HTTP::AIAResolver` -- a standalone module that resolves missing intermediate certificates via the AIA (Authority Information Access) extension in X.509 certificates.
22
-
23
- ## Tasks
24
-
25
- ### Task 1: Create lib/source_monitor/http/aia_resolver.rb
26
-
27
- Create new module `SourceMonitor::HTTP::AIAResolver` with class methods:
28
-
29
- **Public API:**
30
- - `resolve(hostname, port: 443)` -- Entry point. Checks cache first, then: fetch leaf cert -> extract AIA URL -> download intermediate. Returns `OpenSSL::X509::Certificate` or `nil`.
31
- - `enhanced_cert_store(additional_certs)` -- Builds `OpenSSL::X509::Store` with `set_default_paths` plus extra certs from the array.
32
- - `clear_cache!` -- Clears the hostname cache (for testing).
33
- - `cache_size` -- Returns number of cached entries (for testing).
34
-
35
- **Private methods:**
36
- - `fetch_leaf_certificate(hostname, port)` -- TCP+SSL connect with `VERIFY_NONE` to get the server's leaf cert. 5s connect timeout. Uses `ssl_socket.hostname=` for SNI.
37
- - `extract_aia_url(cert)` -- Uses Ruby's built-in `cert.ca_issuer_uris` method. Returns first URI string or nil.
38
- - `download_certificate(url)` -- Plain HTTP GET (AIA URLs are always HTTP, not HTTPS). 5s timeout. Parses DER body as `OpenSSL::X509::Certificate`, falls back to PEM on failure.
39
-
40
- **Cache:** `Mutex` + `Hash` keyed by hostname. Each entry stores `{ cert:, expires_at: }` with 1-hour TTL.
41
-
42
- **Safety:** All methods rescue `StandardError` and return `nil`. This is best-effort -- never makes things worse.
43
-
44
- ### Task 2: Create test/lib/source_monitor/http/aia_resolver_test.rb
45
-
46
- Unit tests:
47
- - `extract_aia_url` with cert that has AIA extension returns URL
48
- - `extract_aia_url` with cert without AIA returns nil
49
- - `download_certificate` with DER body parses correctly (WebMock stub)
50
- - `download_certificate` returns nil on HTTP 404 (WebMock)
51
- - `download_certificate` returns nil on timeout (WebMock)
52
- - `enhanced_cert_store` returns store with added certs
53
- - `enhanced_cert_store` handles empty array gracefully
54
- - Cache: resolve stores result, second call returns cached
55
- - Cache: expired entries are re-fetched
56
- - `clear_cache!` empties the cache
57
- - `resolve` returns nil when hostname unreachable (stub fetch_leaf_certificate)
58
-
59
- ## Files
60
-
61
- | Action | Path |
62
- |--------|------|
63
- | CREATE | `lib/source_monitor/http/aia_resolver.rb` |
64
- | CREATE | `test/lib/source_monitor/http/aia_resolver_test.rb` |
65
-
66
- ## Verification
67
-
68
- ```bash
69
- PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/http/aia_resolver_test.rb
70
- bin/rubocop lib/source_monitor/http/aia_resolver.rb test/lib/source_monitor/http/aia_resolver_test.rb
71
- ```
@@ -1,16 +0,0 @@
1
- ---
2
- phase: 1
3
- plan: 2
4
- status: complete
5
- commit: f60e9bf
6
- ---
7
-
8
- ## What Was Built
9
- - Added `cert_store:` keyword parameter to `HTTP.client` for custom OpenSSL cert stores
10
- - Added `autoload :AIAResolver` to HTTP module
11
- - Plumbed cert_store through `configure_request` -> `configure_ssl` with fallback to `default_cert_store`
12
- - 2 new tests: custom cert_store usage, ssl_ca_file takes precedence over cert_store
13
-
14
- ## Files Modified
15
- - `lib/source_monitor/http.rb` — autoload, cert_store param, SSL plumbing
16
- - `test/lib/source_monitor/http_test.rb` — 2 new cert_store tests
@@ -1,56 +0,0 @@
1
- ---
2
- phase: 1
3
- plan: 2
4
- title: "HTTP Module cert_store Parameter"
5
- wave: 1
6
- depends_on: []
7
- must_haves:
8
- - Add autoload :AIAResolver to module HTTP
9
- - Add cert_store keyword to client method
10
- - Pass cert_store through configure_request to configure_ssl
11
- - configure_ssl uses cert_store when no ssl_ca_file/ssl_ca_path
12
- - Tests for cert_store parameter usage
13
- ---
14
-
15
- # Plan 02: HTTP Module cert_store Parameter
16
-
17
- ## Goal
18
-
19
- Extend `SourceMonitor::HTTP.client` to accept an optional `cert_store:` parameter, enabling callers (like FeedFetcher's AIA retry) to provide a custom `OpenSSL::X509::Store` with additional certificates.
20
-
21
- ## Tasks
22
-
23
- ### Task 1: Modify lib/source_monitor/http.rb
24
-
25
- 1. Add autoload inside `module HTTP` (after RETRY_STATUSES):
26
- ```ruby
27
- autoload :AIAResolver, "source_monitor/http/aia_resolver"
28
- ```
29
-
30
- 2. Add `cert_store: nil` keyword to `client` method signature.
31
-
32
- 3. Pass `cert_store:` through `configure_request` to `configure_ssl`:
33
- - Add `cert_store:` parameter to `configure_request`
34
- - Pass it to `configure_ssl(connection, settings, cert_store:)`
35
-
36
- 4. In `configure_ssl`: when no `ssl_ca_file` or `ssl_ca_path` is set, use `cert_store || default_cert_store`.
37
-
38
- ### Task 2: Add tests to test/lib/source_monitor/http_test.rb
39
-
40
- Add 2 tests:
41
- - `cert_store: param is used when no ssl_ca_file or ssl_ca_path` -- pass a custom store, verify `connection.ssl.cert_store` is the custom store
42
- - `cert_store: is ignored when ssl_ca_file is set` -- configure ssl_ca_file, pass cert_store, verify ca_file takes precedence
43
-
44
- ## Files
45
-
46
- | Action | Path |
47
- |--------|------|
48
- | MODIFY | `lib/source_monitor/http.rb` |
49
- | MODIFY | `test/lib/source_monitor/http_test.rb` |
50
-
51
- ## Verification
52
-
53
- ```bash
54
- PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/http_test.rb
55
- bin/rubocop lib/source_monitor/http.rb test/lib/source_monitor/http_test.rb
56
- ```
@@ -1,17 +0,0 @@
1
- ---
2
- phase: 1
3
- plan: 3
4
- status: complete
5
- commit: 9c38bc3
6
- ---
7
-
8
- ## What Was Built
9
- - Wired AIA certificate resolution into FeedFetcher's SSL error handling
10
- - On `Faraday::SSLError`, attempts intermediate cert recovery via `AIAResolver.resolve` before raising
11
- - Guard flag `@aia_attempted` prevents infinite recursion; `rescue StandardError => nil` ensures recovery never makes things worse
12
- - Tags `instrumentation_payload[:aia_resolved] = true` on successful AIA recovery
13
- - 3 integration tests: success retry path, nil fallback to ConnectionError, non-SSL skip
14
-
15
- ## Files Modified
16
- - `lib/source_monitor/fetching/feed_fetcher.rb` — split SSL rescue, add `attempt_aia_recovery`
17
- - `test/lib/source_monitor/fetching/feed_fetcher_test.rb` — 3 AIA resolution tests
@@ -1,98 +0,0 @@
1
- ---
2
- phase: 1
3
- plan: 3
4
- title: "FeedFetcher AIA Retry Integration"
5
- wave: 2
6
- depends_on: [1, 2]
7
- must_haves:
8
- - Separate Faraday::SSLError rescue from Faraday::ConnectionFailed
9
- - On SSLError attempt AIA resolution once (aia_attempted flag)
10
- - Parse hostname from source.feed_url for AIA resolve
11
- - If intermediate found rebuild connection with enhanced cert store and retry
12
- - If nil raise ConnectionError as before
13
- - Tag successful recoveries with aia_resolved in instrumentation
14
- - Integration tests for all AIA retry paths
15
- - Full test suite passes (1003+ tests)
16
- - RuboCop zero offenses
17
- - Brakeman zero warnings
18
- ---
19
-
20
- # Plan 03: FeedFetcher AIA Retry Integration
21
-
22
- ## Goal
23
-
24
- Wire AIA resolution into FeedFetcher's error handling so SSL failures automatically attempt intermediate certificate recovery before giving up.
25
-
26
- ## Tasks
27
-
28
- ### Task 1: Modify lib/source_monitor/fetching/feed_fetcher.rb
29
-
30
- Modify `perform_fetch` (lines 77-90):
31
-
32
- 1. **Split rescue clause:** Separate `Faraday::SSLError` from `Faraday::ConnectionFailed` into its own rescue:
33
- ```ruby
34
- rescue Faraday::ConnectionFailed => error
35
- raise ConnectionError.new(error.message, original_error: error)
36
- rescue Faraday::SSLError => error
37
- attempt_aia_recovery(error) || raise(ConnectionError.new(error.message, original_error: error))
38
- ```
39
-
40
- 2. **Add `attempt_aia_recovery` private method:**
41
- - Guard: return nil if `@aia_attempted` is true (prevents recursion)
42
- - Set `@aia_attempted = true`
43
- - Parse hostname from `URI.parse(source.feed_url).host`
44
- - Call `SourceMonitor::HTTP::AIAResolver.resolve(hostname)`
45
- - If intermediate found:
46
- - Build enhanced cert store via `AIAResolver.enhanced_cert_store([intermediate])`
47
- - Rebuild `@connection = SourceMonitor::HTTP.client(cert_store: store, headers: request_headers)`
48
- - Return `perform_request` (the retry)
49
- - If nil: return nil (caller raises ConnectionError)
50
- - Rescue StandardError -> nil (never make retry worse)
51
-
52
- 3. **Tag instrumentation:** In the `handle_response` path after successful AIA retry, the `instrumentation_payload[:aia_resolved] = true` will naturally flow through since `perform_fetch` calls `handle_response` on the retried response.
53
-
54
- ### Task 2: Add tests to test/lib/source_monitor/fetching/feed_fetcher_test.rb
55
-
56
- Add 3 tests under a new section `# -- AIA Certificate Resolution --`:
57
-
58
- 1. **SSL error + AIA resolve succeeds -> fetch succeeds:**
59
- - First stub: raise `Faraday::SSLError`
60
- - Stub `AIAResolver.resolve` to return a mock certificate
61
- - Stub `AIAResolver.enhanced_cert_store` to return a store
62
- - Second stub (after retry): return 200 with RSS body
63
- - Assert result.status == :fetched
64
-
65
- 2. **SSL error + AIA resolve returns nil -> ConnectionError:**
66
- - Stub to raise `Faraday::SSLError`
67
- - Stub `AIAResolver.resolve` to return nil
68
- - Assert result.status == :failed
69
- - Assert result.error is ConnectionError
70
-
71
- 3. **Non-SSL ConnectionError -> AIA not attempted:**
72
- - Stub to raise `Faraday::ConnectionFailed`
73
- - Verify `AIAResolver.resolve` was NOT called
74
- - Assert result.status == :failed
75
- - Assert result.error is ConnectionError
76
-
77
- ### Task 3: Run full verification
78
-
79
- 1. `PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb`
80
- 2. `bin/rails test` (full suite)
81
- 3. `bin/rubocop`
82
- 4. `bin/brakeman --no-pager`
83
-
84
- ## Files
85
-
86
- | Action | Path |
87
- |--------|------|
88
- | MODIFY | `lib/source_monitor/fetching/feed_fetcher.rb` |
89
- | MODIFY | `test/lib/source_monitor/fetching/feed_fetcher_test.rb` |
90
-
91
- ## Verification
92
-
93
- ```bash
94
- PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb
95
- bin/rails test
96
- bin/rubocop
97
- bin/brakeman --no-pager
98
- ```
@@ -1,75 +0,0 @@
1
- ## Phase 02 Context
2
-
3
- ### Goal
4
- Not available
5
-
6
- ### Codebase Map Available
7
- Codebase mapping exists in `.vbw-planning/codebase/`. Key files:
8
- - `ARCHITECTURE.md`
9
- - `CONCERNS.md`
10
- - `PATTERNS.md`
11
- - `DEPENDENCIES.md`
12
- - `STRUCTURE.md`
13
- - `CONVENTIONS.md`
14
- - `TESTING.md`
15
- - `STACK.md`
16
-
17
- Read CONVENTIONS.md, PATTERNS.md, STRUCTURE.md, and DEPENDENCIES.md first to bootstrap codebase understanding.
18
-
19
- ### Research Findings
20
- # Phase 2: Test Performance - Research
21
-
22
- ## Investigation Method
23
- 3-agent competing hypothesis team (H1: DB/parallelism, H2: HTTP/VCR, H3: test helpers)
24
-
25
- ## Findings
26
-
27
- ### Time Budget (133s total, 1031 tests)
28
- | Component | Time | % | Root Cause |
29
- |-----------|------|---|------------|
30
- | FeedFetcherTest (71 tests) | 84.8s | 64% | Monolithic class, CPU-bound (Feedjira parse, SHA256, content extraction) |
31
- | Integration tests (9 tests) | 30.8s | 23% | Subprocess spawning (bundle exec rails g) |
32
- | System tests (22 tests) | ~5s | 4% | Selenium/Chrome |
33
- | Debug log IO | 5-15s | 8% | 95MB :debug output to disk |
34
- | Everything else (~930 tests) | ~15s | 11% | Fast |
35
-
36
- ### Why Parallelism Doesn't Help Today
37
- - Minitest distributes by CLASS, not by test
38
- - FeedFetcherTest = 1 class with 71 tests = 1 worker gets all 84.8s
39
- - With 8 workers: max(84.8, 25.6, ~3) ≈ 85s + overhead
40
-
41
- ### DB Is NOT the Bottleneck
42
- - SQL: 3.639s of 69.4s (5.25%) via EventProf
43
- - schema.rb used (fast load)
44
- - Transactional tests enabled (proper rollback)
45
-
46
- ### HTTP Mocking Is NOT the Bottleneck
47
- - Only 4 VCR cassettes (456KB total), 83 WebMock stubs across 9 files
48
- - WebMock configured once globally, auto-resets
49
- - Only finding: default_cert_store recreated per connection (~0.5s total)
50
-
51
- ### Test Helpers Are Cheap
52
- - reset_configuration!: microseconds per call (pure Ruby)
53
- - create_source!: 1 INSERT per call, 158 calls total
54
- - with_inline_jobs: 4 calls total, negligible
55
-
56
- ### Profiling Data (tmp/test_prof/)
57
- - TagProf: lib tests 38.0s (55%), integration 28.5s (41%), controllers 0.97s, jobs 0.53s, models 0.47s
58
- - EventProf: FeedFetcherTest 0.888s SQL in 34.3s total (2.6% SQL)
59
-
60
- ## Relevant Patterns
61
- - TestProf before_all available but used in only 1 of 54 eligible files
62
- - Thread-based parallelism already proven in coverage mode
63
- - PG fork segfault only on single-file runs, NOT full suite
64
-
65
- ## Risks
66
- - Thread safety: reset_configuration! may need thread-local config or mutex
67
- - Test isolation: splitting FeedFetcherTest must preserve test independence
68
- - before_all: only safe for tests that don't mutate shared data
69
-
70
- ## Recommendations (by impact)
71
- 1. Split FeedFetcherTest into 5-6 classes → enables parallelism, -50s
72
- 2. Set config.log_level = :warn → -5-15s
73
- 3. Tag/skip integration tests in dev → -30s dev experience
74
- 4. Switch to thread-based parallelism → enables splitting benefit
75
- 5. Adopt before_all in DB-heavy test files → -3-5s
@@ -1,89 +0,0 @@
1
- ## Phase 02 Context (Compiled)
2
-
3
- ### Goal
4
- Not available
5
-
6
- ### Success Criteria
7
- Not available
8
-
9
- ### Requirements (Not available)
10
- No matching requirements found
11
-
12
-
13
- ### Active Decisions
14
-
15
- | Decision | Date | Context |
16
- |----------|------|---------|
17
- | Single-phase milestone for AIA fix | 2026-02-17 | Complete plan already validated; no scoping needed |
18
- | 3 plans with wave parallelism | 2026-02-17 | Plans 01+02 (wave 1, disjoint files), Plan 03 (wave 2, integration) |
19
-
20
- ### Codebase Map Available
21
- Codebase mapping exists in `.vbw-planning/codebase/`. Key files:
22
- - `ARCHITECTURE.md`
23
- - `CONCERNS.md`
24
- - `PATTERNS.md`
25
- - `DEPENDENCIES.md`
26
- - `STRUCTURE.md`
27
- - `CONVENTIONS.md`
28
- - `TESTING.md`
29
- - `STACK.md`
30
-
31
- Read ARCHITECTURE.md, CONCERNS.md, and STRUCTURE.md first to bootstrap codebase understanding.
32
-
33
- ### Research Findings
34
- # Phase 2: Test Performance - Research
35
-
36
- ## Investigation Method
37
- 3-agent competing hypothesis team (H1: DB/parallelism, H2: HTTP/VCR, H3: test helpers)
38
-
39
- ## Findings
40
-
41
- ### Time Budget (133s total, 1031 tests)
42
- | Component | Time | % | Root Cause |
43
- |-----------|------|---|------------|
44
- | FeedFetcherTest (71 tests) | 84.8s | 64% | Monolithic class, CPU-bound (Feedjira parse, SHA256, content extraction) |
45
- | Integration tests (9 tests) | 30.8s | 23% | Subprocess spawning (bundle exec rails g) |
46
- | System tests (22 tests) | ~5s | 4% | Selenium/Chrome |
47
- | Debug log IO | 5-15s | 8% | 95MB :debug output to disk |
48
- | Everything else (~930 tests) | ~15s | 11% | Fast |
49
-
50
- ### Why Parallelism Doesn't Help Today
51
- - Minitest distributes by CLASS, not by test
52
- - FeedFetcherTest = 1 class with 71 tests = 1 worker gets all 84.8s
53
- - With 8 workers: max(84.8, 25.6, ~3) ≈ 85s + overhead
54
-
55
- ### DB Is NOT the Bottleneck
56
- - SQL: 3.639s of 69.4s (5.25%) via EventProf
57
- - schema.rb used (fast load)
58
- - Transactional tests enabled (proper rollback)
59
-
60
- ### HTTP Mocking Is NOT the Bottleneck
61
- - Only 4 VCR cassettes (456KB total), 83 WebMock stubs across 9 files
62
- - WebMock configured once globally, auto-resets
63
- - Only finding: default_cert_store recreated per connection (~0.5s total)
64
-
65
- ### Test Helpers Are Cheap
66
- - reset_configuration!: microseconds per call (pure Ruby)
67
- - create_source!: 1 INSERT per call, 158 calls total
68
- - with_inline_jobs: 4 calls total, negligible
69
-
70
- ### Profiling Data (tmp/test_prof/)
71
- - TagProf: lib tests 38.0s (55%), integration 28.5s (41%), controllers 0.97s, jobs 0.53s, models 0.47s
72
- - EventProf: FeedFetcherTest 0.888s SQL in 34.3s total (2.6% SQL)
73
-
74
- ## Relevant Patterns
75
- - TestProf before_all available but used in only 1 of 54 eligible files
76
- - Thread-based parallelism already proven in coverage mode
77
- - PG fork segfault only on single-file runs, NOT full suite
78
-
79
- ## Risks
80
- - Thread safety: reset_configuration! may need thread-local config or mutex
81
- - Test isolation: splitting FeedFetcherTest must preserve test independence
82
- - before_all: only safe for tests that don't mutate shared data
83
-
84
- ## Recommendations (by impact)
85
- 1. Split FeedFetcherTest into 5-6 classes → enables parallelism, -50s
86
- 2. Set config.log_level = :warn → -5-15s
87
- 3. Tag/skip integration tests in dev → -30s dev experience
88
- 4. Switch to thread-based parallelism → enables splitting benefit
89
- 5. Adopt before_all in DB-heavy test files → -3-5s
@@ -1,23 +0,0 @@
1
- ## Phase 02 Verification Context
2
-
3
- ### Goal
4
- Not available
5
-
6
- ### Success Criteria
7
- Not available
8
-
9
- ### Requirements to Verify
10
- No matching requirements found
11
-
12
- ### Codebase Map Available
13
- Codebase mapping exists in `.vbw-planning/codebase/`. Key files:
14
- - `ARCHITECTURE.md`
15
- - `CONCERNS.md`
16
- - `PATTERNS.md`
17
- - `DEPENDENCIES.md`
18
- - `STRUCTURE.md`
19
- - `CONVENTIONS.md`
20
- - `TESTING.md`
21
- - `STACK.md`
22
-
23
- Read TESTING.md, CONCERNS.md, and ARCHITECTURE.md first to bootstrap codebase understanding.
@@ -1,56 +0,0 @@
1
- # Phase 2: Test Performance - Research
2
-
3
- ## Investigation Method
4
- 3-agent competing hypothesis team (H1: DB/parallelism, H2: HTTP/VCR, H3: test helpers)
5
-
6
- ## Findings
7
-
8
- ### Time Budget (133s total, 1031 tests)
9
- | Component | Time | % | Root Cause |
10
- |-----------|------|---|------------|
11
- | FeedFetcherTest (71 tests) | 84.8s | 64% | Monolithic class, CPU-bound (Feedjira parse, SHA256, content extraction) |
12
- | Integration tests (9 tests) | 30.8s | 23% | Subprocess spawning (bundle exec rails g) |
13
- | System tests (22 tests) | ~5s | 4% | Selenium/Chrome |
14
- | Debug log IO | 5-15s | 8% | 95MB :debug output to disk |
15
- | Everything else (~930 tests) | ~15s | 11% | Fast |
16
-
17
- ### Why Parallelism Doesn't Help Today
18
- - Minitest distributes by CLASS, not by test
19
- - FeedFetcherTest = 1 class with 71 tests = 1 worker gets all 84.8s
20
- - With 8 workers: max(84.8, 25.6, ~3) ≈ 85s + overhead
21
-
22
- ### DB Is NOT the Bottleneck
23
- - SQL: 3.639s of 69.4s (5.25%) via EventProf
24
- - schema.rb used (fast load)
25
- - Transactional tests enabled (proper rollback)
26
-
27
- ### HTTP Mocking Is NOT the Bottleneck
28
- - Only 4 VCR cassettes (456KB total), 83 WebMock stubs across 9 files
29
- - WebMock configured once globally, auto-resets
30
- - Only finding: default_cert_store recreated per connection (~0.5s total)
31
-
32
- ### Test Helpers Are Cheap
33
- - reset_configuration!: microseconds per call (pure Ruby)
34
- - create_source!: 1 INSERT per call, 158 calls total
35
- - with_inline_jobs: 4 calls total, negligible
36
-
37
- ### Profiling Data (tmp/test_prof/)
38
- - TagProf: lib tests 38.0s (55%), integration 28.5s (41%), controllers 0.97s, jobs 0.53s, models 0.47s
39
- - EventProf: FeedFetcherTest 0.888s SQL in 34.3s total (2.6% SQL)
40
-
41
- ## Relevant Patterns
42
- - TestProf before_all available but used in only 1 of 54 eligible files
43
- - Thread-based parallelism already proven in coverage mode
44
- - PG fork segfault only on single-file runs, NOT full suite
45
-
46
- ## Risks
47
- - Thread safety: reset_configuration! may need thread-local config or mutex
48
- - Test isolation: splitting FeedFetcherTest must preserve test independence
49
- - before_all: only safe for tests that don't mutate shared data
50
-
51
- ## Recommendations (by impact)
52
- 1. Split FeedFetcherTest into 5-6 classes → enables parallelism, -50s
53
- 2. Set config.log_level = :warn → -5-15s
54
- 3. Tag/skip integration tests in dev → -30s dev experience
55
- 4. Switch to thread-based parallelism → enables splitting benefit
56
- 5. Adopt before_all in DB-heavy test files → -3-5s
@@ -1,51 +0,0 @@
1
- # Phase 02 Verification: Test Performance
2
-
3
- **Verdict: PASS**
4
- **Date:** 2026-02-18
5
- **Tier:** Deep
6
-
7
- ## Requirements Verification
8
-
9
- | Requirement | Status | Evidence |
10
- |-------------|--------|----------|
11
- | REQ-PERF-01: FeedFetcherTest split into 5+ classes | PASS | 6 files created in test/lib/source_monitor/fetching/ (success, error_handling, adaptive_interval, retry_circuit, entry_processing, utilities) |
12
- | REQ-PERF-02: Test log level set to :warn | PASS | config.log_level = :warn in test/dummy/config/environments/test.rb |
13
- | REQ-PERF-03: Integration tests excludable | PASS | lib/tasks/test_fast.rake provides test:fast task (1022 tests vs 1033 full) |
14
- | REQ-PERF-04: Default parallelism switched to threads | PASS | parallelize(workers: worker_count, with: :threads) in test_helper.rb |
15
- | REQ-PERF-05: DB-heavy files use setup_once/before_all | PASS | 5 files now use setup_once (up from 1) |
16
-
17
- ## Checks Performed
18
-
19
- ### Structural Checks
20
- - [x] 6 split test files exist in test/lib/source_monitor/fetching/
21
- - [x] Original feed_fetcher_test.rb deleted
22
- - [x] Shared helper module exists (feed_fetcher_test_helper.rb)
23
- - [x] Each split file includes FeedFetcherTestHelper
24
- - [x] Test count preserved: 71 tests across 6 files (5+13+6+6+8+33)
25
- - [x] config.log_level = :warn present in test.rb
26
- - [x] lib/tasks/test_fast.rake exists and is valid Ruby
27
- - [x] parallelize uses with: :threads for all modes
28
- - [x] setup_once used in 5 files (sources_index_metrics, source_activity_rates, source_fetch_interval_distribution, upcoming_fetch_schedule, query_test)
29
-
30
- ### Runtime Checks
31
- - [x] Full test suite: 1033 tests, 0 failures, 0 errors
32
- - [x] Single-file runs work without PARALLEL_WORKERS=1 (PG segfault eliminated)
33
- - [x] Each split file passes individually with PARALLEL_WORKERS=1
34
- - [x] test:fast runs 1022 tests (excludes 11 integration/system tests)
35
- - [x] Two consecutive full suite runs: 0 flaky failures
36
-
37
- ### Quality Checks
38
- - [x] RuboCop: 0 offenses across all modified files
39
- - [x] Brakeman: 0 warnings
40
- - [x] No test isolation regressions
41
-
42
- ### Commits
43
- - 912665f: perf(02-03): adopt setup_once/before_all in DB-heavy test files
44
- - a951c95: feat(02-01): split FeedFetcherTest into 6 concern-based test classes
45
- - edbfe23: perf(02-02): reduce test log IO and add test:fast rake task
46
- - eceb06d: perf(test): switch default parallelism from forks to threads
47
-
48
- ## Notes
49
-
50
- - TestProf emits cosmetic warning "before_all is not implemented for parallalization with threads" — does not affect correctness. before_all/setup_once works correctly because single-file runs stay below parallelization threshold, and full suite distributes by class.
51
- - PLAN-02 deviated from plan: --exclude-pattern is not a Minitest feature, so Dir glob approach was used instead. Functionally equivalent.
@@ -1,37 +0,0 @@
1
- ---
2
- status: complete
3
- phase: "02"
4
- plan: "01"
5
- title: "Split FeedFetcherTest into Concern-Based Classes"
6
- commits:
7
- - hash: a951c95
8
- message: "feat(02-01): split FeedFetcherTest into 6 concern-based test classes"
9
- tasks_completed: 4
10
- tasks_total: 4
11
- deviations: []
12
- ---
13
-
14
- ## What Was Built
15
-
16
- Split the monolithic `FeedFetcherTest` (71 tests, 1350 lines, single class) into 6 independent test classes plus a shared helper module. Each file is independently runnable with `PARALLEL_WORKERS=1`. All 1033 tests pass, RuboCop zero offenses.
17
-
18
- Test distribution:
19
- - `FeedFetcherSuccessTest`: 5 tests (RSS/Atom/JSON fetching, ETag/304, Netflix cassette)
20
- - `FeedFetcherErrorHandlingTest`: 13 tests (Faraday error wrapping, AIA resolution, failure recording)
21
- - `FeedFetcherAdaptiveIntervalTest`: 6 tests (interval increase/decrease, min/max bounds, disabled mode)
22
- - `FeedFetcherRetryCircuitTest`: 6 tests (retry state, circuit breaker, policy error handling)
23
- - `FeedFetcherEntryProcessingTest`: 8 tests (entry creation/update tracking, error normalization, digest fallbacks)
24
- - `FeedFetcherUtilitiesTest`: 33 tests (headers, jitter, metadata, signature, config helpers, parsing)
25
-
26
- ## Files Modified
27
-
28
- | Action | Path |
29
- |--------|------|
30
- | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_test_helper.rb` |
31
- | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_success_test.rb` |
32
- | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_error_handling_test.rb` |
33
- | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_adaptive_interval_test.rb` |
34
- | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_retry_circuit_test.rb` |
35
- | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_entry_processing_test.rb` |
36
- | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_utilities_test.rb` |
37
- | DELETE | `test/lib/source_monitor/fetching/feed_fetcher_test.rb` |