source_monitor 0.11.1 → 0.12.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (123) hide show
  1. checksums.yaml +4 -4
  2. data/.claude/commands/rails-audit.md +77 -0
  3. data/.claude/commands/release.md +70 -47
  4. data/.claude/skills/sm-architecture/reference/module-map.md +32 -0
  5. data/.claude/skills/sm-domain-model/reference/model-graph.md +42 -1
  6. data/.claude/skills/sm-job/reference/job-conventions.md +46 -40
  7. data/.claude/skills/sm-upgrade/reference/version-history.md +19 -0
  8. data/CHANGELOG.md +57 -0
  9. data/CLAUDE.md +2 -2
  10. data/Gemfile.lock +7 -20
  11. data/RAILS_AUDIT.md +424 -0
  12. data/README.md +6 -3
  13. data/VERSION +1 -1
  14. data/app/assets/builds/source_monitor/application.css +4 -24
  15. data/app/assets/builds/source_monitor/application.js +57 -89
  16. data/app/assets/builds/source_monitor/application.js.map +4 -4
  17. data/app/assets/javascripts/source_monitor/application.js +3 -6
  18. data/app/assets/javascripts/source_monitor/controllers/dropdown_controller.js +6 -86
  19. data/app/assets/javascripts/source_monitor/controllers/filter_submit_controller.js +13 -0
  20. data/app/assets/javascripts/source_monitor/controllers/modal_controller.js +56 -0
  21. data/app/assets/javascripts/source_monitor/controllers/notification_controller.js +3 -13
  22. data/app/components/source_monitor/application_component.rb +10 -0
  23. data/app/components/source_monitor/filter_dropdown_component.rb +62 -0
  24. data/app/components/source_monitor/icon_component.rb +140 -0
  25. data/app/components/source_monitor/status_badge_component.html.erb +8 -0
  26. data/app/components/source_monitor/status_badge_component.rb +96 -0
  27. data/app/controllers/concerns/source_monitor/sanitizes_search_params.rb +4 -0
  28. data/app/controllers/concerns/source_monitor/set_source.rb +13 -0
  29. data/app/controllers/source_monitor/application_controller.rb +17 -0
  30. data/app/controllers/source_monitor/bulk_scrape_enablements_controller.rb +6 -10
  31. data/app/controllers/source_monitor/dashboard_controller.rb +5 -1
  32. data/app/controllers/source_monitor/import_history_dismissals_controller.rb +1 -1
  33. data/app/controllers/source_monitor/import_sessions_controller.rb +30 -9
  34. data/app/controllers/source_monitor/item_scrapes_controller.rb +70 -0
  35. data/app/controllers/source_monitor/items_controller.rb +2 -69
  36. data/app/controllers/source_monitor/source_bulk_scrapes_controller.rb +1 -4
  37. data/app/controllers/source_monitor/source_favicon_fetches_controller.rb +2 -12
  38. data/app/controllers/source_monitor/source_fetches_controller.rb +1 -6
  39. data/app/controllers/source_monitor/source_health_checks_controller.rb +9 -16
  40. data/app/controllers/source_monitor/source_health_resets_controller.rb +1 -6
  41. data/app/controllers/source_monitor/source_retries_controller.rb +1 -6
  42. data/app/controllers/source_monitor/source_scrape_tests_controller.rb +2 -4
  43. data/app/controllers/source_monitor/source_turbo_responses.rb +1 -3
  44. data/app/controllers/source_monitor/sources_controller.rb +15 -20
  45. data/app/helpers/source_monitor/application_helper.rb +15 -31
  46. data/app/helpers/source_monitor/health_badge_helper.rb +8 -0
  47. data/app/jobs/source_monitor/download_content_images_job.rb +1 -59
  48. data/app/jobs/source_monitor/favicon_fetch_job.rb +1 -58
  49. data/app/jobs/source_monitor/fetch_feed_job.rb +2 -52
  50. data/app/jobs/source_monitor/import_opml_job.rb +6 -145
  51. data/app/jobs/source_monitor/import_session_health_check_job.rb +15 -76
  52. data/app/jobs/source_monitor/item_cleanup_job.rb +5 -0
  53. data/app/jobs/source_monitor/log_cleanup_job.rb +13 -2
  54. data/app/jobs/source_monitor/schedule_fetches_job.rb +8 -0
  55. data/app/jobs/source_monitor/scrape_item_job.rb +6 -52
  56. data/app/jobs/source_monitor/source_health_check_job.rb +1 -72
  57. data/app/models/concerns/source_monitor/loggable.rb +12 -0
  58. data/app/models/source_monitor/fetch_log.rb +0 -8
  59. data/app/models/source_monitor/health_check_log.rb +0 -8
  60. data/app/models/source_monitor/import_history.rb +14 -0
  61. data/app/models/source_monitor/import_session.rb +2 -0
  62. data/app/models/source_monitor/item.rb +15 -0
  63. data/app/models/source_monitor/item_content.rb +4 -3
  64. data/app/models/source_monitor/scrape_log.rb +4 -6
  65. data/app/models/source_monitor/source.rb +28 -19
  66. data/app/presenters/source_monitor/base_presenter.rb +19 -0
  67. data/app/presenters/source_monitor/source_details_presenter.rb +61 -0
  68. data/app/presenters/source_monitor/sources_filter_presenter.rb +61 -0
  69. data/app/views/source_monitor/dashboard/_recent_activity.html.erb +3 -3
  70. data/app/views/source_monitor/dashboard/_stat_card.html.erb +2 -1
  71. data/app/views/source_monitor/dashboard/_stats.html.erb +5 -7
  72. data/app/views/source_monitor/items/_details.html.erb +11 -14
  73. data/app/views/source_monitor/items/index.html.erb +10 -35
  74. data/app/views/source_monitor/logs/index.html.erb +20 -41
  75. data/app/views/source_monitor/shared/_form_errors.html.erb +14 -0
  76. data/app/views/source_monitor/source_scrape_tests/_result.html.erb +1 -29
  77. data/app/views/source_monitor/source_scrape_tests/_result_content.html.erb +33 -0
  78. data/app/views/source_monitor/source_scrape_tests/show.html.erb +1 -29
  79. data/app/views/source_monitor/sources/_bulk_scrape_enable_modal.html.erb +2 -2
  80. data/app/views/source_monitor/sources/_bulk_scrape_modal.html.erb +7 -5
  81. data/app/views/source_monitor/sources/_details.html.erb +24 -52
  82. data/app/views/source_monitor/sources/_health_status_badge.html.erb +4 -6
  83. data/app/views/source_monitor/sources/_row.html.erb +7 -18
  84. data/app/views/source_monitor/sources/edit.html.erb +1 -10
  85. data/app/views/source_monitor/sources/index.html.erb +26 -46
  86. data/app/views/source_monitor/sources/new.html.erb +1 -10
  87. data/config/routes.rb +1 -1
  88. data/db/migrate/20260313120000_add_composite_indexes_to_log_tables.rb +14 -0
  89. data/db/migrate/20260314120000_align_health_status_default.rb +11 -0
  90. data/docs/setup.md +2 -2
  91. data/docs/upgrade.md +23 -0
  92. data/lib/source_monitor/analytics/sources_index_metrics.rb +15 -0
  93. data/lib/source_monitor/dashboard/queries/recent_activity_query.rb +10 -4
  94. data/lib/source_monitor/dashboard/turbo_broadcaster.rb +21 -5
  95. data/lib/source_monitor/favicons/fetcher.rb +86 -0
  96. data/lib/source_monitor/fetching/cloudflare_bypass.rb +14 -5
  97. data/lib/source_monitor/fetching/completion/event_publisher.rb +12 -0
  98. data/lib/source_monitor/fetching/completion/follow_up_handler.rb +15 -2
  99. data/lib/source_monitor/fetching/completion/retention_handler.rb +11 -3
  100. data/lib/source_monitor/fetching/feed_fetcher.rb +2 -21
  101. data/lib/source_monitor/fetching/fetch_runner.rb +12 -3
  102. data/lib/source_monitor/fetching/retry_orchestrator.rb +102 -0
  103. data/lib/source_monitor/fetching/stalled_fetch_reconciler.rb +9 -0
  104. data/lib/source_monitor/health/source_health_check_orchestrator.rb +95 -0
  105. data/lib/source_monitor/health.rb +1 -0
  106. data/lib/source_monitor/images/downloader.rb +6 -7
  107. data/lib/source_monitor/images/processor.rb +98 -0
  108. data/lib/source_monitor/import_sessions/health_check_updater.rb +95 -0
  109. data/lib/source_monitor/import_sessions/opml_importer.rb +163 -0
  110. data/lib/source_monitor/items/item_creator.rb +0 -21
  111. data/lib/source_monitor/logs/query.rb +20 -0
  112. data/lib/source_monitor/queries/scrape_candidates_query.rb +30 -0
  113. data/lib/source_monitor/queries.rb +7 -0
  114. data/lib/source_monitor/scheduler.rb +5 -0
  115. data/lib/source_monitor/scraping/bulk_result_presenter.rb +11 -8
  116. data/lib/source_monitor/scraping/runner.rb +52 -0
  117. data/lib/source_monitor/scraping/scheduler.rb +5 -0
  118. data/lib/source_monitor/scraping/state.rb +4 -2
  119. data/lib/source_monitor/security/parameter_sanitizer.rb +7 -0
  120. data/lib/source_monitor/version.rb +1 -1
  121. data/lib/source_monitor.rb +7 -0
  122. data/source_monitor.gemspec +1 -0
  123. metadata +47 -1
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 9f700fdb3be3b317a91a05f04e32cd285fcb13d1acb39d3a79fb5da1da4781a0
4
- data.tar.gz: 5304f346c91cb7435663e2d8cef0f12743c334d1218cf2f6fb830df127a404ba
3
+ metadata.gz: d460ce01dd1453813fe455bde15c1b8fc2eb6954ca20ff4bf7599b55480c98d5
4
+ data.tar.gz: 16e5903aa483ba06cc092b08a8148edc3050a3830cd4dda42abfbca3d1cab25f
5
5
  SHA512:
6
- metadata.gz: 9689c88dc3dd352c86d91dc2e48a1555e9f2fd8cdde11ba20288532ae3287994fd844ab07dbda5e63fc4435dca9f5900008efb9fbf4d394d5dcfc2b139e5bbb1
7
- data.tar.gz: 99850f35657ee948728143e17f07c69ebe4591a41612cbe3ae637113d1ec2a5f7c569158a350eaf6c5f3f05e0024103a013f3ec479210f0efa291db8841b399c
6
+ metadata.gz: 20a4b8a80f4a861d8ea7e543e4cd90e31709bf9a4d504cf40059375fa439f5d5dd3803bb6ad133d50c75dfe50ee42cfd0913cae66ea41422df1473710493bb5c
7
+ data.tar.gz: ae82ac7575c31e93839bbe48c309a71f57cc1ac91ecd67a4f7690d978be459459abcdafb554fad36e98ff0cdcc424133f6a77f4f756fdfbaf6e11b22c8c9ce27
@@ -0,0 +1,77 @@
1
+ # Rails Best Practices Audit
2
+
3
+ Perform a comprehensive Rails best practices audit of the entire codebase. Use the rails-specific skills and agents to identify opportunities to simplify, refactor, and align with Rails conventions ("the Rails Way").
4
+
5
+ ## Usage
6
+
7
+ ```
8
+ /rails-audit # Full audit of entire codebase
9
+ /rails-audit models only # Scope to specific layer
10
+ /rails-audit --changed-only # Only audit changed files (git diff)
11
+ ```
12
+
13
+ ## Instructions
14
+
15
+ Launch a team of agents in parallel to explore the codebase across multiple dimensions. Each agent should use the relevant rails skills (see CLAUDE.md Skill Catalog) and search the web for best practices when skills are insufficient.
16
+
17
+ ### Agent 1: Models & Concerns (rails-review agent)
18
+ - Fat-model anti-patterns: models doing too much vs. missing scopes/validations
19
+ - Concern hygiene: single-purpose? overused? should any be model methods?
20
+ - Business logic placement: logic in controllers/services that belongs in models
21
+ - ActiveRecord anti-patterns: raw SQL where scopes suffice, missing `includes`/`preload`
22
+ - Missing validations, unnecessary callbacks, state management patterns
23
+ - Check for state-as-records pattern compliance (booleans vs. state records)
24
+
25
+ ### Agent 2: Controllers & Routes (rails-review agent)
26
+ - Everything-is-CRUD compliance: custom actions that should be separate resources
27
+ - Business logic leaking into controllers
28
+ - Strong parameters, before_actions, proper response handling
29
+ - RESTful route compliance (no custom `member`/`collection` verbs)
30
+ - Controller concerns: well-focused or kitchen-sink?
31
+
32
+ ### Agent 3: Services, Jobs & Pipeline (rails-review agent)
33
+ - Service objects: single-responsibility, Result pattern compliance
34
+ - Job shallowness: jobs should only deserialize + delegate, no business logic
35
+ - Service objects that should be model methods or concerns (< 3 models = model method)
36
+ - Pipeline stages: consolidation opportunities, error handling
37
+ - Query objects: are complex queries properly extracted?
38
+
39
+ ### Agent 4: Views, Frontend & Hotwire (rails-review agent)
40
+ - Turbo Frame/Stream best practices
41
+ - Stimulus controllers: small, focused, one behavior each
42
+ - View logic that should be presenters (SimpleDelegator) or ViewComponents
43
+ - Tailwind CSS patterns: repeated utility groups that should be components
44
+ - Partial organization and reuse
45
+
46
+ ### Agent 5: Testing & Quality (rails-review agent)
47
+ - Test DRYness: repeated setup that should be helpers
48
+ - Factory helper consistency (`create_source!`, etc.)
49
+ - Testing behavior vs. implementation
50
+ - Missing coverage patterns (validations, scopes, edge cases)
51
+ - Test isolation and parallel-safety
52
+
53
+ ## Output
54
+
55
+ Produce a markdown file at `RAILS_AUDIT.md` in the project root with:
56
+
57
+ ```markdown
58
+ # Rails Best Practices Audit — [date]
59
+
60
+ ## Executive Summary
61
+ [High-level findings count by severity]
62
+
63
+ ## Findings by Category
64
+
65
+ ### Category Name
66
+ #### Finding Title
67
+ - **Severity:** high/medium/low
68
+ - **File(s):** `path/to/file.rb:line_number`
69
+ - **Current:** [what it does now]
70
+ - **Recommended:** [what it should do, with code example if helpful]
71
+ - **Rationale:** [why this is better, link to Rails convention]
72
+ - **Effort:** quick (< 30 min) / short (< 2 hrs) / medium (< 1 day) / large (> 1 day)
73
+ ```
74
+
75
+ Sort findings within each category by severity (high first), then by effort (quick first).
76
+
77
+ $ARGUMENTS
@@ -26,7 +26,8 @@ These are real issues encountered in previous releases. Each step below accounts
26
26
  9. **ESLint browser globals**: Any JS file using browser APIs (MutationObserver, requestAnimationFrame, cancelAnimationFrame, IntersectionObserver, etc.) MUST declare them with a `/* global ... */` comment at the top. ESLint's `no-undef` rule in CI will reject them otherwise.
27
27
  10. **Diff coverage rescue paths**: Every `rescue`/fallback/error handling branch in changed source code needs test coverage. Common blind spots: `rescue StandardError => e` logging, `rescue URI::InvalidURIError` returning nil, fallback `false` returns. Write targeted tests for these BEFORE creating the release commit.
28
28
  11. **Zsh glob nomatch**: Commands like `rm -f *.gem` fail in zsh when no files match. Always use `rm -f *.gem 2>/dev/null || true` or check existence first with `ls`.
29
- 12. **Documentation drift**: Features, config options, and behavioral changes often land across milestone work without corresponding doc updates. The Documentation Audit step (Step 4) catches this -- check `docs/`, `README.md`, skill reference files (`sm-*/reference/`), and the initializer template against the actual source code. In v0.9.x, 14 files needed updates that would have been missed without this step.
29
+ 12. **Documentation drift**: Features, config options, and behavioral changes often land across milestone work without corresponding doc updates. Step 4 now spawns agents to both audit AND fix all stale docs automatically. In v0.9.x, 14 files needed updates; in v0.12.0, 7 files were stale (README version refs, docs/setup version refs, docs/upgrade missing section, 4 skill references). Always assume docs are stale.
30
+ 13. **test-coverage script health_suite**: The `bin/test-coverage` script runs a second "health_suite" pass with specific test files. If you add new test files for health/service classes, you MUST add them to the `TARGETED_TESTS` array in `bin/test-coverage` or SimpleCov's merged report will show 0% coverage for those files even though the main suite covers them. In v0.12.0, `source_health_check_orchestrator_test.rb` showed 31% coverage on CI despite 100% locally because it wasn't in the health_suite list.
30
31
 
31
32
  ## Step 1: Git Hygiene
32
33
 
@@ -114,54 +115,76 @@ The changelog follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) f
114
115
  - Insert the new versioned entry immediately after the `## [Unreleased]` block and before the previous release entry.
115
116
  - Preserve all existing entries below.
116
117
 
117
- ## Step 4: Documentation Audit
118
+ ## Step 4: Documentation Update (Mandatory)
118
119
 
119
- Verify that all project documentation reflects the current state of the codebase. Changes made since the last release (or during milestone work) may have introduced features, configuration options, bug fixes, or behavioral changes that need to be documented.
120
+ **This step is NON-OPTIONAL.** Every release MUST update all documentation to reflect the current codebase state. Documentation drift is the #1 blind spot in milestone-based work in v0.9.x, 14 files needed updates; in v0.12.0, 7 files were stale. Always assume docs are stale and verify.
120
121
 
121
- 1. **Gather what changed** since the last release tag:
122
- ```
123
- git diff vPREVIOUS..HEAD --name-only -- lib/ app/ config/
124
- ```
125
- This shows which source files changed. Use this to identify features/fixes that may need documentation.
126
-
127
- 2. **Check these documentation files against the changes:**
128
-
129
- | File | What to verify |
130
- |------|---------------|
131
- | `CHANGELOG.md` | Has an `[Unreleased]` or versioned entry covering all user-facing changes |
132
- | `README.md` | Version references match, feature descriptions current, gem version in install instructions |
133
- | `docs/configuration.md` | All config options documented, new settings included, env vars listed |
134
- | `docs/deployment.md` | Worker/queue descriptions match current queues and job assignments |
135
- | `docs/troubleshooting.md` | Covers known failure modes from recent changes |
136
- | `docs/upgrade.md` | Has upgrade section for this version with action items |
137
- | `docs/setup.md` | Setup steps still accurate |
138
-
139
- 3. **Check skills reference files** (engine-specific documentation for Claude Code):
140
-
141
- | Skill Reference | What to verify |
142
- |----------------|---------------|
143
- | `sm-configure/reference/configuration-reference.md` | All config settings and their defaults |
144
- | `sm-configuration-setting/reference/settings-catalog.md` | Settings catalog with types, defaults, descriptions |
145
- | `sm-job/reference/job-conventions.md` | Queue names, job assignments, concurrency defaults |
146
- | `sm-pipeline-stage/reference/completion-handlers.md` | Pipeline handler code matches actual implementation |
147
- | `sm-upgrade/reference/version-history.md` | Version transition notes for the new release |
148
- | `sm-host-setup/reference/initializer-template.md` | Initializer template shows all configurable options |
149
-
150
- 4. **For each file that is stale or missing coverage**:
151
- - Update it to reflect the current codebase behavior.
152
- - For config docs: read the actual settings classes in `lib/source_monitor/configuration/` to verify defaults.
153
- - For job docs: read `app/jobs/source_monitor/` to verify queue assignments.
154
- - For upgrade notes: summarize breaking changes, new config, and action items.
155
-
156
- 5. **If all documentation is already up to date**, report:
157
- ```
158
- Documentation Audit: All files current. No updates needed.
159
- ```
160
- If updates were made, report:
161
- ```
162
- Documentation Audit: Updated N files.
163
- - <file>: <what was updated>
164
- ```
122
+ ### 4a. Gather what changed
123
+
124
+ ```
125
+ git log vPREVIOUS..HEAD --oneline --no-merges
126
+ git diff vPREVIOUS..HEAD --name-only -- lib/ app/ config/
127
+ ```
128
+
129
+ Summarize the key changes: new classes, extracted services, new components, new model methods, new migrations, config changes, breaking changes.
130
+
131
+ ### 4b. Spawn Explore agent to audit all docs
132
+
133
+ Spawn an Explore agent (subagent_type: "Explore", thoroughness: "very thorough") to compare documentation against the actual source code. The agent MUST check ALL of these files:
134
+
135
+ **Project docs:**
136
+
137
+ | File | What to verify |
138
+ |------|---------------|
139
+ | `README.md` | Version references match new version, feature descriptions current, install instructions accurate |
140
+ | `docs/setup.md` | Version references match, setup steps accurate |
141
+ | `docs/upgrade.md` | Has upgrade section for THIS version with migrations, action items, breaking changes |
142
+ | `docs/configuration.md` | All config options documented, defaults match source code |
143
+ | `docs/deployment.md` | Worker/queue descriptions match current queues and job assignments |
144
+ | `docs/troubleshooting.md` | Covers known failure modes from recent changes |
145
+
146
+ **Skills reference files** (engine-specific documentation for Claude Code):
147
+
148
+ | Skill Reference | What to verify |
149
+ |----------------|---------------|
150
+ | `sm-configure/reference/configuration-reference.md` | All config settings and their defaults |
151
+ | `sm-configuration-setting/reference/settings-catalog.md` | Settings catalog with types, defaults, descriptions |
152
+ | `sm-job/reference/job-conventions.md` | Queue names, job assignments, concurrency defaults, service class delegation |
153
+ | `sm-pipeline-stage/reference/completion-handlers.md` | Pipeline handler code matches actual implementation |
154
+ | `sm-upgrade/reference/version-history.md` | Version transition notes for the new release |
155
+ | `sm-host-setup/reference/initializer-template.md` | Initializer template shows all configurable options |
156
+ | `sm-architecture/reference/module-map.md` | All modules, classes, components, presenters listed |
157
+ | `sm-domain-model/reference/model-graph.md` | All model methods, validations, scopes, associations |
158
+
159
+ The Explore agent should output a structured report: files that are current, files that need updates (with specific stale content), and what each update should contain.
160
+
161
+ ### 4c. Spawn Docs agent to fix all stale files
162
+
163
+ Based on the Explore agent's report, spawn a Docs agent (subagent_type: "vbw:vbw-docs", model: "sonnet") to update ALL stale files. The Docs agent MUST:
164
+
165
+ - Read each file before editing (targeted edits, not full rewrites)
166
+ - Update version references to the new release version
167
+ - Add upgrade section for this version (migrations, action items, breaking changes)
168
+ - Update module maps, model graphs, job conventions to reflect current source code
169
+ - Add entries for any new classes, components, concerns, or model methods
170
+
171
+ ### 4d. Verify and report
172
+
173
+ After the Docs agent completes, verify all files were updated:
174
+ ```
175
+ git diff --name-only # Should show the updated doc files
176
+ ```
177
+
178
+ Report:
179
+ ```
180
+ Documentation Update: Updated N files.
181
+ - <file>: <what was updated>
182
+ ```
183
+
184
+ If the Explore agent found 0 stale files, report:
185
+ ```
186
+ Documentation Update: All files current. No updates needed.
187
+ ```
165
188
 
166
189
  Do NOT commit documentation updates separately -- they will be included in the single release commit in Step 7.
167
190
 
@@ -79,6 +79,7 @@ Complete module tree with each module's responsibility.
79
79
  |--------|------|----------------|
80
80
  | `EntryNormalizer` | `import_sessions/entry_normalizer.rb` | Normalize OPML entries to standard format |
81
81
  | `HealthCheckBroadcaster` | `import_sessions/health_check_broadcaster.rb` | Broadcast health check progress via Turbo Streams |
82
+ | `HealthCheckUpdater` | `import_sessions/health_check_updater.rb` | Service delegated to by ImportSessionHealthCheckJob; runs checks and broadcasts |
82
83
 
83
84
  ### Jobs
84
85
 
@@ -104,6 +105,7 @@ Complete module tree with each module's responsibility.
104
105
  |--------|------|----------------|
105
106
  | `Sanitizable` | `models/sanitizable.rb` | `sanitizes_string_attributes`, `sanitizes_hash_attributes` class methods |
106
107
  | `UrlNormalizable` | `models/url_normalizable.rb` | `normalizes_urls`, `validates_url_format` class methods |
108
+ | `SetSource` | `models/set_source.rb` | Controller concern: resolves and assigns `@source` from params before actions |
107
109
 
108
110
  ### Scrapers (Scraper Adapters)
109
111
 
@@ -123,6 +125,7 @@ Complete module tree with each module's responsibility.
123
125
  | `ItemScraper` | `scraping/item_scraper.rb` | Scrape a single item |
124
126
  | `ItemScraper::AdapterResolver` | `scraping/item_scraper/adapter_resolver.rb` | Select scraper adapter for a source |
125
127
  | `ItemScraper::Persistence` | `scraping/item_scraper/persistence.rb` | Save scrape results to ItemContent |
128
+ | `Runner` | `scraping/runner.rb` | Service delegated to by ScrapeItemJob; orchestrates state, scraping, and logging |
126
129
  | `BulkSourceScraper` | `scraping/bulk_source_scraper.rb` | Scrape all pending items for a source |
127
130
  | `BulkResultPresenter` | `scraping/bulk_result_presenter.rb` | Format bulk scrape results |
128
131
  | `State` | `scraping/state.rb` | Track scraping state per source |
@@ -150,6 +153,7 @@ Complete module tree with each module's responsibility.
150
153
  |--------|------|----------------|
151
154
  | `SourceHealthMonitor` | `health/source_health_monitor.rb` | Calculate rolling success rate, update health_status |
152
155
  | `SourceHealthCheck` | `health/source_health_check.rb` | Perform HTTP health check on a source |
156
+ | `SourceHealthCheckOrchestrator` | `health/source_health_check_orchestrator.rb` | Service delegated to by SourceHealthCheckJob; coordinates check, logging, and broadcasting |
153
157
  | `SourceHealthReset` | `health/source_health_reset.rb` | Reset health state for a source |
154
158
  | `ImportSourceHealthCheck` | `health/import_source_health_check.rb` | Health check for import session sources |
155
159
 
@@ -182,6 +186,34 @@ Complete module tree with each module's responsibility.
182
186
  | `Verification::ActionCableVerifier` | `setup/verification/action_cable_verifier.rb` | Verify Action Cable setup |
183
187
  | `Verification::TelemetryLogger` | `setup/verification/telemetry_logger.rb` | Log setup telemetry |
184
188
 
189
+ ### Favicons
190
+
191
+ | Module | File | Responsibility |
192
+ |--------|------|----------------|
193
+ | `Favicons::Discoverer` | `favicons/discoverer.rb` | Multi-strategy favicon discovery (direct, HTML parsing, Google API) |
194
+ | `Favicons::Fetcher` | `favicons/fetcher.rb` | Service delegated to by FaviconFetchJob; guards, discovers, and attaches favicon |
195
+
196
+ ### Images
197
+
198
+ | Module | File | Responsibility |
199
+ |--------|------|----------------|
200
+ | `Images::Processor` | `images/processor.rb` | Service delegated to by DownloadContentImagesJob; downloads and attaches content images via Active Storage |
201
+
202
+ ### ViewComponents
203
+
204
+ | Component | File | Responsibility |
205
+ |-----------|------|----------------|
206
+ | `StatusBadgeComponent` | `app/components/source_monitor/status_badge_component.rb` | Renders a colored status badge for health_status values |
207
+ | `IconComponent` | `app/components/source_monitor/icon_component.rb` | Renders consistent inline SVG or CSS icons |
208
+ | `FilterDropdownComponent` | `app/components/source_monitor/filter_dropdown_component.rb` | Renders filter dropdown menus for sources/items index |
209
+
210
+ ### Presenters
211
+
212
+ | Presenter | File | Responsibility |
213
+ |-----------|------|----------------|
214
+ | `SourceDetailsPresenter` | `app/presenters/source_monitor/source_details_presenter.rb` | View-specific formatting for source detail pages (SimpleDelegator over Source) |
215
+ | `SourcesFilterPresenter` | `app/presenters/source_monitor/sources_filter_presenter.rb` | Formats filter state and option lists for the sources index |
216
+
185
217
  ### Other
186
218
 
187
219
  | Module | File | Responsibility |
@@ -112,4 +112,45 @@ Source has a `type` column for potential STI subclassing but it is not actively
112
112
  |--------|--------|-------|-------|
113
113
  | Source | `items_count` | Item | Only counts active (non-deleted) items |
114
114
 
115
- The counter is decremented manually during `soft_delete!` and can be recalculated via `reset_items_counter!`.
115
+ The counter is decremented manually during `soft_delete!` and **incremented** during `restore!`. Both operations update `items_count` atomically. Use `reset_items_counter!` to recalculate from scratch if the counter drifts.
116
+
117
+ ## Model Methods (v0.12.0+)
118
+
119
+ ### Source.enable_scraping!(ids)
120
+
121
+ Bulk-enables scraping for a list of source IDs:
122
+
123
+ ```ruby
124
+ Source.enable_scraping!([1, 2, 3])
125
+ # Sets scraping_enabled = true for each source in the list
126
+ ```
127
+
128
+ Use case: enabling scraping on a filtered set of sources from the dashboard or a rake task.
129
+
130
+ ### Item#restore!
131
+
132
+ Reverses a soft delete. Symmetric counterpart to `soft_delete!`:
133
+
134
+ ```ruby
135
+ item.restore!
136
+ # Clears deleted_at, increments source.items_count
137
+ ```
138
+
139
+ After `restore!`, the item re-enters the `:items` (active) association and the source counter cache reflects the change.
140
+
141
+ ### health_status Validation
142
+
143
+ As of v0.12.0, `health_status` is validated against the four permitted values. Assigning an unrecognized value raises a validation error:
144
+
145
+ ```ruby
146
+ source.health_status = "healthy" # invalid -- was removed in 0.11.0
147
+ source.valid? # => false
148
+ source.errors[:health_status] # => ["is not included in the list"]
149
+ ```
150
+
151
+ Permitted values: `"working"`, `"declining"`, `"improving"`, `"failing"`.
152
+
153
+ ## Schema Notes (v0.12.0)
154
+
155
+ - **Composite indexes on log tables:** `sourcemon_fetch_logs`, `sourcemon_scrape_logs`, and `sourcemon_health_check_logs` now have composite indexes on `(source_id, created_at)` to improve log query performance on busy sources.
156
+ - **health_status default alignment:** The `health_status` column default on `sourcemon_sources` was updated to `"working"` (previously `"healthy"`, a removed value) to match the four-value enum added in v0.11.0.
@@ -113,7 +113,7 @@ Notable patterns:
113
113
 
114
114
  ### Worker Job (ScrapeItemJob)
115
115
 
116
- Demonstrates lifecycle logging:
116
+ Demonstrates shallow delegation to a service class:
117
117
 
118
118
  ```ruby
119
119
  class ScrapeItemJob < ApplicationJob
@@ -121,35 +121,18 @@ class ScrapeItemJob < ApplicationJob
121
121
  discard_on ActiveJob::DeserializationError
122
122
 
123
123
  def perform(item_id)
124
- log("job:start", item_id: item_id)
125
124
  item = Item.includes(:source).find_by(id: item_id)
126
125
  return unless item
127
126
 
128
- source = item.source
129
- unless source&.scraping_enabled?
130
- log("job:skipped_scraping_disabled", item: item)
131
- Scraping::State.clear_inflight!(item)
132
- return
133
- end
134
-
135
- Scraping::State.mark_processing!(item)
136
- Scraping::ItemScraper.new(item:, source:).call
137
- log("job:completed", item: item, status: item.scrape_status)
138
- rescue StandardError => error
139
- log("job:error", item: item, error: error.message)
140
- Scraping::State.mark_failed!(item)
141
- raise
142
- ensure
143
- Scraping::State.clear_inflight!(item) if item
127
+ Scraping::Runner.new(item: item).call
144
128
  end
145
129
  end
146
130
  ```
147
131
 
148
132
  Notable patterns:
149
- - `includes(:source)` prevents N+1 query
150
- - Lifecycle state management (`mark_processing!`, `clear_inflight!`)
151
- - Error re-raise after state cleanup
152
- - Structured JSON logging at each stage
133
+ - Job body is deserialization + delegation only
134
+ - All business logic (state management, scraping, logging) lives in `Scraping::Runner`
135
+ - `includes(:source)` prevents N+1 query before handing off to the service
153
136
 
154
137
  ### Scheduling Job (ScheduleFetchesJob)
155
138
 
@@ -168,7 +151,7 @@ end
168
151
 
169
152
  ### Lightweight Fetch Job (FaviconFetchJob)
170
153
 
171
- Demonstrates multi-strategy cascade with guard clauses:
154
+ Demonstrates shallow delegation with guard clause:
172
155
 
173
156
  ```ruby
174
157
  class FaviconFetchJob < ApplicationJob
@@ -178,23 +161,19 @@ class FaviconFetchJob < ApplicationJob
178
161
  def perform(source_id)
179
162
  source = Source.find_by(id: source_id)
180
163
  return unless source
181
- return unless should_fetch?(source)
182
164
 
183
- result = Favicons::Discoverer.new(source: source).call
184
- attach_favicon(source, result) if result.success?
165
+ Favicons::Fetcher.new(source: source).call
185
166
  end
186
167
  end
187
168
  ```
188
169
 
189
170
  Notable patterns:
190
- - Multiple guard clauses: source exists, Active Storage defined, no existing favicon, outside cooldown period
191
- - Uses `Favicons::Discoverer` service with 3-strategy cascade (direct `/favicon.ico`, HTML parsing, Google API)
192
- - Failed attempts tracked in source `metadata` JSONB (`favicon_last_attempted_at`) for retry cooldown
171
+ - Job contains only lookup + delegation guard clauses and discovery strategy cascade live in `Favicons::Fetcher`
193
172
  - Graceful degradation: host apps without Active Storage never enqueue this job
194
173
 
195
174
  ### Broadcast Job (SourceHealthCheckJob)
196
175
 
197
- Demonstrates result broadcasting:
176
+ Demonstrates shallow delegation with result broadcasting:
198
177
 
199
178
  ```ruby
200
179
  class SourceHealthCheckJob < ApplicationJob
@@ -205,21 +184,48 @@ class SourceHealthCheckJob < ApplicationJob
205
184
  source = Source.find_by(id: source_id)
206
185
  return unless source
207
186
 
208
- result = Health::SourceHealthCheck.new(source: source).call
209
- broadcast_outcome(source, result)
210
- result
211
- rescue StandardError => error
212
- record_unexpected_failure(source, error) if source
213
- broadcast_outcome(source, nil, error) if source
214
- nil
187
+ Health::SourceHealthCheckOrchestrator.new(source: source).call
215
188
  end
216
189
  end
217
190
  ```
218
191
 
219
192
  Notable patterns:
220
- - Always broadcasts UI update (success or failure)
221
- - Creates log record even for unexpected failures
222
- - Returns nil on error instead of re-raising (health checks are non-critical)
193
+ - Job body is lookup + delegation only
194
+ - Broadcasting, logging, and error handling live in `Health::SourceHealthCheckOrchestrator`
195
+ - Returns nil on missing source; orchestrator handles nil-on-error (health checks are non-critical)
196
+
197
+ ## Shallow Delegation Pattern
198
+
199
+ As of v0.12.0, five maintenance jobs delegate entirely to dedicated service classes. Jobs contain only deserialization and delegation — no business logic.
200
+
201
+ | Job | Service Class |
202
+ |-----|---------------|
203
+ | `ScrapeItemJob` | `Scraping::Runner` |
204
+ | `DownloadContentImagesJob` | `Images::Processor` |
205
+ | `FaviconFetchJob` | `Favicons::Fetcher` |
206
+ | `SourceHealthCheckJob` | `Health::SourceHealthCheckOrchestrator` |
207
+ | `ImportSessionHealthCheckJob` | `ImportSessions::HealthCheckUpdater` |
208
+
209
+ **Why this pattern:**
210
+ - Jobs are the transport mechanism, not the behavior container.
211
+ - Service classes are unit-testable without ActiveJob infrastructure.
212
+ - Future pipeline changes (e.g., calling the same logic synchronously) require no job changes.
213
+
214
+ **Template for new jobs:**
215
+
216
+ ```ruby
217
+ class MyJob < ApplicationJob
218
+ source_monitor_queue :maintenance
219
+ discard_on ActiveJob::DeserializationError
220
+
221
+ def perform(record_id)
222
+ record = MyModel.find_by(id: record_id)
223
+ return unless record
224
+
225
+ MyNamespace::MyService.new(record: record).call
226
+ end
227
+ end
228
+ ```
223
229
 
224
230
  ## _later / _now Naming Convention
225
231
 
@@ -2,6 +2,25 @@
2
2
 
3
3
  Version-specific migration notes for each major/minor version transition. Agents should reference this file when guiding users through multi-version upgrades.
4
4
 
5
+ ## 0.11.x to 0.12.0
6
+
7
+ **Key changes:**
8
+ - 2 new migrations: composite indexes on `sourcemon_fetch_logs`, `sourcemon_scrape_logs`, and `sourcemon_health_check_logs` (on `source_id, created_at`), and `health_status` column default corrected to `"working"`.
9
+ - 5 background jobs extracted to service classes: `ScrapeItemJob` -> `Scraping::Runner`, `DownloadContentImagesJob` -> `Images::Processor`, `FaviconFetchJob` -> `Favicons::Fetcher`, `SourceHealthCheckJob` -> `Health::SourceHealthCheckOrchestrator`, `ImportSessionHealthCheckJob` -> `ImportSessions::HealthCheckUpdater`. Job arguments and queue assignments are unchanged.
10
+ - New ViewComponents: `StatusBadgeComponent`, `IconComponent`, `FilterDropdownComponent`.
11
+ - New presenters: `SourceDetailsPresenter`, `SourcesFilterPresenter`.
12
+ - New model methods: `Source.enable_scraping!(ids)`, `Item#restore!`, `health_status` validation against the 4 permitted values.
13
+
14
+ **Action items:**
15
+ 1. Copy and run migrations:
16
+ ```bash
17
+ bin/rails source_monitor:install:migrations
18
+ bin/rails db:migrate
19
+ ```
20
+ 2. No breaking changes -- all existing initializer configuration and job interfaces remain valid.
21
+ 3. No configuration changes required.
22
+ 4. ViewComponents and presenters are available for use in custom views but are not required.
23
+
5
24
  ## 0.10.2 to 0.11.0
6
25
 
7
26
  **Key changes:**
data/CHANGELOG.md CHANGED
@@ -15,6 +15,63 @@ All notable changes to this project are documented below. The format follows [Ke
15
15
 
16
16
  - No unreleased changes yet.
17
17
 
18
+ ## [0.12.1] - 2026-03-15
19
+
20
+ ### Documentation
21
+ - Updated README, docs/setup, docs/upgrade with v0.12.0 version references and upgrade guide
22
+ - Updated 4 skill references (sm-job, sm-architecture, sm-domain-model, sm-upgrade) for service class extractions, new components, and model methods
23
+ - Improved `/release` command: Step 4 now spawns agents to automatically audit AND fix all stale docs before every release
24
+
25
+ ## [0.12.0] - 2026-03-15
26
+
27
+ ### Added
28
+ - `StatusBadgeComponent` for unified status badge rendering across all views
29
+ - `IconComponent` with named icon registry replacing inline SVGs
30
+ - `SourceDetailsPresenter` and `SourcesFilterPresenter` for view formatting
31
+ - `FilterDropdownComponent` for reusable filter dropdowns
32
+ - `SetSource` controller concern eliminating duplication across 7 controllers
33
+ - `rescue_from ActiveRecord::RecordNotFound` with Turbo-aware 404 handling
34
+ - `Source.enable_scraping!` class method for bulk enablement
35
+ - `Item#restore!` with counter cache symmetry to `soft_delete!`
36
+ - `Source` health_status inclusion validation and scraping scopes
37
+ - Modal accessibility: `role="dialog"`, `aria-modal`, focus trapping via `inert`
38
+ - Logs index Turbo Frame for partial page updates on filter/pagination
39
+ - Missing controller tests for FetchLogsController and ScrapeLogsController
40
+ - Date range scopes and composite indexes on Loggable concern
41
+ - Result structs for completion handlers (EventPublisher, RetentionHandler, FollowUpHandler)
42
+ - Transient vs fatal error classification in FaviconFetchJob and DownloadContentImagesJob
43
+ - SharedLoggableTests module for consistent log model testing
44
+ - TEST_CONVENTIONS.md documenting testing patterns
45
+
46
+ ### Fixed
47
+ - LogCleanupJob now cascades deletes to LogEntry records preventing orphaned rows
48
+ - `health_status` default aligned between model (`working`) and database
49
+ - Swallowed exceptions in `Scraping::State` and pipeline rescue blocks now log warnings
50
+ - Pagination tests use `clean_source_monitor_tables!` instead of `Source.destroy_all` for parallel safety
51
+ - `fallback_user_id` guarded behind `Rails.env.development?` to prevent engine writing to host-app tables
52
+ - Dashboard GC pressure reduced from SolidQueue data accumulation
53
+ - Deadlock rescue added to 4 jobs
54
+
55
+ ### Changed
56
+ - **5 jobs extracted to service classes** (shallow delegation pattern):
57
+ - `ScrapeItemJob` → `Scraping::Runner` (73→21 lines)
58
+ - `DownloadContentImagesJob` → `Images::Processor` (95→16 lines)
59
+ - `FaviconFetchJob` → `Favicons::Fetcher` (94→16 lines)
60
+ - `SourceHealthCheckJob` → `Health::SourceHealthCheckOrchestrator` (85→16 lines)
61
+ - `ImportSessionHealthCheckJob` → `ImportSessions::HealthCheckUpdater` (99→32 lines)
62
+ - Duplicated scrape rate-limiting consolidated (removed from ScrapeItemJob, kept in Enqueuer)
63
+ - `sync_log_entry` callback consolidated into Loggable concern (removed from 3 models)
64
+ - `ItemScrapesController` extracted from Items#scrape for CRUD compliance
65
+ - FetchFeedJob retry/circuit-breaker logic extracted to RetryOrchestrator service
66
+ - Favicon cooldown logic moved from controller to Source model
67
+ - Import step dispatch refactored to handler registry pattern
68
+ - 30 backward-compatibility forwarding methods removed from FeedFetcher and ItemCreator
69
+ - `Images::Downloader` uses `HTTP.client` instead of raw Faraday (respects proxy/SSL config)
70
+ - `CloudflareBypass` limited to 1 UA rotation attempt with per-attempt timeout
71
+ - Dashboard stats split into per-stat Turbo Streams for granular updates
72
+ - Test factory helpers centralized into ModelFactories module
73
+ - System tests refactored to shared base class with SystemTestHelpers
74
+
18
75
  ## [0.11.1] - 2026-03-13
19
76
 
20
77
  ### Fixed
data/CLAUDE.md CHANGED
@@ -4,8 +4,8 @@
4
4
 
5
5
  ## Active Context
6
6
 
7
- **Last shipped:** ui-fixes-and-smart-scraping (5 phases, 22 plans)
8
- **Next action:** /vbw:vibe to plan Phase 01 (Ultimate Turbo Modal Integration)
7
+ **Last shipped:** rails-audit-and-refactoring (7 phases, 30 plans)
8
+ **Next action:** /vbw:vibe to start new work
9
9
 
10
10
  ## Key Decisions
11
11