source_monitor 0.11.1 → 0.12.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.claude/commands/rails-audit.md +77 -0
- data/.claude/commands/release.md +70 -47
- data/.claude/skills/sm-architecture/reference/module-map.md +32 -0
- data/.claude/skills/sm-domain-model/reference/model-graph.md +42 -1
- data/.claude/skills/sm-job/reference/job-conventions.md +46 -40
- data/.claude/skills/sm-upgrade/reference/version-history.md +19 -0
- data/CHANGELOG.md +57 -0
- data/CLAUDE.md +2 -2
- data/Gemfile.lock +7 -20
- data/RAILS_AUDIT.md +424 -0
- data/README.md +6 -3
- data/VERSION +1 -1
- data/app/assets/builds/source_monitor/application.css +4 -24
- data/app/assets/builds/source_monitor/application.js +57 -89
- data/app/assets/builds/source_monitor/application.js.map +4 -4
- data/app/assets/javascripts/source_monitor/application.js +3 -6
- data/app/assets/javascripts/source_monitor/controllers/dropdown_controller.js +6 -86
- data/app/assets/javascripts/source_monitor/controllers/filter_submit_controller.js +13 -0
- data/app/assets/javascripts/source_monitor/controllers/modal_controller.js +56 -0
- data/app/assets/javascripts/source_monitor/controllers/notification_controller.js +3 -13
- data/app/components/source_monitor/application_component.rb +10 -0
- data/app/components/source_monitor/filter_dropdown_component.rb +62 -0
- data/app/components/source_monitor/icon_component.rb +140 -0
- data/app/components/source_monitor/status_badge_component.html.erb +8 -0
- data/app/components/source_monitor/status_badge_component.rb +96 -0
- data/app/controllers/concerns/source_monitor/sanitizes_search_params.rb +4 -0
- data/app/controllers/concerns/source_monitor/set_source.rb +13 -0
- data/app/controllers/source_monitor/application_controller.rb +17 -0
- data/app/controllers/source_monitor/bulk_scrape_enablements_controller.rb +6 -10
- data/app/controllers/source_monitor/dashboard_controller.rb +5 -1
- data/app/controllers/source_monitor/import_history_dismissals_controller.rb +1 -1
- data/app/controllers/source_monitor/import_sessions_controller.rb +30 -9
- data/app/controllers/source_monitor/item_scrapes_controller.rb +70 -0
- data/app/controllers/source_monitor/items_controller.rb +2 -69
- data/app/controllers/source_monitor/source_bulk_scrapes_controller.rb +1 -4
- data/app/controllers/source_monitor/source_favicon_fetches_controller.rb +2 -12
- data/app/controllers/source_monitor/source_fetches_controller.rb +1 -6
- data/app/controllers/source_monitor/source_health_checks_controller.rb +9 -16
- data/app/controllers/source_monitor/source_health_resets_controller.rb +1 -6
- data/app/controllers/source_monitor/source_retries_controller.rb +1 -6
- data/app/controllers/source_monitor/source_scrape_tests_controller.rb +2 -4
- data/app/controllers/source_monitor/source_turbo_responses.rb +1 -3
- data/app/controllers/source_monitor/sources_controller.rb +15 -20
- data/app/helpers/source_monitor/application_helper.rb +15 -31
- data/app/helpers/source_monitor/health_badge_helper.rb +8 -0
- data/app/jobs/source_monitor/download_content_images_job.rb +1 -59
- data/app/jobs/source_monitor/favicon_fetch_job.rb +1 -58
- data/app/jobs/source_monitor/fetch_feed_job.rb +2 -52
- data/app/jobs/source_monitor/import_opml_job.rb +6 -145
- data/app/jobs/source_monitor/import_session_health_check_job.rb +15 -76
- data/app/jobs/source_monitor/item_cleanup_job.rb +5 -0
- data/app/jobs/source_monitor/log_cleanup_job.rb +13 -2
- data/app/jobs/source_monitor/schedule_fetches_job.rb +8 -0
- data/app/jobs/source_monitor/scrape_item_job.rb +6 -52
- data/app/jobs/source_monitor/source_health_check_job.rb +1 -72
- data/app/models/concerns/source_monitor/loggable.rb +12 -0
- data/app/models/source_monitor/fetch_log.rb +0 -8
- data/app/models/source_monitor/health_check_log.rb +0 -8
- data/app/models/source_monitor/import_history.rb +14 -0
- data/app/models/source_monitor/import_session.rb +2 -0
- data/app/models/source_monitor/item.rb +15 -0
- data/app/models/source_monitor/item_content.rb +4 -3
- data/app/models/source_monitor/scrape_log.rb +4 -6
- data/app/models/source_monitor/source.rb +28 -19
- data/app/presenters/source_monitor/base_presenter.rb +19 -0
- data/app/presenters/source_monitor/source_details_presenter.rb +61 -0
- data/app/presenters/source_monitor/sources_filter_presenter.rb +61 -0
- data/app/views/source_monitor/dashboard/_recent_activity.html.erb +3 -3
- data/app/views/source_monitor/dashboard/_stat_card.html.erb +2 -1
- data/app/views/source_monitor/dashboard/_stats.html.erb +5 -7
- data/app/views/source_monitor/items/_details.html.erb +11 -14
- data/app/views/source_monitor/items/index.html.erb +10 -35
- data/app/views/source_monitor/logs/index.html.erb +20 -41
- data/app/views/source_monitor/shared/_form_errors.html.erb +14 -0
- data/app/views/source_monitor/source_scrape_tests/_result.html.erb +1 -29
- data/app/views/source_monitor/source_scrape_tests/_result_content.html.erb +33 -0
- data/app/views/source_monitor/source_scrape_tests/show.html.erb +1 -29
- data/app/views/source_monitor/sources/_bulk_scrape_enable_modal.html.erb +2 -2
- data/app/views/source_monitor/sources/_bulk_scrape_modal.html.erb +7 -5
- data/app/views/source_monitor/sources/_details.html.erb +24 -52
- data/app/views/source_monitor/sources/_health_status_badge.html.erb +4 -6
- data/app/views/source_monitor/sources/_row.html.erb +7 -18
- data/app/views/source_monitor/sources/edit.html.erb +1 -10
- data/app/views/source_monitor/sources/index.html.erb +26 -46
- data/app/views/source_monitor/sources/new.html.erb +1 -10
- data/config/routes.rb +1 -1
- data/db/migrate/20260313120000_add_composite_indexes_to_log_tables.rb +14 -0
- data/db/migrate/20260314120000_align_health_status_default.rb +11 -0
- data/docs/setup.md +2 -2
- data/docs/upgrade.md +23 -0
- data/lib/source_monitor/analytics/sources_index_metrics.rb +15 -0
- data/lib/source_monitor/dashboard/queries/recent_activity_query.rb +10 -4
- data/lib/source_monitor/dashboard/turbo_broadcaster.rb +21 -5
- data/lib/source_monitor/favicons/fetcher.rb +86 -0
- data/lib/source_monitor/fetching/cloudflare_bypass.rb +14 -5
- data/lib/source_monitor/fetching/completion/event_publisher.rb +12 -0
- data/lib/source_monitor/fetching/completion/follow_up_handler.rb +15 -2
- data/lib/source_monitor/fetching/completion/retention_handler.rb +11 -3
- data/lib/source_monitor/fetching/feed_fetcher.rb +2 -21
- data/lib/source_monitor/fetching/fetch_runner.rb +12 -3
- data/lib/source_monitor/fetching/retry_orchestrator.rb +102 -0
- data/lib/source_monitor/fetching/stalled_fetch_reconciler.rb +9 -0
- data/lib/source_monitor/health/source_health_check_orchestrator.rb +95 -0
- data/lib/source_monitor/health.rb +1 -0
- data/lib/source_monitor/images/downloader.rb +6 -7
- data/lib/source_monitor/images/processor.rb +98 -0
- data/lib/source_monitor/import_sessions/health_check_updater.rb +95 -0
- data/lib/source_monitor/import_sessions/opml_importer.rb +163 -0
- data/lib/source_monitor/items/item_creator.rb +0 -21
- data/lib/source_monitor/logs/query.rb +20 -0
- data/lib/source_monitor/queries/scrape_candidates_query.rb +30 -0
- data/lib/source_monitor/queries.rb +7 -0
- data/lib/source_monitor/scheduler.rb +5 -0
- data/lib/source_monitor/scraping/bulk_result_presenter.rb +11 -8
- data/lib/source_monitor/scraping/runner.rb +52 -0
- data/lib/source_monitor/scraping/scheduler.rb +5 -0
- data/lib/source_monitor/scraping/state.rb +4 -2
- data/lib/source_monitor/security/parameter_sanitizer.rb +7 -0
- data/lib/source_monitor/version.rb +1 -1
- data/lib/source_monitor.rb +7 -0
- data/source_monitor.gemspec +1 -0
- metadata +47 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: d460ce01dd1453813fe455bde15c1b8fc2eb6954ca20ff4bf7599b55480c98d5
|
|
4
|
+
data.tar.gz: 16e5903aa483ba06cc092b08a8148edc3050a3830cd4dda42abfbca3d1cab25f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 20a4b8a80f4a861d8ea7e543e4cd90e31709bf9a4d504cf40059375fa439f5d5dd3803bb6ad133d50c75dfe50ee42cfd0913cae66ea41422df1473710493bb5c
|
|
7
|
+
data.tar.gz: ae82ac7575c31e93839bbe48c309a71f57cc1ac91ecd67a4f7690d978be459459abcdafb554fad36e98ff0cdcc424133f6a77f4f756fdfbaf6e11b22c8c9ce27
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Rails Best Practices Audit
|
|
2
|
+
|
|
3
|
+
Perform a comprehensive Rails best practices audit of the entire codebase. Use the rails-specific skills and agents to identify opportunities to simplify, refactor, and align with Rails conventions ("the Rails Way").
|
|
4
|
+
|
|
5
|
+
## Usage
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
/rails-audit # Full audit of entire codebase
|
|
9
|
+
/rails-audit models only # Scope to specific layer
|
|
10
|
+
/rails-audit --changed-only # Only audit changed files (git diff)
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
## Instructions
|
|
14
|
+
|
|
15
|
+
Launch a team of agents in parallel to explore the codebase across multiple dimensions. Each agent should use the relevant rails skills (see CLAUDE.md Skill Catalog) and search the web for best practices when skills are insufficient.
|
|
16
|
+
|
|
17
|
+
### Agent 1: Models & Concerns (rails-review agent)
|
|
18
|
+
- Fat-model anti-patterns: models doing too much vs. missing scopes/validations
|
|
19
|
+
- Concern hygiene: single-purpose? overused? should any be model methods?
|
|
20
|
+
- Business logic placement: logic in controllers/services that belongs in models
|
|
21
|
+
- ActiveRecord anti-patterns: raw SQL where scopes suffice, missing `includes`/`preload`
|
|
22
|
+
- Missing validations, unnecessary callbacks, state management patterns
|
|
23
|
+
- Check for state-as-records pattern compliance (booleans vs. state records)
|
|
24
|
+
|
|
25
|
+
### Agent 2: Controllers & Routes (rails-review agent)
|
|
26
|
+
- Everything-is-CRUD compliance: custom actions that should be separate resources
|
|
27
|
+
- Business logic leaking into controllers
|
|
28
|
+
- Strong parameters, before_actions, proper response handling
|
|
29
|
+
- RESTful route compliance (no custom `member`/`collection` verbs)
|
|
30
|
+
- Controller concerns: well-focused or kitchen-sink?
|
|
31
|
+
|
|
32
|
+
### Agent 3: Services, Jobs & Pipeline (rails-review agent)
|
|
33
|
+
- Service objects: single-responsibility, Result pattern compliance
|
|
34
|
+
- Job shallowness: jobs should only deserialize + delegate, no business logic
|
|
35
|
+
- Service objects that should be model methods or concerns (< 3 models = model method)
|
|
36
|
+
- Pipeline stages: consolidation opportunities, error handling
|
|
37
|
+
- Query objects: are complex queries properly extracted?
|
|
38
|
+
|
|
39
|
+
### Agent 4: Views, Frontend & Hotwire (rails-review agent)
|
|
40
|
+
- Turbo Frame/Stream best practices
|
|
41
|
+
- Stimulus controllers: small, focused, one behavior each
|
|
42
|
+
- View logic that should be presenters (SimpleDelegator) or ViewComponents
|
|
43
|
+
- Tailwind CSS patterns: repeated utility groups that should be components
|
|
44
|
+
- Partial organization and reuse
|
|
45
|
+
|
|
46
|
+
### Agent 5: Testing & Quality (rails-review agent)
|
|
47
|
+
- Test DRYness: repeated setup that should be helpers
|
|
48
|
+
- Factory helper consistency (`create_source!`, etc.)
|
|
49
|
+
- Testing behavior vs. implementation
|
|
50
|
+
- Missing coverage patterns (validations, scopes, edge cases)
|
|
51
|
+
- Test isolation and parallel-safety
|
|
52
|
+
|
|
53
|
+
## Output
|
|
54
|
+
|
|
55
|
+
Produce a markdown file at `RAILS_AUDIT.md` in the project root with:
|
|
56
|
+
|
|
57
|
+
```markdown
|
|
58
|
+
# Rails Best Practices Audit — [date]
|
|
59
|
+
|
|
60
|
+
## Executive Summary
|
|
61
|
+
[High-level findings count by severity]
|
|
62
|
+
|
|
63
|
+
## Findings by Category
|
|
64
|
+
|
|
65
|
+
### Category Name
|
|
66
|
+
#### Finding Title
|
|
67
|
+
- **Severity:** high/medium/low
|
|
68
|
+
- **File(s):** `path/to/file.rb:line_number`
|
|
69
|
+
- **Current:** [what it does now]
|
|
70
|
+
- **Recommended:** [what it should do, with code example if helpful]
|
|
71
|
+
- **Rationale:** [why this is better, link to Rails convention]
|
|
72
|
+
- **Effort:** quick (< 30 min) / short (< 2 hrs) / medium (< 1 day) / large (> 1 day)
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
Sort findings within each category by severity (high first), then by effort (quick first).
|
|
76
|
+
|
|
77
|
+
$ARGUMENTS
|
data/.claude/commands/release.md
CHANGED
|
@@ -26,7 +26,8 @@ These are real issues encountered in previous releases. Each step below accounts
|
|
|
26
26
|
9. **ESLint browser globals**: Any JS file using browser APIs (MutationObserver, requestAnimationFrame, cancelAnimationFrame, IntersectionObserver, etc.) MUST declare them with a `/* global ... */` comment at the top. ESLint's `no-undef` rule in CI will reject them otherwise.
|
|
27
27
|
10. **Diff coverage rescue paths**: Every `rescue`/fallback/error handling branch in changed source code needs test coverage. Common blind spots: `rescue StandardError => e` logging, `rescue URI::InvalidURIError` returning nil, fallback `false` returns. Write targeted tests for these BEFORE creating the release commit.
|
|
28
28
|
11. **Zsh glob nomatch**: Commands like `rm -f *.gem` fail in zsh when no files match. Always use `rm -f *.gem 2>/dev/null || true` or check existence first with `ls`.
|
|
29
|
-
12. **Documentation drift**: Features, config options, and behavioral changes often land across milestone work without corresponding doc updates.
|
|
29
|
+
12. **Documentation drift**: Features, config options, and behavioral changes often land across milestone work without corresponding doc updates. Step 4 now spawns agents to both audit AND fix all stale docs automatically. In v0.9.x, 14 files needed updates; in v0.12.0, 7 files were stale (README version refs, docs/setup version refs, docs/upgrade missing section, 4 skill references). Always assume docs are stale.
|
|
30
|
+
13. **test-coverage script health_suite**: The `bin/test-coverage` script runs a second "health_suite" pass with specific test files. If you add new test files for health/service classes, you MUST add them to the `TARGETED_TESTS` array in `bin/test-coverage` or SimpleCov's merged report will show 0% coverage for those files even though the main suite covers them. In v0.12.0, `source_health_check_orchestrator_test.rb` showed 31% coverage on CI despite 100% locally because it wasn't in the health_suite list.
|
|
30
31
|
|
|
31
32
|
## Step 1: Git Hygiene
|
|
32
33
|
|
|
@@ -114,54 +115,76 @@ The changelog follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) f
|
|
|
114
115
|
- Insert the new versioned entry immediately after the `## [Unreleased]` block and before the previous release entry.
|
|
115
116
|
- Preserve all existing entries below.
|
|
116
117
|
|
|
117
|
-
## Step 4: Documentation
|
|
118
|
+
## Step 4: Documentation Update (Mandatory)
|
|
118
119
|
|
|
119
|
-
|
|
120
|
+
**This step is NON-OPTIONAL.** Every release MUST update all documentation to reflect the current codebase state. Documentation drift is the #1 blind spot in milestone-based work — in v0.9.x, 14 files needed updates; in v0.12.0, 7 files were stale. Always assume docs are stale and verify.
|
|
120
121
|
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
122
|
+
### 4a. Gather what changed
|
|
123
|
+
|
|
124
|
+
```
|
|
125
|
+
git log vPREVIOUS..HEAD --oneline --no-merges
|
|
126
|
+
git diff vPREVIOUS..HEAD --name-only -- lib/ app/ config/
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
Summarize the key changes: new classes, extracted services, new components, new model methods, new migrations, config changes, breaking changes.
|
|
130
|
+
|
|
131
|
+
### 4b. Spawn Explore agent to audit all docs
|
|
132
|
+
|
|
133
|
+
Spawn an Explore agent (subagent_type: "Explore", thoroughness: "very thorough") to compare documentation against the actual source code. The agent MUST check ALL of these files:
|
|
134
|
+
|
|
135
|
+
**Project docs:**
|
|
136
|
+
|
|
137
|
+
| File | What to verify |
|
|
138
|
+
|------|---------------|
|
|
139
|
+
| `README.md` | Version references match new version, feature descriptions current, install instructions accurate |
|
|
140
|
+
| `docs/setup.md` | Version references match, setup steps accurate |
|
|
141
|
+
| `docs/upgrade.md` | Has upgrade section for THIS version with migrations, action items, breaking changes |
|
|
142
|
+
| `docs/configuration.md` | All config options documented, defaults match source code |
|
|
143
|
+
| `docs/deployment.md` | Worker/queue descriptions match current queues and job assignments |
|
|
144
|
+
| `docs/troubleshooting.md` | Covers known failure modes from recent changes |
|
|
145
|
+
|
|
146
|
+
**Skills reference files** (engine-specific documentation for Claude Code):
|
|
147
|
+
|
|
148
|
+
| Skill Reference | What to verify |
|
|
149
|
+
|----------------|---------------|
|
|
150
|
+
| `sm-configure/reference/configuration-reference.md` | All config settings and their defaults |
|
|
151
|
+
| `sm-configuration-setting/reference/settings-catalog.md` | Settings catalog with types, defaults, descriptions |
|
|
152
|
+
| `sm-job/reference/job-conventions.md` | Queue names, job assignments, concurrency defaults, service class delegation |
|
|
153
|
+
| `sm-pipeline-stage/reference/completion-handlers.md` | Pipeline handler code matches actual implementation |
|
|
154
|
+
| `sm-upgrade/reference/version-history.md` | Version transition notes for the new release |
|
|
155
|
+
| `sm-host-setup/reference/initializer-template.md` | Initializer template shows all configurable options |
|
|
156
|
+
| `sm-architecture/reference/module-map.md` | All modules, classes, components, presenters listed |
|
|
157
|
+
| `sm-domain-model/reference/model-graph.md` | All model methods, validations, scopes, associations |
|
|
158
|
+
|
|
159
|
+
The Explore agent should output a structured report: files that are current, files that need updates (with specific stale content), and what each update should contain.
|
|
160
|
+
|
|
161
|
+
### 4c. Spawn Docs agent to fix all stale files
|
|
162
|
+
|
|
163
|
+
Based on the Explore agent's report, spawn a Docs agent (subagent_type: "vbw:vbw-docs", model: "sonnet") to update ALL stale files. The Docs agent MUST:
|
|
164
|
+
|
|
165
|
+
- Read each file before editing (targeted edits, not full rewrites)
|
|
166
|
+
- Update version references to the new release version
|
|
167
|
+
- Add upgrade section for this version (migrations, action items, breaking changes)
|
|
168
|
+
- Update module maps, model graphs, job conventions to reflect current source code
|
|
169
|
+
- Add entries for any new classes, components, concerns, or model methods
|
|
170
|
+
|
|
171
|
+
### 4d. Verify and report
|
|
172
|
+
|
|
173
|
+
After the Docs agent completes, verify all files were updated:
|
|
174
|
+
```
|
|
175
|
+
git diff --name-only # Should show the updated doc files
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Report:
|
|
179
|
+
```
|
|
180
|
+
Documentation Update: Updated N files.
|
|
181
|
+
- <file>: <what was updated>
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
If the Explore agent found 0 stale files, report:
|
|
185
|
+
```
|
|
186
|
+
Documentation Update: All files current. No updates needed.
|
|
187
|
+
```
|
|
165
188
|
|
|
166
189
|
Do NOT commit documentation updates separately -- they will be included in the single release commit in Step 7.
|
|
167
190
|
|
|
@@ -79,6 +79,7 @@ Complete module tree with each module's responsibility.
|
|
|
79
79
|
|--------|------|----------------|
|
|
80
80
|
| `EntryNormalizer` | `import_sessions/entry_normalizer.rb` | Normalize OPML entries to standard format |
|
|
81
81
|
| `HealthCheckBroadcaster` | `import_sessions/health_check_broadcaster.rb` | Broadcast health check progress via Turbo Streams |
|
|
82
|
+
| `HealthCheckUpdater` | `import_sessions/health_check_updater.rb` | Service delegated to by ImportSessionHealthCheckJob; runs checks and broadcasts |
|
|
82
83
|
|
|
83
84
|
### Jobs
|
|
84
85
|
|
|
@@ -104,6 +105,7 @@ Complete module tree with each module's responsibility.
|
|
|
104
105
|
|--------|------|----------------|
|
|
105
106
|
| `Sanitizable` | `models/sanitizable.rb` | `sanitizes_string_attributes`, `sanitizes_hash_attributes` class methods |
|
|
106
107
|
| `UrlNormalizable` | `models/url_normalizable.rb` | `normalizes_urls`, `validates_url_format` class methods |
|
|
108
|
+
| `SetSource` | `models/set_source.rb` | Controller concern: resolves and assigns `@source` from params before actions |
|
|
107
109
|
|
|
108
110
|
### Scrapers (Scraper Adapters)
|
|
109
111
|
|
|
@@ -123,6 +125,7 @@ Complete module tree with each module's responsibility.
|
|
|
123
125
|
| `ItemScraper` | `scraping/item_scraper.rb` | Scrape a single item |
|
|
124
126
|
| `ItemScraper::AdapterResolver` | `scraping/item_scraper/adapter_resolver.rb` | Select scraper adapter for a source |
|
|
125
127
|
| `ItemScraper::Persistence` | `scraping/item_scraper/persistence.rb` | Save scrape results to ItemContent |
|
|
128
|
+
| `Runner` | `scraping/runner.rb` | Service delegated to by ScrapeItemJob; orchestrates state, scraping, and logging |
|
|
126
129
|
| `BulkSourceScraper` | `scraping/bulk_source_scraper.rb` | Scrape all pending items for a source |
|
|
127
130
|
| `BulkResultPresenter` | `scraping/bulk_result_presenter.rb` | Format bulk scrape results |
|
|
128
131
|
| `State` | `scraping/state.rb` | Track scraping state per source |
|
|
@@ -150,6 +153,7 @@ Complete module tree with each module's responsibility.
|
|
|
150
153
|
|--------|------|----------------|
|
|
151
154
|
| `SourceHealthMonitor` | `health/source_health_monitor.rb` | Calculate rolling success rate, update health_status |
|
|
152
155
|
| `SourceHealthCheck` | `health/source_health_check.rb` | Perform HTTP health check on a source |
|
|
156
|
+
| `SourceHealthCheckOrchestrator` | `health/source_health_check_orchestrator.rb` | Service delegated to by SourceHealthCheckJob; coordinates check, logging, and broadcasting |
|
|
153
157
|
| `SourceHealthReset` | `health/source_health_reset.rb` | Reset health state for a source |
|
|
154
158
|
| `ImportSourceHealthCheck` | `health/import_source_health_check.rb` | Health check for import session sources |
|
|
155
159
|
|
|
@@ -182,6 +186,34 @@ Complete module tree with each module's responsibility.
|
|
|
182
186
|
| `Verification::ActionCableVerifier` | `setup/verification/action_cable_verifier.rb` | Verify Action Cable setup |
|
|
183
187
|
| `Verification::TelemetryLogger` | `setup/verification/telemetry_logger.rb` | Log setup telemetry |
|
|
184
188
|
|
|
189
|
+
### Favicons
|
|
190
|
+
|
|
191
|
+
| Module | File | Responsibility |
|
|
192
|
+
|--------|------|----------------|
|
|
193
|
+
| `Favicons::Discoverer` | `favicons/discoverer.rb` | Multi-strategy favicon discovery (direct, HTML parsing, Google API) |
|
|
194
|
+
| `Favicons::Fetcher` | `favicons/fetcher.rb` | Service delegated to by FaviconFetchJob; guards, discovers, and attaches favicon |
|
|
195
|
+
|
|
196
|
+
### Images
|
|
197
|
+
|
|
198
|
+
| Module | File | Responsibility |
|
|
199
|
+
|--------|------|----------------|
|
|
200
|
+
| `Images::Processor` | `images/processor.rb` | Service delegated to by DownloadContentImagesJob; downloads and attaches content images via Active Storage |
|
|
201
|
+
|
|
202
|
+
### ViewComponents
|
|
203
|
+
|
|
204
|
+
| Component | File | Responsibility |
|
|
205
|
+
|-----------|------|----------------|
|
|
206
|
+
| `StatusBadgeComponent` | `app/components/source_monitor/status_badge_component.rb` | Renders a colored status badge for health_status values |
|
|
207
|
+
| `IconComponent` | `app/components/source_monitor/icon_component.rb` | Renders consistent inline SVG or CSS icons |
|
|
208
|
+
| `FilterDropdownComponent` | `app/components/source_monitor/filter_dropdown_component.rb` | Renders filter dropdown menus for sources/items index |
|
|
209
|
+
|
|
210
|
+
### Presenters
|
|
211
|
+
|
|
212
|
+
| Presenter | File | Responsibility |
|
|
213
|
+
|-----------|------|----------------|
|
|
214
|
+
| `SourceDetailsPresenter` | `app/presenters/source_monitor/source_details_presenter.rb` | View-specific formatting for source detail pages (SimpleDelegator over Source) |
|
|
215
|
+
| `SourcesFilterPresenter` | `app/presenters/source_monitor/sources_filter_presenter.rb` | Formats filter state and option lists for the sources index |
|
|
216
|
+
|
|
185
217
|
### Other
|
|
186
218
|
|
|
187
219
|
| Module | File | Responsibility |
|
|
@@ -112,4 +112,45 @@ Source has a `type` column for potential STI subclassing but it is not actively
|
|
|
112
112
|
|--------|--------|-------|-------|
|
|
113
113
|
| Source | `items_count` | Item | Only counts active (non-deleted) items |
|
|
114
114
|
|
|
115
|
-
The counter is decremented manually during `soft_delete!` and
|
|
115
|
+
The counter is decremented manually during `soft_delete!` and **incremented** during `restore!`. Both operations update `items_count` atomically. Use `reset_items_counter!` to recalculate from scratch if the counter drifts.
|
|
116
|
+
|
|
117
|
+
## Model Methods (v0.12.0+)
|
|
118
|
+
|
|
119
|
+
### Source.enable_scraping!(ids)
|
|
120
|
+
|
|
121
|
+
Bulk-enables scraping for a list of source IDs:
|
|
122
|
+
|
|
123
|
+
```ruby
|
|
124
|
+
Source.enable_scraping!([1, 2, 3])
|
|
125
|
+
# Sets scraping_enabled = true for each source in the list
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
Use case: enabling scraping on a filtered set of sources from the dashboard or a rake task.
|
|
129
|
+
|
|
130
|
+
### Item#restore!
|
|
131
|
+
|
|
132
|
+
Reverses a soft delete. Symmetric counterpart to `soft_delete!`:
|
|
133
|
+
|
|
134
|
+
```ruby
|
|
135
|
+
item.restore!
|
|
136
|
+
# Clears deleted_at, increments source.items_count
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
After `restore!`, the item re-enters the `:items` (active) association and the source counter cache reflects the change.
|
|
140
|
+
|
|
141
|
+
### health_status Validation
|
|
142
|
+
|
|
143
|
+
As of v0.12.0, `health_status` is validated against the four permitted values. Assigning an unrecognized value raises a validation error:
|
|
144
|
+
|
|
145
|
+
```ruby
|
|
146
|
+
source.health_status = "healthy" # invalid -- was removed in 0.11.0
|
|
147
|
+
source.valid? # => false
|
|
148
|
+
source.errors[:health_status] # => ["is not included in the list"]
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
Permitted values: `"working"`, `"declining"`, `"improving"`, `"failing"`.
|
|
152
|
+
|
|
153
|
+
## Schema Notes (v0.12.0)
|
|
154
|
+
|
|
155
|
+
- **Composite indexes on log tables:** `sourcemon_fetch_logs`, `sourcemon_scrape_logs`, and `sourcemon_health_check_logs` now have composite indexes on `(source_id, created_at)` to improve log query performance on busy sources.
|
|
156
|
+
- **health_status default alignment:** The `health_status` column default on `sourcemon_sources` was updated to `"working"` (previously `"healthy"`, a removed value) to match the four-value enum added in v0.11.0.
|
|
@@ -113,7 +113,7 @@ Notable patterns:
|
|
|
113
113
|
|
|
114
114
|
### Worker Job (ScrapeItemJob)
|
|
115
115
|
|
|
116
|
-
Demonstrates
|
|
116
|
+
Demonstrates shallow delegation to a service class:
|
|
117
117
|
|
|
118
118
|
```ruby
|
|
119
119
|
class ScrapeItemJob < ApplicationJob
|
|
@@ -121,35 +121,18 @@ class ScrapeItemJob < ApplicationJob
|
|
|
121
121
|
discard_on ActiveJob::DeserializationError
|
|
122
122
|
|
|
123
123
|
def perform(item_id)
|
|
124
|
-
log("job:start", item_id: item_id)
|
|
125
124
|
item = Item.includes(:source).find_by(id: item_id)
|
|
126
125
|
return unless item
|
|
127
126
|
|
|
128
|
-
|
|
129
|
-
unless source&.scraping_enabled?
|
|
130
|
-
log("job:skipped_scraping_disabled", item: item)
|
|
131
|
-
Scraping::State.clear_inflight!(item)
|
|
132
|
-
return
|
|
133
|
-
end
|
|
134
|
-
|
|
135
|
-
Scraping::State.mark_processing!(item)
|
|
136
|
-
Scraping::ItemScraper.new(item:, source:).call
|
|
137
|
-
log("job:completed", item: item, status: item.scrape_status)
|
|
138
|
-
rescue StandardError => error
|
|
139
|
-
log("job:error", item: item, error: error.message)
|
|
140
|
-
Scraping::State.mark_failed!(item)
|
|
141
|
-
raise
|
|
142
|
-
ensure
|
|
143
|
-
Scraping::State.clear_inflight!(item) if item
|
|
127
|
+
Scraping::Runner.new(item: item).call
|
|
144
128
|
end
|
|
145
129
|
end
|
|
146
130
|
```
|
|
147
131
|
|
|
148
132
|
Notable patterns:
|
|
149
|
-
-
|
|
150
|
-
-
|
|
151
|
-
-
|
|
152
|
-
- Structured JSON logging at each stage
|
|
133
|
+
- Job body is deserialization + delegation only
|
|
134
|
+
- All business logic (state management, scraping, logging) lives in `Scraping::Runner`
|
|
135
|
+
- `includes(:source)` prevents N+1 query before handing off to the service
|
|
153
136
|
|
|
154
137
|
### Scheduling Job (ScheduleFetchesJob)
|
|
155
138
|
|
|
@@ -168,7 +151,7 @@ end
|
|
|
168
151
|
|
|
169
152
|
### Lightweight Fetch Job (FaviconFetchJob)
|
|
170
153
|
|
|
171
|
-
Demonstrates
|
|
154
|
+
Demonstrates shallow delegation with guard clause:
|
|
172
155
|
|
|
173
156
|
```ruby
|
|
174
157
|
class FaviconFetchJob < ApplicationJob
|
|
@@ -178,23 +161,19 @@ class FaviconFetchJob < ApplicationJob
|
|
|
178
161
|
def perform(source_id)
|
|
179
162
|
source = Source.find_by(id: source_id)
|
|
180
163
|
return unless source
|
|
181
|
-
return unless should_fetch?(source)
|
|
182
164
|
|
|
183
|
-
|
|
184
|
-
attach_favicon(source, result) if result.success?
|
|
165
|
+
Favicons::Fetcher.new(source: source).call
|
|
185
166
|
end
|
|
186
167
|
end
|
|
187
168
|
```
|
|
188
169
|
|
|
189
170
|
Notable patterns:
|
|
190
|
-
-
|
|
191
|
-
- Uses `Favicons::Discoverer` service with 3-strategy cascade (direct `/favicon.ico`, HTML parsing, Google API)
|
|
192
|
-
- Failed attempts tracked in source `metadata` JSONB (`favicon_last_attempted_at`) for retry cooldown
|
|
171
|
+
- Job contains only lookup + delegation — guard clauses and discovery strategy cascade live in `Favicons::Fetcher`
|
|
193
172
|
- Graceful degradation: host apps without Active Storage never enqueue this job
|
|
194
173
|
|
|
195
174
|
### Broadcast Job (SourceHealthCheckJob)
|
|
196
175
|
|
|
197
|
-
Demonstrates result broadcasting:
|
|
176
|
+
Demonstrates shallow delegation with result broadcasting:
|
|
198
177
|
|
|
199
178
|
```ruby
|
|
200
179
|
class SourceHealthCheckJob < ApplicationJob
|
|
@@ -205,21 +184,48 @@ class SourceHealthCheckJob < ApplicationJob
|
|
|
205
184
|
source = Source.find_by(id: source_id)
|
|
206
185
|
return unless source
|
|
207
186
|
|
|
208
|
-
|
|
209
|
-
broadcast_outcome(source, result)
|
|
210
|
-
result
|
|
211
|
-
rescue StandardError => error
|
|
212
|
-
record_unexpected_failure(source, error) if source
|
|
213
|
-
broadcast_outcome(source, nil, error) if source
|
|
214
|
-
nil
|
|
187
|
+
Health::SourceHealthCheckOrchestrator.new(source: source).call
|
|
215
188
|
end
|
|
216
189
|
end
|
|
217
190
|
```
|
|
218
191
|
|
|
219
192
|
Notable patterns:
|
|
220
|
-
-
|
|
221
|
-
-
|
|
222
|
-
- Returns nil on
|
|
193
|
+
- Job body is lookup + delegation only
|
|
194
|
+
- Broadcasting, logging, and error handling live in `Health::SourceHealthCheckOrchestrator`
|
|
195
|
+
- Returns nil on missing source; orchestrator handles nil-on-error (health checks are non-critical)
|
|
196
|
+
|
|
197
|
+
## Shallow Delegation Pattern
|
|
198
|
+
|
|
199
|
+
As of v0.12.0, five maintenance jobs delegate entirely to dedicated service classes. Jobs contain only deserialization and delegation — no business logic.
|
|
200
|
+
|
|
201
|
+
| Job | Service Class |
|
|
202
|
+
|-----|---------------|
|
|
203
|
+
| `ScrapeItemJob` | `Scraping::Runner` |
|
|
204
|
+
| `DownloadContentImagesJob` | `Images::Processor` |
|
|
205
|
+
| `FaviconFetchJob` | `Favicons::Fetcher` |
|
|
206
|
+
| `SourceHealthCheckJob` | `Health::SourceHealthCheckOrchestrator` |
|
|
207
|
+
| `ImportSessionHealthCheckJob` | `ImportSessions::HealthCheckUpdater` |
|
|
208
|
+
|
|
209
|
+
**Why this pattern:**
|
|
210
|
+
- Jobs are the transport mechanism, not the behavior container.
|
|
211
|
+
- Service classes are unit-testable without ActiveJob infrastructure.
|
|
212
|
+
- Future pipeline changes (e.g., calling the same logic synchronously) require no job changes.
|
|
213
|
+
|
|
214
|
+
**Template for new jobs:**
|
|
215
|
+
|
|
216
|
+
```ruby
|
|
217
|
+
class MyJob < ApplicationJob
|
|
218
|
+
source_monitor_queue :maintenance
|
|
219
|
+
discard_on ActiveJob::DeserializationError
|
|
220
|
+
|
|
221
|
+
def perform(record_id)
|
|
222
|
+
record = MyModel.find_by(id: record_id)
|
|
223
|
+
return unless record
|
|
224
|
+
|
|
225
|
+
MyNamespace::MyService.new(record: record).call
|
|
226
|
+
end
|
|
227
|
+
end
|
|
228
|
+
```
|
|
223
229
|
|
|
224
230
|
## _later / _now Naming Convention
|
|
225
231
|
|
|
@@ -2,6 +2,25 @@
|
|
|
2
2
|
|
|
3
3
|
Version-specific migration notes for each major/minor version transition. Agents should reference this file when guiding users through multi-version upgrades.
|
|
4
4
|
|
|
5
|
+
## 0.11.x to 0.12.0
|
|
6
|
+
|
|
7
|
+
**Key changes:**
|
|
8
|
+
- 2 new migrations: composite indexes on `sourcemon_fetch_logs`, `sourcemon_scrape_logs`, and `sourcemon_health_check_logs` (on `source_id, created_at`), and `health_status` column default corrected to `"working"`.
|
|
9
|
+
- 5 background jobs extracted to service classes: `ScrapeItemJob` -> `Scraping::Runner`, `DownloadContentImagesJob` -> `Images::Processor`, `FaviconFetchJob` -> `Favicons::Fetcher`, `SourceHealthCheckJob` -> `Health::SourceHealthCheckOrchestrator`, `ImportSessionHealthCheckJob` -> `ImportSessions::HealthCheckUpdater`. Job arguments and queue assignments are unchanged.
|
|
10
|
+
- New ViewComponents: `StatusBadgeComponent`, `IconComponent`, `FilterDropdownComponent`.
|
|
11
|
+
- New presenters: `SourceDetailsPresenter`, `SourcesFilterPresenter`.
|
|
12
|
+
- New model methods: `Source.enable_scraping!(ids)`, `Item#restore!`, `health_status` validation against the 4 permitted values.
|
|
13
|
+
|
|
14
|
+
**Action items:**
|
|
15
|
+
1. Copy and run migrations:
|
|
16
|
+
```bash
|
|
17
|
+
bin/rails source_monitor:install:migrations
|
|
18
|
+
bin/rails db:migrate
|
|
19
|
+
```
|
|
20
|
+
2. No breaking changes -- all existing initializer configuration and job interfaces remain valid.
|
|
21
|
+
3. No configuration changes required.
|
|
22
|
+
4. ViewComponents and presenters are available for use in custom views but are not required.
|
|
23
|
+
|
|
5
24
|
## 0.10.2 to 0.11.0
|
|
6
25
|
|
|
7
26
|
**Key changes:**
|
data/CHANGELOG.md
CHANGED
|
@@ -15,6 +15,63 @@ All notable changes to this project are documented below. The format follows [Ke
|
|
|
15
15
|
|
|
16
16
|
- No unreleased changes yet.
|
|
17
17
|
|
|
18
|
+
## [0.12.1] - 2026-03-15
|
|
19
|
+
|
|
20
|
+
### Documentation
|
|
21
|
+
- Updated README, docs/setup, docs/upgrade with v0.12.0 version references and upgrade guide
|
|
22
|
+
- Updated 4 skill references (sm-job, sm-architecture, sm-domain-model, sm-upgrade) for service class extractions, new components, and model methods
|
|
23
|
+
- Improved `/release` command: Step 4 now spawns agents to automatically audit AND fix all stale docs before every release
|
|
24
|
+
|
|
25
|
+
## [0.12.0] - 2026-03-15
|
|
26
|
+
|
|
27
|
+
### Added
|
|
28
|
+
- `StatusBadgeComponent` for unified status badge rendering across all views
|
|
29
|
+
- `IconComponent` with named icon registry replacing inline SVGs
|
|
30
|
+
- `SourceDetailsPresenter` and `SourcesFilterPresenter` for view formatting
|
|
31
|
+
- `FilterDropdownComponent` for reusable filter dropdowns
|
|
32
|
+
- `SetSource` controller concern eliminating duplication across 7 controllers
|
|
33
|
+
- `rescue_from ActiveRecord::RecordNotFound` with Turbo-aware 404 handling
|
|
34
|
+
- `Source.enable_scraping!` class method for bulk enablement
|
|
35
|
+
- `Item#restore!` with counter cache symmetry to `soft_delete!`
|
|
36
|
+
- `Source` health_status inclusion validation and scraping scopes
|
|
37
|
+
- Modal accessibility: `role="dialog"`, `aria-modal`, focus trapping via `inert`
|
|
38
|
+
- Logs index Turbo Frame for partial page updates on filter/pagination
|
|
39
|
+
- Missing controller tests for FetchLogsController and ScrapeLogsController
|
|
40
|
+
- Date range scopes and composite indexes on Loggable concern
|
|
41
|
+
- Result structs for completion handlers (EventPublisher, RetentionHandler, FollowUpHandler)
|
|
42
|
+
- Transient vs fatal error classification in FaviconFetchJob and DownloadContentImagesJob
|
|
43
|
+
- SharedLoggableTests module for consistent log model testing
|
|
44
|
+
- TEST_CONVENTIONS.md documenting testing patterns
|
|
45
|
+
|
|
46
|
+
### Fixed
|
|
47
|
+
- LogCleanupJob now cascades deletes to LogEntry records preventing orphaned rows
|
|
48
|
+
- `health_status` default aligned between model (`working`) and database
|
|
49
|
+
- Swallowed exceptions in `Scraping::State` and pipeline rescue blocks now log warnings
|
|
50
|
+
- Pagination tests use `clean_source_monitor_tables!` instead of `Source.destroy_all` for parallel safety
|
|
51
|
+
- `fallback_user_id` guarded behind `Rails.env.development?` to prevent engine writing to host-app tables
|
|
52
|
+
- Dashboard GC pressure reduced from SolidQueue data accumulation
|
|
53
|
+
- Deadlock rescue added to 4 jobs
|
|
54
|
+
|
|
55
|
+
### Changed
|
|
56
|
+
- **5 jobs extracted to service classes** (shallow delegation pattern):
|
|
57
|
+
- `ScrapeItemJob` → `Scraping::Runner` (73→21 lines)
|
|
58
|
+
- `DownloadContentImagesJob` → `Images::Processor` (95→16 lines)
|
|
59
|
+
- `FaviconFetchJob` → `Favicons::Fetcher` (94→16 lines)
|
|
60
|
+
- `SourceHealthCheckJob` → `Health::SourceHealthCheckOrchestrator` (85→16 lines)
|
|
61
|
+
- `ImportSessionHealthCheckJob` → `ImportSessions::HealthCheckUpdater` (99→32 lines)
|
|
62
|
+
- Duplicated scrape rate-limiting consolidated (removed from ScrapeItemJob, kept in Enqueuer)
|
|
63
|
+
- `sync_log_entry` callback consolidated into Loggable concern (removed from 3 models)
|
|
64
|
+
- `ItemScrapesController` extracted from Items#scrape for CRUD compliance
|
|
65
|
+
- FetchFeedJob retry/circuit-breaker logic extracted to RetryOrchestrator service
|
|
66
|
+
- Favicon cooldown logic moved from controller to Source model
|
|
67
|
+
- Import step dispatch refactored to handler registry pattern
|
|
68
|
+
- 30 backward-compatibility forwarding methods removed from FeedFetcher and ItemCreator
|
|
69
|
+
- `Images::Downloader` uses `HTTP.client` instead of raw Faraday (respects proxy/SSL config)
|
|
70
|
+
- `CloudflareBypass` limited to 1 UA rotation attempt with per-attempt timeout
|
|
71
|
+
- Dashboard stats split into per-stat Turbo Streams for granular updates
|
|
72
|
+
- Test factory helpers centralized into ModelFactories module
|
|
73
|
+
- System tests refactored to shared base class with SystemTestHelpers
|
|
74
|
+
|
|
18
75
|
## [0.11.1] - 2026-03-13
|
|
19
76
|
|
|
20
77
|
### Fixed
|
data/CLAUDE.md
CHANGED
|
@@ -4,8 +4,8 @@
|
|
|
4
4
|
|
|
5
5
|
## Active Context
|
|
6
6
|
|
|
7
|
-
**Last shipped:**
|
|
8
|
-
**Next action:** /vbw:vibe to
|
|
7
|
+
**Last shipped:** rails-audit-and-refactoring (7 phases, 30 plans)
|
|
8
|
+
**Next action:** /vbw:vibe to start new work
|
|
9
9
|
|
|
10
10
|
## Key Decisions
|
|
11
11
|
|