source_monitor 0.3.3 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.claude/commands/release.md +101 -58
- data/.claude/skills/sm-configure/SKILL.md +13 -2
- data/.claude/skills/sm-configure/reference/configuration-reference.md +33 -0
- data/.claude/skills/sm-host-setup/SKILL.md +18 -2
- data/.claude/skills/sm-host-setup/reference/setup-checklist.md +33 -0
- data/.claude/skills/sm-job/SKILL.md +1 -1
- data/.claude/skills/sm-upgrade/SKILL.md +102 -0
- data/.claude/skills/sm-upgrade/reference/upgrade-workflow.md +92 -0
- data/.claude/skills/sm-upgrade/reference/version-history.md +68 -0
- data/.vbw-planning/SHIPPED.md +35 -0
- data/.vbw-planning/config.json +24 -1
- data/.vbw-planning/discovery.json +3 -1
- data/.vbw-planning/{REQUIREMENTS.md → milestones/generator-enhancements/REQUIREMENTS.md} +22 -0
- data/.vbw-planning/milestones/generator-enhancements/ROADMAP.md +125 -0
- data/.vbw-planning/milestones/generator-enhancements/SHIPPED.md +40 -0
- data/.vbw-planning/milestones/generator-enhancements/STATE.md +43 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/01-generator-steps/01-CONTEXT.md +33 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/01-generator-steps/01-VERIFICATION.md +86 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/01-generator-steps/PLAN-01-SUMMARY.md +61 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/01-generator-steps/PLAN-01.md +380 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/02-verification/02-VERIFICATION.md +78 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/02-verification/PLAN-01-SUMMARY.md +46 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/02-verification/PLAN-01.md +500 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/03-docs-alignment/03-VERIFICATION.md +89 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/03-docs-alignment/PLAN-01-SUMMARY.md +48 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/03-docs-alignment/PLAN-01.md +456 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/04-dashboard-ux/04-VERIFICATION.md +129 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/04-dashboard-ux/PLAN-01-SUMMARY.md +70 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/04-dashboard-ux/PLAN-01.md +747 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/05-active-storage-images/05-VERIFICATION.md +156 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/05-active-storage-images/PLAN-01-SUMMARY.md +69 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/05-active-storage-images/PLAN-01.md +455 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/05-active-storage-images/PLAN-02-SUMMARY.md +39 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/05-active-storage-images/PLAN-02.md +488 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/06-netflix-feed-fix/06-VERIFICATION.md +100 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/06-netflix-feed-fix/PLAN-01-SUMMARY.md +37 -0
- data/.vbw-planning/milestones/generator-enhancements/phases/06-netflix-feed-fix/PLAN-01.md +345 -0
- data/.vbw-planning/milestones/upgrade-assurance/REQUIREMENTS.md +80 -0
- data/.vbw-planning/milestones/upgrade-assurance/ROADMAP.md +75 -0
- data/.vbw-planning/milestones/upgrade-assurance/STATE.md +29 -0
- data/.vbw-planning/milestones/upgrade-assurance/phases/01-upgrade-command/01-VERIFICATION.md +144 -0
- data/.vbw-planning/milestones/upgrade-assurance/phases/01-upgrade-command/PLAN-01-SUMMARY.md +43 -0
- data/.vbw-planning/milestones/upgrade-assurance/phases/01-upgrade-command/PLAN-01.md +405 -0
- data/.vbw-planning/milestones/upgrade-assurance/phases/02-config-deprecation/PLAN-01-SUMMARY.md +27 -0
- data/.vbw-planning/milestones/upgrade-assurance/phases/02-config-deprecation/PLAN-01.md +303 -0
- data/.vbw-planning/milestones/upgrade-assurance/phases/03-upgrade-skill-docs/03-VERIFICATION.md +380 -0
- data/.vbw-planning/milestones/upgrade-assurance/phases/03-upgrade-skill-docs/PLAN-01-SUMMARY.md +36 -0
- data/.vbw-planning/milestones/upgrade-assurance/phases/03-upgrade-skill-docs/PLAN-01.md +652 -0
- data/CHANGELOG.md +48 -0
- data/CLAUDE.md +5 -3
- data/Gemfile.lock +1 -1
- data/VERSION +1 -1
- data/app/assets/builds/source_monitor/application.css +9 -0
- data/app/helpers/source_monitor/application_helper.rb +38 -0
- data/app/jobs/source_monitor/download_content_images_job.rb +72 -0
- data/app/models/source_monitor/item_content.rb +2 -0
- data/app/views/source_monitor/dashboard/_recent_activity.html.erb +9 -0
- data/app/views/source_monitor/items/_details.html.erb +2 -2
- data/app/views/source_monitor/logs/index.html.erb +9 -0
- data/app/views/source_monitor/sources/_details.html.erb +2 -2
- data/app/views/source_monitor/sources/_row.html.erb +1 -1
- data/docs/setup.md +10 -1
- data/docs/troubleshooting.md +38 -7
- data/docs/upgrade.md +140 -0
- data/lib/generators/source_monitor/install/install_generator.rb +101 -0
- data/lib/source_monitor/configuration/deprecation_registry.rb +237 -0
- data/lib/source_monitor/configuration/http_settings.rb +7 -1
- data/lib/source_monitor/configuration/images_settings.rb +37 -0
- data/lib/source_monitor/configuration.rb +11 -1
- data/lib/source_monitor/dashboard/queries/recent_activity_query.rb +16 -7
- data/lib/source_monitor/dashboard/recent_activity.rb +1 -0
- data/lib/source_monitor/dashboard/recent_activity_presenter.rb +15 -2
- data/lib/source_monitor/fetching/feed_fetcher/entry_processor.rb +13 -0
- data/lib/source_monitor/http.rb +23 -0
- data/lib/source_monitor/images/content_rewriter.rb +81 -0
- data/lib/source_monitor/images/downloader.rb +82 -0
- data/lib/source_monitor/logs/table_presenter.rb +25 -0
- data/lib/source_monitor/setup/cli.rb +7 -0
- data/lib/source_monitor/setup/procfile_patcher.rb +31 -0
- data/lib/source_monitor/setup/queue_config_patcher.rb +84 -0
- data/lib/source_monitor/setup/skills_installer.rb +1 -0
- data/lib/source_monitor/setup/upgrade_command.rb +59 -0
- data/lib/source_monitor/setup/verification/pending_migrations_verifier.rb +92 -0
- data/lib/source_monitor/setup/verification/recurring_schedule_verifier.rb +102 -0
- data/lib/source_monitor/setup/verification/runner.rb +1 -1
- data/lib/source_monitor/setup/verification/solid_queue_verifier.rb +1 -1
- data/lib/source_monitor/setup/workflow.rb +10 -0
- data/lib/source_monitor/version.rb +1 -1
- data/lib/source_monitor.rb +11 -0
- metadata +51 -2
data/CHANGELOG.md
CHANGED
|
@@ -15,6 +15,54 @@ All notable changes to this project are documented below. The format follows [Ke
|
|
|
15
15
|
|
|
16
16
|
- No unreleased changes yet.
|
|
17
17
|
|
|
18
|
+
## [0.5.0] - 2026-02-13
|
|
19
|
+
|
|
20
|
+
### Added
|
|
21
|
+
|
|
22
|
+
- `bin/source_monitor upgrade` command: detects version changes since last install, copies new migrations, re-runs the generator, runs verification, and reports what changed. Uses a `.source_monitor_version` marker file for version tracking.
|
|
23
|
+
- `PendingMigrationsVerifier` checks for unmigrated SourceMonitor tables in the verification suite, integrated into both `bin/source_monitor verify` and the upgrade flow.
|
|
24
|
+
- Configuration deprecation framework: engine developers can register deprecated config options with `DeprecationRegistry.register`. At boot time, stale options trigger `:warning` (renamed) or `:error` (removed) messages with actionable replacement paths.
|
|
25
|
+
- `sm-upgrade` AI skill guides agents through post-update workflows: CHANGELOG parsing, running the upgrade command, interpreting verification results, and handling deprecation warnings.
|
|
26
|
+
- `docs/upgrade.md` versioned upgrade guide with general steps, version-specific notes (0.1.x through 0.4.x), and troubleshooting.
|
|
27
|
+
- `sm-host-setup` skill cross-references the upgrade workflow.
|
|
28
|
+
|
|
29
|
+
### Testing
|
|
30
|
+
|
|
31
|
+
- 1,003 tests, 0 failures (up from 973 in 0.4.0).
|
|
32
|
+
- RuboCop: 397 files, 0 offenses.
|
|
33
|
+
- Brakeman: 0 warnings.
|
|
34
|
+
|
|
35
|
+
## [0.4.0] - 2026-02-12
|
|
36
|
+
|
|
37
|
+
### Added
|
|
38
|
+
|
|
39
|
+
- Install generator now auto-patches `Procfile.dev` with Solid Queue `jobs:` entry and `queue.yml` with `recurring_schedule` dispatcher wiring (idempotent, skip if already present).
|
|
40
|
+
- `RecurringScheduleVerifier` checks that recurring tasks are registered with Solid Queue dispatchers; `SolidQueueVerifier` remediation now mentions `Procfile.dev` for `bin/dev` users.
|
|
41
|
+
- Dashboard fetch log entries display source URL (domain for RSS, item URL for scrapes) alongside existing summary.
|
|
42
|
+
- External links across dashboard, logs, sources, and items open in new tab with visual indicator icon.
|
|
43
|
+
- Configurable Active Storage image downloads: `config.images.download_to_active_storage` (default `false`) detects inline images in feed content, downloads them via background job, and rewrites `<img>` src attributes with Active Storage URLs.
|
|
44
|
+
- `Images::ContentRewriter` extracts and rewrites image URLs from HTML content using Nokolexbor.
|
|
45
|
+
- `Images::Downloader` service validates content type and size before downloading images.
|
|
46
|
+
- `DownloadContentImagesJob` orchestrates the download/attach/rewrite pipeline per item.
|
|
47
|
+
- SSL certificate store configuration: every Faraday connection gets an `OpenSSL::X509::Store` initialized with `set_default_paths`, resolving "unable to get local issuer certificate" errors on systems with incomplete CA bundles.
|
|
48
|
+
- Configurable SSL options in `HTTPSettings`: `ssl_ca_file`, `ssl_ca_path`, `ssl_verify` for non-standard certificate environments.
|
|
49
|
+
- Netflix Tech Blog VCR cassette regression test proving Medium-hosted RSS feeds parse correctly with the SSL fix.
|
|
50
|
+
|
|
51
|
+
### Fixed
|
|
52
|
+
|
|
53
|
+
- SSL certificate verification failures for feeds hosted on services requiring intermediate CAs (e.g., Netflix Tech Blog via Medium/AWS).
|
|
54
|
+
- Setup documentation now includes `Procfile.dev` and `recurring_schedule` guidance.
|
|
55
|
+
|
|
56
|
+
### Changed
|
|
57
|
+
|
|
58
|
+
- Updated `sm-host-setup`, `sm-configure`, and setup documentation to reflect that the generator handles Procfile.dev and recurring_schedule automatically.
|
|
59
|
+
|
|
60
|
+
### Testing
|
|
61
|
+
|
|
62
|
+
- 973 tests, 3,114 assertions, 0 failures (up from 841 tests in 0.3.3).
|
|
63
|
+
- RuboCop: 389 files, 0 offenses.
|
|
64
|
+
- Brakeman: 0 warnings.
|
|
65
|
+
|
|
18
66
|
## [0.3.3] - 2026-02-11
|
|
19
67
|
|
|
20
68
|
### Fixed
|
data/CLAUDE.md
CHANGED
|
@@ -4,9 +4,10 @@
|
|
|
4
4
|
|
|
5
5
|
## Active Context
|
|
6
6
|
|
|
7
|
-
**Milestone:** none
|
|
8
|
-
**Last shipped:**
|
|
9
|
-
**
|
|
7
|
+
**Milestone:** (none active)
|
|
8
|
+
**Last shipped:** upgrade-assurance (2026-02-13) -- 3 phases, 14 tasks, 12 commits
|
|
9
|
+
**Previous:** generator-enhancements (2026-02-12) -- v0.4.0
|
|
10
|
+
**Next action:** /vbw:vibe to start a new milestone
|
|
10
11
|
|
|
11
12
|
## Key Decisions
|
|
12
13
|
|
|
@@ -192,6 +193,7 @@ Engine-specific skills (`sm-*` prefix). Consumer skills install by default; cont
|
|
|
192
193
|
| `sm-event-handler` | Lifecycle callbacks (after_item_created, etc.) |
|
|
193
194
|
| `sm-model-extension` | Extend engine models from host app |
|
|
194
195
|
| `sm-dashboard-widget` | Dashboard queries, presenters, Turbo broadcasts |
|
|
196
|
+
| `sm-upgrade` | Gem upgrade workflow with CHANGELOG parsing |
|
|
195
197
|
|
|
196
198
|
### Contributor Skills (opt-in)
|
|
197
199
|
|
data/Gemfile.lock
CHANGED
data/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
0.
|
|
1
|
+
0.5.0
|
|
@@ -705,6 +705,10 @@ video {
|
|
|
705
705
|
margin-right: 0.25rem;
|
|
706
706
|
}
|
|
707
707
|
|
|
708
|
+
.fm-admin .mt-0\.5 {
|
|
709
|
+
margin-top: 0.125rem;
|
|
710
|
+
}
|
|
711
|
+
|
|
708
712
|
.fm-admin .mt-1 {
|
|
709
713
|
margin-top: 0.25rem;
|
|
710
714
|
}
|
|
@@ -1789,6 +1793,11 @@ video {
|
|
|
1789
1793
|
--tw-ring-color: transparent;
|
|
1790
1794
|
}
|
|
1791
1795
|
|
|
1796
|
+
.fm-admin .invert {
|
|
1797
|
+
--tw-invert: invert(100%);
|
|
1798
|
+
filter: var(--tw-blur) var(--tw-brightness) var(--tw-contrast) var(--tw-grayscale) var(--tw-hue-rotate) var(--tw-invert) var(--tw-saturate) var(--tw-sepia) var(--tw-drop-shadow);
|
|
1799
|
+
}
|
|
1800
|
+
|
|
1792
1801
|
.fm-admin .filter {
|
|
1793
1802
|
filter: var(--tw-blur) var(--tw-brightness) var(--tw-contrast) var(--tw-grayscale) var(--tw-hue-rotate) var(--tw-invert) var(--tw-saturate) var(--tw-sepia) var(--tw-drop-shadow);
|
|
1794
1803
|
}
|
|
@@ -212,8 +212,46 @@ module SourceMonitor
|
|
|
212
212
|
end
|
|
213
213
|
end
|
|
214
214
|
|
|
215
|
+
# Renders a clickable link that opens in a new tab with an external-link icon.
|
|
216
|
+
# Returns the label as plain text if the URL is blank.
|
|
217
|
+
def external_link_to(label, url, **options)
|
|
218
|
+
return label if url.blank?
|
|
219
|
+
|
|
220
|
+
css = options.delete(:class) || "text-blue-600 hover:text-blue-500"
|
|
221
|
+
link_to(url, target: "_blank", rel: "noopener noreferrer", class: css, title: url, **options) do
|
|
222
|
+
safe_join([ label, " ", external_link_icon ])
|
|
223
|
+
end
|
|
224
|
+
end
|
|
225
|
+
|
|
226
|
+
# Extracts the domain from a URL, returning nil if parsing fails.
|
|
227
|
+
def domain_from_url(url)
|
|
228
|
+
return nil if url.blank?
|
|
229
|
+
|
|
230
|
+
URI.parse(url.to_s).host
|
|
231
|
+
rescue URI::InvalidURIError
|
|
232
|
+
nil
|
|
233
|
+
end
|
|
234
|
+
|
|
215
235
|
private
|
|
216
236
|
|
|
237
|
+
def external_link_icon
|
|
238
|
+
tag.svg(
|
|
239
|
+
class: "inline-block h-3 w-3 text-slate-400",
|
|
240
|
+
xmlns: "http://www.w3.org/2000/svg",
|
|
241
|
+
fill: "none",
|
|
242
|
+
viewBox: "0 0 24 24",
|
|
243
|
+
stroke_width: "2",
|
|
244
|
+
stroke: "currentColor",
|
|
245
|
+
aria: { hidden: "true" }
|
|
246
|
+
) do
|
|
247
|
+
tag.path(
|
|
248
|
+
stroke_linecap: "round",
|
|
249
|
+
stroke_linejoin: "round",
|
|
250
|
+
d: "M13.5 6H5.25A2.25 2.25 0 003 8.25v10.5A2.25 2.25 0 005.25 21h10.5A2.25 2.25 0 0018 18.75V10.5m-10.5 6L21 3m0 0h-5.25M21 3v5.25"
|
|
251
|
+
)
|
|
252
|
+
end
|
|
253
|
+
end
|
|
254
|
+
|
|
217
255
|
def derive_item_scrape_status(item:, source: nil)
|
|
218
256
|
return "idle" unless item
|
|
219
257
|
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SourceMonitor
|
|
4
|
+
class DownloadContentImagesJob < ApplicationJob
|
|
5
|
+
source_monitor_queue :fetch
|
|
6
|
+
|
|
7
|
+
discard_on ActiveJob::DeserializationError
|
|
8
|
+
|
|
9
|
+
def perform(item_id)
|
|
10
|
+
item = SourceMonitor::Item.find_by(id: item_id)
|
|
11
|
+
return unless item
|
|
12
|
+
return unless SourceMonitor.config.images.download_enabled?
|
|
13
|
+
|
|
14
|
+
html = item.content
|
|
15
|
+
return if html.blank?
|
|
16
|
+
|
|
17
|
+
# Build or find item_content for attachment storage
|
|
18
|
+
item_content = item.item_content || item.build_item_content
|
|
19
|
+
|
|
20
|
+
# Skip if images already attached (idempotency)
|
|
21
|
+
return if item_content.persisted? && item_content.images.attached?
|
|
22
|
+
|
|
23
|
+
base_url = item.url
|
|
24
|
+
rewriter = SourceMonitor::Images::ContentRewriter.new(html, base_url: base_url)
|
|
25
|
+
image_urls = rewriter.image_urls
|
|
26
|
+
return if image_urls.empty?
|
|
27
|
+
|
|
28
|
+
# Save item_content first so we can attach blobs to it
|
|
29
|
+
item_content.save! unless item_content.persisted?
|
|
30
|
+
|
|
31
|
+
# Download images and build URL mapping
|
|
32
|
+
url_mapping = download_images(item_content, image_urls)
|
|
33
|
+
return if url_mapping.empty?
|
|
34
|
+
|
|
35
|
+
# Rewrite HTML with Active Storage URLs
|
|
36
|
+
rewritten_html = rewriter.rewrite do |original_url|
|
|
37
|
+
url_mapping[original_url]
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
# Update the item content with rewritten HTML
|
|
41
|
+
item.update!(content: rewritten_html)
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
private
|
|
45
|
+
|
|
46
|
+
def download_images(item_content, image_urls)
|
|
47
|
+
url_mapping = {}
|
|
48
|
+
settings = SourceMonitor.config.images
|
|
49
|
+
|
|
50
|
+
image_urls.each do |image_url|
|
|
51
|
+
result = SourceMonitor::Images::Downloader.new(image_url, settings: settings).call
|
|
52
|
+
next unless result
|
|
53
|
+
|
|
54
|
+
blob = ActiveStorage::Blob.create_and_upload!(
|
|
55
|
+
io: result.io,
|
|
56
|
+
filename: result.filename,
|
|
57
|
+
content_type: result.content_type
|
|
58
|
+
)
|
|
59
|
+
item_content.images.attach(blob)
|
|
60
|
+
|
|
61
|
+
# Generate a serving URL for the blob
|
|
62
|
+
url_mapping[image_url] = Rails.application.routes.url_helpers.rails_blob_path(blob, only_path: true)
|
|
63
|
+
rescue StandardError
|
|
64
|
+
# Individual image failure should not block others.
|
|
65
|
+
# Original URL will be preserved (graceful fallback).
|
|
66
|
+
next
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
url_mapping
|
|
70
|
+
end
|
|
71
|
+
end
|
|
72
|
+
end
|
|
@@ -21,6 +21,15 @@
|
|
|
21
21
|
<div class="mt-1 text-xs text-slate-500">
|
|
22
22
|
<%= event[:description].presence || "No additional details recorded." %>
|
|
23
23
|
</div>
|
|
24
|
+
<% if event[:url_display].present? %>
|
|
25
|
+
<div class="mt-0.5 text-xs text-slate-400 truncate max-w-sm" data-testid="event-url-display">
|
|
26
|
+
<% if event[:url_href].present? %>
|
|
27
|
+
<%= external_link_to event[:url_display], event[:url_href], class: "text-slate-400 hover:text-blue-500" %>
|
|
28
|
+
<% else %>
|
|
29
|
+
<%= event[:url_display] %>
|
|
30
|
+
<% end %>
|
|
31
|
+
</div>
|
|
32
|
+
<% end %>
|
|
24
33
|
</div>
|
|
25
34
|
<div class="text-right">
|
|
26
35
|
<span class="inline-flex items-center rounded-full px-3 py-1 text-xs font-semibold <%= event[:status] == :success ? "bg-green-100 text-green-700" : "bg-rose-100 text-rose-700" %>">
|
|
@@ -53,8 +53,8 @@
|
|
|
53
53
|
<% details = {
|
|
54
54
|
"GUID" => item.guid,
|
|
55
55
|
"Content Fingerprint" => item.content_fingerprint || "—",
|
|
56
|
-
"URL" => item.url,
|
|
57
|
-
"Canonical URL" => item.canonical_url
|
|
56
|
+
"URL" => (item.url.present? ? external_link_to(item.url, item.url, class: "text-slate-900 hover:text-blue-500") : "\u2014"),
|
|
57
|
+
"Canonical URL" => (item.canonical_url.present? ? external_link_to(item.canonical_url, item.canonical_url, class: "text-slate-900 hover:text-blue-500") : "\u2014"),
|
|
58
58
|
"Author" => item.author || "—",
|
|
59
59
|
"Published At" => (item.published_at&.strftime("%b %d, %Y %H:%M %Z") || "—"),
|
|
60
60
|
"Updated At (Source)" => (item.updated_at_source&.strftime("%b %d, %Y %H:%M %Z") || "—"),
|
|
@@ -134,6 +134,15 @@
|
|
|
134
134
|
<% else %>
|
|
135
135
|
<%= row.primary_label %>
|
|
136
136
|
<% end %>
|
|
137
|
+
<% if row.url_label.present? %>
|
|
138
|
+
<div class="mt-0.5 text-xs text-slate-400 truncate max-w-xs">
|
|
139
|
+
<% if row.url_href.present? %>
|
|
140
|
+
<%= external_link_to row.url_label, row.url_href, class: "text-slate-400 hover:text-blue-500" %>
|
|
141
|
+
<% else %>
|
|
142
|
+
<%= row.url_label %>
|
|
143
|
+
<% end %>
|
|
144
|
+
</div>
|
|
145
|
+
<% end %>
|
|
137
146
|
</td>
|
|
138
147
|
<td class="px-6 py-4 text-sm">
|
|
139
148
|
<% if row.source_path %>
|
|
@@ -25,7 +25,7 @@
|
|
|
25
25
|
</span>
|
|
26
26
|
<% end %>
|
|
27
27
|
</div>
|
|
28
|
-
<p class="mt-2 text-sm text-slate-500">Feed URL: <%= source.feed_url %></p>
|
|
28
|
+
<p class="mt-2 text-sm text-slate-500">Feed URL: <%= external_link_to source.feed_url, source.feed_url, class: "text-slate-500 hover:text-blue-500" %></p>
|
|
29
29
|
</div>
|
|
30
30
|
<div class="flex flex-wrap items-center justify-end gap-3">
|
|
31
31
|
<% fetch_disabled = %w[queued fetching].include?(source.fetch_status) %>
|
|
@@ -137,7 +137,7 @@
|
|
|
137
137
|
end
|
|
138
138
|
|
|
139
139
|
details = {
|
|
140
|
-
"Website" => (source.website_url.
|
|
140
|
+
"Website" => (source.website_url.present? ? external_link_to(source.website_url, source.website_url, class: "text-slate-900 hover:text-blue-500") : "\u2014"),
|
|
141
141
|
"Fetch interval" => "#{source.fetch_interval_minutes} minutes (~#{interval_hours} hours)",
|
|
142
142
|
"Adaptive interval" => source.adaptive_fetching_enabled? ? "Auto" : "Fixed",
|
|
143
143
|
"Scraper" => source.scraper_adapter,
|
|
@@ -29,7 +29,7 @@
|
|
|
29
29
|
class: "text-slate-900 hover:text-blue-600 focus:outline-none focus:ring-2 focus:ring-blue-500 focus:ring-offset-2",
|
|
30
30
|
data: { turbo_frame: "_top" } %>
|
|
31
31
|
</div>
|
|
32
|
-
<div class="text-xs text-slate-500 truncate max-w-xs"><%= source.feed_url %></div>
|
|
32
|
+
<div class="text-xs text-slate-500 truncate max-w-xs"><%= external_link_to source.feed_url, source.feed_url, class: "text-slate-500 hover:text-blue-500" %></div>
|
|
33
33
|
</td>
|
|
34
34
|
<td class="px-6 py-4">
|
|
35
35
|
<div class="flex flex-col gap-2 text-xs">
|
data/docs/setup.md
CHANGED
|
@@ -49,7 +49,12 @@ This ensures Bundler can load SourceMonitor so the commands below are available.
|
|
|
49
49
|
```bash
|
|
50
50
|
bin/rails solid_queue:start
|
|
51
51
|
```
|
|
52
|
-
|
|
52
|
+
The install generator automatically handles all worker configuration:
|
|
53
|
+
- **Recurring jobs** are configured in `config/recurring.yml` (fetch scheduling, scraping, cleanup).
|
|
54
|
+
- **Procfile.dev** is patched with a `jobs:` entry so `bin/dev` starts Solid Queue alongside the web server.
|
|
55
|
+
- **Queue dispatcher** is patched with `recurring_schedule: config/recurring.yml` in `config/queue.yml` so recurring jobs load on startup.
|
|
56
|
+
|
|
57
|
+
All three steps are idempotent. If any configuration is missing, re-run: `bin/rails generate source_monitor:install`
|
|
53
58
|
|
|
54
59
|
4. **Visit the dashboard** at the chosen mount path, create a source, and trigger “Fetch Now” to validate realtime updates and Solid Queue processing.
|
|
55
60
|
|
|
@@ -87,6 +92,8 @@ Prefer to script each step or plug SourceMonitor into an existing deployment che
|
|
|
87
92
|
| 4 | `bin/rails railties:install:migrations FROM=source_monitor` | Copy engine migrations (idempotent) |
|
|
88
93
|
| 5 | `bin/rails db:migrate` | Apply schema updates, including Solid Queue tables |
|
|
89
94
|
| 6 | `bin/rails solid_queue:start` | Ensure jobs process via Solid Queue |
|
|
95
|
+
| 6a | Handled by generator (patches `Procfile.dev`) | Ensure `bin/dev` starts Solid Queue workers |
|
|
96
|
+
| 6b | Handled by generator (patches `config/queue.yml`) | Wire recurring jobs into Solid Queue dispatcher |
|
|
90
97
|
| 7 | `bin/jobs --recurring_schedule_file=config/recurring.yml` | Start recurring scheduler (optional but recommended) |
|
|
91
98
|
| 8 | `bin/source_monitor verify` | Confirm Solid Queue/Action Cable readiness and emit telemetry |
|
|
92
99
|
|
|
@@ -100,6 +107,8 @@ Prefer to script each step or plug SourceMonitor into an existing deployment che
|
|
|
100
107
|
4. **Apply database changes** using `bin/rails db:migrate`. If your host already installed Solid Queue migrations manually, delete duplicate files before migrating.
|
|
101
108
|
5. **Wire Action Cable** if necessary. SourceMonitor defaults to Solid Cable; confirm `ApplicationCable::Connection`/`Channel` exist and that `config/initializers/source_monitor.rb` uses the adapter you expect. To switch to Redis, set `config.realtime.adapter = :redis` and `config.realtime.redis_url`.
|
|
102
109
|
6. **Start workers** with `bin/rails solid_queue:start` (or your process manager). The install generator automatically configures recurring jobs in `config/recurring.yml` for fetch scheduling, scraping, and cleanup. They'll run with `bin/dev` or `bin/jobs`.
|
|
110
|
+
- **Procfile.dev:** The generator automatically patches `Procfile.dev` with a `jobs:` entry for Solid Queue. Verify the file contains `jobs: bundle exec rake solid_queue:start` after running the generator.
|
|
111
|
+
- **Recurring schedule:** The generator automatically patches `config/queue.yml` dispatchers with `recurring_schedule: config/recurring.yml`. Verify the key is present after running the generator.
|
|
103
112
|
7. **Review the initializer** and tune queue names, HTTP timeouts, scraping adapters, retention limits, authentication hooks, and Mission Control integration. The [configuration reference](configuration.md) details every option.
|
|
104
113
|
8. **Verify the install**: run `bin/source_monitor verify` to ensure Solid Queue workers and Action Cable are healthy, then visit the mount path to trigger a fetch manually. Enable telemetry if you want JSON logs recorded for support.
|
|
105
114
|
|
data/docs/troubleshooting.md
CHANGED
|
@@ -20,45 +20,76 @@ This guide lists common issues you might encounter while installing, upgrading,
|
|
|
20
20
|
- Ensure at least one Solid Queue worker is running; the dashboard reads visibility data via `SourceMonitor::Jobs::Visibility`.
|
|
21
21
|
- When using mission control integration, keep `config.mission_control_dashboard_path` pointing at a valid route helper; otherwise the dashboard hides the link.
|
|
22
22
|
|
|
23
|
-
## 4.
|
|
23
|
+
## 4. Recurring Jobs Not Running
|
|
24
|
+
|
|
25
|
+
- **Symptoms:** Fetch scheduling, scrape scheduling, and cleanup jobs never fire. Sources never auto-fetch on their configured intervals.
|
|
26
|
+
- **Primary fix:** Re-run the install generator, which automatically patches the dispatcher config:
|
|
27
|
+
```bash
|
|
28
|
+
bin/rails generate source_monitor:install
|
|
29
|
+
```
|
|
30
|
+
- **Diagnostics:** Run `bin/source_monitor verify` to check recurring task registration. The RecurringScheduleVerifier will report whether SourceMonitor recurring tasks are loaded into Solid Queue.
|
|
31
|
+
- **Manual check:** Verify `config/queue.yml` includes `recurring_schedule: config/recurring.yml` under the `dispatchers:` section. Without this key, Solid Queue's dispatcher will not load the recurring schedule even though `config/recurring.yml` exists.
|
|
32
|
+
- **Manual fix (if generator cannot patch):**
|
|
33
|
+
```yaml
|
|
34
|
+
dispatchers:
|
|
35
|
+
- polling_interval: 1
|
|
36
|
+
batch_size: 500
|
|
37
|
+
recurring_schedule: config/recurring.yml
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
## 5. Jobs Not Processing with bin/dev
|
|
41
|
+
|
|
42
|
+
- **Symptoms:** `bin/dev` starts the web server but jobs never run. Running `bin/rails solid_queue:start` manually works fine.
|
|
43
|
+
- **Primary fix:** Re-run the install generator, which automatically patches `Procfile.dev`:
|
|
44
|
+
```bash
|
|
45
|
+
bin/rails generate source_monitor:install
|
|
46
|
+
```
|
|
47
|
+
- **Diagnostics:** Run `bin/source_monitor verify` to check Solid Queue worker status. The SolidQueueVerifier will suggest Procfile.dev if no workers are detected.
|
|
48
|
+
- **Manual check:** Verify `Procfile.dev` includes a `jobs:` line:
|
|
49
|
+
```
|
|
50
|
+
jobs: bundle exec rake solid_queue:start
|
|
51
|
+
```
|
|
52
|
+
- Most Rails 8 apps use foreman or overmind via `bin/dev`. Without a `jobs:` entry, the process manager only starts the web server and asset watchers -- Solid Queue workers are not launched.
|
|
53
|
+
|
|
54
|
+
## 6. Realtime Updates Do Not Stream
|
|
24
55
|
|
|
25
56
|
- Confirm Action Cable is mounted and `ApplicationCable` classes exist (see installation guide).
|
|
26
57
|
- In production, verify WebSocket proxy settings allow the `/cable` endpoint.
|
|
27
58
|
- When switching to Redis, add `config.realtime.adapter = :redis` and `config.realtime.redis_url` in the initializer, then restart web and worker processes.
|
|
28
59
|
- For Solid Cable, check that the `solid_cable_messages` table exists and that no other process clears it unexpectedly.
|
|
29
60
|
|
|
30
|
-
##
|
|
61
|
+
## 7. Fetch Jobs Keep Failing
|
|
31
62
|
|
|
32
63
|
- Review the most recent fetch log entry for the source; it stores the HTTP status, error class, and error message.
|
|
33
64
|
- Increase `config.http.timeout` or `config.http.retry_max` if the feed is slow or prone to transient errors.
|
|
34
65
|
- Supply custom headers or basic auth credentials via the source form when feeds require authentication.
|
|
35
66
|
- Check for TLS issues on self-signed feeds; you may need to configure Faraday with custom SSL options.
|
|
36
67
|
|
|
37
|
-
##
|
|
68
|
+
## 8. Scraping Returns "Failed"
|
|
38
69
|
|
|
39
70
|
- Confirm the source has scraping enabled and the configured adapter exists.
|
|
40
71
|
- Override selectors in the source's scrape settings if the default Readability extraction misses key elements.
|
|
41
72
|
- Inspect the scrape log to see the adapter status and content length. Logs store the HTTP status and any exception raised by the adapter.
|
|
42
73
|
- Retry manually from the item detail page after fixing selectors.
|
|
43
74
|
|
|
44
|
-
##
|
|
75
|
+
## 9. Cleanup Rake Tasks Fail
|
|
45
76
|
|
|
46
77
|
- Pass numeric values for `FETCH_LOG_DAYS` or `SCRAPE_LOG_DAYS` environment variables (e.g., `FETCH_LOG_DAYS=30`).
|
|
47
78
|
- Ensure workers or the console environment have permission to soft delete (`SOFT_DELETE=true`) if you expect tombstones.
|
|
48
79
|
- If job classes cannot load, verify `SourceMonitor.configure` ran before calling `rake source_monitor:cleanup:*`.
|
|
49
80
|
|
|
50
|
-
##
|
|
81
|
+
## 10. Test Suite Cannot Launch a Browser
|
|
51
82
|
|
|
52
83
|
- System tests rely on Selenium + Chrome. Install Chrome/Chromium and set `SELENIUM_CHROME_BINARY` if the binary lives in a non-standard path.
|
|
53
84
|
- You can run `rbenv exec bin/test-coverage --verbose` to inspect failures with additional logging.
|
|
54
85
|
|
|
55
|
-
##
|
|
86
|
+
## 11. Mission Control Jobs Link Returns 404
|
|
56
87
|
|
|
57
88
|
- Mount `MissionControl::Jobs::Engine` in your host routes (for example, `mount MissionControl::Jobs::Engine, at: "/mission_control"`).
|
|
58
89
|
- Keep `config.mission_control_enabled = true` **and** `config.mission_control_dashboard_path` pointing at that mounted route helper. Call `SourceMonitor.mission_control_dashboard_path` in the Rails console to confirm it resolves.
|
|
59
90
|
- When hosting Mission Control in a separate app, provide a full URL instead of a route helper and ensure CORS/WebSocket settings allow the dashboard iframe.
|
|
60
91
|
|
|
61
|
-
##
|
|
92
|
+
## 12. Tailwind Build Fails or Admin UI Loads Without Styles
|
|
62
93
|
|
|
63
94
|
- Running `test/dummy/bin/dev` before configuring the bundling pipeline will serve the admin UI without Tailwind styles or Stimulus behaviours. This happens because the engine no longer ships precompiled assets; see `.ai/engine-asset-configuration.md:11-44` for the required npm setup.
|
|
64
95
|
- Fix by running `npm install` followed by `npm run build` inside the engine root so that `app/assets/builds/source_monitor/application.css` and `application.js` exist. The Rake task `app:source_monitor:assets:build` wraps the same scripts for CI usage.
|
data/docs/upgrade.md
ADDED
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
# SourceMonitor Upgrade Guide
|
|
2
|
+
|
|
3
|
+
This guide covers upgrading SourceMonitor to a new gem version in your host Rails application.
|
|
4
|
+
|
|
5
|
+
## General Upgrade Steps
|
|
6
|
+
|
|
7
|
+
1. Review the [CHANGELOG](../CHANGELOG.md) for changes between your current and target versions
|
|
8
|
+
2. Update your Gemfile version constraint and run `bundle update source_monitor`
|
|
9
|
+
3. Run the upgrade command: `bin/source_monitor upgrade`
|
|
10
|
+
4. Apply database migrations if new ones were copied: `bin/rails db:migrate`
|
|
11
|
+
5. Address any deprecation warnings in your initializer (see Deprecation Handling below)
|
|
12
|
+
6. Run verification: `bin/source_monitor verify`
|
|
13
|
+
7. Restart your web server and background workers
|
|
14
|
+
|
|
15
|
+
## Quick Upgrade (Most Cases)
|
|
16
|
+
|
|
17
|
+
```bash
|
|
18
|
+
# 1. Update the gem
|
|
19
|
+
bundle update source_monitor
|
|
20
|
+
|
|
21
|
+
# 2. Run the upgrade command (handles migrations, generator, verification)
|
|
22
|
+
bin/source_monitor upgrade
|
|
23
|
+
|
|
24
|
+
# 3. Migrate if needed
|
|
25
|
+
bin/rails db:migrate
|
|
26
|
+
|
|
27
|
+
# 4. Restart
|
|
28
|
+
# (restart web server and Solid Queue workers)
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
## Deprecation Handling
|
|
32
|
+
|
|
33
|
+
When upgrading, you may see deprecation warnings in your Rails log:
|
|
34
|
+
|
|
35
|
+
```
|
|
36
|
+
[SourceMonitor] DEPRECATION: 'http.old_option' was deprecated in v0.5.0 and replaced by 'http.new_option'.
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
To resolve:
|
|
40
|
+
1. Open `config/initializers/source_monitor.rb`
|
|
41
|
+
2. Find the deprecated option (e.g., `config.http.old_option = value`)
|
|
42
|
+
3. Replace with the new option from the warning message (e.g., `config.http.new_option = value`)
|
|
43
|
+
4. Restart and verify the warning is gone
|
|
44
|
+
|
|
45
|
+
If a removed option raises an error (`SourceMonitor::DeprecatedOptionError`), you must update the initializer before the app can boot.
|
|
46
|
+
|
|
47
|
+
## Version-Specific Notes
|
|
48
|
+
|
|
49
|
+
### Upgrading to 0.4.0 (from 0.3.x)
|
|
50
|
+
|
|
51
|
+
**Released:** 2026-02-12
|
|
52
|
+
|
|
53
|
+
**What changed:**
|
|
54
|
+
- Install generator now auto-patches `Procfile.dev` with a Solid Queue `jobs:` entry
|
|
55
|
+
- Install generator now patches `config/queue.yml` dispatcher with `recurring_schedule: config/recurring.yml`
|
|
56
|
+
- Active Storage image download feature added (opt-in)
|
|
57
|
+
- SSL certificate configuration added to HTTP settings
|
|
58
|
+
- Enhanced verification messages for SolidQueue and RecurringSchedule verifiers
|
|
59
|
+
|
|
60
|
+
**Upgrade steps:**
|
|
61
|
+
```bash
|
|
62
|
+
bundle update source_monitor
|
|
63
|
+
bin/source_monitor upgrade
|
|
64
|
+
bin/rails db:migrate
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
**Notes:**
|
|
68
|
+
- No breaking changes. All existing configuration remains valid.
|
|
69
|
+
- Re-running the generator (`bin/rails generate source_monitor:install`) will add missing `Procfile.dev` and `queue.yml` entries without overwriting existing config.
|
|
70
|
+
- New optional features: `config.images.download_to_active_storage = true`, `config.http.ssl_ca_file`, `config.http.ssl_ca_path`, `config.http.ssl_verify`.
|
|
71
|
+
|
|
72
|
+
### Upgrading to 0.3.0 (from 0.2.x)
|
|
73
|
+
|
|
74
|
+
**Released:** 2026-02-10
|
|
75
|
+
|
|
76
|
+
**What changed:**
|
|
77
|
+
- Internal refactoring: FeedFetcher, Configuration, ImportSessionsController, and ItemCreator extracted into smaller modules
|
|
78
|
+
- Eager requires replaced with Ruby autoload
|
|
79
|
+
- Skills system added (14 `sm-*` Claude Code skills)
|
|
80
|
+
|
|
81
|
+
**Upgrade steps:**
|
|
82
|
+
```bash
|
|
83
|
+
bundle update source_monitor
|
|
84
|
+
bin/source_monitor upgrade
|
|
85
|
+
bin/rails db:migrate
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
**Notes:**
|
|
89
|
+
- No breaking changes to the public API.
|
|
90
|
+
- If you referenced internal classes directly (e.g., `SourceMonitor::FeedFetcher` internals), verify your code against the new module structure.
|
|
91
|
+
- Optionally install AI skills: `bin/rails source_monitor:skills:install`
|
|
92
|
+
|
|
93
|
+
### Upgrading to 0.2.0 (from 0.1.x)
|
|
94
|
+
|
|
95
|
+
**Released:** 2025-11-25
|
|
96
|
+
|
|
97
|
+
**What changed:**
|
|
98
|
+
- OPML import wizard with multi-step flow
|
|
99
|
+
- New `ImportHistory` model and associated migrations
|
|
100
|
+
|
|
101
|
+
**Upgrade steps:**
|
|
102
|
+
```bash
|
|
103
|
+
bundle update source_monitor
|
|
104
|
+
bin/rails railties:install:migrations FROM=source_monitor
|
|
105
|
+
bin/rails db:migrate
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
**Notes:**
|
|
109
|
+
- New database tables required. Run migrations after updating.
|
|
110
|
+
- No configuration changes needed.
|
|
111
|
+
|
|
112
|
+
## Troubleshooting
|
|
113
|
+
|
|
114
|
+
### "Already up to date" but I expected changes
|
|
115
|
+
- Verify the gem version actually changed: `bundle show source_monitor`
|
|
116
|
+
- Check `Gemfile.lock` for the resolved version
|
|
117
|
+
- If the `.source_monitor_version` marker was manually edited, delete it and re-run upgrade
|
|
118
|
+
|
|
119
|
+
### Migrations fail with duplicate timestamps
|
|
120
|
+
- Remove the duplicate migration file from `db/migrate/` (keep the newer one)
|
|
121
|
+
- Re-run `bin/rails db:migrate`
|
|
122
|
+
|
|
123
|
+
### Deprecation error prevents boot
|
|
124
|
+
- Read the error message for the replacement option
|
|
125
|
+
- Update your initializer before restarting
|
|
126
|
+
- If unsure which option to use, consult [Configuration Reference](configuration.md)
|
|
127
|
+
|
|
128
|
+
### Verification failures after upgrade
|
|
129
|
+
- **PendingMigrations:** Run `bin/rails db:migrate`
|
|
130
|
+
- **SolidQueue:** Ensure workers are running. Check `Procfile.dev` for a `jobs:` entry.
|
|
131
|
+
- **RecurringSchedule:** Re-run `bin/rails generate source_monitor:install` to patch `config/queue.yml`
|
|
132
|
+
- **ActionCable:** Configure Solid Cable or Redis adapter
|
|
133
|
+
|
|
134
|
+
For additional help, see [Troubleshooting](troubleshooting.md).
|
|
135
|
+
|
|
136
|
+
## See Also
|
|
137
|
+
- [Setup Guide](setup.md) -- Initial installation
|
|
138
|
+
- [Configuration Reference](configuration.md) -- All configuration options
|
|
139
|
+
- [Troubleshooting](troubleshooting.md) -- Common issues and fixes
|
|
140
|
+
- [CHANGELOG](../CHANGELOG.md) -- Full version history
|