source_monitor 0.3.3 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (91) hide show
  1. checksums.yaml +4 -4
  2. data/.claude/commands/release.md +101 -58
  3. data/.claude/skills/sm-configure/SKILL.md +13 -2
  4. data/.claude/skills/sm-configure/reference/configuration-reference.md +33 -0
  5. data/.claude/skills/sm-host-setup/SKILL.md +18 -2
  6. data/.claude/skills/sm-host-setup/reference/setup-checklist.md +33 -0
  7. data/.claude/skills/sm-job/SKILL.md +1 -1
  8. data/.claude/skills/sm-upgrade/SKILL.md +102 -0
  9. data/.claude/skills/sm-upgrade/reference/upgrade-workflow.md +92 -0
  10. data/.claude/skills/sm-upgrade/reference/version-history.md +68 -0
  11. data/.vbw-planning/SHIPPED.md +35 -0
  12. data/.vbw-planning/config.json +24 -1
  13. data/.vbw-planning/discovery.json +3 -1
  14. data/.vbw-planning/{REQUIREMENTS.md → milestones/generator-enhancements/REQUIREMENTS.md} +22 -0
  15. data/.vbw-planning/milestones/generator-enhancements/ROADMAP.md +125 -0
  16. data/.vbw-planning/milestones/generator-enhancements/SHIPPED.md +40 -0
  17. data/.vbw-planning/milestones/generator-enhancements/STATE.md +43 -0
  18. data/.vbw-planning/milestones/generator-enhancements/phases/01-generator-steps/01-CONTEXT.md +33 -0
  19. data/.vbw-planning/milestones/generator-enhancements/phases/01-generator-steps/01-VERIFICATION.md +86 -0
  20. data/.vbw-planning/milestones/generator-enhancements/phases/01-generator-steps/PLAN-01-SUMMARY.md +61 -0
  21. data/.vbw-planning/milestones/generator-enhancements/phases/01-generator-steps/PLAN-01.md +380 -0
  22. data/.vbw-planning/milestones/generator-enhancements/phases/02-verification/02-VERIFICATION.md +78 -0
  23. data/.vbw-planning/milestones/generator-enhancements/phases/02-verification/PLAN-01-SUMMARY.md +46 -0
  24. data/.vbw-planning/milestones/generator-enhancements/phases/02-verification/PLAN-01.md +500 -0
  25. data/.vbw-planning/milestones/generator-enhancements/phases/03-docs-alignment/03-VERIFICATION.md +89 -0
  26. data/.vbw-planning/milestones/generator-enhancements/phases/03-docs-alignment/PLAN-01-SUMMARY.md +48 -0
  27. data/.vbw-planning/milestones/generator-enhancements/phases/03-docs-alignment/PLAN-01.md +456 -0
  28. data/.vbw-planning/milestones/generator-enhancements/phases/04-dashboard-ux/04-VERIFICATION.md +129 -0
  29. data/.vbw-planning/milestones/generator-enhancements/phases/04-dashboard-ux/PLAN-01-SUMMARY.md +70 -0
  30. data/.vbw-planning/milestones/generator-enhancements/phases/04-dashboard-ux/PLAN-01.md +747 -0
  31. data/.vbw-planning/milestones/generator-enhancements/phases/05-active-storage-images/05-VERIFICATION.md +156 -0
  32. data/.vbw-planning/milestones/generator-enhancements/phases/05-active-storage-images/PLAN-01-SUMMARY.md +69 -0
  33. data/.vbw-planning/milestones/generator-enhancements/phases/05-active-storage-images/PLAN-01.md +455 -0
  34. data/.vbw-planning/milestones/generator-enhancements/phases/05-active-storage-images/PLAN-02-SUMMARY.md +39 -0
  35. data/.vbw-planning/milestones/generator-enhancements/phases/05-active-storage-images/PLAN-02.md +488 -0
  36. data/.vbw-planning/milestones/generator-enhancements/phases/06-netflix-feed-fix/06-VERIFICATION.md +100 -0
  37. data/.vbw-planning/milestones/generator-enhancements/phases/06-netflix-feed-fix/PLAN-01-SUMMARY.md +37 -0
  38. data/.vbw-planning/milestones/generator-enhancements/phases/06-netflix-feed-fix/PLAN-01.md +345 -0
  39. data/.vbw-planning/milestones/upgrade-assurance/REQUIREMENTS.md +80 -0
  40. data/.vbw-planning/milestones/upgrade-assurance/ROADMAP.md +75 -0
  41. data/.vbw-planning/milestones/upgrade-assurance/STATE.md +29 -0
  42. data/.vbw-planning/milestones/upgrade-assurance/phases/01-upgrade-command/01-VERIFICATION.md +144 -0
  43. data/.vbw-planning/milestones/upgrade-assurance/phases/01-upgrade-command/PLAN-01-SUMMARY.md +43 -0
  44. data/.vbw-planning/milestones/upgrade-assurance/phases/01-upgrade-command/PLAN-01.md +405 -0
  45. data/.vbw-planning/milestones/upgrade-assurance/phases/02-config-deprecation/PLAN-01-SUMMARY.md +27 -0
  46. data/.vbw-planning/milestones/upgrade-assurance/phases/02-config-deprecation/PLAN-01.md +303 -0
  47. data/.vbw-planning/milestones/upgrade-assurance/phases/03-upgrade-skill-docs/03-VERIFICATION.md +380 -0
  48. data/.vbw-planning/milestones/upgrade-assurance/phases/03-upgrade-skill-docs/PLAN-01-SUMMARY.md +36 -0
  49. data/.vbw-planning/milestones/upgrade-assurance/phases/03-upgrade-skill-docs/PLAN-01.md +652 -0
  50. data/CHANGELOG.md +48 -0
  51. data/CLAUDE.md +5 -3
  52. data/Gemfile.lock +1 -1
  53. data/VERSION +1 -1
  54. data/app/assets/builds/source_monitor/application.css +9 -0
  55. data/app/helpers/source_monitor/application_helper.rb +38 -0
  56. data/app/jobs/source_monitor/download_content_images_job.rb +72 -0
  57. data/app/models/source_monitor/item_content.rb +2 -0
  58. data/app/views/source_monitor/dashboard/_recent_activity.html.erb +9 -0
  59. data/app/views/source_monitor/items/_details.html.erb +2 -2
  60. data/app/views/source_monitor/logs/index.html.erb +9 -0
  61. data/app/views/source_monitor/sources/_details.html.erb +2 -2
  62. data/app/views/source_monitor/sources/_row.html.erb +1 -1
  63. data/docs/setup.md +10 -1
  64. data/docs/troubleshooting.md +38 -7
  65. data/docs/upgrade.md +140 -0
  66. data/lib/generators/source_monitor/install/install_generator.rb +101 -0
  67. data/lib/source_monitor/configuration/deprecation_registry.rb +237 -0
  68. data/lib/source_monitor/configuration/http_settings.rb +7 -1
  69. data/lib/source_monitor/configuration/images_settings.rb +37 -0
  70. data/lib/source_monitor/configuration.rb +11 -1
  71. data/lib/source_monitor/dashboard/queries/recent_activity_query.rb +16 -7
  72. data/lib/source_monitor/dashboard/recent_activity.rb +1 -0
  73. data/lib/source_monitor/dashboard/recent_activity_presenter.rb +15 -2
  74. data/lib/source_monitor/fetching/feed_fetcher/entry_processor.rb +13 -0
  75. data/lib/source_monitor/http.rb +23 -0
  76. data/lib/source_monitor/images/content_rewriter.rb +81 -0
  77. data/lib/source_monitor/images/downloader.rb +82 -0
  78. data/lib/source_monitor/logs/table_presenter.rb +25 -0
  79. data/lib/source_monitor/setup/cli.rb +7 -0
  80. data/lib/source_monitor/setup/procfile_patcher.rb +31 -0
  81. data/lib/source_monitor/setup/queue_config_patcher.rb +84 -0
  82. data/lib/source_monitor/setup/skills_installer.rb +1 -0
  83. data/lib/source_monitor/setup/upgrade_command.rb +59 -0
  84. data/lib/source_monitor/setup/verification/pending_migrations_verifier.rb +92 -0
  85. data/lib/source_monitor/setup/verification/recurring_schedule_verifier.rb +102 -0
  86. data/lib/source_monitor/setup/verification/runner.rb +1 -1
  87. data/lib/source_monitor/setup/verification/solid_queue_verifier.rb +1 -1
  88. data/lib/source_monitor/setup/workflow.rb +10 -0
  89. data/lib/source_monitor/version.rb +1 -1
  90. data/lib/source_monitor.rb +11 -0
  91. metadata +51 -2
data/CHANGELOG.md CHANGED
@@ -15,6 +15,54 @@ All notable changes to this project are documented below. The format follows [Ke
15
15
 
16
16
  - No unreleased changes yet.
17
17
 
18
+ ## [0.5.0] - 2026-02-13
19
+
20
+ ### Added
21
+
22
+ - `bin/source_monitor upgrade` command: detects version changes since last install, copies new migrations, re-runs the generator, runs verification, and reports what changed. Uses a `.source_monitor_version` marker file for version tracking.
23
+ - `PendingMigrationsVerifier` checks for unmigrated SourceMonitor tables in the verification suite, integrated into both `bin/source_monitor verify` and the upgrade flow.
24
+ - Configuration deprecation framework: engine developers can register deprecated config options with `DeprecationRegistry.register`. At boot time, stale options trigger `:warning` (renamed) or `:error` (removed) messages with actionable replacement paths.
25
+ - `sm-upgrade` AI skill guides agents through post-update workflows: CHANGELOG parsing, running the upgrade command, interpreting verification results, and handling deprecation warnings.
26
+ - `docs/upgrade.md` versioned upgrade guide with general steps, version-specific notes (0.1.x through 0.4.x), and troubleshooting.
27
+ - `sm-host-setup` skill cross-references the upgrade workflow.
28
+
29
+ ### Testing
30
+
31
+ - 1,003 tests, 0 failures (up from 973 in 0.4.0).
32
+ - RuboCop: 397 files, 0 offenses.
33
+ - Brakeman: 0 warnings.
34
+
35
+ ## [0.4.0] - 2026-02-12
36
+
37
+ ### Added
38
+
39
+ - Install generator now auto-patches `Procfile.dev` with Solid Queue `jobs:` entry and `queue.yml` with `recurring_schedule` dispatcher wiring (idempotent, skip if already present).
40
+ - `RecurringScheduleVerifier` checks that recurring tasks are registered with Solid Queue dispatchers; `SolidQueueVerifier` remediation now mentions `Procfile.dev` for `bin/dev` users.
41
+ - Dashboard fetch log entries display source URL (domain for RSS, item URL for scrapes) alongside existing summary.
42
+ - External links across dashboard, logs, sources, and items open in new tab with visual indicator icon.
43
+ - Configurable Active Storage image downloads: `config.images.download_to_active_storage` (default `false`) detects inline images in feed content, downloads them via background job, and rewrites `<img>` src attributes with Active Storage URLs.
44
+ - `Images::ContentRewriter` extracts and rewrites image URLs from HTML content using Nokolexbor.
45
+ - `Images::Downloader` service validates content type and size before downloading images.
46
+ - `DownloadContentImagesJob` orchestrates the download/attach/rewrite pipeline per item.
47
+ - SSL certificate store configuration: every Faraday connection gets an `OpenSSL::X509::Store` initialized with `set_default_paths`, resolving "unable to get local issuer certificate" errors on systems with incomplete CA bundles.
48
+ - Configurable SSL options in `HTTPSettings`: `ssl_ca_file`, `ssl_ca_path`, `ssl_verify` for non-standard certificate environments.
49
+ - Netflix Tech Blog VCR cassette regression test proving Medium-hosted RSS feeds parse correctly with the SSL fix.
50
+
51
+ ### Fixed
52
+
53
+ - SSL certificate verification failures for feeds hosted on services requiring intermediate CAs (e.g., Netflix Tech Blog via Medium/AWS).
54
+ - Setup documentation now includes `Procfile.dev` and `recurring_schedule` guidance.
55
+
56
+ ### Changed
57
+
58
+ - Updated `sm-host-setup`, `sm-configure`, and setup documentation to reflect that the generator handles Procfile.dev and recurring_schedule automatically.
59
+
60
+ ### Testing
61
+
62
+ - 973 tests, 3,114 assertions, 0 failures (up from 841 tests in 0.3.3).
63
+ - RuboCop: 389 files, 0 offenses.
64
+ - Brakeman: 0 warnings.
65
+
18
66
  ## [0.3.3] - 2026-02-11
19
67
 
20
68
  ### Fixed
data/CLAUDE.md CHANGED
@@ -4,9 +4,10 @@
4
4
 
5
5
  ## Active Context
6
6
 
7
- **Milestone:** none (archived)
8
- **Last shipped:** default (2026-02-10) -- 4 phases, 14 plans, 841 tests
9
- **Next action:** /vbw:plan to start new milestone
7
+ **Milestone:** (none active)
8
+ **Last shipped:** upgrade-assurance (2026-02-13) -- 3 phases, 14 tasks, 12 commits
9
+ **Previous:** generator-enhancements (2026-02-12) -- v0.4.0
10
+ **Next action:** /vbw:vibe to start a new milestone
10
11
 
11
12
  ## Key Decisions
12
13
 
@@ -192,6 +193,7 @@ Engine-specific skills (`sm-*` prefix). Consumer skills install by default; cont
192
193
  | `sm-event-handler` | Lifecycle callbacks (after_item_created, etc.) |
193
194
  | `sm-model-extension` | Extend engine models from host app |
194
195
  | `sm-dashboard-widget` | Dashboard queries, presenters, Turbo broadcasts |
196
+ | `sm-upgrade` | Gem upgrade workflow with CHANGELOG parsing |
195
197
 
196
198
  ### Contributor Skills (opt-in)
197
199
 
data/Gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- source_monitor (0.3.3)
4
+ source_monitor (0.5.0)
5
5
  cssbundling-rails (~> 1.4)
6
6
  faraday (~> 2.9)
7
7
  faraday-follow_redirects (~> 0.4)
data/VERSION CHANGED
@@ -1 +1 @@
1
- 0.3.3
1
+ 0.5.0
@@ -705,6 +705,10 @@ video {
705
705
  margin-right: 0.25rem;
706
706
  }
707
707
 
708
+ .fm-admin .mt-0\.5 {
709
+ margin-top: 0.125rem;
710
+ }
711
+
708
712
  .fm-admin .mt-1 {
709
713
  margin-top: 0.25rem;
710
714
  }
@@ -1789,6 +1793,11 @@ video {
1789
1793
  --tw-ring-color: transparent;
1790
1794
  }
1791
1795
 
1796
+ .fm-admin .invert {
1797
+ --tw-invert: invert(100%);
1798
+ filter: var(--tw-blur) var(--tw-brightness) var(--tw-contrast) var(--tw-grayscale) var(--tw-hue-rotate) var(--tw-invert) var(--tw-saturate) var(--tw-sepia) var(--tw-drop-shadow);
1799
+ }
1800
+
1792
1801
  .fm-admin .filter {
1793
1802
  filter: var(--tw-blur) var(--tw-brightness) var(--tw-contrast) var(--tw-grayscale) var(--tw-hue-rotate) var(--tw-invert) var(--tw-saturate) var(--tw-sepia) var(--tw-drop-shadow);
1794
1803
  }
@@ -212,8 +212,46 @@ module SourceMonitor
212
212
  end
213
213
  end
214
214
 
215
+ # Renders a clickable link that opens in a new tab with an external-link icon.
216
+ # Returns the label as plain text if the URL is blank.
217
+ def external_link_to(label, url, **options)
218
+ return label if url.blank?
219
+
220
+ css = options.delete(:class) || "text-blue-600 hover:text-blue-500"
221
+ link_to(url, target: "_blank", rel: "noopener noreferrer", class: css, title: url, **options) do
222
+ safe_join([ label, " ", external_link_icon ])
223
+ end
224
+ end
225
+
226
+ # Extracts the domain from a URL, returning nil if parsing fails.
227
+ def domain_from_url(url)
228
+ return nil if url.blank?
229
+
230
+ URI.parse(url.to_s).host
231
+ rescue URI::InvalidURIError
232
+ nil
233
+ end
234
+
215
235
  private
216
236
 
237
+ def external_link_icon
238
+ tag.svg(
239
+ class: "inline-block h-3 w-3 text-slate-400",
240
+ xmlns: "http://www.w3.org/2000/svg",
241
+ fill: "none",
242
+ viewBox: "0 0 24 24",
243
+ stroke_width: "2",
244
+ stroke: "currentColor",
245
+ aria: { hidden: "true" }
246
+ ) do
247
+ tag.path(
248
+ stroke_linecap: "round",
249
+ stroke_linejoin: "round",
250
+ d: "M13.5 6H5.25A2.25 2.25 0 003 8.25v10.5A2.25 2.25 0 005.25 21h10.5A2.25 2.25 0 0018 18.75V10.5m-10.5 6L21 3m0 0h-5.25M21 3v5.25"
251
+ )
252
+ end
253
+ end
254
+
217
255
  def derive_item_scrape_status(item:, source: nil)
218
256
  return "idle" unless item
219
257
 
@@ -0,0 +1,72 @@
1
+ # frozen_string_literal: true
2
+
3
+ module SourceMonitor
4
+ class DownloadContentImagesJob < ApplicationJob
5
+ source_monitor_queue :fetch
6
+
7
+ discard_on ActiveJob::DeserializationError
8
+
9
+ def perform(item_id)
10
+ item = SourceMonitor::Item.find_by(id: item_id)
11
+ return unless item
12
+ return unless SourceMonitor.config.images.download_enabled?
13
+
14
+ html = item.content
15
+ return if html.blank?
16
+
17
+ # Build or find item_content for attachment storage
18
+ item_content = item.item_content || item.build_item_content
19
+
20
+ # Skip if images already attached (idempotency)
21
+ return if item_content.persisted? && item_content.images.attached?
22
+
23
+ base_url = item.url
24
+ rewriter = SourceMonitor::Images::ContentRewriter.new(html, base_url: base_url)
25
+ image_urls = rewriter.image_urls
26
+ return if image_urls.empty?
27
+
28
+ # Save item_content first so we can attach blobs to it
29
+ item_content.save! unless item_content.persisted?
30
+
31
+ # Download images and build URL mapping
32
+ url_mapping = download_images(item_content, image_urls)
33
+ return if url_mapping.empty?
34
+
35
+ # Rewrite HTML with Active Storage URLs
36
+ rewritten_html = rewriter.rewrite do |original_url|
37
+ url_mapping[original_url]
38
+ end
39
+
40
+ # Update the item content with rewritten HTML
41
+ item.update!(content: rewritten_html)
42
+ end
43
+
44
+ private
45
+
46
+ def download_images(item_content, image_urls)
47
+ url_mapping = {}
48
+ settings = SourceMonitor.config.images
49
+
50
+ image_urls.each do |image_url|
51
+ result = SourceMonitor::Images::Downloader.new(image_url, settings: settings).call
52
+ next unless result
53
+
54
+ blob = ActiveStorage::Blob.create_and_upload!(
55
+ io: result.io,
56
+ filename: result.filename,
57
+ content_type: result.content_type
58
+ )
59
+ item_content.images.attach(blob)
60
+
61
+ # Generate a serving URL for the blob
62
+ url_mapping[image_url] = Rails.application.routes.url_helpers.rails_blob_path(blob, only_path: true)
63
+ rescue StandardError
64
+ # Individual image failure should not block others.
65
+ # Original URL will be preserved (graceful fallback).
66
+ next
67
+ end
68
+
69
+ url_mapping
70
+ end
71
+ end
72
+ end
@@ -6,6 +6,8 @@ module SourceMonitor
6
6
 
7
7
  validates :item, presence: true
8
8
 
9
+ has_many_attached :images if defined?(ActiveStorage)
10
+
9
11
  SourceMonitor::ModelExtensions.register(self, :item_content)
10
12
  end
11
13
  end
@@ -21,6 +21,15 @@
21
21
  <div class="mt-1 text-xs text-slate-500">
22
22
  <%= event[:description].presence || "No additional details recorded." %>
23
23
  </div>
24
+ <% if event[:url_display].present? %>
25
+ <div class="mt-0.5 text-xs text-slate-400 truncate max-w-sm" data-testid="event-url-display">
26
+ <% if event[:url_href].present? %>
27
+ <%= external_link_to event[:url_display], event[:url_href], class: "text-slate-400 hover:text-blue-500" %>
28
+ <% else %>
29
+ <%= event[:url_display] %>
30
+ <% end %>
31
+ </div>
32
+ <% end %>
24
33
  </div>
25
34
  <div class="text-right">
26
35
  <span class="inline-flex items-center rounded-full px-3 py-1 text-xs font-semibold <%= event[:status] == :success ? "bg-green-100 text-green-700" : "bg-rose-100 text-rose-700" %>">
@@ -53,8 +53,8 @@
53
53
  <% details = {
54
54
  "GUID" => item.guid,
55
55
  "Content Fingerprint" => item.content_fingerprint || "—",
56
- "URL" => item.url,
57
- "Canonical URL" => item.canonical_url || "",
56
+ "URL" => (item.url.present? ? external_link_to(item.url, item.url, class: "text-slate-900 hover:text-blue-500") : "\u2014"),
57
+ "Canonical URL" => (item.canonical_url.present? ? external_link_to(item.canonical_url, item.canonical_url, class: "text-slate-900 hover:text-blue-500") : "\u2014"),
58
58
  "Author" => item.author || "—",
59
59
  "Published At" => (item.published_at&.strftime("%b %d, %Y %H:%M %Z") || "—"),
60
60
  "Updated At (Source)" => (item.updated_at_source&.strftime("%b %d, %Y %H:%M %Z") || "—"),
@@ -134,6 +134,15 @@
134
134
  <% else %>
135
135
  <%= row.primary_label %>
136
136
  <% end %>
137
+ <% if row.url_label.present? %>
138
+ <div class="mt-0.5 text-xs text-slate-400 truncate max-w-xs">
139
+ <% if row.url_href.present? %>
140
+ <%= external_link_to row.url_label, row.url_href, class: "text-slate-400 hover:text-blue-500" %>
141
+ <% else %>
142
+ <%= row.url_label %>
143
+ <% end %>
144
+ </div>
145
+ <% end %>
137
146
  </td>
138
147
  <td class="px-6 py-4 text-sm">
139
148
  <% if row.source_path %>
@@ -25,7 +25,7 @@
25
25
  </span>
26
26
  <% end %>
27
27
  </div>
28
- <p class="mt-2 text-sm text-slate-500">Feed URL: <%= source.feed_url %></p>
28
+ <p class="mt-2 text-sm text-slate-500">Feed URL: <%= external_link_to source.feed_url, source.feed_url, class: "text-slate-500 hover:text-blue-500" %></p>
29
29
  </div>
30
30
  <div class="flex flex-wrap items-center justify-end gap-3">
31
31
  <% fetch_disabled = %w[queued fetching].include?(source.fetch_status) %>
@@ -137,7 +137,7 @@
137
137
  end
138
138
 
139
139
  details = {
140
- "Website" => (source.website_url.presence || ""),
140
+ "Website" => (source.website_url.present? ? external_link_to(source.website_url, source.website_url, class: "text-slate-900 hover:text-blue-500") : "\u2014"),
141
141
  "Fetch interval" => "#{source.fetch_interval_minutes} minutes (~#{interval_hours} hours)",
142
142
  "Adaptive interval" => source.adaptive_fetching_enabled? ? "Auto" : "Fixed",
143
143
  "Scraper" => source.scraper_adapter,
@@ -29,7 +29,7 @@
29
29
  class: "text-slate-900 hover:text-blue-600 focus:outline-none focus:ring-2 focus:ring-blue-500 focus:ring-offset-2",
30
30
  data: { turbo_frame: "_top" } %>
31
31
  </div>
32
- <div class="text-xs text-slate-500 truncate max-w-xs"><%= source.feed_url %></div>
32
+ <div class="text-xs text-slate-500 truncate max-w-xs"><%= external_link_to source.feed_url, source.feed_url, class: "text-slate-500 hover:text-blue-500" %></div>
33
33
  </td>
34
34
  <td class="px-6 py-4">
35
35
  <div class="flex flex-col gap-2 text-xs">
data/docs/setup.md CHANGED
@@ -49,7 +49,12 @@ This ensures Bundler can load SourceMonitor so the commands below are available.
49
49
  ```bash
50
50
  bin/rails solid_queue:start
51
51
  ```
52
- Recurring jobs (fetch scheduling, scraping, cleanup) are automatically configured in `config/recurring.yml` by the install generator. They'll run automatically with `bin/dev` or `bin/jobs`.
52
+ The install generator automatically handles all worker configuration:
53
+ - **Recurring jobs** are configured in `config/recurring.yml` (fetch scheduling, scraping, cleanup).
54
+ - **Procfile.dev** is patched with a `jobs:` entry so `bin/dev` starts Solid Queue alongside the web server.
55
+ - **Queue dispatcher** is patched with `recurring_schedule: config/recurring.yml` in `config/queue.yml` so recurring jobs load on startup.
56
+
57
+ All three steps are idempotent. If any configuration is missing, re-run: `bin/rails generate source_monitor:install`
53
58
 
54
59
  4. **Visit the dashboard** at the chosen mount path, create a source, and trigger “Fetch Now” to validate realtime updates and Solid Queue processing.
55
60
 
@@ -87,6 +92,8 @@ Prefer to script each step or plug SourceMonitor into an existing deployment che
87
92
  | 4 | `bin/rails railties:install:migrations FROM=source_monitor` | Copy engine migrations (idempotent) |
88
93
  | 5 | `bin/rails db:migrate` | Apply schema updates, including Solid Queue tables |
89
94
  | 6 | `bin/rails solid_queue:start` | Ensure jobs process via Solid Queue |
95
+ | 6a | Handled by generator (patches `Procfile.dev`) | Ensure `bin/dev` starts Solid Queue workers |
96
+ | 6b | Handled by generator (patches `config/queue.yml`) | Wire recurring jobs into Solid Queue dispatcher |
90
97
  | 7 | `bin/jobs --recurring_schedule_file=config/recurring.yml` | Start recurring scheduler (optional but recommended) |
91
98
  | 8 | `bin/source_monitor verify` | Confirm Solid Queue/Action Cable readiness and emit telemetry |
92
99
 
@@ -100,6 +107,8 @@ Prefer to script each step or plug SourceMonitor into an existing deployment che
100
107
  4. **Apply database changes** using `bin/rails db:migrate`. If your host already installed Solid Queue migrations manually, delete duplicate files before migrating.
101
108
  5. **Wire Action Cable** if necessary. SourceMonitor defaults to Solid Cable; confirm `ApplicationCable::Connection`/`Channel` exist and that `config/initializers/source_monitor.rb` uses the adapter you expect. To switch to Redis, set `config.realtime.adapter = :redis` and `config.realtime.redis_url`.
102
109
  6. **Start workers** with `bin/rails solid_queue:start` (or your process manager). The install generator automatically configures recurring jobs in `config/recurring.yml` for fetch scheduling, scraping, and cleanup. They'll run with `bin/dev` or `bin/jobs`.
110
+ - **Procfile.dev:** The generator automatically patches `Procfile.dev` with a `jobs:` entry for Solid Queue. Verify the file contains `jobs: bundle exec rake solid_queue:start` after running the generator.
111
+ - **Recurring schedule:** The generator automatically patches `config/queue.yml` dispatchers with `recurring_schedule: config/recurring.yml`. Verify the key is present after running the generator.
103
112
  7. **Review the initializer** and tune queue names, HTTP timeouts, scraping adapters, retention limits, authentication hooks, and Mission Control integration. The [configuration reference](configuration.md) details every option.
104
113
  8. **Verify the install**: run `bin/source_monitor verify` to ensure Solid Queue workers and Action Cable are healthy, then visit the mount path to trigger a fetch manually. Enable telemetry if you want JSON logs recorded for support.
105
114
 
@@ -20,45 +20,76 @@ This guide lists common issues you might encounter while installing, upgrading,
20
20
  - Ensure at least one Solid Queue worker is running; the dashboard reads visibility data via `SourceMonitor::Jobs::Visibility`.
21
21
  - When using mission control integration, keep `config.mission_control_dashboard_path` pointing at a valid route helper; otherwise the dashboard hides the link.
22
22
 
23
- ## 4. Realtime Updates Do Not Stream
23
+ ## 4. Recurring Jobs Not Running
24
+
25
+ - **Symptoms:** Fetch scheduling, scrape scheduling, and cleanup jobs never fire. Sources never auto-fetch on their configured intervals.
26
+ - **Primary fix:** Re-run the install generator, which automatically patches the dispatcher config:
27
+ ```bash
28
+ bin/rails generate source_monitor:install
29
+ ```
30
+ - **Diagnostics:** Run `bin/source_monitor verify` to check recurring task registration. The RecurringScheduleVerifier will report whether SourceMonitor recurring tasks are loaded into Solid Queue.
31
+ - **Manual check:** Verify `config/queue.yml` includes `recurring_schedule: config/recurring.yml` under the `dispatchers:` section. Without this key, Solid Queue's dispatcher will not load the recurring schedule even though `config/recurring.yml` exists.
32
+ - **Manual fix (if generator cannot patch):**
33
+ ```yaml
34
+ dispatchers:
35
+ - polling_interval: 1
36
+ batch_size: 500
37
+ recurring_schedule: config/recurring.yml
38
+ ```
39
+
40
+ ## 5. Jobs Not Processing with bin/dev
41
+
42
+ - **Symptoms:** `bin/dev` starts the web server but jobs never run. Running `bin/rails solid_queue:start` manually works fine.
43
+ - **Primary fix:** Re-run the install generator, which automatically patches `Procfile.dev`:
44
+ ```bash
45
+ bin/rails generate source_monitor:install
46
+ ```
47
+ - **Diagnostics:** Run `bin/source_monitor verify` to check Solid Queue worker status. The SolidQueueVerifier will suggest Procfile.dev if no workers are detected.
48
+ - **Manual check:** Verify `Procfile.dev` includes a `jobs:` line:
49
+ ```
50
+ jobs: bundle exec rake solid_queue:start
51
+ ```
52
+ - Most Rails 8 apps use foreman or overmind via `bin/dev`. Without a `jobs:` entry, the process manager only starts the web server and asset watchers -- Solid Queue workers are not launched.
53
+
54
+ ## 6. Realtime Updates Do Not Stream
24
55
 
25
56
  - Confirm Action Cable is mounted and `ApplicationCable` classes exist (see installation guide).
26
57
  - In production, verify WebSocket proxy settings allow the `/cable` endpoint.
27
58
  - When switching to Redis, add `config.realtime.adapter = :redis` and `config.realtime.redis_url` in the initializer, then restart web and worker processes.
28
59
  - For Solid Cable, check that the `solid_cable_messages` table exists and that no other process clears it unexpectedly.
29
60
 
30
- ## 5. Fetch Jobs Keep Failing
61
+ ## 7. Fetch Jobs Keep Failing
31
62
 
32
63
  - Review the most recent fetch log entry for the source; it stores the HTTP status, error class, and error message.
33
64
  - Increase `config.http.timeout` or `config.http.retry_max` if the feed is slow or prone to transient errors.
34
65
  - Supply custom headers or basic auth credentials via the source form when feeds require authentication.
35
66
  - Check for TLS issues on self-signed feeds; you may need to configure Faraday with custom SSL options.
36
67
 
37
- ## 6. Scraping Returns "Failed"
68
+ ## 8. Scraping Returns "Failed"
38
69
 
39
70
  - Confirm the source has scraping enabled and the configured adapter exists.
40
71
  - Override selectors in the source's scrape settings if the default Readability extraction misses key elements.
41
72
  - Inspect the scrape log to see the adapter status and content length. Logs store the HTTP status and any exception raised by the adapter.
42
73
  - Retry manually from the item detail page after fixing selectors.
43
74
 
44
- ## 7. Cleanup Rake Tasks Fail
75
+ ## 9. Cleanup Rake Tasks Fail
45
76
 
46
77
  - Pass numeric values for `FETCH_LOG_DAYS` or `SCRAPE_LOG_DAYS` environment variables (e.g., `FETCH_LOG_DAYS=30`).
47
78
  - Ensure workers or the console environment have permission to soft delete (`SOFT_DELETE=true`) if you expect tombstones.
48
79
  - If job classes cannot load, verify `SourceMonitor.configure` ran before calling `rake source_monitor:cleanup:*`.
49
80
 
50
- ## 8. Test Suite Cannot Launch a Browser
81
+ ## 10. Test Suite Cannot Launch a Browser
51
82
 
52
83
  - System tests rely on Selenium + Chrome. Install Chrome/Chromium and set `SELENIUM_CHROME_BINARY` if the binary lives in a non-standard path.
53
84
  - You can run `rbenv exec bin/test-coverage --verbose` to inspect failures with additional logging.
54
85
 
55
- ## 9. Mission Control Jobs Link Returns 404
86
+ ## 11. Mission Control Jobs Link Returns 404
56
87
 
57
88
  - Mount `MissionControl::Jobs::Engine` in your host routes (for example, `mount MissionControl::Jobs::Engine, at: "/mission_control"`).
58
89
  - Keep `config.mission_control_enabled = true` **and** `config.mission_control_dashboard_path` pointing at that mounted route helper. Call `SourceMonitor.mission_control_dashboard_path` in the Rails console to confirm it resolves.
59
90
  - When hosting Mission Control in a separate app, provide a full URL instead of a route helper and ensure CORS/WebSocket settings allow the dashboard iframe.
60
91
 
61
- ## 10. Tailwind Build Fails or Admin UI Loads Without Styles
92
+ ## 12. Tailwind Build Fails or Admin UI Loads Without Styles
62
93
 
63
94
  - Running `test/dummy/bin/dev` before configuring the bundling pipeline will serve the admin UI without Tailwind styles or Stimulus behaviours. This happens because the engine no longer ships precompiled assets; see `.ai/engine-asset-configuration.md:11-44` for the required npm setup.
64
95
  - Fix by running `npm install` followed by `npm run build` inside the engine root so that `app/assets/builds/source_monitor/application.css` and `application.js` exist. The Rake task `app:source_monitor:assets:build` wraps the same scripts for CI usage.
data/docs/upgrade.md ADDED
@@ -0,0 +1,140 @@
1
+ # SourceMonitor Upgrade Guide
2
+
3
+ This guide covers upgrading SourceMonitor to a new gem version in your host Rails application.
4
+
5
+ ## General Upgrade Steps
6
+
7
+ 1. Review the [CHANGELOG](../CHANGELOG.md) for changes between your current and target versions
8
+ 2. Update your Gemfile version constraint and run `bundle update source_monitor`
9
+ 3. Run the upgrade command: `bin/source_monitor upgrade`
10
+ 4. Apply database migrations if new ones were copied: `bin/rails db:migrate`
11
+ 5. Address any deprecation warnings in your initializer (see Deprecation Handling below)
12
+ 6. Run verification: `bin/source_monitor verify`
13
+ 7. Restart your web server and background workers
14
+
15
+ ## Quick Upgrade (Most Cases)
16
+
17
+ ```bash
18
+ # 1. Update the gem
19
+ bundle update source_monitor
20
+
21
+ # 2. Run the upgrade command (handles migrations, generator, verification)
22
+ bin/source_monitor upgrade
23
+
24
+ # 3. Migrate if needed
25
+ bin/rails db:migrate
26
+
27
+ # 4. Restart
28
+ # (restart web server and Solid Queue workers)
29
+ ```
30
+
31
+ ## Deprecation Handling
32
+
33
+ When upgrading, you may see deprecation warnings in your Rails log:
34
+
35
+ ```
36
+ [SourceMonitor] DEPRECATION: 'http.old_option' was deprecated in v0.5.0 and replaced by 'http.new_option'.
37
+ ```
38
+
39
+ To resolve:
40
+ 1. Open `config/initializers/source_monitor.rb`
41
+ 2. Find the deprecated option (e.g., `config.http.old_option = value`)
42
+ 3. Replace with the new option from the warning message (e.g., `config.http.new_option = value`)
43
+ 4. Restart and verify the warning is gone
44
+
45
+ If a removed option raises an error (`SourceMonitor::DeprecatedOptionError`), you must update the initializer before the app can boot.
46
+
47
+ ## Version-Specific Notes
48
+
49
+ ### Upgrading to 0.4.0 (from 0.3.x)
50
+
51
+ **Released:** 2026-02-12
52
+
53
+ **What changed:**
54
+ - Install generator now auto-patches `Procfile.dev` with a Solid Queue `jobs:` entry
55
+ - Install generator now patches `config/queue.yml` dispatcher with `recurring_schedule: config/recurring.yml`
56
+ - Active Storage image download feature added (opt-in)
57
+ - SSL certificate configuration added to HTTP settings
58
+ - Enhanced verification messages for SolidQueue and RecurringSchedule verifiers
59
+
60
+ **Upgrade steps:**
61
+ ```bash
62
+ bundle update source_monitor
63
+ bin/source_monitor upgrade
64
+ bin/rails db:migrate
65
+ ```
66
+
67
+ **Notes:**
68
+ - No breaking changes. All existing configuration remains valid.
69
+ - Re-running the generator (`bin/rails generate source_monitor:install`) will add missing `Procfile.dev` and `queue.yml` entries without overwriting existing config.
70
+ - New optional features: `config.images.download_to_active_storage = true`, `config.http.ssl_ca_file`, `config.http.ssl_ca_path`, `config.http.ssl_verify`.
71
+
72
+ ### Upgrading to 0.3.0 (from 0.2.x)
73
+
74
+ **Released:** 2026-02-10
75
+
76
+ **What changed:**
77
+ - Internal refactoring: FeedFetcher, Configuration, ImportSessionsController, and ItemCreator extracted into smaller modules
78
+ - Eager requires replaced with Ruby autoload
79
+ - Skills system added (14 `sm-*` Claude Code skills)
80
+
81
+ **Upgrade steps:**
82
+ ```bash
83
+ bundle update source_monitor
84
+ bin/source_monitor upgrade
85
+ bin/rails db:migrate
86
+ ```
87
+
88
+ **Notes:**
89
+ - No breaking changes to the public API.
90
+ - If you referenced internal classes directly (e.g., `SourceMonitor::FeedFetcher` internals), verify your code against the new module structure.
91
+ - Optionally install AI skills: `bin/rails source_monitor:skills:install`
92
+
93
+ ### Upgrading to 0.2.0 (from 0.1.x)
94
+
95
+ **Released:** 2025-11-25
96
+
97
+ **What changed:**
98
+ - OPML import wizard with multi-step flow
99
+ - New `ImportHistory` model and associated migrations
100
+
101
+ **Upgrade steps:**
102
+ ```bash
103
+ bundle update source_monitor
104
+ bin/rails railties:install:migrations FROM=source_monitor
105
+ bin/rails db:migrate
106
+ ```
107
+
108
+ **Notes:**
109
+ - New database tables required. Run migrations after updating.
110
+ - No configuration changes needed.
111
+
112
+ ## Troubleshooting
113
+
114
+ ### "Already up to date" but I expected changes
115
+ - Verify the gem version actually changed: `bundle show source_monitor`
116
+ - Check `Gemfile.lock` for the resolved version
117
+ - If the `.source_monitor_version` marker was manually edited, delete it and re-run upgrade
118
+
119
+ ### Migrations fail with duplicate timestamps
120
+ - Remove the duplicate migration file from `db/migrate/` (keep the newer one)
121
+ - Re-run `bin/rails db:migrate`
122
+
123
+ ### Deprecation error prevents boot
124
+ - Read the error message for the replacement option
125
+ - Update your initializer before restarting
126
+ - If unsure which option to use, consult [Configuration Reference](configuration.md)
127
+
128
+ ### Verification failures after upgrade
129
+ - **PendingMigrations:** Run `bin/rails db:migrate`
130
+ - **SolidQueue:** Ensure workers are running. Check `Procfile.dev` for a `jobs:` entry.
131
+ - **RecurringSchedule:** Re-run `bin/rails generate source_monitor:install` to patch `config/queue.yml`
132
+ - **ActionCable:** Configure Solid Cable or Redis adapter
133
+
134
+ For additional help, see [Troubleshooting](troubleshooting.md).
135
+
136
+ ## See Also
137
+ - [Setup Guide](setup.md) -- Initial installation
138
+ - [Configuration Reference](configuration.md) -- All configuration options
139
+ - [Troubleshooting](troubleshooting.md) -- Common issues and fixes
140
+ - [CHANGELOG](../CHANGELOG.md) -- Full version history