source_monitor 0.1.1 → 0.1.2

data/docs/setup-validation-log.md

@@ -0,0 +1,36 @@
+# Setup Workflow Validation Notes
+
+Date: 2025-11-13
+Engineer: Codex agent
+
+## Scenario A – Fresh Rails Host (cloned from dummy app)
+
+Steps:
+1. Copied `test/dummy` to `tmp/setup_validation` to simulate a pristine host.
+2. Ran `../../bin/source_monitor install --yes` from the copied directory.
+
+Findings:
+- The first pass surfaced two actionable items:
+  - Gemfile duplication (`eval_gemfile '../../Gemfile'` already includes `source_monitor`). Deleted the appended line to unblock bundler.
+  - Final verification exited with `WARNING` because no Solid Queue workers were running yet. Inserted a synthetic `SolidQueue::Process` row via `bundle exec rails runner` to mimic a heartbeat before re-running verify.
+- Subsequent `../../bin/source_monitor verify` returned:
+  ```
+  Verification summary (OK):
+  - Solid Queue: OK - Solid Queue workers are reporting heartbeats
+  - Action Cable: OK - Solid Cable tables detected and the gem is loaded
+  ```
+- Re-running the installer after the fixes proved idempotent and completed with an `OK` summary.
+
+## Scenario B – Existing Host (same copy, post-install)
+
+Steps:
+1. With the same working copy (representing an already-configured app), ran `../../bin/source_monitor install --yes` again.
+2. Verified that no files were changed beyond timestamp updates and the workflow skipped migration copies/Devise hooks as expected.
+
+Findings:
+- Second run finished cleanly with the same `OK` verification summary. Demonstrates safe re-entry for upgrades/CI.
+
+## Follow-ups
+
+- The Gemfile duplication edge case only applies to internal test harnesses that `eval_gemfile '../../Gemfile'`. Production hosts should add the gem once, but we may consider enhancing `GemfileEditor` to detect `eval_gemfile` references to avoid duplicate insertion.
+- A running (or synthetic) Solid Queue process is required for the verification step to return `OK`. Documented expectations in `docs/setup.md` and highlighted the remediation string emitted by the verifier.

data/docs/setup.md

@@ -0,0 +1,144 @@
+# SourceMonitor Setup Workflow
+
+This guide consolidates the new guided installer, verification commands, and rollback steps so teams can onboard the engine into either a fresh Rails host or an existing application without missing prerequisites.
+
+## Prerequisites
+
+| Requirement | Minimum | Notes |
+| --- | --- | --- |
+| Ruby | 3.4.4 | Use rbenv and match the engine's `.ruby-version`. |
+| Rails | 8.0.2.1 | Run `bin/rails about` inside the host to confirm. |
+| PostgreSQL | 14+ | Required for Solid Queue tables and item storage. |
+| Node.js | 18+ | Needed for Tailwind/esbuild assets when the host owns node tooling. |
+| Background jobs | Solid Queue (>= 0.3, < 3.0) | Add `solid_queue` to the host Gemfile if not present. |
+| Realtime | Solid Cable (>= 3.0) or Redis | Solid Cable is the default; Redis requires `config.realtime.adapter = :redis`. |
+
+## Guided Setup (Recommended)
+
+1. **Check prerequisites** (optional but fast):
+   ```bash
+   bin/rails source_monitor:setup:check
+   ```
+   This invokes the dependency checker added in Phase 10.04 and surfaces remediation text when versions or adapters are missing.
+
+2. **Run the guided installer:**
+   ```bash
+   bin/source_monitor install
+   ```
+   - Prompts for the mount path (defaults to `/source_monitor`).
+   - Ensures `gem "source_monitor"` is in the host Gemfile and runs `bundle install` via rbenv shims.
+   - Runs `npm install` when `package.json` exists.
+   - Executes `bin/rails generate source_monitor:install --mount-path=...`.
+   - Copies migrations, deduplicates duplicate Solid Queue migrations, and reruns `bin/rails db:migrate`.
+   - Updates `config/initializers/source_monitor.rb` with a navigation hint and, when desired, Devise hooks.
+   - Writes a verification report at the end (same as `bin/source_monitor verify`).
+
+3. **Start background workers:**
+   ```bash
+   bin/rails solid_queue:start
+   bin/jobs --recurring_schedule_file=config/recurring.yml # optional recurring scheduler
+   ```
+
+4. **Visit the dashboard** at the chosen mount path, create a source, and trigger “Fetch Now” to validate realtime updates and Solid Queue processing.
+
+### Fully Non-Interactive Install
+
+Use `--yes` to accept defaults (mount path `/source_monitor`, Devise hooks enabled if Devise detected):
+```bash
+bin/source_monitor install --yes
+```
+
+### Verification & Telemetry
+
+- Re-run verification anytime:
+  ```bash
+  bin/source_monitor verify
+  ```
+  or
+  ```bash
+  bin/rails source_monitor:setup:verify
+  ```
+- Results show human-friendly lines plus a JSON blob; exit status is non-zero when any check fails.
+- To persist telemetry for support, set `SOURCE_MONITOR_SETUP_TELEMETRY=true`. Logs append to `log/source_monitor_setup.log`.
+
+## Manual Installation (Advanced)
+
+Prefer to script each step or plug SourceMonitor into an existing deployment checklist? Use the manual flow below. It mirrors the guided CLI internals while keeping every command explicit.
+
+### Quick Reference
+
+| Step | Command | Purpose |
+| --- | --- | --- |
+| 1 | `gem "source_monitor", github: "dchuk/source_monitor"` | Add the engine to your Gemfile (skip if already present) |
+| 2 | `bundle install` | Install Ruby dependencies |
+| 3 | `bin/rails generate source_monitor:install --mount-path=/source_monitor` | Mount the engine and create the initializer |
+| 4 | `bin/rails railties:install:migrations FROM=source_monitor` | Copy engine migrations (idempotent) |
+| 5 | `bin/rails db:migrate` | Apply schema updates, including Solid Queue tables |
+| 6 | `bin/rails solid_queue:start` | Ensure jobs process via Solid Queue |
+| 7 | `bin/jobs --recurring_schedule_file=config/recurring.yml` | Start recurring scheduler (optional but recommended) |
+| 8 | `bin/source_monitor verify` | Confirm Solid Queue/Action Cable readiness and emit telemetry |
+
+> Tip: You can drop these commands directly into your CI pipeline or release scripts. The CLI uses the same services, so mixing and matching is safe.
+
+### Step-by-step Details
+
+1. **Add the gem** to the host `Gemfile` (GitHub edge or released version) and run `bundle install`. If your host manages node tooling, run `npm install` also.
+2. **Install the engine** via `bin/rails generate source_monitor:install --mount-path=/source_monitor`. The generator mounts the engine, creates `config/initializers/source_monitor.rb`, and prints follow-up instructions. Re-running the generator is safe; it detects existing mounts/initializers.
+3. **Copy migrations** with `bin/rails railties:install:migrations FROM=source_monitor`. This brings in the SourceMonitor tables plus Solid Cable/Queue schema when needed. The command is idempotent—run it again after upgrading the gem.
+4. **Apply database changes** using `bin/rails db:migrate`. If your host already installed Solid Queue migrations manually, delete duplicate files before migrating.
+5. **Wire Action Cable** if necessary. SourceMonitor defaults to Solid Cable; confirm `ApplicationCable::Connection`/`Channel` exist and that `config/initializers/source_monitor.rb` uses the adapter you expect. To switch to Redis, set `config.realtime.adapter = :redis` and `config.realtime.redis_url`.
+6. **Start workers** with `bin/rails solid_queue:start` (or your process manager). Add a recurring process via `bin/jobs --recurring_schedule_file=config/recurring.yml` when you need fetch/scrape schedules.
+7. **Review the initializer** and tune queue names, HTTP timeouts, scraping adapters, retention limits, authentication hooks, and Mission Control integration. The [configuration reference](configuration.md) details every option.
+8. **Verify the install**: run `bin/source_monitor verify` to ensure Solid Queue workers and Action Cable are healthy, then visit the mount path to trigger a fetch manually. Enable telemetry if you want JSON logs recorded for support.
+
+### Host Compatibility Matrix
+
+| Host Scenario | Status | Notes |
+| --- | --- | --- |
+| Rails 8 full-stack app | ✅ Supported | Use the guided workflow or the manual generator steps above |
+| Rails 8 API-only app (`--api`) | ✅ Supported | Generator mounts engine; provide your own UI entry point if needed |
+| Dedicated Solid Queue database | ✅ Supported | Run `bin/rails solid_queue:install` in the host app before copying SourceMonitor migrations |
+| Redis-backed Action Cable | ✅ Supported | Set `config.realtime.adapter = :redis` and provide `config.realtime.redis_url`; existing `config/cable.yml` entries are preserved |
+
+## Rollback Steps
+
+If you need to revert the integration in a host app:
+
+1. Remove `gem "source_monitor"` from the host Gemfile and rerun `bundle install`.
+2. Delete the engine initializer (`config/initializers/source_monitor.rb`) and any navigation links referencing the mount path.
+3. Remove the mount entry from `config/routes.rb` (the install generator adds a comment to help locate it).
+4. Drop SourceMonitor tables if they are no longer needed:
+   ```bash
+   bin/rails db:migrate:down VERSION=<timestamp> # repeat for each engine migration
+   ```
+5. Remove Solid Queue / Solid Cable migrations only if no other components rely on them.
+
+Document each removal in the host application's changelog to keep future upgrades predictable.
+
+## Optional Devise System Test Template
+
+Add a guardrail test in the host app (or dummy) to make sure authentication protects the dashboard after upgrades:
+
+```ruby
+# test/system/source_monitor_setup_test.rb
+require "application_system_test_case"
+
+class SourceMonitorSetupTest < ApplicationSystemTestCase
+  test "signed in admin can reach SourceMonitor" do
+    user = users(:admin)
+    sign_in user
+
+    visit "/source_monitor"
+    assert_text "SourceMonitor Dashboard"
+  end
+end
+```
+
+- Swap `sign_in user` for your Devise helper (`login_as`, etc.).
+- Use fixtures or factories that guarantee the user is authorized per the initializer’s `authorize_with` hook.
+
+## Additional Notes
+
+- Re-running `bin/source_monitor install` is idempotent: if migrations already exist or Devise hooks are present, the workflow skips creation and only re-verifies prerequisites.
+- The CLI wraps the same services the rake tasks use, so CI can call `bin/source_monitor verify` directly after migrations to catch worker/cable misconfigurations before deploying.
+- Keep this document aligned with the PRD (`tasks/prd-setup-workflow-streamlining.md`) and active task list (`tasks/tasks-setup-workflow-streamlining.md`).

data/lib/generators/source_monitor/install/install_generator.rb

@@ -34,7 +34,7 @@ module SourceMonitor
 
       def print_next_steps
         say_status :info,
-          "Next steps: review docs/installation.md for install walkthroughs and docs/troubleshooting.md for common fixes.",
+          "Next steps: review docs/setup.md for the guided + manual install walkthrough and docs/troubleshooting.md for common fixes.",
           :green
       end
 

data/lib/source_monitor/setup/bundle_installer.rb

@@ -0,0 +1,17 @@
+module SourceMonitor
+  module Setup
+    class BundleInstaller
+      def initialize(shell: ShellRunner.new)
+        @shell = shell
+      end
+
+      def install
+        shell.run("bundle", "install")
+      end
+
+      private
+
+      attr_reader :shell
+    end
+  end
+end

data/lib/source_monitor/setup/cli.rb

@@ -0,0 +1,51 @@
+require "thor"
+
+module SourceMonitor
+  module Setup
+    class CLI < Thor
+      class_option :yes, type: :boolean, default: false, desc: "Accept all defaults"
+
+      desc "install", "Run the guided SourceMonitor setup workflow"
+      option :mount_path, type: :string, default: Workflow::DEFAULT_MOUNT_PATH
+      def install
+        workflow = Workflow.new(
+          prompter: Prompter.new(shell: shell, auto_yes: options[:yes])
+        )
+        summary = workflow.run
+        handle_summary(summary)
+      end
+
+      desc "verify", "Verify queue workers and Action Cable configuration"
+      def verify
+        summary = verification_runner.call
+        handle_summary(summary)
+      end
+
+      private
+
+      def handle_summary(summary)
+        printer.print(summary)
+        emit_telemetry(summary)
+        exit(1) unless summary.ok?
+      end
+
+      def printer
+        Verification::Printer.new(shell: shell)
+      end
+
+      def verification_runner
+        Verification::Runner.new
+      end
+
+      def emit_telemetry(summary)
+        return unless telemetry_enabled?
+
+        Verification::TelemetryLogger.new.log(summary)
+      end
+
+      def telemetry_enabled?
+        ENV["SOURCE_MONITOR_SETUP_TELEMETRY"].to_s.casecmp("true").zero?
+      end
+    end
+  end
+end

data/lib/source_monitor/setup/dependency_checker.rb

@@ -0,0 +1,165 @@
+module SourceMonitor
+  module Setup
+    class DependencyChecker
+      Dependency = Struct.new(
+        :key,
+        :name,
+        :requirement,
+        :detector,
+        :remediation,
+        keyword_init: true
+      )
+
+      Result = Struct.new(
+        :key,
+        :name,
+        :status,
+        :current,
+        :expected,
+        :remediation,
+        keyword_init: true
+      ) do
+        def ok?
+          status == :ok
+        end
+
+        def warning?
+          status == :warning
+        end
+
+        def error?
+          status == :error
+        end
+
+        def missing?
+          status == :missing
+        end
+      end
+
+      class Summary
+        attr_reader :results
+
+        def initialize(results)
+          @results = results
+        end
+
+        def overall_status
+          return :error if errors?
+          return :warning if warnings?
+
+          :ok
+        end
+
+        def ok?
+          overall_status == :ok
+        end
+
+        def errors?
+          results.any? { |result| result.error? || result.missing? }
+        end
+
+        def warnings?
+          results.any?(&:warning?)
+        end
+
+        def errors
+          results.select { |result| result.error? || result.missing? }
+        end
+
+        def warnings
+          results.select(&:warning?)
+        end
+      end
+
+      def initialize(dependencies: default_dependencies)
+        @dependencies = dependencies
+      end
+
+      def call
+        Summary.new(@dependencies.map { |dependency| evaluate_dependency(dependency) })
+      end
+
+      private
+
+      def evaluate_dependency(dependency)
+        current = safe_detect { dependency.detector.call }
+        expected = requirement_expected(dependency.requirement)
+        status = classify_status(dependency.requirement, current)
+
+        Result.new(
+          key: dependency.key,
+          name: dependency.name,
+          status: status,
+          current: normalize(dependency.requirement, current),
+          expected: expected,
+          remediation: dependency.remediation
+        )
+      end
+
+      def safe_detect
+        yield
+      rescue StandardError
+        nil
+      end
+
+      def classify_status(requirement, current)
+        return :missing if current.nil?
+        return :ok if requirement.respond_to?(:satisfied?) && requirement.satisfied?(current)
+
+        :error
+      end
+
+      def normalize(requirement, value)
+        if requirement.respond_to?(:normalize)
+          requirement.normalize(value)
+        else
+          value
+        end
+      end
+
+      def requirement_expected(requirement)
+        requirement.respond_to?(:expected) ? requirement.expected : nil
+      end
+
+      def default_dependencies
+        [
+          Dependency.new(
+            key: :ruby,
+            name: "Ruby",
+            requirement: Requirements::Version.new(">= 3.4.4"),
+            detector: -> { Detectors.ruby_version },
+            remediation: "Upgrade Ruby to >= 3.4.4"
+          ),
+          Dependency.new(
+            key: :rails,
+            name: "Rails",
+            requirement: Requirements::Version.new(">= 8.0.2.1"),
+            detector: -> { Detectors.rails_version },
+            remediation: "Upgrade Rails to >= 8.0.2.1"
+          ),
+          Dependency.new(
+            key: :node,
+            name: "Node.js",
+            requirement: Requirements::Version.new(">= 18.0.0"),
+            detector: -> { Detectors.node_version },
+            remediation: "Install Node.js 18 or newer"
+          ),
+          Dependency.new(
+            key: :postgres,
+            name: "PostgreSQL Adapter",
+            requirement: Requirements::Adapter.new("postgresql"),
+            detector: -> { Detectors.postgres_adapter },
+            remediation: "Configure the host app to use PostgreSQL before installing SourceMonitor"
+          ),
+          Dependency.new(
+            key: :solid_queue,
+            name: "Solid Queue",
+            requirement: Requirements::Version.new([ ">= 0.3.0", "< 3.0" ]),
+            detector: -> { Detectors.solid_queue_version },
+            remediation: "Add the solid_queue gem (>= 0.3, < 3.0) to your host app"
+          )
+        ]
+      end
+    end
+  end
+end

data/lib/source_monitor/setup/detectors.rb

@@ -0,0 +1,67 @@
+module SourceMonitor
+  module Setup
+    module Detectors
+      extend self
+
+      def ruby_version
+        Gem::Version.new(RUBY_VERSION)
+      end
+
+      def rails_version
+        return unless defined?(Rails)
+
+        Rails.gem_version
+      end
+
+      def node_version(shell: ShellRunner.new)
+        output = shell.run("node", "--version")
+        return if output.blank?
+
+        Gem::Version.new(output.strip.sub(/^v/i, ""))
+      rescue ArgumentError
+        nil
+      end
+
+      def postgres_adapter(config: nil, fallback: nil)
+        config ||= connection_db_config
+        fallback ||= primary_config
+
+        config_adapter(config) || config_adapter(fallback)
+      rescue StandardError
+        nil
+      end
+
+      def solid_queue_version
+        spec = Gem.loaded_specs["solid_queue"]
+        spec&.version
+      end
+
+      private
+
+      def connection_db_config
+        return unless defined?(ActiveRecord::Base)
+
+        ActiveRecord::Base.connection_db_config
+      rescue ActiveRecord::ConnectionNotEstablished
+        nil
+      end
+
+      def primary_config
+        return unless defined?(Rails)
+
+        ActiveRecord::Base.configurations.configs_for(env_name: Rails.env, name: "primary")
+          &.first
+      rescue StandardError
+        nil
+      end
+
+      def config_adapter(config)
+        return if config.nil?
+
+        config.respond_to?(:adapter) ? config.adapter : config[:adapter]
+      rescue StandardError
+        nil
+      end
+    end
+  end
+end

data/lib/source_monitor/setup/gemfile_editor.rb

@@ -0,0 +1,29 @@
+require "pathname"
+
+module SourceMonitor
+  module Setup
+    class GemfileEditor
+      attr_reader :path
+
+      def initialize(path: "Gemfile")
+        @path = Pathname.new(path)
+      end
+
+      def ensure_entry
+        return false unless path.exist?
+
+        contents = path.read
+        return false if contents.match?(/gem\s+['"]source_monitor['"]/)
+
+        path.open("a") do |file|
+          file.write(<<~RUBY)
+
+            gem "source_monitor"
+          RUBY
+        end
+
+        true
+      end
+    end
+  end
+end

data/lib/source_monitor/setup/initializer_patcher.rb

@@ -0,0 +1,59 @@
+require "pathname"
+
+module SourceMonitor
+  module Setup
+    class InitializerPatcher
+      AUTH_MARKER = "# ---- Authentication".freeze
+
+      def initialize(path: "config/initializers/source_monitor.rb")
+        @path = Pathname.new(path)
+      end
+
+      def ensure_navigation_hint(mount_path: Workflow::DEFAULT_MOUNT_PATH)
+        return false unless path.exist?
+
+        comment = "# Mount SourceMonitor at #{mount_path} so admins can find the dashboard."
+        contents = path.read
+        return false if contents.include?(comment)
+
+        updated = contents.sub("# SourceMonitor engine configuration.", "# SourceMonitor engine configuration.\n#{comment}")
+        path.write(updated)
+        true
+      end
+
+      def ensure_devise_hooks
+        return false unless path.exist?
+
+        contents = path.read
+        return false if contents.include?("config.authentication.authenticate_with :authenticate_user!")
+
+        snippet = indent(devise_snippet)
+        updated = if contents.include?(AUTH_MARKER)
+          contents.sub(AUTH_MARKER, "#{AUTH_MARKER}\n#{snippet}")
+        else
+          contents.sub(/SourceMonitor.configure do \|config\|/m, "\\0\n#{snippet}")
+        end
+
+        path.write(updated)
+        true
+      end
+
+      private
+
+      attr_reader :path
+
+      def indent(text)
+        text.lines.map { |line| line.strip.empty? ? "\n" : "  #{line}" }.join
+      end
+
+      def devise_snippet
+        <<~RUBY
+          config.authentication.authenticate_with :authenticate_user!
+          config.authentication.authorize_with ->(controller) { controller.current_user&.respond_to?(:admin?) ? controller.current_user.admin? : true }
+          config.authentication.current_user_method = :current_user
+          config.authentication.user_signed_in_method = :user_signed_in?
+        RUBY
+      end
+    end
+  end
+end

data/lib/source_monitor/setup/install_generator.rb

@@ -0,0 +1,22 @@
+module SourceMonitor
+  module Setup
+    class InstallGenerator
+      def initialize(shell: ShellRunner.new)
+        @shell = shell
+      end
+
+      def run(mount_path: Workflow::DEFAULT_MOUNT_PATH)
+        shell.run(
+          "bin/rails",
+          "generate",
+          "source_monitor:install",
+          "--mount-path=#{mount_path}"
+        )
+      end
+
+      private
+
+      attr_reader :shell
+    end
+  end
+end

data/lib/source_monitor/setup/migration_installer.rb

@@ -0,0 +1,40 @@
+require "fileutils"
+require "pathname"
+
+module SourceMonitor
+  module Setup
+    class MigrationInstaller
+      SOLID_QUEUE_PATTERN = "*_create_solid_queue_tables.rb"
+
+      def initialize(shell: ShellRunner.new, migrations_path: "db/migrate")
+        @shell = shell
+        @migrations_path = Pathname.new(migrations_path)
+      end
+
+      def install
+        copy_migrations
+        deduplicate_solid_queue
+        run_migrations
+      end
+
+      private
+
+      attr_reader :shell, :migrations_path
+
+      def copy_migrations
+        shell.run("bin/rails", "railties:install:migrations", "FROM=source_monitor")
+      end
+
+      def deduplicate_solid_queue
+        files = Dir[migrations_path.join(SOLID_QUEUE_PATTERN).to_s].sort
+        return if files.size <= 1
+
+        files[1..].each { |path| FileUtils.rm_f(path) }
+      end
+
+      def run_migrations
+        shell.run("bin/rails", "db:migrate")
+      end
+    end
+  end
+end

data/lib/source_monitor/setup/node_installer.rb

@@ -0,0 +1,27 @@
+require "pathname"
+
+module SourceMonitor
+  module Setup
+    class NodeInstaller
+      def initialize(root: Pathname.pwd.to_s, shell: ShellRunner.new)
+        @root = Pathname.new(root)
+        @shell = shell
+      end
+
+      def install_if_needed
+        return false unless package_json?
+
+        shell.run("npm", "install")
+        true
+      end
+
+      private
+
+      attr_reader :root, :shell
+
+      def package_json?
+        root.join("package.json").exist?
+      end
+    end
+  end
+end