source_monitor 0.1.1 → 0.1.2

data/tasks/tasks-setup-workflow-streamlining.md

@@ -0,0 +1,51 @@
+## Relevant Files
+
+- `lib/source_monitor/setup` - Entry point for setup orchestration services/Thor commands.
+- `lib/tasks/source_monitor.rake` - Rake interfaces for setup/verification tasks.
+- `bin/source_monitor` - CLI wrapper for invoking setup workflow.
+- `config/initializers/source_monitor.rb` - Installer-generated initializer that may need automated edits.
+- `docs/setup.md` - Developer-facing setup checklist documentation.
+- `test/lib/source_monitor/setup` - Unit tests for setup orchestration services.
+- `test/tasks/source_monitor_setup_test.rb` - Tests covering rake task behavior.
+
+### Notes
+
+- Add/adjust unit tests alongside new services and rake tasks to keep coverage high.
+- Prefer Thor/Rails generator patterns already used in the engine for consistent prompts.
+
+## Instructions for Completing Tasks
+
+IMPORTANT: As you complete each task, check it off by changing `- [ ]` to `- [x]`. Update after every sub-task once they are added.
+
+## Tasks
+
+- [x] 0.0 Create feature branch
+  - [x] 0.1 Create and checkout `feature/setup-workflow-streamlining`
+- [ ] 1.0 Build prerequisite detection and dependency helpers
+  - [x] 1.1 Design dependency checker interface (Ruby/Rails/Postgres/Node/Solid Queue)
+  - [x] 1.2 Implement version detection services with unit tests (TDD)
+  - [x] 1.3 Add remediation guidance mapping (error messages) with tests
+  - [x] 1.4 Wire helpers into CLI task to block/skip steps appropriately
+- [x] 2.0 Implement guided setup command/workflow
+  - [x] 2.1 Scaffold Thor/Rails task entry point with prompts, ensuring specs cover CLI flow
+  - [x] 2.2 Automate Gemfile injection + `bundle install` and cover via integration-style tests/mocks
+  - [x] 2.3 Automate `npm install` detection/execution with tests for both asset pipeline modes
+  - [x] 2.4 Invoke install generator + mount path confirmation, validated by tests against dummy app
+  - [x] 2.5 Copy migrations and deduplicate Solid Queue tables with regression tests
+  - [x] 2.6 Automate initializer patching (Devise hooks optional) with unit tests covering idempotency
+  - [x] 2.7 Provide guided prompts for Devise wiring and ensure tests cover conditional behavior
+- [x] 3.0 Add verification and telemetry tooling
+  - [x] 3.1 Implement Solid Queue worker verification service with tests simulating worker availability
+  - [x] 3.2 Implement Action Cable adapter verification (Solid Cable default, Redis optional) with tests
+  - [x] 3.3 Add reusable `source_monitor:setup:verify` task leveraging verification services with coverage
+  - [x] 3.4 Emit structured JSON + human-readable summaries; test serialization and logging
+  - [x] 3.5 Optional telemetry output (file/webhook) guarded by feature flag with tests
+- [x] 4.0 Refresh documentation and onboarding assets
+  - [x] 4.1 Update `docs/setup.md` to mirror automated workflow, referencing new commands
+  - [x] 4.2 Document rollback steps and optional Devise system test template
+  - [x] 4.3 Ensure `.ai/tasks.md` references this slice and link to PRD/tasks documents
+- [x] 5.0 Validate workflow end-to-end and define rollout
+  - [x] 5.1 Run setup workflow inside fresh Rails dummy app; record findings/logs
+  - [x] 5.2 Run workflow inside existing host scenario (dummy app variations); capture diffs
+  - [x] 5.3 Execute full test suite (`bin/rails test`, targeted setup tests, linters) and document results
+  - [x] 5.4 Draft release notes + rollout checklist (include CI verification task adoption plan)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: source_monitor
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.2
 platform: ruby
 authors:
 - dchuk
@@ -351,7 +351,8 @@ files:
 - docs/configuration.md
 - docs/deployment.md
 - docs/gh-cli-workflow.md
-- docs/installation.md
+- docs/setup-validation-log.md
+- docs/setup.md
 - docs/troubleshooting.md
 - eslint.config.mjs
 - lib/generators/source_monitor/install/install_generator.rb
@@ -426,11 +427,31 @@ files:
 - lib/source_monitor/scraping/state.rb
 - lib/source_monitor/security/authentication.rb
 - lib/source_monitor/security/parameter_sanitizer.rb
+- lib/source_monitor/setup/bundle_installer.rb
+- lib/source_monitor/setup/cli.rb
+- lib/source_monitor/setup/dependency_checker.rb
+- lib/source_monitor/setup/detectors.rb
+- lib/source_monitor/setup/gemfile_editor.rb
+- lib/source_monitor/setup/initializer_patcher.rb
+- lib/source_monitor/setup/install_generator.rb
+- lib/source_monitor/setup/migration_installer.rb
+- lib/source_monitor/setup/node_installer.rb
+- lib/source_monitor/setup/prompter.rb
+- lib/source_monitor/setup/requirements.rb
+- lib/source_monitor/setup/shell_runner.rb
+- lib/source_monitor/setup/verification/action_cable_verifier.rb
+- lib/source_monitor/setup/verification/printer.rb
+- lib/source_monitor/setup/verification/result.rb
+- lib/source_monitor/setup/verification/runner.rb
+- lib/source_monitor/setup/verification/solid_queue_verifier.rb
+- lib/source_monitor/setup/verification/telemetry_logger.rb
+- lib/source_monitor/setup/workflow.rb
 - lib/source_monitor/sources/turbo_stream_presenter.rb
 - lib/source_monitor/turbo_streams/stream_responder.rb
 - lib/source_monitor/version.rb
 - lib/tasks/recover_stalled_fetches.rake
 - lib/tasks/source_monitor_assets.rake
+- lib/tasks/source_monitor_setup.rake
 - lib/tasks/source_monitor_tasks.rake
 - lib/tasks/test_smoke.rake
 - package-lock.json
@@ -438,6 +459,8 @@ files:
 - postcss.config.js
 - source_monitor.gemspec
 - stylelint.config.js
+- tasks/prd-setup-workflow-streamlining.md
+- tasks/tasks-setup-workflow-streamlining.md
 homepage: https://github.com/dchuk/source_monitor
 licenses:
 - MIT

data/docs/installation.md

@@ -1,144 +0,0 @@
-# Installation Guide
-
-SourceMonitor installs like any other Rails engine, but it ships enough infrastructure (background jobs, realtime broadcasting, configuration DSL) that it is worth walking through the full setup. This guide assumes you are adding the engine to an existing Rails 8 host application.
-
-## Prerequisites
-
-- Ruby 3.4.4 (we recommend [rbenv](https://github.com/rbenv/rbenv) for local development: `rbenv install 3.4.4 && rbenv local 3.4.4`, but asdf, chruby, rvm, or container-managed Ruby all work equally well—choose whatever fits your environment)
-- Rails 8.0.2.1 or newer
-- PostgreSQL 13 or newer (the engine migrations rely on JSONB, SKIP LOCKED, and advisory locks)
-- Node.js 18+ and npm or Yarn for asset linting/builds
-- Solid Queue and Solid Cable gems available (they ship with Rails 8, but make sure they are not removed)
-- Optional: Mission Control Jobs if you plan to surface the dashboard shortcut, Redis if you intend to switch realtime adapters
-
-> **Command prefixes:** All commands below are shown without `rbenv exec`. If you use rbenv, prefix `bundle` and `bin/rails` commands with `rbenv exec`. For asdf, no prefix is needed. In Docker/container environments, run commands directly inside the container.
-
-## Quick Reference
-
-| Step | Command | Purpose |
-| --- | --- | --- |
-| 1 | `gem "source_monitor", github: "dchuk/source_monitor"` | Add the engine to your Gemfile |
-| 2 | `bundle install` | Install Ruby dependencies |
-| 3 | `bin/rails generate source_monitor:install --mount-path=/source_monitor` | Mount the engine and create the initializer |
-| 4 | `bin/rails railties:install:migrations FROM=source_monitor` | Copy engine migrations (idempotent) |
-| 5 | `bin/rails db:migrate` | Apply schema updates, including Solid Queue tables |
-| 6 | `bin/rails solid_queue:start` | Ensure jobs process via Solid Queue |
-| 7 | `bin/jobs --recurring_schedule_file=config/recurring.yml` | Start recurring scheduler (optional but recommended) |
-
-## 1. Add the Gem
-
-In your host application's `Gemfile`:
-
-```ruby
-gem "source_monitor", github: "dchuk/source_monitor"
-```
-
-Then install dependencies:
-
-```bash
-bundle install
-```
-
-If you vendor node tooling for linting/assets in the host app, run `npm install` as well.
-
-## 2. Run the Install Generator
-
-The generator mounts the engine and drops an initializer stub for you to configure.
-
-```bash
-bin/rails generate source_monitor:install --mount-path=/source_monitor
-```
-
-Key outputs:
-
-- Adds `mount SourceMonitor::Engine, at: "/source_monitor"` to your routes (change the path with `--mount-path`)
-- Creates `config/initializers/source_monitor.rb` with documented configuration defaults
-- The generator is idempotent: re-running it will detect existing mounts/initializers and skip overwriting your customizations
-- Update your navigation or admin layout to link to the mount path so teammates can discover the dashboard.
-
-## 3. Copy Engine Migrations
-
-SourceMonitor relies on several tables (sources, items, fetch logs, scrape logs, Solid Cable messages, retention helpers) plus an optional Solid Queue schema. Copy them into your host application with:
-
-```bash
-bin/rails railties:install:migrations FROM=source_monitor
-```
-
-This command is idempotent—re-run it when upgrading to pick up new migrations.
-
-## 4. Run Database Migrations
-
-Apply the new schema:
-
-```bash
-bin/rails db:migrate
-```
-
-If you prefer a dedicated database for Solid Queue, run `bin/rails solid_queue:install` beforehand and point the generated config at your queue database. Otherwise the engine-provided migration keeps Solid Queue tables in the primary database.
-
-> Tip: Solid Queue tables must be present before you start the dashboard. If your host app already ran `solid_queue:install`, delete the engine-provided migration before running `db:migrate` to avoid duplication.
-
-## 5. Wire Action Cable (if needed)
-
-SourceMonitor defaults to Solid Cable for realtime. Rails expects `ApplicationCable::Connection` and `ApplicationCable::Channel` to exist in your host app:
-
-```ruby
-# app/channels/application_cable/connection.rb
-module ApplicationCable
-  class Connection < ActionCable::Connection::Base; end
-end
-
-# app/channels/application_cable/channel.rb
-module ApplicationCable
-  class Channel < ActionCable::Channel::Base; end
-end
-```
-
-Verify `config/cable.yml` allows the adapter you choose. To switch to Redis, update `config/initializers/source_monitor.rb` with `config.realtime.adapter = :redis` and provide `config.realtime.redis_url`.
-
-## 6. Configure Background Workers
-
-Solid Queue becomes the default Active Job adapter when the host app still uses `:async`. Keep at least one worker process running:
-
-```bash
-# long-running process
-bin/rails solid_queue:start
-```
-
-SourceMonitor respects explicit queue adapter overrides. If your host sets `config.active_job.queue_adapter` (for example, to `:inline` or `:sidekiq`), the engine leaves that configuration in place.
-
-For recurring schedules, add a process that runs the Solid Queue CLI with the engine's schedule file:
-
-```bash
-bin/jobs --recurring_schedule_file=config/recurring.yml
-```
-
-Set `SOLID_QUEUE_SKIP_RECURRING=true` in environments that manage recurring tasks elsewhere.
-
-## 7. Review Configuration Defaults
-
-Open `config/initializers/source_monitor.rb` and adjust queue namespaces, HTTP timeouts, scraping adapters, retention limits, authentication hooks, and Mission Control integration to match your environment. The [configuration reference](configuration.md) covers every option with examples.
-
-## 8. Verify the Installation
-
-1. Start your Rails server: `bin/rails server`
-2. Ensure a Solid Queue worker is running (see step 6)
-3. Visit the mount path (`/source_monitor` by default)
-4. Create a source and trigger `Fetch Now`—watch fetch logs appear and dashboard metrics update
-
-If you encounter issues, consult the [troubleshooting guide](troubleshooting.md).
-
-## Next Steps
-
-- Explore the admin UI and dashboards
-- Integrate custom scraper adapters or item processors via `SourceMonitor.configure`
-- Set up monitoring for the Solid Queue queues using Mission Control or your preferred observability stack
-
-## Host Compatibility Matrix
-
-| Host Scenario | Status | Notes |
-| --- | --- | --- |
-| Rails 8 full-stack app | ✅ Supported | Default generator flow (mount + initializer) |
-| Rails 8 API-only app (`--api`) | ✅ Supported | Generator mounts engine; ensure you provide a UI entry point if needed |
-| Dedicated Solid Queue database | ✅ Supported | Run `bin/rails solid_queue:install` in the host app before copying SourceMonitor migrations |
-| Redis-backed Action Cable | ✅ Supported | Set `config.realtime.adapter = :redis` and provide `config.realtime.redis_url`; existing `config/cable.yml` entries are preserved |