source_monitor 0.1.3 → 0.2.0

data/tasks/completed/opml-import-wizard/task-04/instructions.md

@@ -0,0 +1,84 @@
+**Context**
+
+- User story: "Health Check for Selected Sources" within the OPML Import Wizard. This Health Check step runs a health check for each selected parsed source, shows live results, and updates selection state before import.
+- Tech brief: health checks must be enqueued as individual Solid Queue jobs, results persisted back into ImportSession.parsed_sources (fields: health_status, error), and updates broadcast in real-time via Turbo Streams. ImportSession uses a standard integer user_id and persists wizard state (parsed_sources, selected_source_ids, current_step, etc.).
+- Existing codebase patterns to follow: Solid Queue job patterns (app/jobs/*), existing source health check logic and controller (source_health_checks_controller), realtime broadcasters (lib/source_monitor/realtime/broadcaster.rb), and Turbo Stream / Turbo Frame UI conventions (lib/source_monitor/turbo_streams/stream_responder.rb).
+
+Goals
+
+- Enqueue one Solid Queue health-check job per selected source when the Health Check step starts.
+- Persist per-source health results into ImportSession.parsed_sources (add/ensure fields health_status and error).
+- Broadcast row-level updates and an overall progress update (completed / total) in real time via Turbo Streams so the Health Check step UI updates as jobs complete.
+- Unselect sources deemed unhealthy by default while allowing the user to re-select them; ensure selection state persists in ImportSession.selected_source_ids.
+- Prevent navigation forward if no healthy sources remain selected (unless the user intentionally re-selects unhealthy ones).
+- Implement server-side cancellation semantics so jobs do not update or broadcast for an expired/cancelled ImportSession.
+
+Technical Guidelines
+
+- Job orchestration
+  - Enqueue a Solid Queue job per selected parsed source. Job payload must include at minimum: import_session_id and an identifier for the parsed source (index or parsed_source_id/UUID as stored in ImportSession.parsed_sources).
+  - Reuse existing feed health check logic where available (inspect source_health_checks controller / health-check service used elsewhere) rather than reimplementing HTTP/Feed checks.
+  - Name or implement the job in line with existing job conventions (app/jobs/source_monitor/*). The job should:
+    - Load the ImportSession (guard by integer user_id scope).
+    - Verify the ImportSession still exists and is in the Health Check step (or has a health_checks_active flag). If the session has been cancelled/expired, exit without broadcasting.
+    - Run the health check for the feed URL.
+    - Persist result into ImportSession.parsed_sources for the specific parsed entry: set health_status (values: healthy/unhealthy/unknown) and error (string or nil).
+    - Update ImportSession.selected_source_ids to remove/unselect the entry if health_status == unhealthy (so UI reflects unselected by default). Do not destroy historical parsed data.
+    - Broadcast a Turbo Stream update for the row and a progress update (completed count) after persisting.
+
+- Persistence and data model
+  - Store health results inside ImportSession.parsed_sources entries (JSONB). Each parsed source entry must include:
+    - id/key (must be addressable by job),
+    - feed_url (used for duplicate detection),
+    - health_status (nullable string enum),
+    - health_error (nullable text).
+  - Selection state must be authoritative in ImportSession.selected_source_ids (JSONB array of parsed entry ids). Health job changes must update that array atomically (transaction).
+  - Ensure ImportSession model uses integer user_id references per the tech brief.
+
+- Broadcast / UI contract
+  - Use Turbo Streams for all live updates. Follow existing engine broadcaster/presenter patterns:
+    - Broadcast a row replacement partial (Turbo Stream) for the specific parsed source row showing updated health_status icon and selection checkbox state.
+    - Broadcast a separate Turbo Stream or stream fragment with overall progress (completed / total). The progress UI in the Health Check view should subscribe to this stream.
+  - Choose a Turbo stream target scope tied to the ImportSession (e.g., stream name or DOM target that includes import_session id) so updates only reach the appropriate wizard instance. Ensure consistent naming with other engine broadcasters.
+  - Keep UI updates non-destructive: rows should preserve user ability to re-select unhealthy sources.
+
+- UI-side behavior to enforce
+  - After a health check job marks a source unhealthy, the UI must show it as unselected by default and display the error inline on the row (tooltip or error cell) while permitting re-selection.
+  - The progress indicator must show completed / total and update as job results arrive.
+  - Navigation to the next step must be disabled when selected healthy sources count is zero. Re-enabling should occur when the user re-selects at least one source manually.
+
+- Cancellation semantics
+  - When the user navigates away from the Health Check step or cancels the wizard, mark the ImportSession (e.g., set health_checks_active: false or current_step != 'health_check'). HealthCheck jobs must check this flag and not broadcast or mutate UI-visible session state if the session is no longer active.
+  - Jobs may still persist logs/errors for auditing but must not broadcast to the now-expired wizard UI.
+
+- Concurrency and safety
+  - Jobs updating ImportSession should be resilient to concurrent updates: use transactions and row-level locking (or optimistic update with retries) to modify parsed_sources and selected_source_ids safely.
+  - Handle cases where a duplicate source is discovered or a source is removed from selected_source_ids mid-check gracefully: persist the health result but ensure selected_source_ids and UI state remain consistent.
+
+- Follow SourceMonitor conventions
+  - Adhere to engine style: controller and job naming, Solid Queue enqueue patterns, Turbo Stream rendering, Tailwind/Turbo Frame based partials, accessibility.
+  - Enforce admin-only access around enqueuing health checks via existing authentication hooks.
+  - Strong-params/sanitization for any controller endpoints that trigger health checks or cancel them.
+
+Out of scope
+
+- Implementing retry logic for failed health checks (explicitly out of scope in PRD).
+- Building or changing the visual design of the Health Check UI beyond status icon/error and progress updates.
+- Global realtime channel/refactor: use engine broadcaster patterns rather than introducing an unrelated pubsub mechanism.
+- Changes to sourcemon_sources table or source creation logic—duplicate detection remains feed-URL-only and uses existing source records.
+
+Suggested research
+
+- Inspect existing health check code and controller:
+  - app/controllers/source_monitor/source_health_checks_controller.rb (how single-source health checks are performed and broadcast).
+  - Any health-check services used for per-source health (search for "health" related services or logs).
+- Review Solid Queue job patterns used in repo for enqueueing and job implementation:
+  - app/jobs/source_monitor/* (FetchFeedJob, ScrapeItemJob) for job structure, error handling, and broadcasting style.
+- Review realtime/Turbo broadcaster and stream responder conventions:
+  - lib/source_monitor/realtime/broadcaster.rb
+  - lib/source_monitor/turbo_streams/stream_responder.rb
+  - existing Turbo stream presenters for row updates (e.g., sources presenters under lib/*).
+- Inspect ImportSession model and migration (app/models/source_monitor/import_session.rb or equivalent) to confirm parsed_sources and selected_source_ids shapes and keys used to address parsed entries.
+- Find existing wizard/ImportSession views or step partials to see expected Turbo Frame names and where to render row partials and the progress indicator (app/views/source_monitor/imports/* or similar).
+
+Use these guidelines to implement the Health Check step backend jobs and Turbo Stream UI updates; leave view markup and job wiring details to the implementer following the repository's conventions.

data/tasks/completed/opml-import-wizard/task-04/requirements.md

@@ -0,0 +1,17 @@
+#### Goals  
+
+- Provide a Health Check step that enqueues an individual background health-check job for each selected source.  
+- Display a live-updating results table with status icons (healthy/unhealthy) updated via Turbo Streams as jobs complete.  
+- Show a progress indicator (counts completed/total) and unselect unhealthy sources by default while allowing users to re-select them.  
+- Block navigation forward if no healthy sources remain selected (unless user manually re-selects an unhealthy source).
+
+#### Technical Considerations  
+
+- Use Solid Queue for background jobs; create or reuse a HealthCheckJob that runs existing feed health check logic and broadcasts results on a Turbo Stream channel/topic for the ImportSession or user.  
+- Store health check results back into ImportSession.parsed_sources entries (health_status, error) so step state is persistent and viewable if refreshed while session exists. Ensure ImportSession is implemented with a standard integer-based user reference (user_id as integer).  
+- Implement server-side cancellation semantics: if the user navigates away from the health-check step before all jobs finish, mark in ImportSession and ensure any remaining jobs do not update transient UI state or re-broadcast for an expired session.  
+- Ensure progress updates and row updates follow the engine’s Turbo Streams conventions and use existing broadcasters patterns.
+
+#### Dependencies  
+
+- Build Preview Table & Selection Persistence (selected set to health-check)

data/tasks/completed/opml-import-wizard/task-05/instructions.md

@@ -0,0 +1,50 @@
+- **Context**
+  - Implement the "Configure" step in the OPML Import Wizard (Configure step of the multi-step wizard). This step must render the exact same input fields and validation behavior as the single-source creation form, and persist the validated bulk settings to ImportSession.bulk_settings (JSONB) so settings carry forward to Confirm and are applied uniformly to all selected sources.
+  - This task depends on the surrounding wizard shell and ImportSession persistence being present (ImportSession record storing parsed sources / selections). The Configure step is one Turbo Frame view inside the wizard route.
+
+- **Goals**
+  - Render the same input fields, field order, help text and inline validation as the single-source creation form.
+  - Persist validated bulk settings into ImportSession.bulk_settings (JSONB) tied to the ImportSession record for the current user.
+  - Prevent advancing to the Confirm step until required fields are valid; show inline validation errors in the Configure view.
+  - Ensure settings apply uniformly to all selected sources (no per-feed overrides).
+  - Follow SourceMonitor UI conventions (Tailwind, Turbo Frames, accessibility) and enforce admin-only access.
+
+- **Technical Guidelines**
+  - Reuse UI:
+    - Render the existing single-source form partial used by new/edit source flows (use the same partial(s) so HTML structure, labels, help text and classes are identical).
+    - Render the form inside the Configure step Turbo Frame used by the wizard layout and include the same error rendering (inline field errors).
+  - Params & Validation:
+    - Map the submitted form params to the same permitted attributes used for single-source creation (use the same strong-parameter logic as the SourcesController). Reuse SourceMonitor’s existing parameter sanitizer or the SourcesController's source_params as canonical source attributes.
+    - Validate required fields server-side using the same validation rules or service used for single-source creation before persisting to ImportSession.bulk_settings. If the engine has an existing form object/service for validating source attributes, reuse it; otherwise, perform the same attribute-level validations prior to saving the ImportSession.
+    - If validation fails, render the Configure step with the same inline error presentation and prevent progression.
+  - Persistence:
+    - Persist the resulting validated settings JSON into ImportSession.bulk_settings. The ImportSession record must be scoped to the current user (ImportSession.user_id must use integer-based ActiveRecord FK).
+    - Keep settings persisted when the user navigates between wizard steps (updates to ImportSession.bulk_settings replace previous values).
+    - Do not apply settings to sources yet — just store them until the final import job runs.
+  - Access control and security:
+    - Restrict the Configure step to authenticated admin users consistent with engine conventions.
+    - Use existing strong-parameter sanitization utilities. Do not accept arbitrary params; persist only the permitted source attributes structure to bulk_settings.
+  - UI/UX:
+    - Apply the same Tailwind classes and accessibility attributes as the single-source form.
+    - Ensure the Configure step form disables the Next/Confirm action until the server confirms the form is valid (server-side validation); show inline validation messages without toasts.
+    - Do not implement per-feed overrides — the UI must not expose per-source editing here.
+  - Integration shape:
+    - The Configure controller action should update ImportSession.bulk_settings and return a Turbo Frame response so the wizard UI updates in-place.
+    - Ensure any client-side flows follow existing Turbo Frame/Turbo Stream patterns used throughout the engine (no new client-side state store).
+  - Data shape:
+    - Persist settings as JSON that matches the attribute names used by sourcemon_sources creation (so the import job can apply them to each created source).
+  - Logging & errors:
+    - Surface validation errors inline. For unexpected save errors (DB or permission), render an actionable error message on the Configure step and do not advance.
+
+- **Out of scope**
+  - Implementing the final import background job, import history persistence, or broadcasting results.
+  - Per-feed (per-row) settings or overrides.
+  - Health check orchestration or changes to health-check results UI.
+  - Wizard shell routing, ImportSession model/migration creation (assume ImportSession exists and is integer-user-scoped), and Upload/Preview step parsing logic.
+  - Client-side persistence beyond Turbo Frames / ImportSession-backed persistence.
+
+- **Suggested research**
+  - Inspect the single-source form partial(s) used by new/edit source pages (app/views/source_monitor/sources/* — locate _form partial and new/edit templates) to reuse markup and error helpers.
+  - Review SourcesController#source_params and SourceMonitor::Security::ParameterSanitizer (or equivalent) to determine the exact permitted attributes shape to persist.
+  - Inspect the ImportSession model/schema and controller endpoints for updating session state (ImportSession.bulk_settings JSONB field) and confirm user scoping and allowed update patterns.
+  - Find examples in the engine of reusing partials and Turbo Frame responses for form steps (sources index/new flows and other wizard-like views) to match response patterns and accessibility/styling conventions.

data/tasks/completed/opml-import-wizard/task-05/requirements.md

@@ -0,0 +1,17 @@
+#### Goals  
+
+- Implement the Configure step that displays the exact same input fields as the single source creation form, applied uniformly to all selected sources.  
+- Persist validated bulk settings into ImportSession.bulk_settings so they carry forward to the Confirm step and are applied during import.  
+- Validate required fields and prevent progression until the form is valid.
+
+#### Technical Considerations  
+
+- Reuse the existing single-source form partial to render fields and validation messages; adapt controller params mapping to persist the resulting bulk settings JSON into ImportSession.bulk_settings. Ensure ImportSession uses an integer-based user reference (standard Rails user_id integer foreign key).  
+- Use the engine's existing parameter sanitization and validation patterns for source attributes.  
+- Ensure the form is accessible and styled with Tailwind consistent with the rest of the engine.  
+- Disallow per-feed overrides in this release; the stored settings apply uniformly to all selected sources during import.
+
+#### Dependencies  
+
+- Enqueue Health Checks & Turbo Stream UI (health results determine which sources are in the target set)  
+- Implement OPML Import Wizard Shell (wizard rendering and step navigation)

data/tasks/completed/opml-import-wizard/task-06/instructions.md

@@ -0,0 +1,92 @@
+**Context**
+
+- This task implements the Confirm step of the OPML Import Wizard and the background import job for SourceMonitor.
+- Upstream work (wizard shell, OPML parsing, preview/selection, health checks, bulk settings) will have populated an ImportSession record containing parsed_sources, selected_source_ids and bulk_settings.
+- The engine already uses Solid Queue for background jobs, Turbo Streams for realtime UI updates, and has existing source creation logic and Turbo broadcast/responders to follow.
+
+**Goals**
+
+- Render a Confirm step view that shows a final summary: the list of selected sources to be imported and the bulk settings that will be applied.
+- When the user confirms, enqueue a background import job (Solid Queue / ActiveJob) that:
+  - Iterates selected sources, creates sources individually (reusing the engine’s existing source creation logic/service where available).
+  - Records per-source outcomes: successes, failures (with error details), and skipped duplicates (feed URL match).
+  - Persists an ImportHistory AR record containing imported_sources, failed_sources, skipped_duplicates, bulk_settings, started_at, completed_at, and reference to the performing user (integer user_id).
+  - Handles idempotency/races (e.g., ActiveRecord::RecordNotUnique) and records appropriate skipped/failed state.
+- Broadcast import completion via Turbo Streams so a static confirmation (counts and a table of failures) is visible on the Sources index; persist ImportHistory so the confirmation is queryable on subsequent visits.
+
+**Technical Guidelines**
+
+- Models & DB
+  - Add ImportHistory ActiveRecord model + migration.
+    - Use Rails default integer primary keys and integer user reference: e.g. t.references :user, foreign_key: true (integer FK).
+    - Columns: imported_sources (jsonb), failed_sources (jsonb), skipped_duplicates (jsonb), bulk_settings (jsonb), started_at (t.datetime), completed_at (t.datetime), created_at/updated_at.
+    - Index created_at for query performance.
+  - Do not change existing sourcemon_sources table schema. Duplicate detection must use feed_url matching against that table.
+
+- Background Job
+  - Implement an import ActiveJob under app/jobs/source_monitor/import_opml_job.rb (or similar) that uses the Solid Queue adapter (follow existing job naming/queue conventions in repo).
+  - Job responsibilities:
+    - Load ImportSession or be supplied with the ImportSession id and ImportHistory id to update status.
+    - Persist started_at at job start and completed_at at finish to the ImportHistory record.
+    - For each selected parsed source:
+      - Skip any source that already exists by exact feed_url match — mark as skipped duplicate and include feed_url and reason in skipped_duplicates.
+      - Attempt to create the source by calling the engine’s existing source-creation service/object (discover and reuse service; do NOT reimplement creation logic in the job).
+      - On successful create: append a concise representation (id, feed_url, title) to imported_sources.
+      - On exception:
+        - If ActiveRecord::RecordNotUnique (race / concurrent import), treat as skipped duplicate (record to skipped_duplicates) rather than failure.
+        - Otherwise, record failure with feed_url, error class and message in failed_sources.
+      - Ensure partial failures do not stop processing of remaining sources; process all selected sources and collect per-source results.
+    - Save ImportHistory with final arrays and timestamps.
+  - Job must be idempotent for retries: detect already-created feed_url and avoid creating duplicates.
+  - Use transactions cautiously: individual source creations may be wrapped in their own transaction so one source failure doesn’t roll back the whole batch.
+
+- Controller & Confirm Step
+  - Add Confirm step controller action(s) to the wizard controller that:
+    - Renders summary table of ImportSession.selected_source_ids joined with ImportSession.parsed_sources to show feed_url, title, and applied bulk_settings.
+    - On confirm POST, create an ImportHistory record (with user reference and bulk_settings), enqueue the import job (passing ImportSession id and ImportHistory id), and return an immediate Turbo Stream/html response that redirects user back to the Sources index (or shows a confirmation note).
+    - Enforce admin-only access using the engine’s existing authentication hooks (follow existing controller patterns; e.g., inherit SourceMonitor::ApplicationController and use configured authentication).
+    - Sanitize permitted params as per engine conventions.
+
+- Broadcasting & UI persistence
+  - When job completes, broadcast a Turbo Stream message so the Sources index can render a static confirmation section (counts and failures).
+    - Reuse engine’s existing broadcasting patterns (lib/source_monitor/realtime/* and lib/source_monitor/turbo_streams/stream_responder.rb) — create a suitable Turbo Stream target (e.g., turbo_stream_from "source_monitor_import_histories" or reuse "source_monitor_sources") and broadcast a partial that renders ImportHistory summary.
+  - Ensure the ImportHistory record itself is persisted and can be queried from the Sources index UI (Sources index should be able to list recent ImportHistory entries). For this task: ensure ImportHistory model, migration and broadcast exist; UI change for Sources index to consume ImportHistory should read from ImportHistory.where(user: current_user). Order and exact UI placement follow engine conventions.
+
+- Error handling & idempotency
+  - Duplicate detection must be feed_url exact match against sourcemon_sources before attempting create; handle race conditions (RecordNotUnique) gracefully by recording skipped_duplicates.
+  - For any unexpected exceptions record error_class, error_message and minimal backtrace (if desired) in failed_sources; do not leak raw exceptions to clients—render error details in admin UI only.
+  - Ensure the import job cannot create duplicate sources if it is run multiple times (use pre-check + rescue RecordNotUnique).
+
+- Conventions & Integration
+  - Follow existing Rails engine patterns: controllers respond to HTML and turbo_stream, use Turbo Frames for partial updates, Tailwind styling, accessibility.
+  - Use Solid Queue / ActiveJob consistent queue naming conventions (look up SourceMonitor.config.queue_name_for or existing job classes to match queue naming).
+  - Use strong parameter sanitization utilities already present in the engine (SourceMonitor::Security::ParameterSanitizer or controller helpers).
+  - Tests: add unit/functional tests for ImportHistory model and job behavior, and a system/controller test covering confirm POST enqueuing job and final persisted ImportHistory.
+
+**Out of scope**
+
+- Implementing or modifying Preview/Upload/Health steps (assume ImportSession contains necessary state).
+- Per-feed settings overrides (bulk settings apply uniformly).
+- Real-time progress bar for the import job (only final completion broadcast and static confirmation required).
+- Advanced retry/recovery logic for failed imports beyond basic idempotency and recording errors.
+- Large-scale UI redesign—only add Confirm step view and ensure Sources index can display ImportHistory entries.
+
+**Suggested research (files/areas to inspect before implementing)**
+
+- Controller patterns and Turbo Stream responders:
+  - app/controllers/source_monitor/sources_controller.rb
+  - lib/source_monitor/turbo_streams/stream_responder.rb
+  - any existing controllers that render summary/confirmation messages
+- Background job examples and Solid Queue usage:
+  - app/jobs/source_monitor/* (FetchFeedJob, ScrapeItemJob) and how they declare queues
+  - lib/source_monitor/jobs/solid_queue_metrics.rb and job conventions
+- Real-time broadcasting patterns:
+  - lib/source_monitor/realtime/broadcaster.rb
+  - lib/source_monitor/dashboard/turbo_broadcaster.rb
+- ImportSession usage and fields:
+  - the ImportSession model and where parsed_sources, selected_source_ids, bulk_settings are stored (migration and model file from ImportSession task)
+- Source creation logic/service objects:
+  - app/controllers/source_monitor/sources_controller.rb#create and any service objects under lib/source_monitor or app/services related to creating a Source (prefer reuse)
+- Tests demonstrating handling of ActiveRecord::RecordNotUnique or duplicate create patterns (see item creation duplicates in lib/source_monitor/items/item_creator.rb for pattern)
+
+Use the repository’s existing patterns and helpers; do not introduce new auth/permission systems—leverage engine’s existing admin-only checks and Turbo Stream broadcasting conventions.

data/tasks/completed/opml-import-wizard/task-06/requirements.md

@@ -0,0 +1,21 @@
+#### Goals  
+
+- Implement the Confirm step that shows a summary table of selected sources and the applied bulk settings for final review.  
+- When confirmed, enqueue a background import job that creates sources individually, records per-source success/failure and skipped duplicates, and persists an ImportHistory record with results.  
+- After job completion, ensure the user can view a static confirmation message on the Sources index with counts and a table of failures; guarantee that ImportHistory entries are queryable from the UI later.
+
+#### Technical Considerations  
+
+- Use Solid Queue for the import background job; the job should iterate selected sources, create sources individually (reusing Sources creation logic/service objects), capture and persist errors per source, and skip duplicates discovered by feed URL matching.  
+- Create an ImportHistory ActiveRecord model/migration with JSONB columns for imported_sources, failed_sources, skipped_duplicates, bulk_settings, started_at and completed_at. Implement ImportHistory to reference the performing user using standard integer-based ActiveRecord IDs (i.e., user_id as an integer foreign key). Ensure migrations and model associations use Rails default integer id conventions rather than UUIDs.  
+- Broadcast import completion via Turbo Streams (targeting Sources index or an ImportHistory feed) so users see static messaging and results; Ensure the static confirmation is available on next visit to Sources index even if the user navigated away during processing.  
+- Ensure transactional and idempotency considerations for creation logic: handle ActiveRecord::RecordNotUnique or race conditions gracefully and record appropriate failure/skipped state.  
+- Enforce admin-only access and strong parameter sanitization.
+
+#### Dependencies  
+
+- Implement OPML Import Wizard Shell  
+- Add OPML Upload & Synchronous Parsing  
+- Build Preview Table & Selection Persistence  
+- Enqueue Health Checks & Turbo Stream UI  
+- Bulk Settings Form Reusing Single Source Partial

data/tasks/completed/phase_17_01_complexity_audit_2025-10-12.md

@@ -0,0 +1,62 @@
+# Phase 17.01 Complexity Audit (2025-10-12)
+
+## Component Inventory
+
+- **Controllers (7)**: `SourceMonitor::ApplicationController`, `DashboardController`, `SourcesController`, `ItemsController`, `FetchLogsController`, `ScrapeLogsController`, `HealthController`.
+- **Models (6)**: `SourceMonitor::ApplicationRecord`, `Source`, `Item`, `ItemContent`, `FetchLog`, `ScrapeLog`.
+- **Jobs (6)**: `ApplicationJob`, `FetchFeedJob`, `ScrapeItemJob`, `ScheduleFetchesJob`, `ItemCleanupJob`, `LogCleanupJob`.
+- **Service/Support Modules (34)**: Analytics (`Analytics::SourceFetchIntervalDistribution`, `Analytics::SourceActivityRates`), Dashboard (`Dashboard::Queries`, `Dashboard::TurboBroadcaster`, `Dashboard::UpcomingFetchSchedule`), Fetching (`Fetching::FeedFetcher`, `Fetching::FetchRunner`, `Fetching::FetchError`, `Fetching::RetryPolicy`), HTTP (`HTTP`), Instrumentation (`Instrumentation`, `Events`, `Metrics`), Items (`Items::ItemCreator`, `Items::RetentionPruner`), Jobs support (`Jobs::SolidQueueMetrics`, `Jobs::Visibility`), Realtime (`Realtime`, `Realtime::Adapter`, `Realtime::Broadcaster`), Scheduler (`Scheduler`, `Scraping::Scheduler`), Scraping (`Scraping::Enqueuer`, `Scraping::ItemScraper`, `Scrapers::Base`, `Scrapers::Readability`, `Scrapers::Fetchers::HttpFetcher`, `Scrapers::Parsers::ReadabilityParser`), Security (`Security::Authentication`, `Security::ParameterSanitizer`), Configuration (`Configuration`, `ModelExtensions`, `feedjira` extensions, `version`).
+- **View Components**: none defined under `app/components`.
+- **Front-End Assets**: Stimulus bootstrapper `app/assets/javascripts/source_monitor/application.js`, controllers (`controllers/notification_controller.js`, `controllers/async_submit_controller.js`, `controllers/dropdown_controller.js`), transition shim `dropdown_transition_shim.js`, sprockets manifest `application.js`, Tailwind sources `app/assets/tailwind/application.css`, compiled Tailwind bundle `app/assets/builds/tailwind.css` (~1.4k lines), stylesheet manifest `app/assets/stylesheets/source_monitor/application.css`, Turbo-aware notification partials under `app/views/source_monitor/shared/` (not exhaustively listed).
+
+## Controller Review (17.01.02)
+
+- `SourcesController#index` mixes search sanitization, analytics aggregation, and bucket math, making the action broader than RESTful list concerns. Extracting the distribution helpers (`extract_fetch_interval_filter`, `distribution_sources_scope`, `find_matching_bucket`) into a presenter or query object would clarify the controller's responsibility (`app/controllers/source_monitor/sources_controller.rb:11-199`).
+- Both `SourcesController` and `ItemsController` duplicate an identical `sanitized_search_params` implementation; moving it into a concern would reduce coupling to parameter structure (`app/controllers/source_monitor/sources_controller.rb:172-194`, `app/controllers/source_monitor/items_controller.rb:120-140`).
+- Manual pagination in `ItemsController#index` reimplements offset/limit logic and page bounds; adopting a pagination helper like Pagy would cut duplication and guard against off-by-one errors (`app/controllers/source_monitor/items_controller.rb:21-29`).
+- Non-RESTful actions (`fetch`/`retry` on Sources, `scrape` on Items) are correctly routed but include complex Turbo Stream responses inline; extracting to responder objects would improve reusability and testability (`app/controllers/source_monitor/sources_controller.rb:65-168`, `app/controllers/source_monitor/items_controller.rb:38-83`).
+- `FetchLogsController` and `ScrapeLogsController` share similar filtering logic but diverge in integer parsing; consider consolidating to a shared concern to keep future filter changes in one place (`app/controllers/source_monitor/fetch_logs_controller.rb:5-23`, `app/controllers/source_monitor/scrape_logs_controller.rb:5-45`).
+
+## Model and Service Review (17.01.03)
+
+- `SourceMonitor::Source` handles normalization, sanitization, health defaults, and validation in a single model class. The sanitization callbacks (`sanitize_user_inputs`) could move to a concern shared with other models to avoid repeating security rules elsewhere (`app/models/source_monitor/source.rb:47-151`).
+- `SourceMonitor::Item` embeds URL normalization and soft-delete logic. Extracting the normalization routine (`normalize_urls`) to a reusable helper would support other URL-bearing models (`app/models/source_monitor/item.rb:52-133`).
+- `Fetching::FetchRunner` coordinates locking, fetch execution, retention pruning, retry scheduling, and scrape enqueueing. Splitting concurrency/locking into a collaborator would keep the runner focused on orchestration (`lib/source_monitor/fetching/fetch_runner.rb:39-186`).
+- `Scraping::ItemScraper` owns adapter resolution, persistence, logging, and error translation in one class; consider splitting adapter lookup and persistence so new adapters can be added without touching the transaction workflow (`lib/source_monitor/scraping/item_scraper.rb:28-200`).
+- `Items::RetentionPruner` mixes strategy selection with batch execution and counter maintenance; moving strategy-specific behavior into separate classes would make the soft-delete vs destroy paths easier to evolve (`lib/source_monitor/items/retention_pruner.rb:21-162`).
+- `Dashboard::Queries` executes multiple direct ActiveRecord calls every page load; caching or background precomputation for expensive counts (e.g., `FetchLog` and `ScrapeLog` limits) could improve dashboard responsiveness (`lib/source_monitor/dashboard/queries.rb:9-87`).
+
+## Jobs, Workers, and Scheduling (17.01.04)
+
+- `FetchFeedJob` re-raises all errors except concurrency conflicts, delegating retries to ActiveJob defaults. Considering explicit retry/backoff for transient failures would align with Solid Queue best practices (`app/jobs/source_monitor/fetch_feed_job.rb:5-19`).
+- `ScrapeItemJob` performs direct `update_columns` writes inside locks; wrapping these state changes in small service objects (or reusing `Scraping::Enqueuer` helpers) would reduce raw SQL usage and prevent silent attribute drift (`app/jobs/source_monitor/scrape_item_job.rb:21-56`).
+- Cleanup jobs (`ItemCleanupJob`, `LogCleanupJob`) repeat option normalization logic; a shared base utility could centralize casting rules for command-line invocations (`app/jobs/source_monitor/item_cleanup_job.rb:12-74`, `app/jobs/source_monitor/log_cleanup_job.rb:14-78`).
+- `Scheduler.run` only enqueues fetches; adding observability (e.g., instrumentation events around locked IDs) would help diagnose scheduling gaps (`lib/source_monitor/scheduler.rb:7-55`).
+- Recurring schedule references in `config/recurring.yml` rely on Solid Queue CLI defaults; documenting these in `SourceMonitor.configure` and ensuring Mission Control checks respect them would reduce drift across host apps.
+
+## Front-End Asset Review (17.01.05)
+
+- Stimulus controllers register themselves on the global `window` object and expect external UMD bundles (`app/assets/javascripts/source_monitor/application.js:1-32`, `app/assets/javascripts/source_monitor/controllers/notification_controller.js:1-55`). Migrating to Importmap or ESbuild modules would eliminate global leakage and simplify testing.
+- `dropdown_controller.js` assumes a global `StimulusDropdown` shim; when the dependency is absent, dropdowns silently fail. Guarding this with feature detection and graceful degradation (e.g., fallback to CSS-only menus) would improve resilience (`app/assets/javascripts/source_monitor/controllers/dropdown_controller.js:1-14`).
+- The transition shim provides minimal show/hide toggling and bypasses Tailwind’s transition helpers; aligning it with the stimulus-use-transition API would avoid divergence from upstream updates (`app/assets/javascripts/source_monitor/dropdown_transition_shim.js:1-25`).
+- A compiled Tailwind bundle (`app/assets/builds/tailwind.css`) lives in source control; documenting the build step and ensuring the bundle is regenerated during releases will prevent stale utility classes.
+- No dedicated asset linting or bundler checks run in CI; adding `yarn lint` or equivalent (once build tooling is selected) would catch drift early.
+
+## Metrics & Hotspots (17.01.06)
+
+- `bundle exec rubocop --format offenses` reports **366 offenses across 44 files**, overwhelmingly `Layout/SpaceInsideArrayLiteralBrackets` (352 autocorrectable). Cleaning whitespace rules and enabling CI enforcement will quickly improve signal-to-noise.
+- `bundle exec brakeman` could not run because the gem is not in the bundle; adding it to the development group will enable baseline security scanning.
+- SimpleCov is not configured (`test/test_helper.rb` lacks coverage hooks), so historical coverage trends are unavailable; enabling coverage reports will support future regressions analysis.
+- Hotspot summary:
+  - Controllers: duplicated sanitization helper and complex inline Turbo responses (`SourcesController#index`, `ItemsController#scrape`).
+  - Services: `Fetching::FetchRunner` and `Scraping::ItemScraper` concentrate orchestration, logging, and persistence logic in single classes, making them prime candidates for refactoring.
+  - Jobs: Cleanup jobs’ duplicated option parsing invites inconsistencies when adding CLI flags.
+  - Front-end: reliance on global Stimulus objects without build-time enforcement increases risk of runtime errors when assets load out of order.
+
+## Recommended Next Steps
+
+1. Extract shared request-sanitization and pagination helpers to reduce duplication in controllers.
+2. Break down `FetchRunner` and `ItemScraper` into smaller collaborators (e.g., concurrency guard, adapter resolver, persistence handler) to improve testability.
+3. Add Rubocop autocorrect (spacing rules) and wire Rubocop/Brakeman into CI to maintain quality gates.
+4. Decide on an asset pipeline (Importmap vs ESBuild) and refactor Stimulus controllers out of the global namespace, documenting required UMD shims until migration completes.
+5. Enable SimpleCov and capture baseline coverage for future phases, ensuring retention and scraping flows stay covered.

data/tasks/completed/phase_17_02_complexity_findings_2025-10-12.md

@@ -0,0 +1,74 @@
+# Phase 17.02 Complexity Findings Report (2025-10-12)
+
+## 1. Issue Catalog
+
+### Controllers
+- **Duplicated parameter sanitization** — `app/controllers/source_monitor/items_controller.rb:120` and `app/controllers/source_monitor/sources_controller.rb:172` both implement identical `sanitized_search_params`; keeping these in each controller risks divergence in future filters.
+- **Manual pagination and query orchestration** — `app/controllers/source_monitor/items_controller.rb:13-33` mixes pagination math, search setup, and view state assembly, crowding the action and duplicating paging behaviour elsewhere.
+- **Inline analytics aggregation** — `app/controllers/source_monitor/sources_controller.rb:11-27` performs distribution bucketing and analytics queries in-controller, making the action heavy and difficult to unit test.
+- **Turbo Stream composition embedded in controllers** — `app/controllers/source_monitor/items_controller.rb:38-83` and `app/controllers/source_monitor/sources_controller.rb:140-168` contain sizeable arrays of Turbo Stream operations; changes to toast behaviour or DOM IDs now require controller edits.
+- **Parallel filter logic in logs controllers** — `app/controllers/source_monitor/fetch_logs_controller.rb:5-23` and `app/controllers/source_monitor/scrape_logs_controller.rb:5-45` share similar scoping patterns but diverge in parameter casting, inviting subtle inconsistencies.
+
+### Models & Services
+- **`SourceMonitor::Source` doing too much** — `app/models/source_monitor/source.rb:20-150` handles sanitization, URL normalization, defaulting, and health thresholds; the breadth complicates reuse of sanitization rules by other models.
+- **`SourceMonitor::Item` URL normalization** — `app/models/source_monitor/item.rb:52-133` embeds URL parsing logic; other URL-backed models cannot reuse the behaviour without duplication.
+- **`Fetching::FetchRunner` orchestration sprawl** — `lib/source_monitor/fetching/fetch_runner.rb:39-186` owns locking, state transitions, retention pruning, scrape enqueueing, retry scheduling, and instrumentation dispatch in one class.
+- **`Scraping::ItemScraper` responsibilities** — `lib/source_monitor/scraping/item_scraper.rb:28-200` combines adapter discovery, HTTP metadata handling, persistence, logging, and event publication within a single method flow.
+- **`Items::RetentionPruner` strategy branching** — `lib/source_monitor/items/retention_pruner.rb:21-162` interleaves strategy selection with batch execution and counter maintenance, making soft-delete vs destroy paths harder to evolve.
+- **Dashboard query coupling** — `lib/source_monitor/dashboard/queries.rb:17-87` fires multiple fresh queries for every request without caching or batching, and mixes routing helpers with aggregation logic.
+
+### Jobs, Workers, Scheduling
+- **Limited retry semantics** — `app/jobs/source_monitor/fetch_feed_job.rb:5-19` rescues only concurrency errors; transient network failures rely on defaults rather than an explicit retry/backoff plan.
+- **State updates inside jobs** — `app/jobs/source_monitor/scrape_item_job.rb:25-45` issues `update_columns` inside locks; behaviour overlaps with `Scraping::Enqueuer`, risking drift between enqueue-time and job-time state handling.
+- **Duplicated option normalization** — `app/jobs/source_monitor/item_cleanup_job.rb:18-69` and `app/jobs/source_monitor/log_cleanup_job.rb:18-71` repeat the same option-shaping logic, increasing maintenance costs as CLI flags expand.
+- **Scheduler blind spots** — `lib/source_monitor/scheduler.rb:7-49` enqueues fetches but emits no instrumentation or metrics, limiting observability when schedule gaps occur.
+
+### Front-End Assets
+- **Global Stimulus registration** — `app/assets/javascripts/source_monitor/application.js:1-32` depends on global `window.Stimulus`, diverging from modern ES module patterns and risking double registration.
+- **Fragile dropdown dependency** — `app/assets/javascripts/source_monitor/controllers/dropdown_controller.js:1-14` assumes `window.StimulusDropdown` exists; missing UMD bundle breaks dropdowns without fallback.
+- **Transition shim divergence** — `app/assets/javascripts/source_monitor/dropdown_transition_shim.js:1-24` implements a custom `useTransition` that may fall behind the upstream library API.
+- **Compiled Tailwind artefact** — `app/assets/builds/tailwind.css` (~1.4k lines) is committed without guardrails ensuring regeneration during releases.
+- **No asset linting/build verification** — no scripted checks ensure controllers or Tailwind bundles stay current, leaving runtime errors undetected until manual testing.
+
+### Tooling Gaps
+- **Rubocop violations** — `bundle exec rubocop --format offenses` reports 366 issues (352 auto-correctable spacing offenses), indicating style drift.
+- **Brakeman absent from bundle** — Running `bundle exec brakeman` fails because the gem is not declared; security checks are currently blocked.
+- **No coverage tracking** — `test/test_helper.rb` lacks SimpleCov hooks, so coverage regressions are invisible.
+
+## 2. Recommended Remediation (Controllers, Models/Services, Jobs)
+- **Controllers**: Extract shared parameter sanitization and pagination into concerns or service objects; move analytics aggregation into query objects; introduce presenters/responders for Turbo Stream responses.
+- **Models/Services**: Create reusable sanitization utilities (e.g., `SourceMonitor::Sanitization`), encapsulate URL normalization in a dedicated module, break `FetchRunner` into concurrency, state, and follow-up collaborators, and separate `ItemScraper` responsibilities into adapter resolver, persistence handler, and logger.
+- **Jobs**: Align enqueue/job state transitions by reusing shared helpers, define custom retry strategies for transient errors, centralize option parsing for cleanup jobs, and instrument scheduler runs with metrics events.
+
+## 3. DRY & Complexity Reduction Strategies
+1. **Shared sanitization & normalization modules** — Provide mixins for parameter and URL sanitation to reuse across controllers and models.
+2. **Pagination abstraction** — Introduce a pagination service or adopt Pagy/Kaminari to eliminate manual offset math across controllers.
+3. **Command objects for Turbo responses** — Wrap Turbo Stream payload assembly in dedicated builders for reuse between controllers and tests.
+4. **Service decomposition** — Apply single-responsibility classes (lock manager, retention handler, scrape enqueuer) to reduce the size and branching of core orchestration services.
+5. **Instrumentation helpers** — Centralize logging/instrumentation patterns to avoid duplicating JSON payload composition across jobs and services.
+
+## 4. Front-End Package & View-Layer Recommendations
+- Migrate Stimulus controllers to modules (ESBuild or Importmap) while keeping a compatibility shim until host apps adopt the bundle.
+- Provide defensive checks or progressive enhancement fallbacks when optional UMD dependencies (e.g., `StimulusDropdown`) are missing.
+- Replace the custom transition shim with the maintained `@hotwired/stimulus-use` package or document divergence to keep behaviour in sync.
+- Add a lightweight asset build verification step (e.g., `bin/rails test:assets` or `yarn lint`) and document Tailwind rebuild requirements in contributor guides.
+- Audit Turbo Stream partials to ensure DOM IDs referenced in controllers are declared in a single presenter to avoid coupling across views.
+
+## 5. Prioritization & Sequencing
+| Issue | Area | Effort | Impact | Priority |
+| --- | --- | --- | --- | --- |
+| Decompose `FetchRunner` responsibilities | Fetching services | Medium | High | P1 |
+| Shared controller sanitization/pagination helpers | Controllers | Low | High | P1 |
+| Align Scraping enqueue vs job state handling | Jobs/Services | Medium | High | P1 |
+| Formalize Stimulus asset pipeline | Front-end | Medium | Medium | P2 |
+| Consolidate cleanup job option parsing | Background jobs | Low | Medium | P2 |
+| Introduce Rubocop/Brakeman & coverage baselines | Tooling | Low | Medium | P2 |
+| Replace `ItemScraper` adapter/persistence monolith | Scraping services | High | High | P1 |
+| Dashboard query caching | Dashboard | Medium | Medium | P3 |
+| Tailwind rebuild automation | Front-end | Low | Medium | P2 |
+
+P1 items should seed Phase 17.03 workstreams; P2 items follow once shared utilities exist; P3 items can ride along with subsequent dashboard improvements.
+
+## 6. Deliverables
+- Saved this findings report at `.ai/phase_17_02_complexity_findings_2025-10-12.md`.
+- Source material cross-referenced with Rails controller best practices from the official guides.

data/tasks/completed/phase_17_03_refactor_plan_2025-10-12.md

@@ -0,0 +1,37 @@
+# Phase 17.03 Refactoring Execution Plan (2025-10-12)
+
+## Workstreams & Owners (17.03.01)
+- **Controllers & Views** — Owner: D. Demchuk — deliver sanitized search helper, shared pagination, and Turbo presenter layer.
+- **Core Services** — Owner: J. Patel — modularize FetchRunner, ItemScraper, and RetentionPruner responsibilities.
+- **Jobs & Scheduling** — Owner: L. Nguyen — standardize scraping job state transitions, cleanup job option parsing, and scheduler instrumentation/retry policies.
+- **Front-End Pipeline** — Owner: M. Garcia — modernize Stimulus packaging, dropdown fallback, transition shim, and Tailwind rebuild automation.
+- **Tooling & Metrics** — Owner: QA Guild — introduce Rubocop/Brakeman gating, SimpleCov baseline, and asset/test automation.
+
+## Success Criteria & Coverage (17.03.02)
+- **Controllers & Views**: Shared sanitization module reused by sources/items/log controllers; pagination helper with unit tests; Turbo presenter unit specs plus system regression for toast delivery.
+- **Core Services**: FetchRunner split into lock manager + orchestration service with unit coverage and integration test updates; ItemScraper extraction with adapter contract tests; RetentionPruner strategies isolated with unit coverage for destroy/soft delete paths.
+- **Jobs & Scheduling**: ScrapeItemJob leverages shared state helper with job tests; cleanup jobs share option parser module with unit tests; scheduler instrumentation emits event verified by integration test.
+- **Front-End Pipeline**: Stimulus controllers load via import map/build pipeline with smoke test in system suite; dropdown fallback verified with JS integration; Tailwind rebuild command documented and executed in CI.
+- **Tooling & Metrics**: Rubocop enforced in CI (passing run recorded); Brakeman baseline added; SimpleCov coverage target ≥90% for new code; asset lint/test tasks integrated.
+
+## Sequencing & Dependencies (17.03.03)
+1. **Tooling & Metrics** (enabling guardrails before refactors).
+2. **Controllers & Views** (shared helpers reduce duplication ahead of service changes).
+3. **Core Services** (builds on shared helpers for sanitization messaging).
+4. **Jobs & Scheduling** (depends on service decomposition for scrape enqueue alignment).
+5. **Front-End Pipeline** (requires controller presenter outputs stabilized).
+
+## Monitoring & QA Checkpoints (17.03.04)
+- Add instrumentation dashboards for scheduler events before deploying job changes.
+- Run full system test suite after controller/view and front-end updates.
+- Enable CI stages: `lint`, `security`, `test`, `assets` with gating.
+- Schedule post-deploy log review for first run of refactored FetchRunner and ItemScraper.
+
+## Follow-Up Tickets (17.03.05)
+- Log backlog items for dashboard caching refinements and Mission Control integration once core refactors land.
+- Document migration guidance for host apps adopting new helpers/presenters (README + upgrade notes).
+- Track potential Pagy adoption experiment if manual pagination proves insufficient.
+
+## Deliverables
+- Execution plan stored here and referenced from roadmap tasks.
+- Serves as baseline for future 17.x phases.

data/tasks/completed/phase_21_01_log_consolidation_2025-10-15.md

@@ -0,0 +1,30 @@
+# Phase 21.01 – Log Consolidation Notes (2025-10-15)
+
+## Existing UI Inventory
+
+- `app/views/source_monitor/fetch_logs/index.html.erb`
+  - Columns: Started timestamp, Source link, HTTP status, Result badge, item delta summary, view link.
+  - Page-level filter: status (`All`, `Successes`, `Failures`).
+  - Metadata surfaced via badges and counts; table limited to 50 rows.
+- `app/views/source_monitor/fetch_logs/show.html.erb`
+  - Summary stack with source, HTTP, success flag, duration, item counts, timestamps, job id.
+  - Secondary panels for error details (message, backtrace), HTTP headers (pretty JSON), metadata.
+- `app/views/source_monitor/scrape_logs/index.html.erb`
+  - Columns: Started timestamp, Item link, Source link, Result badge, duration, view link.
+  - Same status filter bar; duration replaces item delta metrics.
+- `app/views/source_monitor/scrape_logs/show.html.erb`
+  - Summary with item/source links, adapter, HTTP, success flag, duration, content length, timestamps.
+  - Panels for error details and metadata.
+- Navigation (`app/views/layouts/source_monitor/application.html.erb`) now surfaces a single **Logs** entry pointing at the consolidated index.
+
+## Combined Layout Outline
+
+- Single "Logs" index with shared table:
+  - Columns: Started, Type badge, Subject (item or source), Source link, HTTP/adapter summary, Result badge with inline error text, Metrics (fetch deltas or scrape duration), Detail link.
+  - Rows expose `data-log-row` identifiers (`fetch-<id>` / `scrape-<id>`) for tests and Turbo.
+- Unified filter bar:
+  - Status toggle (All/Successes/Failures) and Log Type toggle (All Logs/Fetch Logs/Scrape Logs).
+  - Search form includes free-text box, timeframe select (24h/7d/30d), explicit start range inputs, and numeric `source_id`/`item_id` filters.
+  - Pagination uses `SourceMonitor::Pagination::Paginator` with Prev/Next links and a page indicator span.
+- Detail views remain type-specific (fetch/scrape) but back links now route through `source_monitor.logs_path` with the appropriate `log_type` preset.
+- Controller composes `SourceMonitor::Logs::Query` + `TablePresenter`; no shared concern is needed.

data/tasks/completed/release_checklist.md

@@ -0,0 +1,23 @@
+# SourceMonitor Release Checklist
+
+## Automated Gates
+- [ ] Confirm the `release_verification` GitHub Actions job succeeded on the target commit (generator smoke test, gem build/unpack, diff coverage).
+- [ ] Ensure the standard CI jobs (`lint`, `security`, `test`) are green.
+
+## Manual Validation
+- [ ] Run `bin/release VERSION` locally and inspect the console output for any skipped steps or warnings.
+- [ ] Inspect `pkg/source_monitor-<VERSION>.gem`:
+  - [ ] Verify file size is within expected bounds compared to the previous release.
+  - [ ] Extract the gem (`gem unpack`) and confirm `lib/` and templates are present.
+- [ ] Review `CHANGELOG.md` entry for the release and ensure the annotated tag message matches.
+- [ ] Preview the README locally with a Markdown renderer to confirm formatting.
+- [ ] Confirm `config/coverage_baseline.json` is up to date (rerun `bin/test-coverage` + `bin/update-coverage-baseline` if functional coverage changed).
+- [ ] Smoke test the disposable host app harness:
+  - [ ] `bin/rails test test/integration/release_packaging_test.rb`
+  - [ ] `bin/rails runner 'SourceMonitor::FetchFeedJob'` in the dummy app if new background job behavior shipped.
+- [ ] Verify any new environment variables or configuration flags are documented in `docs/installation.md` or `docs/deployment.md`.
+
+## Publication
+- [ ] Tag the release (`git tag v<VERSION> -F <TAG_FILE>`) and push the tag.
+- [ ] Publish the gem (`gem push pkg/source_monitor-<VERSION>.gem`).
+- [ ] Announce the release (Slack/email) with highlights and migration notes.