source_monitor 0.3.0 → 0.3.2

data/.claude/skills/sm-dashboard-widget/reference/dashboard-patterns.md

@@ -0,0 +1,304 @@
+# Dashboard Patterns Reference
+
+## Query Patterns
+
+### Single-Value Aggregation (StatsQuery pattern)
+Use `connection.select_value` for scalar results:
+
+```ruby
+def total_items_count
+  SourceMonitor::Item.connection.select_value(
+    "SELECT COUNT(*) FROM #{SourceMonitor::Item.quoted_table_name}"
+  ).to_i
+end
+```
+
+### Multi-Column Aggregation
+Use `connection.exec_query` with conditional aggregation:
+
+```ruby
+def source_counts
+  @source_counts ||= begin
+    SourceMonitor::Source.connection.exec_query(<<~SQL.squish).first || {}
+      SELECT
+        COUNT(*) AS total_sources,
+        SUM(CASE WHEN active THEN 1 ELSE 0 END) AS active_sources,
+        SUM(CASE WHEN (failure_count > 0) THEN 1 ELSE 0 END) AS failed_sources
+      FROM #{SourceMonitor::Source.quoted_table_name}
+    SQL
+  end
+end
+```
+
+### UNION ALL Cross-Table Query (RecentActivityQuery pattern)
+Combine events from multiple tables into a unified feed:
+
+```ruby
+def unified_sql_template
+  <<~SQL
+    SELECT resource_type, resource_id, occurred_at, ...
+    FROM (
+      #{fetch_log_sql}
+      UNION ALL
+      #{scrape_log_sql}
+      UNION ALL
+      #{item_sql}
+    ) AS dashboard_events
+    WHERE occurred_at IS NOT NULL
+    ORDER BY occurred_at DESC
+    LIMIT ?
+  SQL
+end
+```
+
+Each sub-query must:
+- Use the same column names/count (pad with `NULL AS column_name`)
+- Use `quoted_table_name` for all table references
+- Use string constants for type discriminators: `'fetch_log' AS resource_type`
+
+Sanitize with:
+```ruby
+ActiveRecord::Base.send(:sanitize_sql_array, [sql_template, limit])
+```
+
+### Time-Window Grouping (UpcomingFetchSchedule pattern)
+Group records into defined time buckets relative to a reference time:
+
+```ruby
+INTERVAL_DEFINITIONS = [
+  { key: "0-30",  label: "Within 30 minutes", min_minutes: 0,   max_minutes: 30 },
+  { key: "30-60", label: "30-60 minutes",     min_minutes: 30,  max_minutes: 60 },
+  # ...
+].freeze
+
+def definition_for(next_fetch_at)
+  minutes = (next_fetch_at - reference_time) / 60.0
+  INTERVAL_DEFINITIONS.find do |d|
+    minutes >= d[:min_minutes] && (d[:max_minutes].nil? || minutes < d[:max_minutes])
+  end
+end
+```
+
+### ActiveRecord Scope Query
+For simpler queries, use ActiveRecord directly:
+
+```ruby
+def fetches_today_count
+  SourceMonitor::FetchLog.where("started_at >= ?", start_of_day).count
+end
+```
+
+## Presenter Patterns
+
+### Collection Presenter
+Transforms an array of domain objects into view-model hashes:
+
+```ruby
+class RecentActivityPresenter
+  def initialize(events, url_helpers:)
+    @events = events
+    @url_helpers = url_helpers
+  end
+
+  def to_a
+    events.map { |event| build_view_model(event) }
+  end
+
+  private
+
+  attr_reader :events, :url_helpers
+
+  def build_view_model(event)
+    case event.type
+    when :fetch_log then fetch_event(event)
+    when :item      then item_event(event)
+    else                  fallback_event(event)
+    end
+  end
+
+  def fetch_event(event)
+    {
+      label: "Fetch ##{event.id}",
+      description: "#{event.items_created.to_i} created",
+      status: event.success? ? :success : :failure,
+      type: :fetch,
+      time: event.occurred_at,
+      path: url_helpers.fetch_log_path(event.id)
+    }
+  end
+end
+```
+
+### Static Actions Presenter
+Resolves route names to actual paths:
+
+```ruby
+class QuickActionsPresenter
+  def initialize(actions, url_helpers:)
+    @actions = actions
+    @url_helpers = url_helpers
+  end
+
+  def to_a
+    actions.map do |action|
+      {
+        label: action.label,
+        description: action.description,
+        path: url_helpers.public_send(action.route_name)
+      }
+    end
+  end
+end
+```
+
+## Turbo Stream Broadcast Patterns
+
+### Stream Subscription (in view)
+```erb
+<%= turbo_stream_from SourceMonitor::Dashboard::TurboBroadcaster::STREAM_NAME %>
+```
+
+### Broadcasting a Replace
+```ruby
+Turbo::StreamsChannel.broadcast_replace_to(
+  STREAM_NAME,                                    # channel name
+  target: "source_monitor_dashboard_stats",       # DOM element id
+  html: render_partial(                           # rendered HTML
+    "source_monitor/dashboard/stats",
+    stats: queries.stats
+  )
+)
+```
+
+### Rendering Partials Outside Request Context
+```ruby
+def render_partial(partial, locals)
+  SourceMonitor::DashboardController.render(
+    partial:,
+    locals:
+  )
+end
+```
+
+### Event Callback Registration
+```ruby
+def setup!
+  return unless turbo_streams_available?
+
+  register_callback(:after_fetch_completed, fetch_callback)
+  register_callback(:after_item_created, item_callback)
+end
+
+def register_callback(name, callback)
+  callbacks = SourceMonitor.config.events.callbacks_for(name)
+  return if callbacks.include?(callback)
+
+  SourceMonitor.config.events.public_send(name, callback)
+end
+```
+
+### Guard Against Missing Turbo
+```ruby
+def turbo_streams_available?
+  defined?(Turbo::StreamsChannel)
+end
+```
+
+## View Patterns
+
+### Card Container
+```erb
+<div id="source_monitor_dashboard_my_widget"
+     class="rounded-lg border border-slate-200 bg-white shadow-sm">
+  <div class="border-b border-slate-200 px-5 py-4">
+    <h2 class="text-lg font-medium">Title</h2>
+    <p class="mt-1 text-xs text-slate-500">Subtitle.</p>
+  </div>
+  <div class="divide-y divide-slate-100">
+    <!-- content rows -->
+  </div>
+</div>
+```
+
+### Stat Card (Collection Render)
+```erb
+<div class="grid gap-5 sm:grid-cols-2 xl:grid-cols-5">
+  <%= render partial: "stat_card", collection: [
+    { label: "Sources", value: stats[:total_sources], caption: "Total registered" },
+    { label: "Active",  value: stats[:active_sources], caption: "Fetching on schedule" }
+  ] %>
+</div>
+```
+
+Each stat card:
+```erb
+<div class="rounded-lg border border-slate-200 bg-white p-5 shadow-sm">
+  <dt class="text-xs font-medium uppercase tracking-wide text-slate-500"><%= stat_card[:label] %></dt>
+  <dd class="mt-2 text-3xl font-semibold text-slate-900"><%= value %></dd>
+  <p class="mt-1 text-xs text-slate-500"><%= stat_card[:caption] %></p>
+</div>
+```
+
+### Status Badge
+```erb
+<span class="inline-flex items-center rounded-full px-3 py-1 text-xs font-semibold
+  <%= event[:status] == :success ? 'bg-green-100 text-green-700' : 'bg-rose-100 text-rose-700' %>">
+  <%= event[:status] == :success ? "Success" : "Failure" %>
+</span>
+```
+
+### Empty State
+```erb
+<% if data.any? %>
+  <!-- content -->
+<% else %>
+  <div class="px-5 py-6 text-sm text-slate-500">
+    No data available yet.
+  </div>
+<% end %>
+```
+
+### Dashboard Layout Grid
+The dashboard uses a 3-column grid at large breakpoints:
+```erb
+<section class="grid gap-6 lg:grid-cols-3">
+  <div class="lg:col-span-2 space-y-6">
+    <!-- main content (recent activity, fetch schedule) -->
+  </div>
+  <div class="space-y-6">
+    <!-- sidebar (job metrics, quick actions) -->
+  </div>
+</section>
+```
+
+## Metrics Integration
+
+Every query method in the facade emits:
+- `dashboard_{name}_duration_ms` -- execution time gauge
+- `dashboard_{name}_last_run_at_epoch` -- timestamp of last execution
+- Widget-specific gauges (e.g., `dashboard_stats_total_sources`)
+
+```ruby
+def measure(name, metadata = {})
+  started_at = monotonic_time
+  result = yield
+  duration_ms = ((monotonic_time - started_at) * 1000.0).round(2)
+
+  ActiveSupport::Notifications.instrument("source_monitor.dashboard.#{name}", ...)
+  SourceMonitor::Metrics.gauge(:"dashboard_#{name}_duration_ms", duration_ms)
+
+  result
+end
+```
+
+## Naming Conventions
+
+| Layer | Naming Pattern | Example |
+|-------|----------------|---------|
+| Query class | `{Widget}Query` | `StatsQuery` |
+| Presenter | `{Widget}Presenter` | `RecentActivityPresenter` |
+| Struct | `{Widget}::Event` or `{Widget}::Group` | `RecentActivity::Event` |
+| View partial | `_snake_case.html.erb` | `_recent_activity.html.erb` |
+| Turbo target ID | `source_monitor_dashboard_{snake_case}` | `source_monitor_dashboard_stats` |
+| Metric name | `dashboard_{snake_case}_{metric}` | `dashboard_stats_total_sources` |
+| Notification | `source_monitor.dashboard.{name}` | `source_monitor.dashboard.stats` |

data/.claude/skills/sm-domain-model/SKILL.md

@@ -0,0 +1,188 @@
+---
+name: sm-domain-model
+description: Provides SourceMonitor engine domain model context. Use when working with engine models, associations, validations, scopes, database schema, or any model-layer code in the source_monitor namespace.
+allowed-tools: Read, Glob, Grep
+user-invocable: false
+---
+
+# SourceMonitor Domain Model
+
+## Overview
+
+SourceMonitor is a Rails 8 mountable engine for RSS/feed monitoring. All models live under the `SourceMonitor::` namespace and inherit from `SourceMonitor::ApplicationRecord`. Tables use a configurable prefix (default: `sourcemon_`).
+
+## Model Graph
+
+```
+Source (central entity)
+ |-- has_many :items (active only, via scope)
+ |-- has_many :all_items (includes soft-deleted)
+ |-- has_many :fetch_logs
+ |-- has_many :scrape_logs
+ |-- has_many :health_check_logs
+ |-- has_many :log_entries
+ |
+ Item
+ |-- belongs_to :source (counter_cache: true)
+ |-- has_one :item_content (dependent: :destroy, autosave: true)
+ |-- has_many :scrape_logs
+ |-- has_many :log_entries
+ |
+ ItemContent
+ |-- belongs_to :item (touch: true)
+ |
+ FetchLog (includes Loggable)
+ |-- belongs_to :source
+ |-- has_one :log_entry (as: :loggable, polymorphic)
+ |
+ ScrapeLog (includes Loggable)
+ |-- belongs_to :item
+ |-- belongs_to :source
+ |-- has_one :log_entry (as: :loggable, polymorphic)
+ |
+ HealthCheckLog (includes Loggable)
+ |-- belongs_to :source
+ |-- has_one :log_entry (as: :loggable, polymorphic)
+ |
+ LogEntry (delegated_type :loggable)
+ |-- belongs_to :source
+ |-- belongs_to :item (optional)
+ |-- loggable types: FetchLog, ScrapeLog, HealthCheckLog
+ |
+ ImportSession (standalone)
+ |-- user_id (FK to host app)
+ |
+ ImportHistory (standalone)
+ |-- user_id (FK to host app)
+```
+
+## Models Summary
+
+| Model | Table | Purpose |
+|-------|-------|---------|
+| `Source` | `sourcemon_sources` | Feed source with URL, fetch config, health tracking |
+| `Item` | `sourcemon_items` | Individual feed entry/article |
+| `ItemContent` | `sourcemon_item_contents` | Scraped HTML/content (split from items for performance) |
+| `FetchLog` | `sourcemon_fetch_logs` | Record of each feed fetch attempt |
+| `ScrapeLog` | `sourcemon_scrape_logs` | Record of each item scrape attempt |
+| `HealthCheckLog` | `sourcemon_health_check_logs` | Record of each health check |
+| `LogEntry` | `sourcemon_log_entries` | Unified log view via delegated_type (polymorphic) |
+| `ImportSession` | `sourcemon_import_sessions` | OPML import wizard state |
+| `ImportHistory` | `sourcemon_import_histories` | Completed import records |
+
+## Key Concerns
+
+### Loggable (`app/models/concerns/source_monitor/loggable.rb`)
+Shared by FetchLog, ScrapeLog, HealthCheckLog:
+- `attribute :metadata, default: -> { {} }`
+- `validates :started_at, presence: true`
+- `validates :duration_ms, numericality: { >= 0 }, allow_nil: true`
+- Scopes: `recent`, `successful`, `failed`
+
+### Sanitizable (`lib/source_monitor/models/sanitizable.rb`)
+String/hash attribute sanitization. Used by Source.
+
+### UrlNormalizable (`lib/source_monitor/models/url_normalizable.rb`)
+URL normalization and format validation. Used by Source and Item.
+
+## Source Model Details
+
+### State Values
+
+| Field | Values | Notes |
+|-------|--------|-------|
+| `fetch_status` | `idle`, `queued`, `fetching`, `failed`, `invalid` | DB CHECK constraint |
+| `health_status` | `healthy` (default) | String, extensible |
+| `active` | `true`/`false` | Boolean toggle |
+
+### Key Scopes
+
+| Scope | Meaning |
+|-------|---------|
+| `active` | `WHERE active = true` |
+| `failed` | `failure_count > 0 OR last_error IS NOT NULL OR last_error_at IS NOT NULL` |
+| `healthy` | `active AND failure_count = 0 AND last_error IS NULL AND last_error_at IS NULL` |
+| `due_for_fetch(reference_time:)` | Class method. Active sources where `next_fetch_at IS NULL OR <= reference_time` |
+
+### Validations
+
+| Field | Rules |
+|-------|-------|
+| `name` | presence |
+| `feed_url` | presence, uniqueness (case insensitive) |
+| `fetch_interval_minutes` | numericality > 0 |
+| `scraper_adapter` | presence |
+| `items_retention_days` | integer >= 0, allow nil |
+| `max_items` | integer >= 0, allow nil |
+| `fetch_status` | inclusion in FETCH_STATUS_VALUES |
+| `fetch_retry_attempt` | integer >= 0 |
+| `health_auto_pause_threshold` | custom: 0..1 range |
+
+### Notable Methods
+
+- `fetch_interval_hours` / `fetch_interval_hours=` -- convenience accessors converting minutes
+- `fetch_circuit_open?` -- circuit breaker check
+- `auto_paused?` -- health-based auto-pause check
+- `reset_items_counter!` -- recalculate counter cache from active items
+
+## Item Model Details
+
+### Soft Delete Pattern
+Items use soft delete via `deleted_at` column (NOT default_scope):
+- `scope :active` -- `WHERE deleted_at IS NULL`
+- `scope :with_deleted` -- unscopes deleted_at
+- `scope :only_deleted` -- `WHERE deleted_at IS NOT NULL`
+- `soft_delete!(timestamp:)` -- sets deleted_at, decrements counter cache
+- `deleted?` -- checks deleted_at presence
+
+### Key Scopes
+
+| Scope | Meaning |
+|-------|---------|
+| `active` | `WHERE deleted_at IS NULL` |
+| `recent` | Active, ordered by `published_at DESC NULLS LAST, created_at DESC` |
+| `published` | Active, `WHERE published_at IS NOT NULL` |
+| `pending_scrape` | Active, `WHERE scraped_at IS NULL` |
+| `failed_scrape` | Active, `WHERE scrape_status = 'failed'` |
+
+### Validations
+
+| Field | Rules |
+|-------|-------|
+| `source` | presence |
+| `guid` | presence, uniqueness scoped to source_id (case insensitive) |
+| `content_fingerprint` | uniqueness scoped to source_id, allow blank |
+| `url` | presence |
+
+### Content Delegation
+`scraped_html` and `scraped_content` delegate to `ItemContent`. Setting these values auto-creates/destroys the ItemContent association.
+
+## LogEntry Delegated Type
+
+LogEntry uses Rails `delegated_type` to unify FetchLog, ScrapeLog, and HealthCheckLog:
+- `loggable_type` -- polymorphic type column
+- `loggable_id` -- polymorphic ID column
+- Helper methods: `fetch?`, `scrape?`, `health_check?`, `log_type`
+- After-save sync: each log type calls `Logs::EntrySync.call(self)` to keep LogEntry in sync
+
+## ImportSession Wizard Steps
+
+Ordered steps: `upload` -> `preview` -> `health_check` -> `configure` -> `confirm`
+
+Methods: `next_step`, `previous_step`, `health_stream_name`, `health_check_targets`
+
+## ModelExtensions Registry
+
+All models register via `SourceMonitor::ModelExtensions.register(self, :key)`. This system:
+1. Assigns table names using the configured prefix
+2. Applies host-app concerns
+3. Applies host-app validations
+4. Supports `reload!` for configuration changes
+
+## References
+
+- [Model Relationship Graph](reference/model-graph.md) -- Visual model relationships
+- [Table Structure](reference/table-structure.md) -- Complete schema with columns, types, indexes
+- Source files: `app/models/source_monitor/*.rb`
+- Concern: `app/models/concerns/source_monitor/loggable.rb`
+- Migrations: `db/migrate/*.rb`

data/.claude/skills/sm-domain-model/reference/model-graph.md

@@ -0,0 +1,114 @@
+# SourceMonitor Model Relationship Graph
+
+## Core Feed Monitoring
+
+```
+                          SourceMonitor::Source
+                          =====================
+                          Central entity: a feed URL to monitor
+                          Table: sourcemon_sources
+                                    |
+          +-----------+-------------+-------------+-----------+------------+
+          |           |             |             |           |            |
+     has_many    has_many      has_many      has_many    has_many     has_many
+     :items      :all_items   :fetch_logs   :scrape_    :health_     :log_entries
+     (active)    (all)                       logs       check_logs
+          |           |             |             |           |            |
+          v           v             v             v           v            v
+     SourceMonitor::Item       FetchLog      ScrapeLog   HealthCheckLog  LogEntry
+     ===================      ========      =========   ==============  ========
+     Feed entry/article        Fetch         Scrape       Health        Unified
+     Table: sourcemon_items    attempt       attempt      check         log view
+          |                    record        record       record
+          |
+     +----+----+
+     |         |
+   has_one   has_many
+   :item_    :scrape_logs
+   content
+     |         |
+     v         v
+  ItemContent  ScrapeLog
+  ===========  (also belongs_to :source)
+  Scraped
+  content
+```
+
+## Association Details
+
+### Source -> Item
+- `has_many :items` -- Only active (non-deleted) items via `-> { active }` scope
+- `has_many :all_items` -- All items including soft-deleted
+- `dependent: :destroy` on all_items
+- Counter cache: `items_count` on Source (tracks active items)
+
+### Item -> ItemContent
+- `has_one :item_content` -- Lazy-created when scraped content is assigned
+- `dependent: :destroy, autosave: true`
+- `touch: true` on the belongs_to side
+- Content is auto-destroyed when both `scraped_html` and `scraped_content` become blank
+
+### Source -> FetchLog
+- `has_many :fetch_logs, dependent: :destroy`
+- Each FetchLog records one feed fetch attempt with HTTP details, item counts, timing
+
+### Source -> ScrapeLog (and Item -> ScrapeLog)
+- ScrapeLog has dual parents: `belongs_to :source` AND `belongs_to :item`
+- Validates that `item.source_id == source_id` (source must match item's source)
+- Records one content scrape attempt per item
+
+### Source -> HealthCheckLog
+- `has_many :health_check_logs, dependent: :destroy`
+- Records health check results for the source
+
+### LogEntry (Delegated Type)
+- Polymorphic unified view of all log types
+- `delegated_type :loggable, types: [FetchLog, ScrapeLog, HealthCheckLog]`
+- `belongs_to :source` (always present)
+- `belongs_to :item` (optional, present for scrape logs)
+- Synced via `Logs::EntrySync.call()` after each log save
+
+## Import Subsystem (Standalone)
+
+```
+ImportSession                     ImportHistory
+=============                     =============
+OPML import wizard state          Completed import record
+Table: sourcemon_import_sessions  Table: sourcemon_import_histories
+
+Fields:                           Fields:
+- user_id (FK -> users)           - user_id (FK -> users)
+- opml_file_metadata (jsonb)      - imported_sources (jsonb)
+- parsed_sources (jsonb)          - failed_sources (jsonb)
+- selected_source_ids (jsonb)     - skipped_duplicates (jsonb)
+- bulk_settings (jsonb)           - bulk_settings (jsonb)
+- current_step (string)           - started_at / completed_at
+- health_checks_active (bool)
+- health_check_target_ids (jsonb)
+```
+
+These models reference the host app's `users` table and are NOT directly associated with Source/Item models.
+
+## Polymorphic Patterns
+
+### Delegated Type (LogEntry)
+LogEntry uses `delegated_type` (not STI) for the unified log view:
+```ruby
+# LogEntry
+delegated_type :loggable, types: %w[
+  SourceMonitor::FetchLog
+  SourceMonitor::ScrapeLog
+  SourceMonitor::HealthCheckLog
+]
+```
+
+### STI Column on Source
+Source has a `type` column for potential STI subclassing but it is not actively used with subclasses currently.
+
+## Counter Caches
+
+| Parent | Column | Child | Notes |
+|--------|--------|-------|-------|
+| Source | `items_count` | Item | Only counts active (non-deleted) items |
+
+The counter is decremented manually during `soft_delete!` and can be recalculated via `reset_items_counter!`.