source_monitor 0.3.0 → 0.3.2

data/.claude/skills/sm-engine-migration/SKILL.md

@@ -0,0 +1,395 @@
+---
+name: sm-engine-migration
+description: Migration conventions for the Source Monitor engine. Use when creating database migrations, adding columns, indexes, constraints, or modifying the schema for the Source Monitor engine.
+allowed-tools: Read, Write, Edit, Bash, Glob, Grep
+---
+
+# Source Monitor Engine Migrations
+
+## Table Naming Convention
+
+All engine tables use the `sourcemon_` prefix:
+
+| Model | Table Name |
+|-------|-----------|
+| Source | `sourcemon_sources` |
+| Item | `sourcemon_items` |
+| FetchLog | `sourcemon_fetch_logs` |
+| ScrapeLog | `sourcemon_scrape_logs` |
+| LogEntry | `sourcemon_log_entries` |
+| ItemContent | `sourcemon_item_contents` |
+| HealthCheckLog | `sourcemon_health_check_logs` |
+| ImportSession | `sourcemon_import_sessions` |
+| ImportHistory | `sourcemon_import_histories` |
+
+The prefix comes from `SourceMonitor.config.models.table_name_prefix` (default: `"sourcemon_"`).
+
+## Creating a Migration
+
+```bash
+bin/rails generate migration AddFieldToSourcemonSources field:type
+```
+
+### Naming Convention
+
+Migration class names describe the change:
+
+| Pattern | Example |
+|---------|---------|
+| Create table | `CreateSourceMonitorLogEntries` |
+| Add column | `AddAdaptiveFetchingToggleToSources` |
+| Add index | `AddCompositeIndexToLogEntries` |
+| Add constraint | `AddFetchStatusCheckConstraint` |
+| Multi-column | `AddHealthFieldsToSources` |
+| Performance | `OptimizeSourceMonitorDatabasePerformance` |
+| Modify constraint | `RefreshFetchStatusConstraint` |
+
+## Table Creation Pattern
+
+```ruby
+# frozen_string_literal: true
+
+class CreateSourceMonitorWidgets < ActiveRecord::Migration[8.1]
+  def change
+    create_table :sourcemon_widgets do |t|
+      # Foreign keys reference engine tables by name
+      t.references :source, null: false, foreign_key: { to_table: :sourcemon_sources }
+      t.references :item, foreign_key: { to_table: :sourcemon_items }
+
+      # Columns
+      t.string :name, null: false
+      t.boolean :active, null: false, default: true
+      t.integer :count, null: false, default: 0
+      t.jsonb :metadata, null: false, default: {}
+      t.datetime :started_at, null: false
+      t.datetime :completed_at
+
+      t.timestamps
+    end
+
+    # Indexes after create_table
+    add_index :sourcemon_widgets, :name
+    add_index :sourcemon_widgets, :active
+    add_index :sourcemon_widgets, :started_at
+  end
+end
+```
+
+### Dynamic Table Names
+
+Later migrations use `SourceMonitor.table_name_prefix` for consistency:
+
+```ruby
+create_table :"#{SourceMonitor.table_name_prefix}import_sessions" do |t|
+  # ...
+end
+
+add_index :"#{SourceMonitor.table_name_prefix}import_sessions", :current_step
+```
+
+Both hardcoded `sourcemon_` and dynamic prefix are used in the codebase. For new migrations, prefer the dynamic approach.
+
+## Foreign Key Conventions
+
+**Always specify `to_table`** for foreign keys referencing engine tables:
+
+```ruby
+# Engine-to-engine FK
+t.references :source, null: false, foreign_key: { to_table: :sourcemon_sources }
+t.references :item, foreign_key: { to_table: :sourcemon_items }
+
+# Engine-to-host-app FK (references host app's users table)
+t.references :user, null: false, foreign_key: true
+
+# Polymorphic reference (no FK constraint)
+t.references :loggable, polymorphic: true, null: false,
+  index: { name: "index_sourcemon_log_entries_on_loggable" }
+```
+
+## Index Conventions
+
+### Standard Indexes
+
+```ruby
+# Single column
+add_index :sourcemon_sources, :feed_url, unique: true
+add_index :sourcemon_sources, :active
+add_index :sourcemon_sources, :next_fetch_at
+
+# Composite unique index
+add_index :sourcemon_items, [:source_id, :guid], unique: true
+
+# Named index (when auto-generated name is too long)
+add_index :sourcemon_items, %i[source_id published_at created_at],
+  name: "index_sourcemon_items_on_source_and_published_at"
+```
+
+### Concurrent Indexes (for zero-downtime)
+
+```ruby
+class AddCompositeIndexToLogEntries < ActiveRecord::Migration[8.1]
+  disable_ddl_transaction!
+
+  def change
+    add_index :sourcemon_log_entries, [:started_at, :id],
+      order: { started_at: :desc, id: :desc },
+      name: "index_log_entries_on_started_at_desc_id_desc",
+      algorithm: :concurrently
+  end
+end
+```
+
+### Conditional Index Creation
+
+```ruby
+unless index_exists?(:sourcemon_sources, :created_at)
+  add_index :sourcemon_sources, :created_at, name: "index_sourcemon_sources_on_created_at"
+end
+```
+
+## Column Patterns
+
+### JSONB Columns
+
+Always provide `null: false, default: {}` (or `default: []` for arrays):
+
+```ruby
+t.jsonb :metadata, null: false, default: {}
+t.jsonb :scrape_settings, null: false, default: {}
+t.jsonb :categories, null: false, default: []
+t.jsonb :parsed_sources, null: false, default: []
+```
+
+### Boolean Columns
+
+Always provide `null: false, default:`:
+
+```ruby
+t.boolean :active, null: false, default: true
+t.boolean :success, null: false, default: false
+t.boolean :scraping_enabled, null: false, default: false
+```
+
+### Counter Columns
+
+```ruby
+t.integer :items_count, null: false, default: 0
+t.integer :failure_count, null: false, default: 0
+t.integer :comments_count, null: false, default: 0
+```
+
+### Decimal Columns (for rates/thresholds)
+
+```ruby
+t.decimal :rolling_success_rate, precision: 5, scale: 4
+t.decimal :health_auto_pause_threshold, precision: 5, scale: 4
+```
+
+## CHECK Constraints
+
+### Adding a Constraint
+
+```ruby
+class AddFetchStatusCheckConstraint < ActiveRecord::Migration[8.0]
+  def up
+    execute <<-SQL
+      ALTER TABLE sourcemon_sources
+      ADD CONSTRAINT check_fetch_status_values
+      CHECK (fetch_status IN ('idle', 'queued', 'fetching', 'failed'))
+    SQL
+  end
+
+  def down
+    execute <<-SQL
+      ALTER TABLE sourcemon_sources
+      DROP CONSTRAINT check_fetch_status_values
+    SQL
+  end
+end
+```
+
+### Modifying a Constraint
+
+```ruby
+class RefreshFetchStatusConstraint < ActiveRecord::Migration[8.0]
+  ALLOWED_STATUSES = %w[idle queued fetching failed invalid].freeze
+  PREVIOUS_STATUSES = %w[idle queued fetching failed].freeze
+
+  def up
+    replace_constraint(ALLOWED_STATUSES)
+  end
+
+  def down
+    replace_constraint(PREVIOUS_STATUSES)
+  end
+
+  private
+
+  def replace_constraint(statuses)
+    quoted = statuses.map { |s| ActiveRecord::Base.connection.quote(s) }.join(", ")
+
+    execute <<~SQL
+      ALTER TABLE sourcemon_sources DROP CONSTRAINT IF EXISTS check_fetch_status_values
+    SQL
+
+    execute <<~SQL
+      ALTER TABLE sourcemon_sources
+      ADD CONSTRAINT check_fetch_status_values CHECK (fetch_status IN (#{quoted}))
+    SQL
+  end
+end
+```
+
+## Data Migration Pattern
+
+For migrations that backfill data, use anonymous ActiveRecord classes:
+
+```ruby
+reversible do |direction|
+  direction.up do
+    say_with_time "Backfilling sourcemon_log_entries" do
+      source_class = Class.new(ActiveRecord::Base) { self.table_name = "sourcemon_fetch_logs" }
+      target_class = Class.new(ActiveRecord::Base) { self.table_name = "sourcemon_log_entries" }
+
+      source_class.find_each do |record|
+        target_class.create!(
+          source_id: record.source_id,
+          # ... map fields ...
+        )
+      end
+    end
+  end
+end
+```
+
+## Column Extraction Pattern
+
+Moving columns from one table to a new table:
+
+```ruby
+class CreateSourceMonitorItemContents < ActiveRecord::Migration[8.0]
+  def up
+    create_table :sourcemon_item_contents do |t|
+      t.references :item, null: false,
+        foreign_key: { to_table: :sourcemon_items },
+        index: { unique: true }
+      t.text :scraped_html
+      t.text :scraped_content
+      t.timestamps(null: false)
+    end
+
+    # Migrate existing data
+    execute <<~SQL
+      INSERT INTO sourcemon_item_contents (item_id, scraped_html, scraped_content, created_at, updated_at)
+      SELECT id, scraped_html, scraped_content, COALESCE(updated_at, CURRENT_TIMESTAMP), COALESCE(updated_at, CURRENT_TIMESTAMP)
+      FROM sourcemon_items
+      WHERE scraped_html IS NOT NULL OR scraped_content IS NOT NULL
+    SQL
+
+    # Remove old columns
+    remove_column :sourcemon_items, :scraped_html, :text
+    remove_column :sourcemon_items, :scraped_content, :text
+  end
+
+  def down
+    add_column :sourcemon_items, :scraped_html, :text
+    add_column :sourcemon_items, :scraped_content, :text
+
+    execute <<~SQL
+      UPDATE sourcemon_items items
+      SET scraped_html = contents.scraped_html,
+          scraped_content = contents.scraped_content
+      FROM sourcemon_item_contents contents
+      WHERE contents.item_id = items.id
+    SQL
+
+    drop_table :sourcemon_item_contents
+  end
+end
+```
+
+## Adding NOT NULL to Existing Columns
+
+Clean up data before adding constraint:
+
+```ruby
+class AddNotNullConstraintsToItems < ActiveRecord::Migration[8.0]
+  def up
+    # Fix existing NULL values first
+    execute <<~SQL
+      UPDATE sourcemon_items
+      SET guid = COALESCE(content_fingerprint, gen_random_uuid()::text)
+      WHERE guid IS NULL
+    SQL
+
+    change_column_null :sourcemon_items, :guid, false
+  end
+
+  def down
+    change_column_null :sourcemon_items, :guid, true
+  end
+end
+```
+
+## Bulk Column Changes
+
+```ruby
+class AddHealthFieldsToSources < ActiveRecord::Migration[8.0]
+  def change
+    change_table :sourcemon_sources, bulk: true do |t|
+      t.decimal :rolling_success_rate, precision: 5, scale: 4
+      t.string :health_status, null: false, default: "healthy"
+      t.datetime :health_status_changed_at
+      t.datetime :auto_paused_at
+      t.datetime :auto_paused_until
+      t.decimal :health_auto_pause_threshold, precision: 5, scale: 4
+    end
+
+    add_index :sourcemon_sources, :health_status
+    add_index :sourcemon_sources, :auto_paused_until
+  end
+end
+```
+
+## Host App Installation
+
+Engine migrations are installed in the host app via:
+
+```bash
+bin/rails source_monitor:install:migrations
+bin/rails db:migrate
+```
+
+This copies migration files from the engine's `db/migrate/` into the host app's `db/migrate/` directory, preserving timestamps.
+
+## Testing
+
+Test migrations indirectly by testing the models and database constraints they create:
+
+```ruby
+test "database rejects invalid fetch_status values" do
+  source = create_source!
+  error = assert_raises(ActiveRecord::StatementInvalid) do
+    source.update_columns(fetch_status: "bogus")
+  end
+  assert_match(/check_fetch_status_values/i, error.message)
+end
+```
+
+## Checklist
+
+- [ ] Table uses `sourcemon_` prefix
+- [ ] Foreign keys specify `to_table:` for engine tables
+- [ ] JSONB columns have `null: false, default: {}` (or `[]`)
+- [ ] Boolean columns have `null: false, default:`
+- [ ] Counter columns have `null: false, default: 0`
+- [ ] Indexes have explicit names if auto-name would be too long
+- [ ] Migration is reversible (or has explicit `up`/`down`)
+- [ ] Data migrations use anonymous AR classes (not model constants)
+- [ ] Concurrent indexes use `disable_ddl_transaction!` and `algorithm: :concurrently`
+- [ ] CHECK constraints use raw SQL with `execute`
+- [ ] Run: `bin/rails db:migrate && bin/rails db:rollback && bin/rails db:migrate`
+
+## References
+
+- [reference/migration-conventions.md](reference/migration-conventions.md) -- Complete table catalog and naming conventions

data/.claude/skills/sm-engine-migration/reference/migration-conventions.md

@@ -0,0 +1,255 @@
+# Migration Conventions Reference
+
+## Complete Table Catalog
+
+### sourcemon_sources (core)
+
+Created in: `20241008120000_create_source_monitor_sources.rb`
+
+| Column | Type | Constraints | Default |
+|--------|------|------------|---------|
+| `name` | string | NOT NULL | - |
+| `feed_url` | string | NOT NULL, unique index | - |
+| `website_url` | string | - | - |
+| `active` | boolean | NOT NULL | `true` |
+| `feed_format` | string | - | - |
+| `fetch_interval_minutes` | integer | NOT NULL | `360` |
+| `next_fetch_at` | datetime | indexed | - |
+| `last_fetched_at` | datetime | - | - |
+| `last_fetch_duration_ms` | integer | - | - |
+| `last_http_status` | integer | - | - |
+| `last_error` | text | - | - |
+| `last_error_at` | datetime | - | - |
+| `etag` | string | - | - |
+| `last_modified` | datetime | - | - |
+| `failure_count` | integer | NOT NULL | `0` |
+| `backoff_until` | datetime | - | - |
+| `items_count` | integer | NOT NULL | `0` |
+| `scraping_enabled` | boolean | NOT NULL | `false` |
+| `auto_scrape` | boolean | NOT NULL | `false` |
+| `scrape_settings` | jsonb | NOT NULL | `{}` |
+| `scraper_adapter` | string | NOT NULL | `"readability"` |
+| `requires_javascript` | boolean | NOT NULL | `false` |
+| `custom_headers` | jsonb | NOT NULL | `{}` |
+| `items_retention_days` | integer | - | - |
+| `max_items` | integer | - | - |
+| `metadata` | jsonb | NOT NULL | `{}` |
+| `adaptive_fetching_enabled` | boolean | NOT NULL | `true` |
+| `type` | string | - | - |
+| `fetch_status` | string | CHECK constraint | `"idle"` |
+| `fetch_retry_attempt` | integer | - | - |
+| `fetch_circuit_opened_at` | datetime | - | - |
+| `fetch_circuit_until` | datetime | - | - |
+| `rolling_success_rate` | decimal(5,4) | - | - |
+| `health_status` | string | NOT NULL, indexed | `"healthy"` |
+| `health_status_changed_at` | datetime | - | - |
+| `auto_paused_at` | datetime | - | - |
+| `auto_paused_until` | datetime | indexed | - |
+| `health_auto_pause_threshold` | decimal(5,4) | - | - |
+| `feed_content_readability` | string | - | - |
+
+**Indexes:**
+- `feed_url` (unique)
+- `active`
+- `next_fetch_at`
+- `created_at`
+- `health_status`
+- `auto_paused_until`
+
+**CHECK constraints:**
+- `check_fetch_status_values`: `fetch_status IN ('idle', 'queued', 'fetching', 'failed', 'invalid')`
+
+---
+
+### sourcemon_items
+
+Created in: `20241008121000_create_source_monitor_items.rb`
+
+| Column | Type | Constraints | Default |
+|--------|------|------------|---------|
+| `source_id` | reference | NOT NULL, FK | - |
+| `guid` | string | NOT NULL, indexed | - |
+| `content_fingerprint` | string | indexed | - |
+| `title` | string | - | - |
+| `url` | string | NOT NULL, indexed | - |
+| `canonical_url` | string | - | - |
+| `author` | string | - | - |
+| `authors` | jsonb | NOT NULL | `[]` |
+| `summary` | text | - | - |
+| `content` | text | - | - |
+| `scraped_at` | datetime | - | - |
+| `scrape_status` | string | indexed | - |
+| `published_at` | datetime | indexed | - |
+| `updated_at_source` | datetime | - | - |
+| `categories` | jsonb | NOT NULL | `[]` |
+| `tags` | jsonb | NOT NULL | `[]` |
+| `keywords` | jsonb | NOT NULL | `[]` |
+| `enclosures` | jsonb | NOT NULL | `[]` |
+| `media_thumbnail_url` | string | - | - |
+| `media_content` | jsonb | NOT NULL | `[]` |
+| `language` | string | - | - |
+| `copyright` | string | - | - |
+| `comments_url` | string | - | - |
+| `comments_count` | integer | NOT NULL | `0` |
+| `metadata` | jsonb | NOT NULL | `{}` |
+| `deleted_at` | datetime | - | - |
+
+**Indexes:**
+- `[source_id, guid]` (unique composite)
+- `[source_id, content_fingerprint]` (unique composite)
+- `[source_id, published_at, created_at]` (named: `index_sourcemon_items_on_source_and_published_at`)
+- `guid`
+- `content_fingerprint`
+- `url`
+- `scrape_status`
+- `published_at`
+
+---
+
+### sourcemon_fetch_logs
+
+Created in: `20241008122000_create_source_monitor_fetch_logs.rb`
+
+| Column | Type | Constraints | Default |
+|--------|------|------------|---------|
+| `source_id` | reference | NOT NULL, FK | - |
+| `success` | boolean | NOT NULL | `false` |
+| `items_created` | integer | NOT NULL | `0` |
+| `items_updated` | integer | NOT NULL | `0` |
+| `items_failed` | integer | NOT NULL | `0` |
+| `started_at` | datetime | NOT NULL, indexed | - |
+| `completed_at` | datetime | - | - |
+| `duration_ms` | integer | - | - |
+| `http_status` | integer | - | - |
+| `http_response_headers` | jsonb | NOT NULL | `{}` |
+| `error_class` | string | - | - |
+| `error_message` | text | - | - |
+| `error_backtrace` | text | - | - |
+| `feed_size_bytes` | integer | - | - |
+| `items_in_feed` | integer | - | - |
+| `job_id` | string | indexed | - |
+| `metadata` | jsonb | NOT NULL | `{}` |
+
+---
+
+### sourcemon_scrape_logs
+
+Created in: `20241008123000_create_source_monitor_scrape_logs.rb`
+
+References `sourcemon_sources` and `sourcemon_items`.
+
+---
+
+### sourcemon_log_entries (unified log view)
+
+Created in: `20251015100000_create_source_monitor_log_entries.rb`
+
+| Column | Type | Constraints | Default |
+|--------|------|------------|---------|
+| `loggable_type` | string | NOT NULL (polymorphic) | - |
+| `loggable_id` | integer | NOT NULL (polymorphic) | - |
+| `source_id` | reference | NOT NULL, FK | - |
+| `item_id` | reference | FK (nullable) | - |
+| `success` | boolean | NOT NULL | `false` |
+| `started_at` | datetime | NOT NULL, indexed | - |
+| `completed_at` | datetime | - | - |
+| `http_status` | integer | - | - |
+| `duration_ms` | integer | - | - |
+| `items_created` | integer | - | - |
+| `items_updated` | integer | - | - |
+| `items_failed` | integer | - | - |
+| `scraper_adapter` | string | indexed | - |
+| `content_length` | integer | - | - |
+| `error_class` | string | - | - |
+| `error_message` | text | - | - |
+
+**Indexes:**
+- `[loggable_type, loggable_id]` (named: `index_sourcemon_log_entries_on_loggable`)
+- `[started_at, id]` (descending, concurrent)
+- `[loggable_type, started_at, id]` (descending, concurrent)
+- `started_at`
+- `success`
+- `scraper_adapter`
+
+---
+
+### sourcemon_item_contents
+
+Created in: `20251009090000_create_source_monitor_item_contents.rb`
+
+| Column | Type | Constraints |
+|--------|------|------------|
+| `item_id` | reference | NOT NULL, FK, unique index |
+| `scraped_html` | text | - |
+| `scraped_content` | text | - |
+
+---
+
+### sourcemon_health_check_logs
+
+Created in: `20251022100000_create_source_monitor_health_check_logs.rb`
+
+---
+
+### sourcemon_import_sessions
+
+Created in: `20251124090000_create_import_sessions.rb`
+
+Uses dynamic prefix: `:"#{SourceMonitor.table_name_prefix}import_sessions"`
+
+---
+
+### sourcemon_import_histories
+
+Created in: `20251125094500_create_import_histories.rb`
+
+Uses dynamic prefix: `:"#{SourceMonitor.table_name_prefix}import_histories"`
+
+---
+
+## Migration Version History
+
+| Migration | Rails Version | Description |
+|-----------|--------------|-------------|
+| `20241008120000` | 8.0 | Create sources |
+| `20241008121000` | 8.0 | Create items |
+| `20241008122000` | 8.0 | Create fetch_logs |
+| `20241008123000` | 8.0 | Create scrape_logs |
+| `20251008183000` | 8.0 | Change fetch_interval to minutes |
+| `20251009090000` | 8.0 | Extract item_contents |
+| `20251009103000` | 8.0 | Add feed_content_readability |
+| `20251010090000` | 7.1 | Add adaptive_fetching_enabled |
+| `20251010123000` | 8.0 | Add deleted_at to items |
+| `20251010153000` | 8.0 | Add type to sources (STI) |
+| `20251010154500` | 8.0 | Add fetch_status to sources |
+| `20251010160000` | 8.0 | Create solid_cable_messages |
+| `20251011090000` | 8.0 | Add fetch retry state |
+| `20251012090000` | 8.0 | Add health fields |
+| `20251012100000` | 8.0 | Optimize indexes |
+| `20251014064947` | 8.0 | Add NOT NULL to items |
+| `20251014171659` | 8.0 | Add performance indexes |
+| `20251014172525` | 8.0 | Add fetch_status CHECK constraint |
+| `20251015100000` | 7.1 | Create log_entries |
+| `20251022100000` | 8.0 | Create health_check_logs |
+| `20251108120116` | 8.0 | Refresh fetch_status constraint |
+| `20251124090000` | 8.1 | Create import_sessions |
+| `20251124153000` | 8.1 | Add health to import_sessions |
+| `20251125094500` | 8.1 | Create import_histories |
+| `20260210204022` | 8.1 | Add composite index to log_entries |
+
+## Key Patterns Summary
+
+| Pattern | Convention |
+|---------|-----------|
+| Table prefix | `sourcemon_` |
+| FK to engine table | `foreign_key: { to_table: :sourcemon_<table> }` |
+| FK to host table | `foreign_key: true` |
+| JSONB hash | `null: false, default: {}` |
+| JSONB array | `null: false, default: []` |
+| Boolean | `null: false, default: <value>` |
+| Counter | `null: false, default: 0` |
+| Decimal rate | `precision: 5, scale: 4` |
+| Long index name | Use `name:` parameter |
+| Zero-downtime index | `disable_ddl_transaction!` + `algorithm: :concurrently` |
+| Data backfill | Anonymous AR class + `find_each` |
+| Constraint | Raw SQL `execute` with `up`/`down` |