source_monitor 0.2.1 → 0.3.1

data/.claude/skills/rails-service-object/SKILL.md

@@ -0,0 +1,349 @@
+---
+name: rails-service-object
+description: Creates service objects following single-responsibility principle with comprehensive tests. Use when extracting business logic from controllers, creating complex operations, implementing interactors, or when user mentions service objects or POROs.
+allowed-tools: Read, Write, Edit, Bash, Glob, Grep
+---
+
+# Rails Service Object Pattern
+
+## Overview
+
+Service objects encapsulate business logic:
+- Single responsibility (one public method: `#call`)
+- Easy to test in isolation
+- Reusable across controllers, jobs, rake tasks
+- Clear input/output contract
+- Dependency injection for testability
+
+## When to Use Service Objects
+
+| Scenario | Use Service Object? |
+|----------|---------------------|
+| Complex business logic spanning multiple models | Yes |
+| Multiple model interactions in one operation | Yes |
+| External API calls | Yes |
+| Logic shared across controllers/jobs | Yes |
+| Operations with side effects (emails, webhooks) | Yes |
+| Simple CRUD operations | **No** (use model) |
+| Single model validation | **No** (use model) |
+| Simple query/filter | **No** (use scope or query object) |
+| View formatting | **No** (use presenter) |
+| Form handling with validations | **No** (use form object) |
+
+## When NOT to Use Service Objects
+
+**Don't create a service object when:**
+- A model callback does the job (e.g., `after_create :send_welcome_email`)
+- The logic is a single ActiveRecord operation
+- A concern would share the behavior more naturally
+- You're wrapping a single method call (adds indirection for no benefit)
+- The "service" just delegates to one model method
+
+**Rule of thumb:** If your service object's `#call` method is under 5 lines and calls one model method, you don't need it.
+
+## Workflow Checklist
+
+```
+Service Object Progress:
+- [ ] Step 1: Define input/output contract
+- [ ] Step 2: Create service test (RED)
+- [ ] Step 3: Run test (fails - no service)
+- [ ] Step 4: Create service file with empty #call
+- [ ] Step 5: Run test (fails - wrong return)
+- [ ] Step 6: Implement #call method
+- [ ] Step 7: Run test (GREEN)
+- [ ] Step 8: Add error case tests
+- [ ] Step 9: Implement error handling
+- [ ] Step 10: Final test run
+```
+
+## Step 1: Define Contract
+
+```markdown
+## Service: Orders::CreateService
+
+### Purpose
+Creates a new order with inventory validation and payment processing.
+
+### Input
+- user: User (required)
+- items: Array<Hash> (required) - [{product_id:, quantity:}]
+- payment_method_id: Integer (optional)
+
+### Output (Result object)
+Success: { success?: true, data: Order }
+Failure: { success?: false, error: String, code: Symbol }
+
+### Dependencies
+- inventory_service: Checks product availability
+- payment_gateway: Processes payment
+
+### Side Effects
+- Creates Order and OrderItem records
+- Decrements inventory
+- Charges payment method
+- Sends confirmation email (async)
+```
+
+## Step 2: Service Test
+
+Location: `test/services/orders/create_service_test.rb`
+
+```ruby
+# frozen_string_literal: true
+
+require "test_helper"
+
+class Orders::CreateServiceTest < ActiveSupport::TestCase
+  setup do
+    @user = users(:one)
+    @product = products(:available)
+    @items = [{ product_id: @product.id, quantity: 2 }]
+    @service = Orders::CreateService.new
+  end
+
+  test "#call with valid inputs returns success" do
+    result = @service.call(user: @user, items: @items)
+
+    assert result.success?
+    assert_instance_of Order, result.data
+    assert_equal @user, result.data.user
+  end
+
+  test "#call with valid inputs creates an order" do
+    assert_difference("Order.count", 1) do
+      @service.call(user: @user, items: @items)
+    end
+  end
+
+  test "#call with empty items returns failure" do
+    result = @service.call(user: @user, items: [])
+
+    assert result.failure?
+    assert_equal "No items provided", result.error
+  end
+
+  test "#call with insufficient inventory returns failure" do
+    items = [{ product_id: @product.id, quantity: 999_999 }]
+
+    result = @service.call(user: @user, items: items)
+
+    assert result.failure?
+  end
+
+  test "#call with insufficient inventory does not create order" do
+    items = [{ product_id: @product.id, quantity: 999_999 }]
+
+    assert_no_difference("Order.count") do
+      @service.call(user: @user, items: items)
+    end
+  end
+end
+```
+
+## Step 3-6: Implement Service
+
+Location: `app/services/orders/create_service.rb`
+
+```ruby
+# frozen_string_literal: true
+
+module Orders
+  class CreateService
+    def initialize(inventory_service: InventoryService.new,
+                   payment_gateway: PaymentGateway.new)
+      @inventory_service = inventory_service
+      @payment_gateway = payment_gateway
+    end
+
+    def call(user:, items:, payment_method_id: nil)
+      return failure("No items provided", :empty_items) if items.empty?
+      return failure("Insufficient inventory", :insufficient_inventory) unless inventory_available?(items)
+
+      order = create_order(user, items)
+      process_payment(order, payment_method_id) if payment_method_id
+
+      success(order)
+    rescue ActiveRecord::RecordInvalid => e
+      failure(e.message, :validation_failed)
+    end
+
+    private
+
+    attr_reader :inventory_service, :payment_gateway
+
+    def inventory_available?(items)
+      items.all? do |item|
+        inventory_service.available?(item[:product_id], item[:quantity])
+      end
+    end
+
+    def create_order(user, items)
+      ActiveRecord::Base.transaction do
+        order = Order.create!(user: user, status: :pending)
+
+        items.each do |item|
+          order.order_items.create!(
+            product_id: item[:product_id],
+            quantity: item[:quantity]
+          )
+          inventory_service.decrement(item[:product_id], item[:quantity])
+        end
+
+        order
+      end
+    end
+
+    def process_payment(order, payment_method_id)
+      payment_gateway.charge(
+        amount: order.total,
+        payment_method_id: payment_method_id
+      )
+      order.update!(status: :paid)
+    end
+
+    def success(data)
+      Result.new(success: true, data: data)
+    end
+
+    def failure(error, code = :unknown)
+      Result.new(success: false, error: error, code: code)
+    end
+  end
+end
+```
+
+## Result Object
+
+```ruby
+# app/services/result.rb
+# frozen_string_literal: true
+
+class Result
+  attr_reader :data, :error, :code
+
+  def initialize(success:, data: nil, error: nil, code: nil)
+    @success = success
+    @data = data
+    @error = error
+    @code = code
+  end
+
+  def success?
+    @success
+  end
+
+  def failure?
+    !@success
+  end
+
+  def deconstruct_keys(keys)
+    { success: @success, data: @data, error: @error, code: @code }
+  end
+end
+```
+
+## Testing with Mocked Dependencies
+
+```ruby
+class Orders::CreateServiceTest < ActiveSupport::TestCase
+  setup do
+    @inventory_service = Minitest::Mock.new
+    @payment_gateway = Minitest::Mock.new
+    @service = Orders::CreateService.new(
+      inventory_service: @inventory_service,
+      payment_gateway: @payment_gateway
+    )
+  end
+
+  test "calls inventory service to check availability" do
+    @inventory_service.expect(:available?, true, [Integer, Integer])
+    @inventory_service.expect(:decrement, true, [Integer, Integer])
+
+    @service.call(user: users(:one), items: [{ product_id: 1, quantity: 2 }])
+
+    @inventory_service.verify
+  end
+end
+```
+
+## Calling Services
+
+### From Controllers
+
+```ruby
+class OrdersController < ApplicationController
+  def create
+    result = Orders::CreateService.new.call(
+      user: current_user,
+      items: order_params[:items],
+      payment_method_id: order_params[:payment_method_id]
+    )
+
+    if result.success?
+      redirect_to result.data, notice: "Order created"
+    else
+      flash.now[:alert] = result.error
+      render :new, status: :unprocessable_entity
+    end
+  end
+end
+```
+
+### From Jobs
+
+```ruby
+class ProcessOrderJob < ApplicationJob
+  def perform(user_id, items)
+    user = User.find(user_id)
+    result = Orders::CreateService.new.call(user: user, items: items)
+
+    unless result.success?
+      Rails.logger.error("Order failed: #{result.error}")
+    end
+  end
+end
+```
+
+## Directory Structure
+
+```
+app/services/
+  result.rb
+  orders/
+    create_service.rb
+    cancel_service.rb
+  users/
+    register_service.rb
+  payments/
+    charge_service.rb
+```
+
+## Conventions
+
+1. **Naming**: `Namespace::VerbNounService` (e.g., `Orders::CreateService`)
+2. **Location**: `app/services/[namespace]/[name]_service.rb`
+3. **Interface**: Single public method `#call`
+4. **Return**: Always return Result object
+5. **Dependencies**: Inject via constructor
+6. **Errors**: Catch and wrap in Result, don't raise
+
+## Anti-Patterns to Avoid
+
+1. **God service**: Too many responsibilities - split it
+2. **Hidden dependencies**: Using globals instead of injection
+3. **No return contract**: Returning different types
+4. **Raising exceptions**: Use Result objects instead
+5. **Service wrapping one method**: Just call the method directly
+6. **Service with multiple public methods**: Use separate services
+
+## Checklist
+
+- [ ] Contract defined (input/output/side effects)
+- [ ] Test written first (RED)
+- [ ] Single public method `#call`
+- [ ] Returns Result object consistently
+- [ ] Dependencies injected via constructor
+- [ ] Error cases tested
+- [ ] Transaction wraps multi-model operations
+- [ ] All tests GREEN

data/.claude/skills/sm-architecture/SKILL.md

@@ -0,0 +1,233 @@
+---
+name: sm-architecture
+description: Provides SourceMonitor engine architecture context. Use when working with engine internals, lib/ module structure, autoload organization, configuration DSL, pipelines, or any structural/organizational code in the source_monitor namespace.
+allowed-tools: Read, Glob, Grep
+user-invocable: false
+---
+
+# SourceMonitor Architecture
+
+## Overview
+
+SourceMonitor is a Rails 8 mountable engine (`SourceMonitor::Engine`). Code is split between:
+- **`app/`** -- Rails conventions (models, controllers, views, jobs, concerns)
+- **`lib/source_monitor/`** -- Domain logic, configuration, pipelines, utilities
+
+The engine uses **Ruby autoload** (not Zeitwerk) for `lib/` modules, with explicit `require` only for critical boot-time modules.
+
+## Boot Sequence
+
+`lib/source_monitor.rb` loads in this order:
+
+1. **Optional gems** (rescue LoadError): `solid_queue`, `solid_cable`, `turbo-rails`, `ransack`
+2. **Table name prefix** setup via `redefine_method`
+3. **Explicit requires** (11 files -- must load at boot):
+   - `version`, `engine`, `configuration`, `model_extensions`
+   - `events`, `instrumentation`, `metrics`
+   - `health`, `realtime`, `feedjira_extensions`
+4. **Autoload declarations** (71 modules) organized by namespace
+
+## Engine Configuration
+
+`SourceMonitor::Engine` (`lib/source_monitor/engine.rb`):
+- `isolate_namespace SourceMonitor`
+- Table name prefix from `config.models.table_name_prefix`
+- Initializers: assets, metrics subscribers, dashboard streams, jobs/Solid Queue setup
+
+## Module Tree
+
+```
+SourceMonitor (top-level)
+|-- HTTP                    # Faraday HTTP client factory
+|-- Scheduler               # Fetch scheduling coordinator
+|-- Assets                  # Asset path helpers
+|
+|-- Analytics/              # Dashboard analytics queries
+|   |-- SourceFetchIntervalDistribution
+|   |-- SourceActivityRates
+|   |-- SourcesIndexMetrics
+|
+|-- Dashboard/              # Dashboard UI support
+|   |-- QuickAction, QuickActionsPresenter
+|   |-- RecentActivity, RecentActivityPresenter
+|   |-- Queries, TurboBroadcaster
+|   |-- UpcomingFetchSchedule
+|
+|-- Fetching/               # Feed fetch pipeline
+|   |-- FeedFetcher          # Main orchestrator
+|   |   |-- AdaptiveInterval # Interval calculation
+|   |   |-- SourceUpdater    # Source state updates
+|   |   |-- EntryProcessor   # Entry iteration + item creation
+|   |-- FetchRunner          # Job-level fetch coordinator
+|   |-- RetryPolicy          # Retry/circuit-breaker decisions
+|   |-- StalledFetchReconciler
+|   |-- AdvisoryLock         # PG advisory locks
+|   |-- FetchError (+ subclasses)
+|
+|-- Items/                  # Item management
+|   |-- ItemCreator          # Create/update items from entries
+|   |   |-- EntryParser      # Parse feed entries to attributes
+|   |   |-- ContentExtractor # Process content through readability
+|   |-- RetentionPruner      # Age/count-based item cleanup
+|   |-- RetentionStrategies/ # Destroy vs SoftDelete
+|
+|-- ImportSessions/         # OPML import support
+|   |-- EntryNormalizer
+|   |-- HealthCheckBroadcaster
+|
+|-- Jobs/                   # Job infrastructure
+|   |-- CleanupOptions
+|   |-- Visibility           # Queue visibility setup
+|   |-- SolidQueueMetrics
+|   |-- FetchFailureSubscriber
+|
+|-- Logs/                   # Unified log system
+|   |-- EntrySync            # Sync log records to LogEntry
+|   |-- FilterSet, Query, TablePresenter
+|
+|-- Models/                 # Model concerns
+|   |-- Sanitizable          # String/hash sanitization
+|   |-- UrlNormalizable      # URL normalization
+|
+|-- Scrapers/               # Content scraping adapters
+|   |-- Base                 # Scraper interface
+|   |-- Readability          # Default readability adapter
+|   |-- Fetchers/HttpFetcher
+|   |-- Parsers/ReadabilityParser
+|
+|-- Scraping/               # Scraping orchestration
+|   |-- Enqueuer, Scheduler
+|   |-- ItemScraper (+ AdapterResolver, Persistence)
+|   |-- BulkSourceScraper, BulkResultPresenter
+|   |-- State
+|
+|-- Configuration/          # Configuration DSL (12 settings files)
+|   |-- HTTPSettings, FetchingSettings, HealthSettings
+|   |-- ScrapingSettings, RealtimeSettings, RetentionSettings
+|   |-- AuthenticationSettings, ScraperRegistry
+|   |-- Events, ValidationDefinition
+|   |-- ModelDefinition, Models
+|
+|-- Security/               # Security layer
+|   |-- ParameterSanitizer
+|   |-- Authentication
+|
+|-- Setup/                  # Install/setup wizard
+|   |-- CLI, Workflow, Requirements
+|   |-- Detectors, DependencyChecker
+|   |-- GemfileEditor, BundleInstaller, NodeInstaller
+|   |-- InstallGenerator, MigrationInstaller, InitializerPatcher
+|   |-- Verification/ (Result, Runner, Printer, etc.)
+|
+|-- Pagination/Paginator
+|-- Release/ (Changelog, Runner)
+|-- Sources/ (Params, TurboStreamPresenter)
+|-- TurboStreams/StreamResponder
+```
+
+## Key Architectural Patterns
+
+### 1. Configuration DSL
+
+The `Configuration` class composes 12 settings objects:
+
+```ruby
+SourceMonitor.configure do |config|
+  config.http.timeout = 30
+  config.fetching.min_interval_minutes = 5
+  config.health.auto_pause_threshold = 0.3
+  config.retention.strategy = :soft_delete
+  config.scraping.concurrency = 3
+  config.models.table_name_prefix = "sm_"
+end
+```
+
+Each settings class is a standalone PORO with defaults. Configuration is resettable via `reset_configuration!`.
+
+### 2. ModelExtensions Registry
+
+Models register themselves at class load time:
+```ruby
+SourceMonitor::ModelExtensions.register(self, :source)
+```
+
+This enables:
+- Dynamic table name prefixing
+- Host-app concern injection
+- Host-app validation injection
+- Full reload on config change
+
+### 3. Event System
+
+Three lifecycle events dispatched through `SourceMonitor::Events`:
+
+| Event | Fired When | Payload |
+|-------|-----------|---------|
+| `after_item_created` | New item saved | ItemCreatedEvent |
+| `after_item_scraped` | Scrape completed | ItemScrapedEvent |
+| `after_fetch_completed` | Fetch finished | FetchCompletedEvent |
+
+Plus `item_processors` -- callbacks run for every item (created or updated).
+
+Events are registered via `config.events` and dispatched with error isolation per handler.
+
+### 4. Instrumentation (ActiveSupport::Notifications)
+
+| Event | Purpose |
+|-------|---------|
+| `source_monitor.fetch.start` | Fetch beginning |
+| `source_monitor.fetch.finish` | Fetch completed |
+| `source_monitor.items.duplicate` | Duplicate item detected |
+| `source_monitor.items.retention` | Retention pruning |
+
+`Metrics` module subscribes to these and maintains counters/gauges.
+
+### 5. Pipeline Architecture
+
+**Fetch Pipeline:**
+```
+FetchRunner
+  -> AdvisoryLock (PG lock per source)
+    -> FeedFetcher.call
+      -> HTTP request (Faraday)
+      -> Parse feed (Feedjira)
+      -> EntryProcessor.process_feed_entries
+        -> ItemCreator.call (per entry)
+          -> EntryParser.parse
+          -> ContentExtractor.process_feed_content
+        -> Events.run_item_processors
+        -> Events.after_item_created
+      -> SourceUpdater.update_source_for_success
+        -> AdaptiveInterval.apply_adaptive_interval!
+      -> SourceUpdater.create_fetch_log
+      -> Events.after_fetch_completed
+    -> Completion handlers (retention, follow-up scraping)
+```
+
+**Scrape Pipeline:**
+```
+Scraping::Enqueuer
+  -> ItemScraper
+    -> AdapterResolver (select scraper)
+    -> Scrapers::Base subclass
+      -> Fetchers::HttpFetcher
+      -> Parsers::ReadabilityParser
+    -> Persistence (save to ItemContent)
+    -> Events.after_item_scraped
+```
+
+### 6. Health Monitoring
+
+Health module hooks into `after_fetch_completed`:
+- `SourceHealthMonitor` -- calculates rolling success rate
+- `SourceHealthCheck` -- HTTP health probe
+- Auto-pause sources below threshold
+- `SourceHealthReset` -- manual health reset
+
+## References
+
+- [Module Map](reference/module-map.md) -- Full module tree with responsibilities
+- [Extraction Patterns](reference/extraction-patterns.md) -- Refactoring patterns from Phase 3/4
+- Main entry: `lib/source_monitor.rb`
+- Engine: `lib/source_monitor/engine.rb`
+- Configuration: `lib/source_monitor/configuration.rb` + `lib/source_monitor/configuration/*.rb`