source_monitor 0.7.0 → 0.8.0

data/.vbw-planning/codebase/PATTERNS.md

@@ -1,262 +0,0 @@
-# Patterns
-
-Recurring patterns observed across the SourceMonitor codebase.
-
-## 1. Service Object Pattern
-
-**Where**: `lib/source_monitor/fetching/`, `lib/source_monitor/scraping/`, `lib/source_monitor/health/`, `lib/source_monitor/items/`
-
-Service objects encapsulate domain operations. They follow a consistent structure:
-
-```ruby
-class SomeService
-  def initialize(source:, **deps)
-    @source = source
-  end
-
-  def call
-    # orchestrate operation
-    # return a Result struct
-  end
-end
-```
-
-**Examples**:
-- `Fetching::FeedFetcher` -- `#call` returns `Result` struct
-- `Scraping::ItemScraper` -- `#call` returns `Result` struct
-- `Health::SourceHealthMonitor` -- `#call` updates source health
-- `Health::SourceHealthCheck` -- `#call` probes source URL
-- `Items::RetentionPruner` -- `#call` prunes old items
-- `Items::ItemCreator` -- `.call(source:, entry:)` class method
-
-## 2. Struct-Based Result Objects
-
-**Where**: Throughout all service objects
-
-Operations return typed `Struct` instances rather than raw hashes or arrays:
-
-```ruby
-Result = Struct.new(:status, :item, :log, :message, :error, keyword_init: true) do
-  def success?
-    status.to_s != "failed"
-  end
-end
-```
-
-**Examples**:
-- `Scraping::ItemScraper::Result` -- `:status, :item, :log, :message, :error`
-- `Scrapers::Base::Result` -- `:status, :html, :content, :metadata`
-- `Fetching::FeedFetcher::Result` -- `:status, :feed, :response, :body, :error, :item_processing, :retry_decision`
-- `Fetching::FeedFetcher::EntryProcessingResult` -- `:created, :updated, :failed, :items, :errors`
-- `Events::ItemCreatedEvent`, `Events::ItemScrapedEvent`, `Events::FetchCompletedEvent`
-- `Setup::Verification::Result` -- verification outcome
-
-## 3. Adapter/Strategy Pattern
-
-**Where**: `lib/source_monitor/scrapers/`, `lib/source_monitor/realtime/`, `lib/source_monitor/items/retention_strategies/`
-
-Pluggable behavior via abstract base class with `#call` contract:
-
-```ruby
-class Scrapers::Base
-  def call
-    raise NotImplementedError
-  end
-end
-```
-
-**Instances**:
-- **Scraper adapters**: `Scrapers::Base` -> `Scrapers::Readability` (registered in `ScraperRegistry`)
-- **Realtime adapters**: `solid_cable`, `redis`, `async` (configured in `RealtimeSettings`)
-- **Retention strategies**: `:destroy`, `:soft_delete` (in `Items::RetentionStrategies/`)
-
-## 4. Event/Callback System
-
-**Where**: `lib/source_monitor/events.rb`, `lib/source_monitor/configuration.rb`
-
-Event-driven communication between engine components:
-
-```ruby
-# Registration
-SourceMonitor.config.events.after_item_created { |event| ... }
-SourceMonitor.config.events.after_item_scraped { |event| ... }
-SourceMonitor.config.events.after_fetch_completed { |event| ... }
-
-# Dispatch
-SourceMonitor::Events.after_item_created(item:, source:, entry:, result:)
-```
-
-- Typed event structs carry context
-- Error isolation: each handler failure is logged, does not stop other handlers
-- Item processor pipeline: `Events.run_item_processors` runs all registered processors
-- Used by `Health` module to register fetch completion callback
-
-## 5. Configuration DSL with Nested Settings Objects
-
-**Where**: `lib/source_monitor/configuration.rb`
-
-Deeply nested configuration with domain-specific settings classes:
-
-```ruby
-SourceMonitor.configure do |config|
-  config.http.timeout = 30
-  config.fetching.min_interval_minutes = 10
-  config.health.window_size = 50
-  config.scrapers.register(:custom, MyCustomScraper)
-  config.models.source.include_concern SomeConcern
-  config.authentication.authenticate_with :authenticate_admin!
-end
-```
-
-**Pattern traits**:
-- Each settings class has `reset!` for test isolation
-- Constants for defaults (e.g., `DEFAULT_QUEUE_NAMESPACE`)
-- Callable values supported (procs/lambdas) for dynamic resolution
-- Validation in setters (e.g., `RealtimeSettings#adapter=` checks `VALID_ADAPTERS`)
-
-## 6. Model Extension System
-
-**Where**: `lib/source_monitor/model_extensions.rb`, `lib/source_monitor/configuration.rb`
-
-Host apps can dynamically inject concerns and validations into engine models:
-
-```ruby
-config.models.source.include_concern "MyApp::SourceExtensions"
-config.models.source.validate :custom_validator
-config.models.source.validate { |record| record.errors.add(:base, "invalid") if ... }
-```
-
-- `ModelExtensions.register(model_class, key)` called in each model class body
-- `ModelExtensions.reload!` re-applies all extensions on configuration change
-- Manages table name prefix assignment
-- Tracks applied concerns/validations to prevent duplicates
-- Removes old extension validations before re-applying
-
-## 7. Turbo Stream Response Pattern
-
-**Where**: `lib/source_monitor/turbo_streams/stream_responder.rb`, controllers
-
-Controllers build Turbo Stream responses via a `StreamResponder` builder:
-
-```ruby
-responder = SourceMonitor::TurboStreams::StreamResponder.new
-presenter = SourceMonitor::Sources::TurboStreamPresenter.new(source:, responder:)
-presenter.render_deletion(metrics:, query:, ...)
-responder.toast(message:, level: :success)
-render turbo_stream: responder.render(view_context)
-```
-
-- Accumulates stream actions as an array
-- Supports toast notifications, redirects, and custom actions
-- Pairs with Stimulus controllers on the frontend
-
-## 8. Defensive Logging Guard
-
-**Where**: All jobs and service objects
-
-Consistent pattern for safe logging:
-
-```ruby
-def log(stage, **extra)
-  return unless defined?(Rails) && Rails.respond_to?(:logger) && Rails.logger
-  Rails.logger.info("[SourceMonitor::...] #{payload.to_json}")
-rescue StandardError
-  nil
-end
-```
-
-This three-part guard (`defined?`, `respond_to?`, truthy check) prevents errors when running outside Rails (e.g., in tests or standalone scripts).
-
-## 9. ActiveSupport::Notifications Instrumentation
-
-**Where**: `lib/source_monitor/instrumentation.rb`, `lib/source_monitor/metrics.rb`
-
-Standard Rails instrumentation pattern:
-
-```ruby
-# Emit events
-SourceMonitor::Instrumentation.fetch_start(payload)
-SourceMonitor::Instrumentation.fetch_finish(payload)
-
-# Subscribe to events
-ActiveSupport::Notifications.subscribe("source_monitor.fetch.finish") do |...|
-  SourceMonitor::Metrics.increment(:fetch_finished_total)
-end
-```
-
-- Events namespaced as `source_monitor.*`
-- Monotonic clock for duration measurement
-- Metrics module aggregates counters and gauges in memory
-
-## 10. Search/Filter with Ransack
-
-**Where**: Models (`ransackable_attributes`, `ransackable_associations`), `SanitizesSearchParams` concern
-
-```ruby
-class Source < ApplicationRecord
-  def self.ransackable_attributes(_auth_object = nil)
-    %w[name feed_url website_url created_at ...]
-  end
-end
-```
-
-- Explicit whitelisting of searchable attributes (required by Ransack 4+)
-- `SanitizesSearchParams` controller concern sanitizes search inputs
-- Used in `SourcesController#index` and `LogsController#index`
-
-## 11. Circuit Breaker / Retry Policy
-
-**Where**: `lib/source_monitor/fetching/retry_policy.rb`, `lib/source_monitor/fetching/feed_fetcher.rb`, `app/jobs/source_monitor/fetch_feed_job.rb`
-
-Fetch failures trigger an escalating retry policy:
-
-1. **Retry with backoff**: Exponential wait, up to N attempts
-2. **Circuit open**: After exhausting retries, block fetches for a cooldown period
-3. **Circuit close**: Scheduler recovers after cooldown expires
-
-State stored on `Source` model: `fetch_retry_attempt`, `fetch_circuit_opened_at`, `fetch_circuit_until`, `backoff_until`.
-
-## 12. Wizard State Machine (OPML Import)
-
-**Where**: `app/models/source_monitor/import_session.rb`, `app/controllers/source_monitor/import_sessions_controller.rb`
-
-Multi-step wizard with explicit step ordering:
-
-```ruby
-STEP_ORDER = %w[upload preview health_check configure confirm].freeze
-```
-
-- State persisted in `ImportSession` model with JSONB columns
-- Each step has dedicated `handle_*_step` and `prepare_*_context` methods
-- Navigation via `next_step`/`previous_step` model methods
-- Step transitions guarded by validation (e.g., "select at least one source")
-
-## 13. Soft Delete Pattern
-
-**Where**: `app/models/source_monitor/item.rb`
-
-Items use soft deletion via `deleted_at` timestamp rather than physical deletion:
-
-```ruby
-scope :active, -> { where(deleted_at: nil) }
-scope :with_deleted, -> { unscope(where: :deleted_at) }
-scope :only_deleted, -> { where.not(deleted_at: nil) }
-
-def soft_delete!(timestamp: Time.current)
-  update_columns(deleted_at: timestamp, updated_at: timestamp)
-  Source.decrement_counter(:items_count, source_id)
-end
-```
-
-No `default_scope` is used (explicitly noted as avoiding anti-pattern).
-
-## 14. Separate Content Table
-
-**Where**: `app/models/source_monitor/item.rb`, `app/models/source_monitor/item_content.rb`
-
-Large scraped content is stored in a separate `ItemContent` model rather than on the `Item` directly:
-
-- Lazy-loaded via `has_one :item_content`
-- Auto-created when content is assigned, auto-destroyed when both fields become blank
-- Delegates `scraped_html` and `scraped_content` to `ItemContent`
-- Prevents bloating the items table with large text columns

data/.vbw-planning/codebase/STACK.md

@@ -1,101 +0,0 @@
-# Tech Stack
-
-## Core Platform
-
-| Layer | Technology | Version |
-|-------|-----------|---------|
-| Language | Ruby | >= 3.4.0 (CI uses 3.4.4) |
-| Framework | Rails | >= 8.0.3, < 9.0 (locked at 8.1.1) |
-| Database | PostgreSQL | 15 (CI service image) |
-| Background Jobs | Solid Queue | >= 0.3, < 3.0 (locked at 1.2.4) |
-| WebSocket/Realtime | Solid Cable | >= 3.0, < 4.0 (locked at 3.0.12) |
-| Frontend Interactivity | Turbo Rails | ~> 2.0 (locked at 2.0.20) |
-| JS Framework | Stimulus (Hotwired) | ^3.2.2 |
-| CSS Framework | Tailwind CSS | ^3.4.10 |
-| Web Server | Puma | 7.1.0 |
-| Asset Pipeline | Propshaft | 1.3.1 |
-
-## Project Type
-
-Mountable Rails 8 Engine gem (`source_monitor.gemspec`), distributed as a RubyGem. The engine uses `isolate_namespace SourceMonitor` and provides its own models, controllers, views, jobs, and frontend assets.
-
-- **Version**: 0.2.1
-- **Required Ruby**: >= 3.4.0
-- **License**: MIT
-
-## Feed Parsing & HTTP
-
-| Purpose | Gem | Version |
-|---------|-----|---------|
-| RSS/Atom/JSON feed parsing | Feedjira | >= 3.2, < 5.0 (locked 4.0.1) |
-| HTTP client | Faraday | ~> 2.9 (locked 2.14.0) |
-| HTTP retry middleware | faraday-retry | ~> 2.2 |
-| HTTP redirect following | faraday-follow_redirects | ~> 0.4 |
-| HTTP gzip compression | faraday-gzip | ~> 3.0 |
-
-## Content Scraping & Parsing
-
-| Purpose | Gem | Version |
-|---------|-----|---------|
-| HTML parsing (fast, C-based) | Nokolexbor | ~> 0.5 (locked 0.6.2) |
-| HTML parsing (standard) | Nokogiri | 1.18.10 (transitive) |
-| Article content extraction | ruby-readability | ~> 0.7 |
-
-## Search & Querying
-
-| Purpose | Gem | Version |
-|---------|-----|---------|
-| Search/filter forms | Ransack | ~> 4.2 (locked 4.4.1) |
-
-## Frontend Build Pipeline
-
-| Tool | Purpose | Version |
-|------|---------|---------|
-| esbuild | JS bundling | ^0.23.0 |
-| Tailwind CSS | Utility-first CSS | ^3.4.10 |
-| PostCSS | CSS processing | ^8.4.45 |
-| Autoprefixer | CSS vendor prefixes | ^10.4.20 |
-| ESLint | JS linting | ^9.11.0 |
-| Stylelint | CSS linting | ^16.8.0 |
-
-Build orchestration via `package.json` scripts:
-- `npm run build` -- builds both CSS (tailwindcss) and JS (esbuild)
-- `cssbundling-rails` (~> 1.4) and `jsbundling-rails` (~> 1.3) bridge npm builds into the Rails asset pipeline
-
-## JS Dependencies (Runtime)
-
-| Package | Purpose |
-|---------|---------|
-| `@hotwired/stimulus` ^3.2.2 | Stimulus controllers for UI interactions |
-| `stimulus-use` ^0.52.0 | Stimulus composable behaviors library |
-
-## Testing Stack
-
-| Tool | Purpose |
-|------|---------|
-| Minitest | Test framework (Rails default) |
-| Capybara | System/integration test driver |
-| Selenium WebDriver | Browser automation for system tests |
-| WebMock | HTTP request stubbing |
-| VCR | HTTP interaction recording/playback |
-| SimpleCov | Code coverage (branch coverage enabled) |
-| test-prof | Test profiling (TagProf, EventProf) |
-| StackProf | Sampling profiler for performance analysis |
-
-## Code Quality & Security
-
-| Tool | Purpose |
-|------|---------|
-| RuboCop (rails-omakase) | Ruby/Rails linting (omakase style) |
-| Brakeman | Static security analysis |
-| ESLint | JavaScript linting |
-| Stylelint | CSS linting |
-
-## CI/CD
-
-- **GitHub Actions** with 5 jobs: `lint`, `security`, `test`, `release_verification`, `profiling` (scheduled nightly)
-- Ruby 3.4.4, Node 20
-- PostgreSQL 15 as service container
-- Diff coverage enforcement via custom `bin/check-diff-coverage`
-- Test profiling guardrails via `bin/check-test-prof-metrics`
-- Parallel test execution (configurable via `SOURCE_MONITOR_TEST_WORKERS`)

data/.vbw-planning/codebase/STRUCTURE.md

@@ -1,324 +0,0 @@
-# Directory Structure
-
-## Top-Level Layout
-
-```
-source_monitor/
-  app/                    # Rails engine application code
-  bin/                    # Scripts (rubocop, test runners, CI checks)
-  config/                 # Engine configuration (routes, tailwind, feedjira initializer)
-  db/migrate/             # Engine migrations (24 migration files)
-  docs/                   # Documentation
-  examples/               # Example configurations and adapters
-  lib/                    # Engine library code and rake tasks
-  test/                   # Test suite
-  tasks/                  # Rake task definitions (aliases)
-  coverage/               # SimpleCov output (gitignored content)
-  node_modules/           # NPM packages
-  pkg/                    # Gem build output
-  tmp/                    # Temporary files
-```
-
-## App Directory (`app/`)
-
-```
-app/
-  assets/
-    builds/source_monitor/     # Pre-built CSS and JS (committed)
-      application.css          # Built Tailwind CSS output
-      application.js           # Built esbuild JS output
-    config/
-      source_monitor_manifest.js
-    images/source_monitor/     # SVGs and icons
-    javascripts/source_monitor/
-      application.js           # Stimulus app entry point
-      turbo_actions.js         # Custom Turbo Stream actions
-      controllers/
-        async_submit_controller.js
-        confirm_navigation_controller.js
-        dropdown_controller.js
-        modal_controller.js
-        notification_controller.js
-        select_all_controller.js
-    stylesheets/source_monitor/
-      application.tailwind.css # Tailwind input file
-    svgs/source_monitor/       # SVG assets
-  controllers/source_monitor/
-    application_controller.rb
-    dashboard_controller.rb
-    fetch_logs_controller.rb
-    health_controller.rb
-    import_sessions_controller.rb
-    items_controller.rb
-    logs_controller.rb
-    scrape_logs_controller.rb
-    source_bulk_scrapes_controller.rb
-    source_fetches_controller.rb
-    source_health_checks_controller.rb
-    source_health_resets_controller.rb
-    source_retries_controller.rb
-    source_turbo_responses.rb
-    sources_controller.rb
-    concerns/
-      sanitizes_search_params.rb
-  jobs/source_monitor/
-    application_job.rb
-    fetch_feed_job.rb
-    import_opml_job.rb
-    import_session_health_check_job.rb
-    item_cleanup_job.rb
-    log_cleanup_job.rb
-    schedule_fetches_job.rb
-    scrape_item_job.rb
-    source_health_check_job.rb
-  mailers/source_monitor/
-    application_mailer.rb
-  models/source_monitor/
-    application_record.rb
-    fetch_log.rb
-    health_check_log.rb
-    import_history.rb
-    import_session.rb
-    item.rb
-    item_content.rb
-    log_entry.rb
-    scrape_log.rb
-    source.rb
-    concerns/
-      loggable.rb
-  views/source_monitor/
-    dashboard/
-      index.html.erb
-      _fetch_schedule.html.erb
-      _job_metrics.html.erb
-      _recent_activity.html.erb
-      _stat_card.html.erb
-      _stats.html.erb
-    fetch_logs/
-      show.html.erb
-    import_sessions/
-      show.html.erb
-      show.turbo_stream.erb
-      _header.html.erb
-      _sidebar.html.erb
-      health_check/
-        _progress.html.erb
-        _row.html.erb
-      steps/
-        _configure.html.erb
-        _confirm.html.erb
-        _health_check.html.erb
-        _navigation.html.erb
-        _preview.html.erb
-        _upload.html.erb
-    items/
-      index.html.erb
-      show.html.erb
-      _details.html.erb
-      _details_wrapper.html.erb
-    logs/
-      index.html.erb
-    scrape_logs/
-      show.html.erb
-    shared/
-      _toast.html.erb
-    sources/
-      index.html.erb
-      show.html.erb
-      new.html.erb
-      edit.html.erb
-      _bulk_scrape_form.html.erb
-      _bulk_scrape_modal.html.erb
-      _details.html.erb
-      _details_wrapper.html.erb
-      _empty_state_row.html.erb
-      _fetch_interval_heatmap.html.erb
-      _form.html.erb
-      _form_fields.html.erb
-      _health_status_badge.html.erb
-      _import_history_panel.html.erb
-      _row.html.erb
-```
-
-## Lib Directory (`lib/source_monitor/`)
-
-```
-lib/
-  source_monitor.rb               # Main entry point, requires, module definition
-  source_monitor/
-    version.rb                    # VERSION constant (0.2.1)
-    engine.rb                     # Rails::Engine with initializers
-    configuration.rb              # Configuration DSL (655 lines)
-    events.rb                     # Event system (dispatch, callbacks)
-    instrumentation.rb            # ActiveSupport::Notifications wrapper
-    metrics.rb                    # In-memory counters/gauges
-    http.rb                       # Faraday client builder
-    model_extensions.rb           # Dynamic model concern/validation injection
-    scheduler.rb                  # Source fetch scheduling with SKIP LOCKED
-    health.rb                     # Health module setup
-    realtime.rb                   # Realtime broadcasting setup
-    feedjira_extensions.rb        # Feedjira customizations
-    assets.rb                     # Asset management utilities
-
-    analytics/                    # Query objects for dashboard metrics
-      source_activity_rates.rb
-      source_fetch_interval_distribution.rb
-      sources_index_metrics.rb
-    assets/                       # Asset bundling helpers
-    dashboard/                    # Dashboard presenters and queries
-      queries.rb
-      quick_action.rb
-      quick_actions_presenter.rb
-      recent_activity.rb
-      recent_activity_presenter.rb
-      turbo_broadcaster.rb
-      upcoming_fetch_schedule.rb
-    fetching/                     # Feed fetching pipeline
-      feed_fetcher.rb             # Core fetcher (627 lines)
-      fetch_error.rb
-      fetch_runner.rb
-      retry_policy.rb
-      stalled_fetch_reconciler.rb
-    health/                       # Health monitoring
-      import_source_health_check.rb
-      source_health_check.rb
-      source_health_monitor.rb
-      source_health_reset.rb
-    import_sessions/              # OPML import support
-      entry_normalizer.rb
-    items/                        # Item management
-      item_creator.rb
-      retention_pruner.rb
-    jobs/                         # Job support modules
-      cleanup_options.rb
-      fetch_failure_subscriber.rb
-      solid_queue_metrics.rb
-      visibility.rb
-    logs/                         # Unified log system
-      entry_sync.rb
-      filter_set.rb
-      query.rb
-      table_presenter.rb
-    models/                       # Shared model concerns
-      sanitizable.rb
-      url_normalizable.rb
-    pagination/                   # Pagination support
-      paginator.rb
-    realtime/                     # Realtime broadcasting
-      adapter.rb
-      broadcaster.rb
-    release/                      # Release management
-      changelog.rb
-      runner.rb
-    scrapers/                     # Scraper adapters
-      base.rb
-      readability.rb
-      fetchers/
-        http_fetcher.rb
-      parsers/
-        readability_parser.rb
-    scraping/                     # Scraping orchestration
-      bulk_result_presenter.rb
-      bulk_source_scraper.rb
-      enqueuer.rb
-      item_scraper.rb
-      item_scraper/
-        adapter_resolver.rb
-        persistence.rb
-      scheduler.rb
-      state.rb
-    security/                     # Security modules
-      authentication.rb
-      parameter_sanitizer.rb
-    setup/                        # Installation workflow
-      bundle_installer.rb
-      cli.rb
-      dependency_checker.rb
-      detectors.rb
-      gemfile_editor.rb
-      initializer_patcher.rb
-      install_generator.rb
-      migration_installer.rb
-      node_installer.rb
-      prompter.rb
-      requirements.rb
-      shell_runner.rb
-      workflow.rb
-      verification/
-        action_cable_verifier.rb
-        printer.rb
-        result.rb
-        runner.rb
-        solid_queue_verifier.rb
-        telemetry_logger.rb
-    sources/                      # Source-specific support
-      params.rb
-      turbo_stream_presenter.rb
-    turbo_streams/                # Turbo Stream helpers
-      stream_responder.rb
-  tasks/                          # Rake tasks
-    recover_stalled_fetches.rake
-    source_monitor_assets.rake
-    source_monitor_setup.rake
-    source_monitor_tasks.rake
-    test_smoke.rake
-```
-
-## Test Directory (`test/`)
-
-```
-test/
-  test_helper.rb                  # Test configuration and shared helpers
-  test_prof.rb                    # test-prof integration
-  source_monitor_test.rb          # Module-level tests
-  fixtures/                       # Test fixtures
-    vcr_cassettes/                # VCR recorded HTTP interactions
-  dummy/                          # Full Rails dummy app for testing
-    app/
-    bin/
-    config/
-    db/
-  examples/                       # Example integration tests
-    advanced_template_test.rb
-    basic_template_test.rb
-    custom_adapter_example_test.rb
-    docker_config_test.rb
-  integration/                    # Integration tests
-    engine_mounting_test.rb
-    host_install_flow_test.rb
-    navigation_test.rb
-    release_packaging_test.rb
-  lib/source_monitor/             # Unit tests mirroring lib/ structure
-    configuration_test.rb
-    feedjira_configuration_test.rb
-    instrumentation_test.rb
-    health/
-    pagination/
-    release/
-    scraping/
-    security/  (implicitly tested)
-    setup/
-    turbo_streams/
-  mailers/
-  models/source_monitor/          # Model tests
-  system/                         # System/browser tests
-    dashboard_test.rb
-    dropdown_fallback_test.rb
-    items_test.rb
-    logs_test.rb
-    mission_control_test.rb
-    sources_test.rb
-  tasks/                          # Rake task tests
-```
-
-## Key File Counts
-
-| Category | Count |
-|----------|-------|
-| Ruby files (.rb) | ~324 |
-| ERB templates (.erb) | ~48 |
-| JavaScript files (.js) | ~14 |
-| YAML configs (.yml) | ~16 |
-| Test files (*_test.rb) | ~124 |
-| Migrations | 24 |
-| Stimulus controllers | 6 |