source_monitor 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 90320cedd371ba574e94a72530899c9c186ee1e90f6b27dd221dca6e5831a626
+  data.tar.gz: 67db4a9fb817b386cbdffd675c44ec8e32cfa806ad4be284e95a80756180cf82
+SHA512:
+  metadata.gz: 1f63265ddc4b7f5a2989a91c04b61645e573ed71b68aa0348197f0f0afb34d8f36b9d8127613d7c63b25df39d7e20603c31cfac445b9f431188bdaf4724f46ee
+  data.tar.gz: 6ceed78ef9db2de1d61904745c326016529a06f5ad7c56c07f0428bebf4d6c237a769532ec0e6ca5be795970833389e67c9a503b47283e48aa7af524a321d969

data/.gitignore

@@ -0,0 +1,16 @@
+/.bundle/
+/doc/
+/log/*.log
+/pkg/
+/tmp/
+/node_modules/
+/coverage/
+/test/dummy/db/*.sqlite3
+/test/dummy/db/*.sqlite3-*
+/test/dummy/log/*.log*
+/test/dummy/storage/
+/test/dummy/tmp/
+/test/tmp/
+/test/lib/tmp/install_generator/config/routes.rb
+/app/assets/builds/*
+!/app/assets/builds/.keep

data/.rubocop.yml

@@ -0,0 +1,12 @@
+# Omakase Ruby styling for Rails
+inherit_gem: { rubocop-rails-omakase: rubocop.yml }
+
+AllCops:
+  Exclude:
+    - "test/dummy/db/schema.rb"
+
+# Overwrite or add rules to create your own house style
+#
+# # Use `[a, [b, c]]` not `[ a, [ b, c ] ]`
+# Layout/SpaceInsideArrayLiteralBrackets:
+#   Enabled: false

data/.ruby-version

@@ -0,0 +1 @@
+3.4.4

data/AGENTS.md

@@ -0,0 +1,132 @@
+# Repository Guidelines
+
+refer to ./ai/project_overview.md for full scope of this project
+
+use rbenv for all ruby and bundler/gem commands, not the system ruby
+
+## Contribution Workflow
+
+- Treat the engine like any external contributor would: no direct commits to `main`.
+- Before writing code, branch off the latest `origin/main` (use a descriptive `feature/` or `bugfix/` prefix) and open a draft PR on `github.com/dchuk/source_monitor` referencing the active `.ai/tasks.md` slice.
+- Push early and often to that branch so history stays visible; keep commits scoped and rebases local to your branch only.
+- Move the PR out of draft once tests pass and the slice is ready for review; request at least one review and wait for all CI jobs (lint, security, Rails tests + diff coverage) to succeed before merging.
+- The `test` job enforces diff coverage via `bin/check-diff-coverage`; if legitimate gaps remain after new code paths, refresh `config/coverage_baseline.json` by running `bin/test-coverage` followed by `bin/update-coverage-baseline` on the updated branch and commit the regenerated baseline.
+- Merge via the PR UI (squash or rebase as agreed) after approval; avoid rewriting shared history post-push.
+- Tag releases only after the release PR merges, then follow the checklist in `CHANGELOG.md`.
+
+## Library Documentation
+
+- Use Context7 MCP constantly to look up fresh documentation for any task you're looking to complete, especially tasks that rely on using libraries or gems
+
+### Project Dependencies & context7 links
+
+- Feedjirra - https://context7.com/feedjira/feedjira
+
+## Project Structure & Module Organization
+
+The engine follows the Rails mountable layout generated by `rails plugin new source_monitor --mountable`. Runtime code lives under `app/` (controllers, jobs, views) and is namespaced as `SourceMonitor`. Long-lived services, adapters, and instrumentation helpers belong in `lib/source_monitor/`. Generator templates and install scripts reside in `lib/generators/source_monitor/`. Tests sit in `test/`, with fixtures under `test/fixtures/feeds/`, and the dummy host app in `test/dummy/` for integration coverage. Keep shared UI assets under `app/assets/` and engine configuration in `config/initializers/`.
+
+## Build, Test, and Development Commands
+
+Run `bin/setup` to install gems, prepare the dummy database, and compile Tailwind. Use `bin/rails test` (or `bundle exec rake test`) for the full MiniTest suite, including system tests through the dummy app. During feature work, `bin/dev` starts the dummy app with Solid Queue workers and Tailwind watcher. Trigger a feed ingest locally with `bin/rails runner 'SourceMonitor::FetchFeeds.call'` once the fetcher service lands.
+
+## Background Job Defaults
+
+- SourceMonitor ensures Solid Queue is the default adapter when the host app is still using the async adapter, but respects any explicit `ActiveJob` configuration already in place. Override queues/concurrency via `SourceMonitor.configure`.
+- Queue names are namespaced (`source_monitor_fetch`/`source_monitor_scrape` by default) and automatically honor host `queue_name_prefix`. Use `SourceMonitor.queue_name(:fetch)` helpers inside jobs.
+- Dashboard queue metrics read directly from Solid Queue tables via `SourceMonitor::Jobs::SolidQueueMetrics`. Host apps must install the Solid Queue migrations (reuse the engine's `20251009140000_create_solid_queue_tables.rb` or run `rails solid_queue:install`) for the card to surface ready/scheduled/failed counts; otherwise the UI falls back to an availability warning. Mission Control remains optional for deeper drill-downs.
+- The dummy host keeps Solid Queue tables in the primary database via `20251009140000_create_solid_queue_tables.rb`. Real apps can either reuse that migration or run `rails solid_queue:install` to manage a dedicated queue database—Mission Control expects one of those setups before it can surface data.
+- Recurring schedules live in `config/recurring.yml`, scheduling `SourceMonitor::ScheduleFetchesJob` each minute plus the scraping scheduler every two minutes. Override the schedule path with `bin/jobs --recurring_schedule_file=...` (or `SOLID_QUEUE_RECURRING_SCHEDULE_FILE`) and disable recurring runners with `SOLID_QUEUE_SKIP_RECURRING=true` or `bin/jobs --skip-recurring`.
+- Hosts that need to wrap Solid Queue command execution can set `config.recurring_command_job_class` in the generated initializer to point at their custom job class.
+
+## Configuration DSL
+
+- `SourceMonitor.configure` now exposes structured namespaces:
+  - `config.http` for Faraday timeouts, retry policy, proxy, and default headers. Per the [Faraday retry docs](https://github.com/lostisland/faraday/blob/main/docs/middleware/index.md), middleware options map 1:1 to the settings we surface (max retries, interval, backoff, statuses).
+  - `config.scrapers` registers/overrides adapters by name; adapters must inherit from `SourceMonitor::Scrapers::Base` and are discovered before constant lookup.
+  - `config.retention` supplies global defaults for `items_retention_days`, `max_items`, and the pruning strategy (`:destroy` or `:soft_delete`). Runtimes treat blank source fields as “inherit from config”.
+  - `config.models` lets host apps adjust table name prefixes, mix in concerns, and register custom validations per engine model. Use it to bolt on associations or STI-specific rules without monkey patches.
+  - `config.realtime` selects the Action Cable backend (`:solid_cable` by default). Solid Cable keeps Turbo streams in the primary database; set `config.realtime.adapter = :redis` and optionally `config.realtime.redis_url` when hosts prefer Redis.
+- Install generator and dummy initializer list all knobs with comments—update those when slicing future roadmap items.
+
+## Retention Defaults
+
+- Per-source retention settings live on `SourceMonitor::Source` (`items_retention_days` and `max_items`). Negative values are rejected; blank means unlimited.
+- `SourceMonitor::Items::RetentionPruner` runs after every fetch via `SourceMonitor::Fetching::FetchRunner`, pruning stale items and their associated content/logs while keeping counter caches in sync.
+- Age-based rules prune items when their published timestamp (or `created_at` fallback) is older than the configured window. Count-based rules keep the newest N items.
+- `SourceMonitor::ItemCleanupJob` batches retention pruning across sources and can soft delete records (`rake source_monitor:cleanup:items` honours `SOFT_DELETE=true`, `SOURCE_IDS=1,2`). `SourceMonitor::LogCleanupJob` prunes old fetch/scrape logs (`rake source_monitor:cleanup:logs`, override `FETCH_LOG_DAYS` / `SCRAPE_LOG_DAYS`).
+- Nightly recurring entries in `config/recurring.yml` enqueue both cleanup jobs by default; adjust schedules or disable via Solid Queue overrides as needed.
+
+## Coding Style & Naming Conventions
+
+Use two-space indentation and Ruby 3.3 syntax. Keep engine classes under the `SourceMonitor::` namespace; new modules should mirror their directory, e.g., `lib/source_monitor/fetching/pipeline.rb`. Favor service objects ending in `Service`, jobs ending in `Job`, and background channels ending in `Channel`. Rails defaults handle formatting, but run `bundle exec rubocop` (configured via `.rubocop.yml`) before opening a PR. For views, stick with ERB and Tailwind utility classes.
+
+## Testing Guidelines
+
+MiniTest drives coverage (`test/models`, `test/controllers`, `test/system`). Name files with `_test.rb` and wrap suites in `module SourceMonitor`. Add VCR cassettes under `test/vcr_cassettes/` when touching HTTP clients, and record new fixtures with descriptive names like `source_fetch_success.yml`. Target >90% coverage for new services and include regression tests for bug fixes. Use `bin/rails test test/system` before changing UI flows.
+
+## Commit & Pull Request Guidelines
+
+Adopt imperative commit messages in the format `scope: action`, e.g., `sources: enforce URL normalization`. Group unrelated work into separate commits. PRs should describe context, summarise the slice delivered, list validation steps (`bin/rails test`, manual fetch run), and reference roadmap items from `.ai/tasks.md`. Include screenshots or console output when altering UI or background jobs. Request at least one review and ensure CI completes before merge.
+
+## Security & Configuration Tips
+
+Store secrets (API keys, webhook tokens) in `config/credentials/` and never commit plain-text values. When adding HTTP endpoints or webhooks, default to Solid Queue middleware for retries and respect the allowlist in `config/source_monitor.yml`. Document new environment variables in `config/application.yml.sample` and call out any migrations that impact host apps.
+
+# Clean coding guidelines
+
+Object Orientated Design: You should write code that embraces change. Here’s how…
+
+SRP: A class should do the smallest possible useful thing; it should have a single responsibility. A class that has more than one responsibility is difficult to reuse. SRP requires that a class be cohesive, that everything the class does be highly related to its purpose. A class that is easy to reuse will make the application easier to change.
+
+Methods, like classes, should have a single responsibility. All of the same reasons apply, having just one responsibility makes them easy to change and easy to reuse.
+
+DRY: DRY code tolerates change because any change in behaviour can be made by changing code in just one place.
+
+Depend on behaviour not data:
+
+Hide instance variables by wrapping them in a method, an attr_reader. The method changes a variable from data ( which is referenced all over) to behaviour (which is defined once). This means if you need to adjust the data you only have to make a change in one place.
+
+Hide data structures using the Struct class.
+
+Minimise Dependencies: An object depends on another object if, when one object changes, the other might be forced to change in turn. An object has a dependency when it knows:
+
+The name of another class.
+
+The name of a message that it intends to send to someone other than self.
+
+The arguments that a message requires.
+
+The order of those arguments.
+
+Dependency Injection: Addresses the first dependency. It decouples two objects by moving the creation of object_A outside of object_B eg:
+
+object_A.new(1,2,object_B.new)
+
+Encapsulation: Addresses the second dependency. Create wrapper methods so that external messages are isolated in one place. The wrapper method is called throughout the class, rather than sending messages externally.
+
+Use hashes for initialization arguments: this addresses the final dependencies.
+
+Depend on things that change less often than you do
+
+## Agent Notes (2025-10-08)
+
+- Finished roadmap section 05.03 (structured fetch error handling). Added `SourceMonitor::Fetching::FetchError` hierarchy, expanded `FeedFetcher` error handling, and ensured instrumentation payloads include success/error context.
+- Fetch attempts now always create `FetchLog` records and update `Source` failure state; added MiniTest coverage for timeout, HTTP 404, and malformed feed scenarios.
+- Relevant tests: `bundle exec rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb` and full suite via `bundle exec rails test` (both passing).
+- Next roadmap item is 05.04 (not started). Untracked tmp dir `test/lib/tmp` can be safely ignored if it reappears.
+
+## Agent Notes (2025-10-09)
+
+- Introduced `SourceMonitor::Scrapers::Base`, establishing the scraper adapter contract and Result object. Subclasses merge default, source, and invocation settings (HashWithIndifferentAccess) and must return a Result with status/html/content/metadata. Use `SourceMonitor::Scrapers::Base.call(item:, source:, settings:, http:)` to execute adapters.
+- Added Readability scraper adapter leveraging SourceMonitor::HTTP, Nokolexbor, and ruby-readability. Supports override selectors via `scrape_settings[:selectors]`, records extraction metadata including strategy and inferred status, and returns structured failure results on HTTP errors.
+- Extracted scraper HTTP fetching and parsing into dedicated collaborators (`SourceMonitor::Scrapers::Fetchers::HttpFetcher` and `SourceMonitor::Scrapers::Parsers::ReadabilityParser`). The adapter now orchestrates these services, simplifying future parser additions while keeping failure metadata consistent.
+- Added Readability scraper adapter leveraging SourceMonitor::HTTP, Nokolexbor, and ruby-readability. Supports override selectors via `scrape_settings[:selectors]`, records extraction metadata including strategy and inferred status, and returns structured failure results on HTTP errors.
+- Phase 07.03 complete: introduced `SourceMonitor::ItemContent` table with one-to-one association, virtual accessors on `Item`, and migration that backfills existing scraped data while dropping legacy columns. Clearing both fields now removes the content row. Run `bundle exec rails db:migrate` in host apps after upgrading.
+- Phase 07.04 complete: added `SourceMonitor::Scraping::ItemScraper` service with log recording, manual scrape controls in the item admin, and a dedicated scraping configuration section on the source form. Item show page now highlights content comparisons, status badges, and recent scrape errors. System coverage updated to exercise the manual scraping flow.
+- Phase 08.02 complete: created `SourceMonitor::FetchFeedJob`, introduced `SourceMonitor::Fetching::FetchRunner` with Postgres advisory locking, and added a temporary `ScrapeItemJob` stub so new items from auto-scrape sources are queued with `scrape_status` set to `pending`. Manual fetch UI now reuses the runner, and tests cover job retry behavior plus follow-up scraping enqueue (`bundle exec rails test test/lib/source_monitor/fetching/fetch_runner_test.rb`, `bundle exec rails test test/jobs/source_monitor/fetch_feed_job_test.rb`, and `bundle exec rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb`).
+- Phase 08.03 complete: implemented `SourceMonitor::Scraping::Enqueuer` with deduplication/auto-scrape guards, promoted `SourceMonitor::ScrapeItemJob` to execute `ItemScraper`, and updated the manual item view to enqueue jobs instead of running scrapes inline. Added unit coverage for the enqueuer/job plus refreshed the manual scrape system test (`bundle exec rails test test/lib/source_monitor/scraping/enqueuer_test.rb test/jobs/source_monitor/scrape_item_job_test.rb test/system/items_test.rb test/lib/source_monitor/fetching/fetch_runner_test.rb`).
+- Phase 08.04 complete: manual "Fetch Now" now queues `FetchFeedJob` via `FetchRunner.enqueue`, dashboard shows live queue metrics from `SourceMonitor::Jobs::Visibility`, and optional Mission Control link appears when configured. System tests updated to assert job enqueues (`test/system/sources_test.rb`, `test/system/items_test.rb`, `test/system/dashboard_test.rb`); run `bundle exec rails test test/system` after UI tweaks.
+- Install generator now also creates `config/initializers/source_monitor.rb` with documented defaults for queue settings, metrics, and Mission Control hooks. Update the initializer if future configuration options are added.
+- `SourceMonitor.mission_control_dashboard_path` now validates that the configured path exists before rendering a dashboard link, preventing 404s when Mission Control isn't mounted yet. Template comments outline how to mount the engine when enabling it.
+- `test/dummy/bin/dev` now runs the Rails server with `BUNDLE_GEMFILE` pointing at the dummy app, so dummy-only gems (like mission_control-jobs) load correctly while the Tailwind watcher still uses the engine's Gemfile.

data/CHANGELOG.md

@@ -0,0 +1,66 @@
+# Changelog
+
+All notable changes to this project are documented below. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project adheres to Semantic Versioning.
+
+## Release Checklist
+
+1. `rbenv exec bundle exec rails test`
+2. `rbenv exec bundle exec rubocop`
+3. `rbenv exec bundle exec rake app:source_monitor:assets:verify`
+4. `rbenv exec bundle exec gem build source_monitor.gemspec`
+5. Update release notes in this file and tag the release (`git tag vX.Y.Z`)
+6. Push tags and publish the gem (`rbenv exec gem push pkg/source_monitor-X.Y.Z.gem`)
+
+## [Unreleased]
+
+- No unreleased changes yet.
+
+## [0.1.1] - 2025-11-09
+
+### Changed
+
+- Bumped the gem to 0.1.1 so the republished package on RubyGems matches the revamped 0.1.0 release notes without reusing the yanked version number.
+
+### Fixed
+
+- Clarified that the 0.1.0 entry now reflects the authoritative feature overview for the first release, preventing consumers from encountering inconsistent documentation across yanks.
+
+## [0.1.0] - 2025-11-08
+
+### Added
+
+- Shipped the initial SourceMonitor mountable Rails engine with Source and Item models, Tailwind-powered admin UI, Turbo-powered dashboards, and a dummy host app for full-stack validation.
+- Implemented the full feed ingestion pipeline: Feedjira-based fetcher, Faraday HTTP stack with retry/timeout controls, adaptive scheduling, structured error types, retention policies, and fetch log instrumentation surfaced in the UI.
+- Introduced comprehensive scraping support with a scraper adapter base class, Readability parser, dedicated `ItemContent` storage, manual/bulk scrape controls, and queue-backed `ScrapeItemJob` orchestration.
+- Established Solid Queue and Solid Cable defaults, including recurring schedule config, Mission Control hooks, `FetchFeedJob`/`ScheduleFetchesJob`, queue metrics dashboards, and helper APIs for namespaced queue names.
+- Added health monitoring, failure recovery controls, analytics widgets (heatmaps, distribution insights), and notification hooks so operators can triage outages and re-run work with confidence.
+- Delivered install tooling—generator, initializer template, cleanup/retention rake tasks, host harness smoke tests, and example host templates—plus Faraday/HTTP, scraper, retention, realtime, and mission control configuration DSLs.
+
+### Changed
+
+- Rebranded the engine, routes, and namespaces to `SourceMonitor`, aligning configuration defaults, installer output, and docs with the new identity.
+- Modernized the asset and JavaScript pipeline (esbuild, bundler, Stimulus fixes) and widened admin layouts, sortable tables, and bulk action UX for better operator ergonomics.
+- Restructured source member actions into nested REST resources (fetch, retry, bulk scrape) and consolidated log views/analytics for clearer operator workflows.
+
+### Fixed
+
+- Hardened scheduler behavior to avoid duplicate catch-up fetches, ensured stalled fetch recovery paths requeue work, and guaranteed fetch failure callbacks always attach logs/state.
+- Resolved Solid Cable initialization issues, host Action Cable dependencies, and dummy host/environment parity problems so realtime updates function out of the box.
+- Stabilized the host harness across Ruby versions, added Postgres-backed CI services, patched rbenv mismatches, and tightened sqlite shims plus asset/database setup to keep tests green on every platform.
+
+### Documentation
+
+- Published install and upgrade guides, roadmap phase notes, PR workflow requirements, health configuration guidance, and mission control instructions; expanded AGENT guidance for future contributors.
+
+### CI/CD
+
+- Added layered coverage guardrails (diff coverage enforcement, result-set merging, targeted health coverage suites), automated release verification, and artifact preservation across the packaging workflow.
+- Upgraded GitHub Actions dependencies, introduced reusable workflows for test/lint/build jobs, and ensured release verification prepares databases, locks dependencies, and emits the packaged gem.
+
+### Upgrade Notes
+
+1. Add `gem "source_monitor", "~> 0.1.0"` to your host `Gemfile` and run `rbenv exec bundle install`.
+2. Execute `rbenv exec bin/rails railties:install:migrations FROM=source_monitor` followed by `rbenv exec bin/rails db:migrate` to copy and run Solid Queue + SourceMonitor migrations.
+3. Review `config/initializers/source_monitor.rb` for queue, scraping, retention, HTTP, and Mission Control settings; adjust the generated defaults to fit your environment.
+4. If you surface Mission Control Jobs from the dashboard, ensure `mission_control-jobs` stays mounted and `SourceMonitor.mission_control_dashboard_path` resolves correctly.
+5. Restart Solid Queue workers, Solid Cable (or Redis Action Cable), and any recurring job runners to pick up the new engine version.

data/CONTRIBUTING.md

@@ -0,0 +1,31 @@
+# Contributing Guidelines
+
+This project ships each roadmap slice behind a fast, reliable test loop. The notes below summarize the expectations for day-to-day development and code review.
+
+## Test Suite Expectations
+
+- **Fast feedback:** run `bundle exec rake app:test:smoke` for unit, helper, and job coverage before pushing. Use `SOURCE_MONITOR_TEST_WORKERS=1` locally when profiling failures for determinism.
+- **Full validation:** continue to run `bundle exec rails test` (or `bin/test-coverage` in CI) before marking a slice ready for review.
+- **Background jobs:** the default adapter is `:test`. Switch to inline execution only for the precise block that needs it via `with_inline_jobs { ... }`; never flip the global adapter.
+- **Database usage:** prefer transactional fixtures over manual `delete_all`. Reach for `setup_once` (TestProf’s `before_all`) when immutable data can be shared safely across examples.
+- **System tests:** keep them focused on UI workflows. If a test only exercises server logic, convert it to a controller/lib test. Stub external services and avoid long flows.
+
+## Performance Profiling
+
+- **Local commands:**
+  - `TAG_PROF=type SOURCE_MONITOR_TEST_WORKERS=1 bundle exec rails test`
+  - `EVENT_PROF=sql.active_record SOURCE_MONITOR_TEST_WORKERS=1 bundle exec rails test`
+  - `TEST_STACK_PROF=1 SOURCE_MONITOR_TEST_WORKERS=1 bundle exec rails test test/integration`
+- **CI automation:** a scheduled `profiling` workflow runs the commands above nightly, archives `tmp/test_prof`, and applies guardrails via `bin/check-test-prof-metrics` (suite ≤ 80s, integration ≤ 35s, DB time ≤ 5s). Failures block the job so regressions surface quickly.
+
+## Code Review Checklist
+
+Before approval, verify that the changes:
+
+1. Avoid unnecessary database persistence (`build_stubbed`/test doubles over `.create` when assertions allow).
+2. Share heavy setup with `setup_once` or helper memoization instead of per-test rebuilds.
+3. Scope inline job execution to `with_inline_jobs { ... }` blocks—no suite-wide adapter swaps.
+4. Keep new or modified system specs lean, relying on lower-level coverage when UI is not essential.
+5. Include updates to smoke/full-test instructions or profiling docs when new flows require them.
+
+Thanks for keeping the suite fast and predictable. Profile early and often, and treat the guardrails as hard limits when reviewing new slices.

data/Gemfile

@@ -0,0 +1,30 @@
+source "https://rubygems.org"
+
+# Specify your gem's dependencies in source_monitor.gemspec.
+gemspec
+
+gem "puma"
+
+gem "pg"
+
+gem "propshaft"
+
+# Omakase Ruby styling [https://github.com/rails/rubocop-rails-omakase/]
+gem "rubocop-rails-omakase", require: false
+
+group :development, :test do
+  gem "brakeman", require: false
+end
+
+# Start debugger with binding.b [https://github.com/ruby/debug]
+# gem "debug", ">= 1.0.0"
+
+group :test do
+  gem "simplecov", require: false
+  gem "test-prof", require: false
+  gem "stackprof", require: false
+  gem "capybara"
+  gem "webmock"
+  gem "vcr"
+  gem "selenium-webdriver"
+end