source_monitor 0.2.1 → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.claude/agents/rails-concern.md +464 -0
- data/.claude/agents/rails-controller.md +424 -0
- data/.claude/agents/rails-hotwire.md +446 -0
- data/.claude/agents/rails-implement.md +374 -0
- data/.claude/agents/rails-job.md +334 -0
- data/.claude/agents/rails-lint.md +294 -0
- data/.claude/agents/rails-mailer.md +371 -0
- data/.claude/agents/rails-migration.md +449 -0
- data/.claude/agents/rails-model.md +420 -0
- data/.claude/agents/rails-policy.md +443 -0
- data/.claude/agents/rails-presenter.md +427 -0
- data/.claude/agents/rails-query.md +412 -0
- data/.claude/agents/rails-review.md +490 -0
- data/.claude/agents/rails-service.md +458 -0
- data/.claude/agents/rails-state-records.md +465 -0
- data/.claude/agents/rails-tdd.md +314 -0
- data/.claude/agents/rails-test.md +441 -0
- data/.claude/agents/rails-view-component.md +418 -0
- data/.claude/hooks/block-secrets.sh +52 -0
- data/.claude/settings.json +85 -0
- data/.claude/skills/action-cable-patterns/SKILL.md +296 -0
- data/.claude/skills/action-mailer-patterns/SKILL.md +295 -0
- data/.claude/skills/active-storage-setup/SKILL.md +311 -0
- data/.claude/skills/api-versioning/SKILL.md +294 -0
- data/.claude/skills/authentication-flow/SKILL.md +335 -0
- data/.claude/skills/authentication-flow/reference/current.md +248 -0
- data/.claude/skills/authentication-flow/reference/passwordless.md +253 -0
- data/.claude/skills/authentication-flow/reference/sessions.md +201 -0
- data/.claude/skills/authorization-pundit/SKILL.md +462 -0
- data/.claude/skills/caching-strategies/SKILL.md +350 -0
- data/.claude/skills/database-migrations/SKILL.md +354 -0
- data/.claude/skills/form-object-patterns/SKILL.md +399 -0
- data/.claude/skills/hotwire-patterns/SKILL.md +247 -0
- data/.claude/skills/hotwire-patterns/reference/stimulus.md +307 -0
- data/.claude/skills/hotwire-patterns/reference/tailwind-integration.md +112 -0
- data/.claude/skills/hotwire-patterns/reference/turbo-frames.md +158 -0
- data/.claude/skills/hotwire-patterns/reference/turbo-streams.md +218 -0
- data/.claude/skills/i18n-patterns/SKILL.md +320 -0
- data/.claude/skills/install/SKILL.md +367 -0
- data/.claude/skills/performance-optimization/SKILL.md +311 -0
- data/.claude/skills/rails-architecture/SKILL.md +259 -0
- data/.claude/skills/rails-architecture/reference/error-handling.md +333 -0
- data/.claude/skills/rails-architecture/reference/event-tracking.md +142 -0
- data/.claude/skills/rails-architecture/reference/layer-interactions.md +417 -0
- data/.claude/skills/rails-architecture/reference/multi-tenancy.md +152 -0
- data/.claude/skills/rails-architecture/reference/query-patterns.md +342 -0
- data/.claude/skills/rails-architecture/reference/service-patterns.md +286 -0
- data/.claude/skills/rails-architecture/reference/state-records.md +250 -0
- data/.claude/skills/rails-architecture/reference/testing-strategy.md +326 -0
- data/.claude/skills/rails-concern/SKILL.md +399 -0
- data/.claude/skills/rails-controller/SKILL.md +336 -0
- data/.claude/skills/rails-model-generator/SKILL.md +321 -0
- data/.claude/skills/rails-model-generator/reference/validations.md +298 -0
- data/.claude/skills/rails-presenter/SKILL.md +274 -0
- data/.claude/skills/rails-query-object/SKILL.md +289 -0
- data/.claude/skills/rails-service-object/SKILL.md +349 -0
- data/.claude/skills/sm-architecture/SKILL.md +233 -0
- data/.claude/skills/sm-architecture/reference/extraction-patterns.md +192 -0
- data/.claude/skills/sm-architecture/reference/module-map.md +194 -0
- data/.claude/skills/sm-configuration-setting/SKILL.md +264 -0
- data/.claude/skills/sm-configuration-setting/reference/settings-catalog.md +248 -0
- data/.claude/skills/sm-configuration-setting/reference/settings-pattern.md +297 -0
- data/.claude/skills/sm-configure/SKILL.md +153 -0
- data/.claude/skills/sm-configure/reference/configuration-reference.md +321 -0
- data/.claude/skills/sm-dashboard-widget/SKILL.md +344 -0
- data/.claude/skills/sm-dashboard-widget/reference/dashboard-patterns.md +304 -0
- data/.claude/skills/sm-domain-model/SKILL.md +188 -0
- data/.claude/skills/sm-domain-model/reference/model-graph.md +114 -0
- data/.claude/skills/sm-domain-model/reference/table-structure.md +348 -0
- data/.claude/skills/sm-engine-migration/SKILL.md +395 -0
- data/.claude/skills/sm-engine-migration/reference/migration-conventions.md +255 -0
- data/.claude/skills/sm-engine-test/SKILL.md +302 -0
- data/.claude/skills/sm-engine-test/reference/test-helpers.md +259 -0
- data/.claude/skills/sm-engine-test/reference/test-patterns.md +411 -0
- data/.claude/skills/sm-event-handler/SKILL.md +265 -0
- data/.claude/skills/sm-event-handler/reference/events-api.md +229 -0
- data/.claude/skills/sm-health-rule/SKILL.md +327 -0
- data/.claude/skills/sm-health-rule/reference/health-system.md +269 -0
- data/.claude/skills/sm-host-setup/SKILL.md +223 -0
- data/.claude/skills/sm-host-setup/reference/initializer-template.md +195 -0
- data/.claude/skills/sm-host-setup/reference/setup-checklist.md +134 -0
- data/.claude/skills/sm-job/SKILL.md +263 -0
- data/.claude/skills/sm-job/reference/job-conventions.md +245 -0
- data/.claude/skills/sm-model-extension/SKILL.md +287 -0
- data/.claude/skills/sm-model-extension/reference/extension-api.md +317 -0
- data/.claude/skills/sm-pipeline-stage/SKILL.md +254 -0
- data/.claude/skills/sm-pipeline-stage/reference/completion-handlers.md +152 -0
- data/.claude/skills/sm-pipeline-stage/reference/entry-processing.md +191 -0
- data/.claude/skills/sm-pipeline-stage/reference/feed-fetcher-architecture.md +198 -0
- data/.claude/skills/sm-scraper-adapter/SKILL.md +284 -0
- data/.claude/skills/sm-scraper-adapter/reference/adapter-contract.md +167 -0
- data/.claude/skills/sm-scraper-adapter/reference/example-adapter.md +274 -0
- data/.claude/skills/solid-queue-setup/SKILL.md +307 -0
- data/.claude/skills/tdd-cycle/SKILL.md +359 -0
- data/.claude/skills/viewcomponent-patterns/SKILL.md +333 -0
- data/.rubocop.yml +2 -0
- data/.ruby-version +1 -1
- data/.vbw-planning/.notification-log.jsonl +246 -0
- data/.vbw-planning/.session-log.jsonl +992 -0
- data/.vbw-planning/PROJECT.md +51 -0
- data/.vbw-planning/REQUIREMENTS.md +50 -0
- data/.vbw-planning/SHIPPED.md +28 -0
- data/.vbw-planning/codebase/ARCHITECTURE.md +147 -0
- data/.vbw-planning/codebase/CONCERNS.md +99 -0
- data/.vbw-planning/codebase/CONVENTIONS.md +97 -0
- data/.vbw-planning/codebase/DEPENDENCIES.md +100 -0
- data/.vbw-planning/codebase/INDEX.md +86 -0
- data/.vbw-planning/codebase/META.md +42 -0
- data/.vbw-planning/codebase/PATTERNS.md +262 -0
- data/.vbw-planning/codebase/STACK.md +101 -0
- data/.vbw-planning/codebase/STRUCTURE.md +324 -0
- data/.vbw-planning/codebase/TESTING.md +154 -0
- data/.vbw-planning/config.json +12 -0
- data/.vbw-planning/discovery.json +24 -0
- data/.vbw-planning/milestones/default/ROADMAP.md +115 -0
- data/.vbw-planning/milestones/default/STATE.md +83 -0
- data/.vbw-planning/milestones/default/phases/01-coverage-analysis-quick-wins/PLAN-01-SUMMARY.md +56 -0
- data/.vbw-planning/milestones/default/phases/01-coverage-analysis-quick-wins/PLAN-01.md +187 -0
- data/.vbw-planning/milestones/default/phases/01-coverage-analysis-quick-wins/PLAN-02-SUMMARY.md +64 -0
- data/.vbw-planning/milestones/default/phases/01-coverage-analysis-quick-wins/PLAN-02.md +137 -0
- data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-01-SUMMARY.md +67 -0
- data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-01.md +142 -0
- data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-02-SUMMARY.md +64 -0
- data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-02.md +138 -0
- data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-03-SUMMARY.md +85 -0
- data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-03.md +147 -0
- data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-04-SUMMARY.md +63 -0
- data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-04.md +129 -0
- data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-05-SUMMARY.md +74 -0
- data/.vbw-planning/milestones/default/phases/02-critical-path-test-coverage/PLAN-05.md +154 -0
- data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/03-VERIFICATION-wave1.md +303 -0
- data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/03-VERIFICATION.md +510 -0
- data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-01-SUMMARY.md +61 -0
- data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-01.md +161 -0
- data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-02-SUMMARY.md +66 -0
- data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-02.md +132 -0
- data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-03-SUMMARY.md +59 -0
- data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-03.md +171 -0
- data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-04-SUMMARY.md +56 -0
- data/.vbw-planning/milestones/default/phases/03-large-file-refactoring/PLAN-04.md +152 -0
- data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/04-CONTEXT.md +33 -0
- data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/PLAN-01-SUMMARY.md +42 -0
- data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/PLAN-01.md +119 -0
- data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/PLAN-02-SUMMARY.md +52 -0
- data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/PLAN-02.md +195 -0
- data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/PLAN-03-SUMMARY.md +79 -0
- data/.vbw-planning/milestones/default/phases/04-code-quality-conventions-cleanup/PLAN-03.md +130 -0
- data/CHANGELOG.md +37 -0
- data/CLAUDE.md +222 -0
- data/Gemfile +8 -0
- data/Gemfile.lock +132 -120
- data/Rakefile +2 -0
- data/app/controllers/source_monitor/application_controller.rb +2 -0
- data/app/controllers/source_monitor/health_controller.rb +2 -0
- data/app/controllers/source_monitor/import_sessions/bulk_configuration.rb +106 -0
- data/app/controllers/source_monitor/import_sessions/entry_annotation.rb +187 -0
- data/app/controllers/source_monitor/import_sessions/health_check_management.rb +112 -0
- data/app/controllers/source_monitor/import_sessions/opml_parser.rb +130 -0
- data/app/controllers/source_monitor/import_sessions_controller.rb +6 -507
- data/app/controllers/source_monitor/items_controller.rb +2 -0
- data/app/controllers/source_monitor/sources_controller.rb +0 -14
- data/app/helpers/source_monitor/application_helper.rb +4 -112
- data/app/helpers/source_monitor/health_badge_helper.rb +69 -0
- data/app/helpers/source_monitor/table_sort_helper.rb +53 -0
- data/app/jobs/source_monitor/application_job.rb +2 -0
- data/app/models/source_monitor/application_record.rb +2 -0
- data/app/models/source_monitor/log_entry.rb +0 -2
- data/config/coverage_baseline.json +217 -1862
- data/config/routes.rb +2 -0
- data/db/migrate/20251009103000_add_feed_content_readability_to_sources.rb +2 -0
- data/db/migrate/20251014171659_add_performance_indexes.rb +2 -0
- data/db/migrate/20251014172525_add_fetch_status_check_constraint.rb +2 -0
- data/db/migrate/20251108120116_refresh_fetch_status_constraint.rb +2 -0
- data/db/migrate/20260210204022_add_composite_index_to_log_entries.rb +17 -0
- data/lib/source_monitor/assets/bundler.rb +2 -0
- data/lib/source_monitor/assets.rb +2 -0
- data/lib/source_monitor/configuration/authentication_settings.rb +62 -0
- data/lib/source_monitor/configuration/events.rb +60 -0
- data/lib/source_monitor/configuration/fetching_settings.rb +27 -0
- data/lib/source_monitor/configuration/health_settings.rb +27 -0
- data/lib/source_monitor/configuration/http_settings.rb +43 -0
- data/lib/source_monitor/configuration/model_definition.rb +108 -0
- data/lib/source_monitor/configuration/models.rb +36 -0
- data/lib/source_monitor/configuration/realtime_settings.rb +95 -0
- data/lib/source_monitor/configuration/retention_settings.rb +45 -0
- data/lib/source_monitor/configuration/scraper_registry.rb +67 -0
- data/lib/source_monitor/configuration/scraping_settings.rb +39 -0
- data/lib/source_monitor/configuration/validation_definition.rb +32 -0
- data/lib/source_monitor/configuration.rb +12 -579
- data/lib/source_monitor/dashboard/queries/recent_activity_query.rb +138 -0
- data/lib/source_monitor/dashboard/queries/stats_query.rb +71 -0
- data/lib/source_monitor/dashboard/queries.rb +2 -195
- data/lib/source_monitor/engine.rb +2 -0
- data/lib/source_monitor/fetching/feed_fetcher/adaptive_interval.rb +141 -0
- data/lib/source_monitor/fetching/feed_fetcher/entry_processor.rb +89 -0
- data/lib/source_monitor/fetching/feed_fetcher/source_updater.rb +200 -0
- data/lib/source_monitor/fetching/feed_fetcher.rb +37 -379
- data/lib/source_monitor/items/item_creator/content_extractor.rb +113 -0
- data/lib/source_monitor/items/item_creator/entry_parser/media_extraction.rb +96 -0
- data/lib/source_monitor/items/item_creator/entry_parser.rb +294 -0
- data/lib/source_monitor/items/item_creator.rb +28 -455
- data/lib/source_monitor/setup/bundle_installer.rb +2 -0
- data/lib/source_monitor/setup/cli.rb +2 -0
- data/lib/source_monitor/setup/dependency_checker.rb +2 -0
- data/lib/source_monitor/setup/detectors.rb +2 -0
- data/lib/source_monitor/setup/gemfile_editor.rb +2 -0
- data/lib/source_monitor/setup/initializer_patcher.rb +2 -0
- data/lib/source_monitor/setup/install_generator.rb +2 -0
- data/lib/source_monitor/setup/migration_installer.rb +2 -0
- data/lib/source_monitor/setup/node_installer.rb +2 -0
- data/lib/source_monitor/setup/prompter.rb +2 -0
- data/lib/source_monitor/setup/requirements.rb +2 -0
- data/lib/source_monitor/setup/shell_runner.rb +2 -0
- data/lib/source_monitor/setup/verification/action_cable_verifier.rb +2 -0
- data/lib/source_monitor/setup/verification/printer.rb +2 -0
- data/lib/source_monitor/setup/verification/result.rb +2 -0
- data/lib/source_monitor/setup/verification/runner.rb +2 -0
- data/lib/source_monitor/setup/verification/solid_queue_verifier.rb +2 -0
- data/lib/source_monitor/setup/verification/telemetry_logger.rb +2 -0
- data/lib/source_monitor/setup/workflow.rb +19 -2
- data/lib/source_monitor/version.rb +3 -1
- data/lib/source_monitor.rb +140 -58
- data/lib/tasks/source_monitor_assets.rake +2 -0
- data/lib/tasks/source_monitor_setup.rake +60 -0
- data/lib/tasks/source_monitor_tasks.rake +2 -0
- data/source_monitor.gemspec +4 -1
- metadata +177 -4
|
@@ -0,0 +1,192 @@
|
|
|
1
|
+
# SourceMonitor Extraction Patterns
|
|
2
|
+
|
|
3
|
+
Patterns used during Phase 3 and Phase 4 refactoring to decompose large classes into focused sub-modules.
|
|
4
|
+
|
|
5
|
+
## Pattern 1: Sub-Module Extraction (FeedFetcher)
|
|
6
|
+
|
|
7
|
+
**Before:** `FeedFetcher` was 627 lines handling HTTP requests, response parsing, source state updates, adaptive interval calculation, and entry processing.
|
|
8
|
+
|
|
9
|
+
**After:** 285 lines in the main class + 3 sub-modules.
|
|
10
|
+
|
|
11
|
+
### Structure
|
|
12
|
+
|
|
13
|
+
```
|
|
14
|
+
lib/source_monitor/fetching/
|
|
15
|
+
feed_fetcher.rb # 285 lines - orchestrator
|
|
16
|
+
feed_fetcher/
|
|
17
|
+
adaptive_interval.rb # Interval calculation logic
|
|
18
|
+
source_updater.rb # Source state updates + fetch log creation
|
|
19
|
+
entry_processor.rb # Feed entry iteration + ItemCreator calls
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
### Technique
|
|
23
|
+
|
|
24
|
+
1. **Create sub-directory** matching the parent class name
|
|
25
|
+
2. **Extract cohesive responsibilities** into separate classes
|
|
26
|
+
3. **Pass dependencies via constructor** (source, adaptive_interval)
|
|
27
|
+
4. **Lazy accessor pattern** in parent:
|
|
28
|
+
```ruby
|
|
29
|
+
def source_updater
|
|
30
|
+
@source_updater ||= SourceUpdater.new(source: source, adaptive_interval: adaptive_interval)
|
|
31
|
+
end
|
|
32
|
+
```
|
|
33
|
+
5. **Forwarding methods** for backward compatibility with tests:
|
|
34
|
+
```ruby
|
|
35
|
+
def process_feed_entries(feed) = entry_processor.process_feed_entries(feed)
|
|
36
|
+
def jitter_offset(interval_seconds) = adaptive_interval.jitter_offset(interval_seconds)
|
|
37
|
+
```
|
|
38
|
+
6. **Require in parent file** (not autoloaded -- explicit require):
|
|
39
|
+
```ruby
|
|
40
|
+
require "source_monitor/fetching/feed_fetcher/adaptive_interval"
|
|
41
|
+
require "source_monitor/fetching/feed_fetcher/source_updater"
|
|
42
|
+
require "source_monitor/fetching/feed_fetcher/entry_processor"
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
### Key Design Decisions
|
|
46
|
+
|
|
47
|
+
- `AdaptiveInterval` is a pure calculator -- no side effects, receives source for reading config
|
|
48
|
+
- `SourceUpdater` handles all `source.update!` calls and fetch log creation
|
|
49
|
+
- `EntryProcessor` iterates entries and fires events (item_processors, after_item_created)
|
|
50
|
+
- Parent `FeedFetcher` remains the public API (`#call`) and coordinates the pipeline
|
|
51
|
+
|
|
52
|
+
---
|
|
53
|
+
|
|
54
|
+
## Pattern 2: Configuration Decomposition
|
|
55
|
+
|
|
56
|
+
**Before:** `Configuration` was 655 lines with all settings inline.
|
|
57
|
+
|
|
58
|
+
**After:** 87 lines composing 12 standalone settings objects.
|
|
59
|
+
|
|
60
|
+
### Structure
|
|
61
|
+
|
|
62
|
+
```
|
|
63
|
+
lib/source_monitor/configuration.rb # 87 lines - composer
|
|
64
|
+
lib/source_monitor/configuration/
|
|
65
|
+
http_settings.rb
|
|
66
|
+
fetching_settings.rb
|
|
67
|
+
health_settings.rb
|
|
68
|
+
scraping_settings.rb
|
|
69
|
+
realtime_settings.rb
|
|
70
|
+
retention_settings.rb
|
|
71
|
+
authentication_settings.rb
|
|
72
|
+
scraper_registry.rb
|
|
73
|
+
events.rb
|
|
74
|
+
validation_definition.rb
|
|
75
|
+
model_definition.rb
|
|
76
|
+
models.rb
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
### Technique
|
|
80
|
+
|
|
81
|
+
1. **One settings class per domain** (HTTP, fetching, health, etc.)
|
|
82
|
+
2. **Composition via attr_reader** in parent:
|
|
83
|
+
```ruby
|
|
84
|
+
attr_reader :http, :scrapers, :retention, :events, :models,
|
|
85
|
+
:realtime, :fetching, :health, :authentication, :scraping
|
|
86
|
+
```
|
|
87
|
+
3. **Initialize all in constructor**:
|
|
88
|
+
```ruby
|
|
89
|
+
def initialize
|
|
90
|
+
@http = HTTPSettings.new
|
|
91
|
+
@fetching = FetchingSettings.new
|
|
92
|
+
# ...
|
|
93
|
+
end
|
|
94
|
+
```
|
|
95
|
+
4. **Each settings class is a PORO** with `attr_accessor` and sensible defaults
|
|
96
|
+
5. **Explicit require** (not autoloaded) since Configuration is boot-critical
|
|
97
|
+
|
|
98
|
+
### Key Design Decisions
|
|
99
|
+
|
|
100
|
+
- Settings objects are simple POROs, not ActiveModel objects
|
|
101
|
+
- No validation at settings level -- validated at usage point
|
|
102
|
+
- Host app accesses via `config.http.timeout = 30` (dot-chain)
|
|
103
|
+
- Reset via `@config = Configuration.new` (new object, not clearing fields)
|
|
104
|
+
|
|
105
|
+
---
|
|
106
|
+
|
|
107
|
+
## Pattern 3: Controller Concern Extraction (ImportSessionsController)
|
|
108
|
+
|
|
109
|
+
**Before:** `ImportSessionsController` was 792 lines with wizard logic, health checks, and OPML parsing.
|
|
110
|
+
|
|
111
|
+
**After:** 295 lines + 4 concerns.
|
|
112
|
+
|
|
113
|
+
### Structure
|
|
114
|
+
|
|
115
|
+
```
|
|
116
|
+
app/controllers/source_monitor/import_sessions_controller.rb # 295 lines
|
|
117
|
+
app/controllers/concerns/source_monitor/import_sessions/
|
|
118
|
+
step_navigation.rb # Wizard step logic
|
|
119
|
+
health_checking.rb # Health check actions
|
|
120
|
+
source_selection.rb # Source selection/deselection
|
|
121
|
+
import_execution.rb # Final import execution
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
### Technique
|
|
125
|
+
|
|
126
|
+
1. **Group by wizard step/feature** -- each concern handles a coherent set of actions
|
|
127
|
+
2. **Include in controller**:
|
|
128
|
+
```ruby
|
|
129
|
+
include ImportSessions::StepNavigation
|
|
130
|
+
include ImportSessions::HealthChecking
|
|
131
|
+
```
|
|
132
|
+
3. **Share state via controller methods** (e.g., `@import_session`, `current_user`)
|
|
133
|
+
4. **Before-action filters** stay in main controller for clarity
|
|
134
|
+
|
|
135
|
+
---
|
|
136
|
+
|
|
137
|
+
## Pattern 4: Processor Extraction (ItemCreator)
|
|
138
|
+
|
|
139
|
+
**Before:** `ItemCreator` was 601 lines handling entry parsing, content extraction, readability processing, and item persistence.
|
|
140
|
+
|
|
141
|
+
**After:** 174 lines + EntryParser (390 lines) + ContentExtractor (113 lines).
|
|
142
|
+
|
|
143
|
+
### Structure
|
|
144
|
+
|
|
145
|
+
```
|
|
146
|
+
lib/source_monitor/items/
|
|
147
|
+
item_creator.rb # 174 lines - orchestrator
|
|
148
|
+
item_creator/
|
|
149
|
+
entry_parser.rb # 390 lines - parse feed entries
|
|
150
|
+
entry_parser/media_extraction.rb # Media parsing concern
|
|
151
|
+
content_extractor.rb # 113 lines - readability processing
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
### Technique
|
|
155
|
+
|
|
156
|
+
1. **EntryParser** handles all Feedjira entry field extraction:
|
|
157
|
+
- URL extraction, timestamp parsing, author normalization
|
|
158
|
+
- GUID generation, fingerprint calculation
|
|
159
|
+
- Category/tag/keyword parsing, media extraction
|
|
160
|
+
- Includes `MediaExtraction` concern for media-specific parsing
|
|
161
|
+
|
|
162
|
+
2. **ContentExtractor** handles readability content processing:
|
|
163
|
+
- Decision logic for when to process content
|
|
164
|
+
- HTML wrapping for readability parser
|
|
165
|
+
- Result metadata building
|
|
166
|
+
|
|
167
|
+
3. **Parent ItemCreator** remains the public API (`#call`, `.call`) and handles:
|
|
168
|
+
- Duplicate detection (by GUID or fingerprint)
|
|
169
|
+
- Create vs update decision
|
|
170
|
+
- Concurrent duplicate handling (rescue RecordNotUnique)
|
|
171
|
+
|
|
172
|
+
4. **Forwarding methods** for backward compatibility (same as FeedFetcher pattern)
|
|
173
|
+
|
|
174
|
+
---
|
|
175
|
+
|
|
176
|
+
## Common Principles Across All Extractions
|
|
177
|
+
|
|
178
|
+
1. **Public API stays on the parent** -- callers don't need to change
|
|
179
|
+
2. **Backward-compatible forwarding** -- old test callsites keep working
|
|
180
|
+
3. **Constructor injection** -- dependencies passed in, not looked up globally
|
|
181
|
+
4. **Lazy accessors** -- sub-modules created on first use
|
|
182
|
+
5. **Explicit require** for sub-modules (not autoloaded) since parent requires them
|
|
183
|
+
6. **Cohesion over size** -- extract by responsibility, not arbitrary line count
|
|
184
|
+
7. **No inheritance** -- composition via delegation, not subclassing
|
|
185
|
+
|
|
186
|
+
## When to Apply
|
|
187
|
+
|
|
188
|
+
Use sub-module extraction when a class has:
|
|
189
|
+
- 3+ distinct responsibilities that can be named
|
|
190
|
+
- Methods that cluster into groups with different collaborators
|
|
191
|
+
- Test files that are hard to navigate due to mixed concerns
|
|
192
|
+
- A clear "orchestrator" role that coordinates the extracted pieces
|
|
@@ -0,0 +1,194 @@
|
|
|
1
|
+
# SourceMonitor Module Map
|
|
2
|
+
|
|
3
|
+
Complete module tree with each module's responsibility.
|
|
4
|
+
|
|
5
|
+
## Top-Level Modules (Explicit Require)
|
|
6
|
+
|
|
7
|
+
| Module | File | Responsibility |
|
|
8
|
+
|--------|------|----------------|
|
|
9
|
+
| `SourceMonitor` | `lib/source_monitor.rb` | Namespace root, configure/reset API, autoload declarations |
|
|
10
|
+
| `Engine` | `lib/source_monitor/engine.rb` | Rails engine setup, isolate_namespace, initializers |
|
|
11
|
+
| `Configuration` | `lib/source_monitor/configuration.rb` | Central config object, composes 12 settings objects |
|
|
12
|
+
| `ModelExtensions` | `lib/source_monitor/model_extensions.rb` | Dynamic table names, concern/validation injection |
|
|
13
|
+
| `Events` | `lib/source_monitor/events.rb` | Lifecycle event dispatch (item created, scraped, fetch completed) |
|
|
14
|
+
| `Instrumentation` | `lib/source_monitor/instrumentation.rb` | ActiveSupport::Notifications wrapper |
|
|
15
|
+
| `Metrics` | `lib/source_monitor/metrics.rb` | Counter/gauge tracking, notification subscribers |
|
|
16
|
+
| `Health` | `lib/source_monitor/health.rb` | Health monitoring setup, fetch callback registration |
|
|
17
|
+
| `Realtime` | `lib/source_monitor/realtime.rb` | ActionCable/Turbo Streams adapter and broadcaster setup |
|
|
18
|
+
| `FeedJiraExtensions` | `lib/source_monitor/feedjira_extensions.rb` | Feedjira monkey-patches/extensions |
|
|
19
|
+
|
|
20
|
+
## Autoloaded Modules
|
|
21
|
+
|
|
22
|
+
### SourceMonitor (Root Level)
|
|
23
|
+
|
|
24
|
+
| Module | File | Responsibility |
|
|
25
|
+
|--------|------|----------------|
|
|
26
|
+
| `HTTP` | `http.rb` | Faraday client factory with configurable timeouts, user-agent, headers |
|
|
27
|
+
| `Scheduler` | `scheduler.rb` | Coordinates scheduled fetch jobs |
|
|
28
|
+
| `Assets` | `assets.rb` | Asset path resolution helpers |
|
|
29
|
+
|
|
30
|
+
### Analytics
|
|
31
|
+
|
|
32
|
+
| Module | File | Responsibility |
|
|
33
|
+
|--------|------|----------------|
|
|
34
|
+
| `SourceFetchIntervalDistribution` | `analytics/source_fetch_interval_distribution.rb` | Distribution stats for fetch intervals |
|
|
35
|
+
| `SourceActivityRates` | `analytics/source_activity_rates.rb` | Item creation rates per source |
|
|
36
|
+
| `SourcesIndexMetrics` | `analytics/sources_index_metrics.rb` | Aggregate metrics for sources index |
|
|
37
|
+
|
|
38
|
+
### Dashboard
|
|
39
|
+
|
|
40
|
+
| Module | File | Responsibility |
|
|
41
|
+
|--------|------|----------------|
|
|
42
|
+
| `QuickAction` | `dashboard/quick_action.rb` | Quick action data object |
|
|
43
|
+
| `QuickActionsPresenter` | `dashboard/quick_actions_presenter.rb` | Format quick actions for view |
|
|
44
|
+
| `RecentActivity` | `dashboard/recent_activity.rb` | Recent activity query |
|
|
45
|
+
| `RecentActivityPresenter` | `dashboard/recent_activity_presenter.rb` | Format activity for view |
|
|
46
|
+
| `Queries` | `dashboard/queries.rb` | Dashboard aggregate queries |
|
|
47
|
+
| `TurboBroadcaster` | `dashboard/turbo_broadcaster.rb` | Broadcast dashboard updates |
|
|
48
|
+
| `UpcomingFetchSchedule` | `dashboard/upcoming_fetch_schedule.rb` | Next-fetch schedule display |
|
|
49
|
+
|
|
50
|
+
### Fetching (Feed Fetch Pipeline)
|
|
51
|
+
|
|
52
|
+
| Module | File | Responsibility |
|
|
53
|
+
|--------|------|----------------|
|
|
54
|
+
| `FeedFetcher` | `fetching/feed_fetcher.rb` | Main fetch orchestrator: request, parse, process entries, update source |
|
|
55
|
+
| `FeedFetcher::AdaptiveInterval` | `fetching/feed_fetcher/adaptive_interval.rb` | Compute next fetch interval based on content changes |
|
|
56
|
+
| `FeedFetcher::SourceUpdater` | `fetching/feed_fetcher/source_updater.rb` | Update source record after fetch (success/failure/not-modified) |
|
|
57
|
+
| `FeedFetcher::EntryProcessor` | `fetching/feed_fetcher/entry_processor.rb` | Iterate feed entries, call ItemCreator, fire events |
|
|
58
|
+
| `FetchRunner` | `fetching/fetch_runner.rb` | Job-level coordinator: acquire lock, run FeedFetcher, handle completion |
|
|
59
|
+
| `RetryPolicy` | `fetching/retry_policy.rb` | Retry/circuit-breaker decision logic |
|
|
60
|
+
| `StalledFetchReconciler` | `fetching/stalled_fetch_reconciler.rb` | Reset sources stuck in "fetching" status |
|
|
61
|
+
| `AdvisoryLock` | `fetching/advisory_lock.rb` | PostgreSQL advisory lock wrapper |
|
|
62
|
+
| `FetchError` | `fetching/fetch_error.rb` | Error hierarchy (TimeoutError, ConnectionError, HTTPError, ParsingError, UnexpectedResponseError) |
|
|
63
|
+
|
|
64
|
+
### Items
|
|
65
|
+
|
|
66
|
+
| Module | File | Responsibility |
|
|
67
|
+
|--------|------|----------------|
|
|
68
|
+
| `ItemCreator` | `items/item_creator.rb` | Create or update Item from feed entry |
|
|
69
|
+
| `ItemCreator::EntryParser` | `items/item_creator/entry_parser.rb` | Parse Feedjira entry into attribute hash |
|
|
70
|
+
| `ItemCreator::ContentExtractor` | `items/item_creator/content_extractor.rb` | Process content through readability parser |
|
|
71
|
+
| `RetentionPruner` | `items/retention_pruner.rb` | Prune items by age/count per source |
|
|
72
|
+
| `RetentionStrategies` | `items/retention_strategies.rb` | Strategy pattern for retention |
|
|
73
|
+
| `RetentionStrategies::Destroy` | `items/retention_strategies/destroy.rb` | Hard-delete retention strategy |
|
|
74
|
+
| `RetentionStrategies::SoftDelete` | `items/retention_strategies/soft_delete.rb` | Soft-delete retention strategy |
|
|
75
|
+
|
|
76
|
+
### ImportSessions
|
|
77
|
+
|
|
78
|
+
| Module | File | Responsibility |
|
|
79
|
+
|--------|------|----------------|
|
|
80
|
+
| `EntryNormalizer` | `import_sessions/entry_normalizer.rb` | Normalize OPML entries to standard format |
|
|
81
|
+
| `HealthCheckBroadcaster` | `import_sessions/health_check_broadcaster.rb` | Broadcast health check progress via Turbo Streams |
|
|
82
|
+
|
|
83
|
+
### Jobs
|
|
84
|
+
|
|
85
|
+
| Module | File | Responsibility |
|
|
86
|
+
|--------|------|----------------|
|
|
87
|
+
| `CleanupOptions` | `jobs/cleanup_options.rb` | Options for job cleanup tasks |
|
|
88
|
+
| `Visibility` | `jobs/visibility.rb` | Configure queue visibility for Solid Queue |
|
|
89
|
+
| `SolidQueueMetrics` | `jobs/solid_queue_metrics.rb` | Extract metrics from Solid Queue tables |
|
|
90
|
+
| `FetchFailureSubscriber` | `jobs/fetch_failure_subscriber.rb` | ActiveJob error subscriber for fetch failures |
|
|
91
|
+
|
|
92
|
+
### Logs
|
|
93
|
+
|
|
94
|
+
| Module | File | Responsibility |
|
|
95
|
+
|--------|------|----------------|
|
|
96
|
+
| `EntrySync` | `logs/entry_sync.rb` | Sync FetchLog/ScrapeLog/HealthCheckLog to unified LogEntry |
|
|
97
|
+
| `FilterSet` | `logs/filter_set.rb` | Log filtering parameters |
|
|
98
|
+
| `Query` | `logs/query.rb` | Log query builder |
|
|
99
|
+
| `TablePresenter` | `logs/table_presenter.rb` | Format log entries for table display |
|
|
100
|
+
|
|
101
|
+
### Models (Shared Concerns)
|
|
102
|
+
|
|
103
|
+
| Module | File | Responsibility |
|
|
104
|
+
|--------|------|----------------|
|
|
105
|
+
| `Sanitizable` | `models/sanitizable.rb` | `sanitizes_string_attributes`, `sanitizes_hash_attributes` class methods |
|
|
106
|
+
| `UrlNormalizable` | `models/url_normalizable.rb` | `normalizes_urls`, `validates_url_format` class methods |
|
|
107
|
+
|
|
108
|
+
### Scrapers (Scraper Adapters)
|
|
109
|
+
|
|
110
|
+
| Module | File | Responsibility |
|
|
111
|
+
|--------|------|----------------|
|
|
112
|
+
| `Base` | `scrapers/base.rb` | Abstract scraper interface |
|
|
113
|
+
| `Readability` | `scrapers/readability.rb` | Default readability-based scraper |
|
|
114
|
+
| `Fetchers::HttpFetcher` | `scrapers/fetchers/http_fetcher.rb` | HTTP content fetcher for scrapers |
|
|
115
|
+
| `Parsers::ReadabilityParser` | `scrapers/parsers/readability_parser.rb` | Parse HTML to readable content |
|
|
116
|
+
|
|
117
|
+
### Scraping (Scraping Orchestration)
|
|
118
|
+
|
|
119
|
+
| Module | File | Responsibility |
|
|
120
|
+
|--------|------|----------------|
|
|
121
|
+
| `Enqueuer` | `scraping/enqueuer.rb` | Queue scrape jobs for items |
|
|
122
|
+
| `Scheduler` | `scraping/scheduler.rb` | Schedule scraping across sources |
|
|
123
|
+
| `ItemScraper` | `scraping/item_scraper.rb` | Scrape a single item |
|
|
124
|
+
| `ItemScraper::AdapterResolver` | `scraping/item_scraper/adapter_resolver.rb` | Select scraper adapter for a source |
|
|
125
|
+
| `ItemScraper::Persistence` | `scraping/item_scraper/persistence.rb` | Save scrape results to ItemContent |
|
|
126
|
+
| `BulkSourceScraper` | `scraping/bulk_source_scraper.rb` | Scrape all pending items for a source |
|
|
127
|
+
| `BulkResultPresenter` | `scraping/bulk_result_presenter.rb` | Format bulk scrape results |
|
|
128
|
+
| `State` | `scraping/state.rb` | Track scraping state per source |
|
|
129
|
+
|
|
130
|
+
### Configuration (12 Settings Files)
|
|
131
|
+
|
|
132
|
+
| Module | File | Responsibility |
|
|
133
|
+
|--------|------|----------------|
|
|
134
|
+
| `HTTPSettings` | `configuration/http_settings.rb` | HTTP timeouts, user-agent, proxy |
|
|
135
|
+
| `FetchingSettings` | `configuration/fetching_settings.rb` | Adaptive interval params, retry config |
|
|
136
|
+
| `HealthSettings` | `configuration/health_settings.rb` | Health check thresholds, auto-pause config |
|
|
137
|
+
| `ScrapingSettings` | `configuration/scraping_settings.rb` | Scraping concurrency, timeouts |
|
|
138
|
+
| `RealtimeSettings` | `configuration/realtime_settings.rb` | ActionCable/Turbo Streams config |
|
|
139
|
+
| `RetentionSettings` | `configuration/retention_settings.rb` | Item retention strategy, defaults |
|
|
140
|
+
| `AuthenticationSettings` | `configuration/authentication_settings.rb` | Auth callbacks for host app |
|
|
141
|
+
| `ScraperRegistry` | `configuration/scraper_registry.rb` | Register custom scraper adapters |
|
|
142
|
+
| `Events` | `configuration/events.rb` | Event callback storage |
|
|
143
|
+
| `ValidationDefinition` | `configuration/validation_definition.rb` | Host-app validation definitions |
|
|
144
|
+
| `ModelDefinition` | `configuration/model_definition.rb` | Per-model extension definitions |
|
|
145
|
+
| `Models` | `configuration/models.rb` | Model registry and table prefix config |
|
|
146
|
+
|
|
147
|
+
### Health
|
|
148
|
+
|
|
149
|
+
| Module | File | Responsibility |
|
|
150
|
+
|--------|------|----------------|
|
|
151
|
+
| `SourceHealthMonitor` | `health/source_health_monitor.rb` | Calculate rolling success rate, update health_status |
|
|
152
|
+
| `SourceHealthCheck` | `health/source_health_check.rb` | Perform HTTP health check on a source |
|
|
153
|
+
| `SourceHealthReset` | `health/source_health_reset.rb` | Reset health state for a source |
|
|
154
|
+
| `ImportSourceHealthCheck` | `health/import_source_health_check.rb` | Health check for import session sources |
|
|
155
|
+
|
|
156
|
+
### Security
|
|
157
|
+
|
|
158
|
+
| Module | File | Responsibility |
|
|
159
|
+
|--------|------|----------------|
|
|
160
|
+
| `ParameterSanitizer` | `security/parameter_sanitizer.rb` | Sanitize controller parameters |
|
|
161
|
+
| `Authentication` | `security/authentication.rb` | Authentication helper callbacks |
|
|
162
|
+
|
|
163
|
+
### Setup (Install Wizard)
|
|
164
|
+
|
|
165
|
+
| Module | File | Responsibility |
|
|
166
|
+
|--------|------|----------------|
|
|
167
|
+
| `CLI` | `setup/cli.rb` | Command-line interface for setup |
|
|
168
|
+
| `Workflow` | `setup/workflow.rb` | Step-by-step setup orchestration |
|
|
169
|
+
| `Requirements` | `setup/requirements.rb` | System requirements checking |
|
|
170
|
+
| `Detectors` | `setup/detectors.rb` | Detect existing config/gems |
|
|
171
|
+
| `DependencyChecker` | `setup/dependency_checker.rb` | Check gem dependencies |
|
|
172
|
+
| `GemfileEditor` | `setup/gemfile_editor.rb` | Edit host app Gemfile |
|
|
173
|
+
| `BundleInstaller` | `setup/bundle_installer.rb` | Run bundle install |
|
|
174
|
+
| `NodeInstaller` | `setup/node_installer.rb` | Install Node.js dependencies |
|
|
175
|
+
| `InstallGenerator` | `setup/install_generator.rb` | Rails generator for install |
|
|
176
|
+
| `MigrationInstaller` | `setup/migration_installer.rb` | Copy and run migrations |
|
|
177
|
+
| `InitializerPatcher` | `setup/initializer_patcher.rb` | Patch host app initializer |
|
|
178
|
+
| `Verification::Result` | `setup/verification/result.rb` | Verification result + summary |
|
|
179
|
+
| `Verification::Runner` | `setup/verification/runner.rb` | Run all verification checks |
|
|
180
|
+
| `Verification::Printer` | `setup/verification/printer.rb` | Print verification results |
|
|
181
|
+
| `Verification::SolidQueueVerifier` | `setup/verification/solid_queue_verifier.rb` | Verify Solid Queue setup |
|
|
182
|
+
| `Verification::ActionCableVerifier` | `setup/verification/action_cable_verifier.rb` | Verify Action Cable setup |
|
|
183
|
+
| `Verification::TelemetryLogger` | `setup/verification/telemetry_logger.rb` | Log setup telemetry |
|
|
184
|
+
|
|
185
|
+
### Other
|
|
186
|
+
|
|
187
|
+
| Module | File | Responsibility |
|
|
188
|
+
|--------|------|----------------|
|
|
189
|
+
| `Pagination::Paginator` | `pagination/paginator.rb` | Offset-based pagination helper |
|
|
190
|
+
| `Release::Changelog` | `release/changelog.rb` | Generate changelog from git history |
|
|
191
|
+
| `Release::Runner` | `release/runner.rb` | Coordinate gem release process |
|
|
192
|
+
| `Sources::Params` | `sources/params.rb` | Strong parameter definitions |
|
|
193
|
+
| `Sources::TurboStreamPresenter` | `sources/turbo_stream_presenter.rb` | Source Turbo Stream formatting |
|
|
194
|
+
| `TurboStreams::StreamResponder` | `turbo_streams/stream_responder.rb` | Turbo Stream response builder |
|
|
@@ -0,0 +1,264 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: sm-configuration-setting
|
|
3
|
+
description: How to add or modify configuration settings in the Source Monitor engine. Use when adding a new config option, modifying defaults, creating a new settings section, or understanding the configuration architecture.
|
|
4
|
+
allowed-tools: Read, Write, Edit, Bash, Glob, Grep
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Source Monitor Configuration Settings
|
|
8
|
+
|
|
9
|
+
## Architecture Overview
|
|
10
|
+
|
|
11
|
+
The `SourceMonitor::Configuration` class was refactored from 655 lines into a lean 87-line orchestrator plus 12 extracted settings files. Each settings file follows a consistent pattern.
|
|
12
|
+
|
|
13
|
+
```
|
|
14
|
+
lib/source_monitor/configuration.rb # Main class (87 lines)
|
|
15
|
+
lib/source_monitor/configuration/
|
|
16
|
+
authentication_settings.rb # Auth handlers
|
|
17
|
+
events.rb # Callbacks and processors
|
|
18
|
+
fetching_settings.rb # Adaptive interval tuning
|
|
19
|
+
health_settings.rb # Health monitoring thresholds
|
|
20
|
+
http_settings.rb # HTTP client defaults
|
|
21
|
+
model_definition.rb # Concern/validation injection
|
|
22
|
+
models.rb # Model definitions registry
|
|
23
|
+
realtime_settings.rb # ActionCable adapter config
|
|
24
|
+
retention_settings.rb # Item retention policies
|
|
25
|
+
scraper_registry.rb # Scraper adapter registry
|
|
26
|
+
scraping_settings.rb # Scraping concurrency limits
|
|
27
|
+
validation_definition.rb # Validation DSL
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
## How Configuration Works
|
|
31
|
+
|
|
32
|
+
```ruby
|
|
33
|
+
# In host app initializer
|
|
34
|
+
SourceMonitor.configure do |config|
|
|
35
|
+
config.fetching.min_interval_minutes = 10
|
|
36
|
+
config.http.timeout = 30
|
|
37
|
+
config.retention.strategy = :soft_delete
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
# Access at runtime
|
|
41
|
+
SourceMonitor.config.fetching.min_interval_minutes # => 10
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
The `config` object is a `SourceMonitor::Configuration` instance. Sub-sections are accessed via reader methods that return settings objects.
|
|
45
|
+
|
|
46
|
+
## Adding a Setting to an Existing Section
|
|
47
|
+
|
|
48
|
+
### Step 1: Identify the Settings File
|
|
49
|
+
|
|
50
|
+
| Setting Category | File | Class |
|
|
51
|
+
|-----------------|------|-------|
|
|
52
|
+
| HTTP client | `http_settings.rb` | `HTTPSettings` |
|
|
53
|
+
| Adaptive fetching | `fetching_settings.rb` | `FetchingSettings` |
|
|
54
|
+
| Health monitoring | `health_settings.rb` | `HealthSettings` |
|
|
55
|
+
| Scraping limits | `scraping_settings.rb` | `ScrapingSettings` |
|
|
56
|
+
| Item retention | `retention_settings.rb` | `RetentionSettings` |
|
|
57
|
+
| Realtime/cable | `realtime_settings.rb` | `RealtimeSettings` |
|
|
58
|
+
| Authentication | `authentication_settings.rb` | `AuthenticationSettings` |
|
|
59
|
+
| Events/callbacks | `events.rb` | `Events` |
|
|
60
|
+
| Scraper adapters | `scraper_registry.rb` | `ScraperRegistry` |
|
|
61
|
+
| Model extensions | `models.rb` / `model_definition.rb` | `Models` / `ModelDefinition` |
|
|
62
|
+
|
|
63
|
+
### Step 2: Add the Attribute
|
|
64
|
+
|
|
65
|
+
```ruby
|
|
66
|
+
# lib/source_monitor/configuration/fetching_settings.rb
|
|
67
|
+
class FetchingSettings
|
|
68
|
+
attr_accessor :min_interval_minutes,
|
|
69
|
+
:max_interval_minutes,
|
|
70
|
+
:increase_factor,
|
|
71
|
+
:decrease_factor,
|
|
72
|
+
:failure_increase_factor,
|
|
73
|
+
:jitter_percent,
|
|
74
|
+
:my_new_setting # <-- Add here
|
|
75
|
+
|
|
76
|
+
def reset!
|
|
77
|
+
@min_interval_minutes = 5
|
|
78
|
+
@max_interval_minutes = 24 * 60
|
|
79
|
+
@increase_factor = 1.25
|
|
80
|
+
@decrease_factor = 0.75
|
|
81
|
+
@failure_increase_factor = 1.5
|
|
82
|
+
@jitter_percent = 0.1
|
|
83
|
+
@my_new_setting = "default" # <-- Set default here
|
|
84
|
+
end
|
|
85
|
+
end
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
### Step 3: Write Tests
|
|
89
|
+
|
|
90
|
+
```ruby
|
|
91
|
+
# test/lib/source_monitor/configuration_test.rb
|
|
92
|
+
test "my_new_setting has correct default" do
|
|
93
|
+
assert_equal "default", SourceMonitor.config.fetching.my_new_setting
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
test "my_new_setting can be overridden" do
|
|
97
|
+
SourceMonitor.configure do |config|
|
|
98
|
+
config.fetching.my_new_setting = "custom"
|
|
99
|
+
end
|
|
100
|
+
assert_equal "custom", SourceMonitor.config.fetching.my_new_setting
|
|
101
|
+
end
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
### Step 4: Verify Reset
|
|
105
|
+
|
|
106
|
+
Ensure `reset!` restores the default. The test suite calls `SourceMonitor.reset_configuration!` in setup, which recreates the entire Configuration object.
|
|
107
|
+
|
|
108
|
+
## Adding a Setting with Validation
|
|
109
|
+
|
|
110
|
+
For settings that need input normalization or validation, use custom setters:
|
|
111
|
+
|
|
112
|
+
```ruby
|
|
113
|
+
class ScrapingSettings
|
|
114
|
+
attr_accessor :max_in_flight_per_source, :max_bulk_batch_size
|
|
115
|
+
|
|
116
|
+
# Custom setter with normalization
|
|
117
|
+
def max_in_flight_per_source=(value)
|
|
118
|
+
@max_in_flight_per_source = normalize_numeric(value)
|
|
119
|
+
end
|
|
120
|
+
|
|
121
|
+
private
|
|
122
|
+
|
|
123
|
+
def normalize_numeric(value)
|
|
124
|
+
return nil if value.nil?
|
|
125
|
+
return nil if value == ""
|
|
126
|
+
integer = value.respond_to?(:to_i) ? value.to_i : value
|
|
127
|
+
integer.positive? ? integer : nil
|
|
128
|
+
end
|
|
129
|
+
end
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
For enum-style settings with strict validation:
|
|
133
|
+
|
|
134
|
+
```ruby
|
|
135
|
+
class RetentionSettings
|
|
136
|
+
def strategy=(value)
|
|
137
|
+
normalized = normalize_strategy(value)
|
|
138
|
+
@strategy = normalized unless normalized.nil?
|
|
139
|
+
end
|
|
140
|
+
|
|
141
|
+
private
|
|
142
|
+
|
|
143
|
+
def normalize_strategy(value)
|
|
144
|
+
return :destroy if value.nil?
|
|
145
|
+
if value.respond_to?(:to_sym)
|
|
146
|
+
candidate = value.to_sym
|
|
147
|
+
raise ArgumentError, "Invalid retention strategy #{value.inspect}" unless %i[destroy soft_delete].include?(candidate)
|
|
148
|
+
candidate
|
|
149
|
+
else
|
|
150
|
+
raise ArgumentError, "Invalid retention strategy #{value.inspect}"
|
|
151
|
+
end
|
|
152
|
+
end
|
|
153
|
+
end
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
## Creating a New Settings Section
|
|
157
|
+
|
|
158
|
+
### Step 1: Create the Settings Class
|
|
159
|
+
|
|
160
|
+
```ruby
|
|
161
|
+
# lib/source_monitor/configuration/notifications_settings.rb
|
|
162
|
+
# frozen_string_literal: true
|
|
163
|
+
|
|
164
|
+
module SourceMonitor
|
|
165
|
+
class Configuration
|
|
166
|
+
class NotificationsSettings
|
|
167
|
+
attr_accessor :enabled, :channels, :throttle_seconds
|
|
168
|
+
|
|
169
|
+
def initialize
|
|
170
|
+
reset!
|
|
171
|
+
end
|
|
172
|
+
|
|
173
|
+
def reset!
|
|
174
|
+
@enabled = true
|
|
175
|
+
@channels = []
|
|
176
|
+
@throttle_seconds = 60
|
|
177
|
+
end
|
|
178
|
+
end
|
|
179
|
+
end
|
|
180
|
+
end
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
### Step 2: Register in Configuration
|
|
184
|
+
|
|
185
|
+
```ruby
|
|
186
|
+
# lib/source_monitor/configuration.rb
|
|
187
|
+
require "source_monitor/configuration/notifications_settings"
|
|
188
|
+
|
|
189
|
+
class Configuration
|
|
190
|
+
attr_reader :http, :scrapers, :retention, :events, :models,
|
|
191
|
+
:realtime, :fetching, :health, :authentication, :scraping,
|
|
192
|
+
:notifications # <-- Add reader
|
|
193
|
+
|
|
194
|
+
def initialize
|
|
195
|
+
# ... existing initialization ...
|
|
196
|
+
@notifications = NotificationsSettings.new # <-- Initialize
|
|
197
|
+
end
|
|
198
|
+
end
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
### Step 3: Write Tests
|
|
202
|
+
|
|
203
|
+
```ruby
|
|
204
|
+
test "notifications settings have correct defaults" do
|
|
205
|
+
settings = SourceMonitor.config.notifications
|
|
206
|
+
assert_equal true, settings.enabled
|
|
207
|
+
assert_equal [], settings.channels
|
|
208
|
+
assert_equal 60, settings.throttle_seconds
|
|
209
|
+
end
|
|
210
|
+
|
|
211
|
+
test "notifications settings can be configured" do
|
|
212
|
+
SourceMonitor.configure do |config|
|
|
213
|
+
config.notifications.enabled = false
|
|
214
|
+
config.notifications.channels = [:email, :slack]
|
|
215
|
+
config.notifications.throttle_seconds = 30
|
|
216
|
+
end
|
|
217
|
+
|
|
218
|
+
settings = SourceMonitor.config.notifications
|
|
219
|
+
assert_equal false, settings.enabled
|
|
220
|
+
assert_equal [:email, :slack], settings.channels
|
|
221
|
+
assert_equal 30, settings.throttle_seconds
|
|
222
|
+
end
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
## Patterns by Section Type
|
|
226
|
+
|
|
227
|
+
### Simple Accessor Pattern (FetchingSettings, HealthSettings)
|
|
228
|
+
|
|
229
|
+
Plain `attr_accessor` with defaults in `reset!`. No validation.
|
|
230
|
+
|
|
231
|
+
### Normalized Setter Pattern (ScrapingSettings)
|
|
232
|
+
|
|
233
|
+
Custom setter that normalizes input (strings to integers, negatives to nil).
|
|
234
|
+
|
|
235
|
+
### Enum Setter Pattern (RetentionSettings, RealtimeSettings)
|
|
236
|
+
|
|
237
|
+
Custom setter that validates against an allowed list and raises `ArgumentError`.
|
|
238
|
+
|
|
239
|
+
### Handler/Callback Pattern (AuthenticationSettings, Events)
|
|
240
|
+
|
|
241
|
+
Registration methods that accept symbols, lambdas, or blocks.
|
|
242
|
+
|
|
243
|
+
### Registry Pattern (ScraperRegistry)
|
|
244
|
+
|
|
245
|
+
Named registration with lookup, unregistration, and enumeration.
|
|
246
|
+
|
|
247
|
+
### Nested Object Pattern (RealtimeSettings::SolidCableOptions)
|
|
248
|
+
|
|
249
|
+
Sub-objects with their own `reset!` and `to_h` methods.
|
|
250
|
+
|
|
251
|
+
## Testing Checklist
|
|
252
|
+
|
|
253
|
+
- [ ] Default value is correct
|
|
254
|
+
- [ ] Value can be overridden via `SourceMonitor.configure`
|
|
255
|
+
- [ ] `reset!` restores the default (tested via `SourceMonitor.reset_configuration!`)
|
|
256
|
+
- [ ] Validation raises `ArgumentError` for invalid values (if applicable)
|
|
257
|
+
- [ ] String/nil normalization works correctly (if applicable)
|
|
258
|
+
- [ ] Test file: `test/lib/source_monitor/configuration_test.rb`
|
|
259
|
+
- [ ] Run: `PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/configuration_test.rb`
|
|
260
|
+
|
|
261
|
+
## References
|
|
262
|
+
|
|
263
|
+
- [reference/settings-catalog.md](reference/settings-catalog.md) -- All settings sections with their attributes
|
|
264
|
+
- [reference/settings-pattern.md](reference/settings-pattern.md) -- Step-by-step pattern for adding settings
|