source_monitor 0.3.0 → 0.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.claude/skills/sm-architecture/SKILL.md +233 -0
- data/.claude/skills/sm-architecture/reference/extraction-patterns.md +192 -0
- data/.claude/skills/sm-architecture/reference/module-map.md +194 -0
- data/.claude/skills/sm-configuration-setting/SKILL.md +264 -0
- data/.claude/skills/sm-configuration-setting/reference/settings-catalog.md +248 -0
- data/.claude/skills/sm-configuration-setting/reference/settings-pattern.md +297 -0
- data/.claude/skills/sm-configure/SKILL.md +153 -0
- data/.claude/skills/sm-configure/reference/configuration-reference.md +321 -0
- data/.claude/skills/sm-dashboard-widget/SKILL.md +344 -0
- data/.claude/skills/sm-dashboard-widget/reference/dashboard-patterns.md +304 -0
- data/.claude/skills/sm-domain-model/SKILL.md +188 -0
- data/.claude/skills/sm-domain-model/reference/model-graph.md +114 -0
- data/.claude/skills/sm-domain-model/reference/table-structure.md +348 -0
- data/.claude/skills/sm-engine-migration/SKILL.md +395 -0
- data/.claude/skills/sm-engine-migration/reference/migration-conventions.md +255 -0
- data/.claude/skills/sm-engine-test/SKILL.md +302 -0
- data/.claude/skills/sm-engine-test/reference/test-helpers.md +259 -0
- data/.claude/skills/sm-engine-test/reference/test-patterns.md +411 -0
- data/.claude/skills/sm-event-handler/SKILL.md +265 -0
- data/.claude/skills/sm-event-handler/reference/events-api.md +229 -0
- data/.claude/skills/sm-health-rule/SKILL.md +327 -0
- data/.claude/skills/sm-health-rule/reference/health-system.md +269 -0
- data/.claude/skills/sm-host-setup/SKILL.md +223 -0
- data/.claude/skills/sm-host-setup/reference/initializer-template.md +195 -0
- data/.claude/skills/sm-host-setup/reference/setup-checklist.md +134 -0
- data/.claude/skills/sm-job/SKILL.md +263 -0
- data/.claude/skills/sm-job/reference/job-conventions.md +245 -0
- data/.claude/skills/sm-model-extension/SKILL.md +287 -0
- data/.claude/skills/sm-model-extension/reference/extension-api.md +317 -0
- data/.claude/skills/sm-pipeline-stage/SKILL.md +254 -0
- data/.claude/skills/sm-pipeline-stage/reference/completion-handlers.md +152 -0
- data/.claude/skills/sm-pipeline-stage/reference/entry-processing.md +191 -0
- data/.claude/skills/sm-pipeline-stage/reference/feed-fetcher-architecture.md +198 -0
- data/.claude/skills/sm-scraper-adapter/SKILL.md +284 -0
- data/.claude/skills/sm-scraper-adapter/reference/adapter-contract.md +167 -0
- data/.claude/skills/sm-scraper-adapter/reference/example-adapter.md +274 -0
- data/.vbw-planning/.notification-log.jsonl +102 -0
- data/.vbw-planning/.session-log.jsonl +505 -0
- data/AGENTS.md +20 -57
- data/CHANGELOG.md +19 -0
- data/CLAUDE.md +44 -1
- data/CONTRIBUTING.md +5 -5
- data/Gemfile.lock +20 -21
- data/README.md +18 -5
- data/VERSION +1 -0
- data/docs/deployment.md +1 -1
- data/docs/setup.md +4 -4
- data/lib/source_monitor/setup/skills_installer.rb +94 -0
- data/lib/source_monitor/setup/workflow.rb +17 -2
- data/lib/source_monitor/version.rb +1 -1
- data/lib/tasks/source_monitor_setup.rake +58 -0
- data/source_monitor.gemspec +1 -0
- metadata +39 -1
|
@@ -0,0 +1,321 @@
|
|
|
1
|
+
# Configuration Reference
|
|
2
|
+
|
|
3
|
+
Complete reference for every SourceMonitor configuration setting.
|
|
4
|
+
|
|
5
|
+
Source: `lib/source_monitor/configuration.rb` and `lib/source_monitor/configuration/*.rb`
|
|
6
|
+
|
|
7
|
+
## Top-Level Settings
|
|
8
|
+
|
|
9
|
+
Defined on `SourceMonitor::Configuration`:
|
|
10
|
+
|
|
11
|
+
| Setting | Type | Default | Description |
|
|
12
|
+
|---|---|---|---|
|
|
13
|
+
| `queue_namespace` | String | `"source_monitor"` | Prefix for queue names and instrumentation keys |
|
|
14
|
+
| `fetch_queue_name` | String | `"source_monitor_fetch"` | Queue name for fetch jobs |
|
|
15
|
+
| `scrape_queue_name` | String | `"source_monitor_scrape"` | Queue name for scrape jobs |
|
|
16
|
+
| `fetch_queue_concurrency` | Integer | `2` | Advisory concurrency for fetch queue |
|
|
17
|
+
| `scrape_queue_concurrency` | Integer | `2` | Advisory concurrency for scrape queue |
|
|
18
|
+
| `recurring_command_job_class` | String/Class/nil | `nil` | Override Solid Queue recurring task job class |
|
|
19
|
+
| `job_metrics_enabled` | Boolean | `true` | Toggle queue metrics on dashboard |
|
|
20
|
+
| `mission_control_enabled` | Boolean | `false` | Show Mission Control link on dashboard |
|
|
21
|
+
| `mission_control_dashboard_path` | String/Proc/nil | `nil` | Path or callable returning MC URL |
|
|
22
|
+
|
|
23
|
+
### Methods
|
|
24
|
+
|
|
25
|
+
| Method | Signature | Description |
|
|
26
|
+
|---|---|---|
|
|
27
|
+
| `queue_name_for` | `(role) -> String` | Returns resolved queue name with host prefix |
|
|
28
|
+
| `concurrency_for` | `(role) -> Integer` | Returns concurrency for `:fetch` or `:scrape` |
|
|
29
|
+
|
|
30
|
+
---
|
|
31
|
+
|
|
32
|
+
## HTTP Settings (`config.http`)
|
|
33
|
+
|
|
34
|
+
Class: `SourceMonitor::Configuration::HTTPSettings`
|
|
35
|
+
|
|
36
|
+
| Setting | Type | Default | Description |
|
|
37
|
+
|---|---|---|---|
|
|
38
|
+
| `timeout` | Integer | `15` | Total request timeout (seconds) |
|
|
39
|
+
| `open_timeout` | Integer | `5` | Connection open timeout (seconds) |
|
|
40
|
+
| `max_redirects` | Integer | `5` | Maximum redirects to follow |
|
|
41
|
+
| `user_agent` | String | `"SourceMonitor/<version>"` | User-Agent header |
|
|
42
|
+
| `proxy` | String/Hash/nil | `nil` | HTTP proxy configuration |
|
|
43
|
+
| `headers` | Hash | `{}` | Extra headers merged into every request |
|
|
44
|
+
| `retry_max` | Integer | `4` | Maximum retry attempts |
|
|
45
|
+
| `retry_interval` | Float | `0.5` | Initial retry delay (seconds) |
|
|
46
|
+
| `retry_interval_randomness` | Float | `0.5` | Randomness factor for retry jitter |
|
|
47
|
+
| `retry_backoff_factor` | Integer | `2` | Exponential backoff multiplier |
|
|
48
|
+
| `retry_statuses` | Array/nil | `nil` | HTTP status codes to retry (nil = use defaults) |
|
|
49
|
+
|
|
50
|
+
```ruby
|
|
51
|
+
config.http.timeout = 30
|
|
52
|
+
config.http.proxy = "http://proxy.example.com:8080"
|
|
53
|
+
config.http.headers = { "X-Custom" => "value" }
|
|
54
|
+
config.http.retry_statuses = [429, 500, 502, 503, 504]
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
---
|
|
58
|
+
|
|
59
|
+
## Fetching Settings (`config.fetching`)
|
|
60
|
+
|
|
61
|
+
Class: `SourceMonitor::Configuration::FetchingSettings`
|
|
62
|
+
|
|
63
|
+
Controls adaptive fetch scheduling.
|
|
64
|
+
|
|
65
|
+
| Setting | Type | Default | Description |
|
|
66
|
+
|---|---|---|---|
|
|
67
|
+
| `min_interval_minutes` | Integer | `5` | Minimum fetch interval (minutes) |
|
|
68
|
+
| `max_interval_minutes` | Integer | `1440` | Maximum fetch interval (24 hours) |
|
|
69
|
+
| `increase_factor` | Float | `1.25` | Multiplier when source trends slow (no new items) |
|
|
70
|
+
| `decrease_factor` | Float | `0.75` | Multiplier when new items arrive |
|
|
71
|
+
| `failure_increase_factor` | Float | `1.5` | Multiplier on consecutive failures |
|
|
72
|
+
| `jitter_percent` | Float | `0.1` | Random jitter (+/-10%, 0 disables) |
|
|
73
|
+
|
|
74
|
+
```ruby
|
|
75
|
+
config.fetching.min_interval_minutes = 10
|
|
76
|
+
config.fetching.max_interval_minutes = 720 # 12 hours
|
|
77
|
+
config.fetching.jitter_percent = 0.15 # +/-15%
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
## Health Settings (`config.health`)
|
|
83
|
+
|
|
84
|
+
Class: `SourceMonitor::Configuration::HealthSettings`
|
|
85
|
+
|
|
86
|
+
Tunes automatic pause/resume heuristics per source.
|
|
87
|
+
|
|
88
|
+
| Setting | Type | Default | Description |
|
|
89
|
+
|---|---|---|---|
|
|
90
|
+
| `window_size` | Integer | `20` | Number of fetch attempts to evaluate |
|
|
91
|
+
| `healthy_threshold` | Float | `0.8` | Success ratio for "healthy" badge |
|
|
92
|
+
| `warning_threshold` | Float | `0.5` | Success ratio for "warning" badge |
|
|
93
|
+
| `auto_pause_threshold` | Float | `0.2` | Auto-pause source below this ratio |
|
|
94
|
+
| `auto_resume_threshold` | Float | `0.6` | Auto-resume source above this ratio |
|
|
95
|
+
| `auto_pause_cooldown_minutes` | Integer | `60` | Grace period before re-enabling |
|
|
96
|
+
|
|
97
|
+
```ruby
|
|
98
|
+
config.health.window_size = 50
|
|
99
|
+
config.health.auto_pause_threshold = 0.1
|
|
100
|
+
config.health.auto_pause_cooldown_minutes = 120
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
---
|
|
104
|
+
|
|
105
|
+
## Scraper Registry (`config.scrapers`)
|
|
106
|
+
|
|
107
|
+
Class: `SourceMonitor::Configuration::ScraperRegistry`
|
|
108
|
+
|
|
109
|
+
Manages scraper adapter registration. Adapters must inherit from `SourceMonitor::Scrapers::Base`.
|
|
110
|
+
|
|
111
|
+
### Methods
|
|
112
|
+
|
|
113
|
+
| Method | Signature | Description |
|
|
114
|
+
|---|---|---|
|
|
115
|
+
| `register` | `(name, adapter)` | Register adapter by name (symbol/string) |
|
|
116
|
+
| `unregister` | `(name)` | Remove adapter by name |
|
|
117
|
+
| `adapter_for` | `(name) -> Class/nil` | Look up adapter class |
|
|
118
|
+
| `each` | `(&block)` | Iterate registered adapters |
|
|
119
|
+
|
|
120
|
+
Names are normalized to lowercase alphanumeric + underscore.
|
|
121
|
+
|
|
122
|
+
Adapters can be a Class or a String (constantized lazily).
|
|
123
|
+
|
|
124
|
+
```ruby
|
|
125
|
+
config.scrapers.register(:custom, MyApp::Scrapers::Custom)
|
|
126
|
+
config.scrapers.register(:premium, "MyApp::Scrapers::Premium")
|
|
127
|
+
config.scrapers.unregister(:readability)
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
---
|
|
131
|
+
|
|
132
|
+
## Retention Settings (`config.retention`)
|
|
133
|
+
|
|
134
|
+
Class: `SourceMonitor::Configuration::RetentionSettings`
|
|
135
|
+
|
|
136
|
+
Global defaults inherited by sources with blank retention fields.
|
|
137
|
+
|
|
138
|
+
| Setting | Type | Default | Description |
|
|
139
|
+
|---|---|---|---|
|
|
140
|
+
| `items_retention_days` | Integer/nil | `nil` | Prune items older than N days (nil = keep forever) |
|
|
141
|
+
| `max_items` | Integer/nil | `nil` | Keep only N most recent items (nil = unlimited) |
|
|
142
|
+
| `strategy` | Symbol | `:destroy` | Pruning strategy: `:destroy` or `:soft_delete` |
|
|
143
|
+
|
|
144
|
+
```ruby
|
|
145
|
+
config.retention.items_retention_days = 90
|
|
146
|
+
config.retention.max_items = 1000
|
|
147
|
+
config.retention.strategy = :soft_delete
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
---
|
|
151
|
+
|
|
152
|
+
## Scraping Settings (`config.scraping`)
|
|
153
|
+
|
|
154
|
+
Class: `SourceMonitor::Configuration::ScrapingSettings`
|
|
155
|
+
|
|
156
|
+
| Setting | Type | Default | Description |
|
|
157
|
+
|---|---|---|---|
|
|
158
|
+
| `max_in_flight_per_source` | Integer/nil | `25` | Max concurrent scrapes per source |
|
|
159
|
+
| `max_bulk_batch_size` | Integer/nil | `100` | Max items per bulk scrape enqueue |
|
|
160
|
+
|
|
161
|
+
Values are normalized to positive integers. Set to `nil` to disable limits.
|
|
162
|
+
|
|
163
|
+
```ruby
|
|
164
|
+
config.scraping.max_in_flight_per_source = 50
|
|
165
|
+
config.scraping.max_bulk_batch_size = 200
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
---
|
|
169
|
+
|
|
170
|
+
## Events (`config.events`)
|
|
171
|
+
|
|
172
|
+
Class: `SourceMonitor::Configuration::Events`
|
|
173
|
+
|
|
174
|
+
Register lifecycle callbacks. See the `sm-event-handler` skill for full event documentation.
|
|
175
|
+
|
|
176
|
+
### Callback Methods
|
|
177
|
+
|
|
178
|
+
| Method | Signature | Description |
|
|
179
|
+
|---|---|---|
|
|
180
|
+
| `after_item_created` | `(handler=nil, &block)` | Called after a new item is created from a feed entry |
|
|
181
|
+
| `after_item_scraped` | `(handler=nil, &block)` | Called after an item is scraped for content |
|
|
182
|
+
| `after_fetch_completed` | `(handler=nil, &block)` | Called after a feed fetch finishes |
|
|
183
|
+
| `register_item_processor` | `(processor=nil, &block)` | Register a post-processing pipeline step |
|
|
184
|
+
| `callbacks_for` | `(name) -> Array` | Retrieve callbacks for an event |
|
|
185
|
+
| `item_processors` | `-> Array` | Retrieve registered item processors |
|
|
186
|
+
| `reset!` | `-> void` | Clear all callbacks and processors |
|
|
187
|
+
|
|
188
|
+
Handlers must respond to `#call`. Can be a Proc, lambda, or any callable object.
|
|
189
|
+
|
|
190
|
+
```ruby
|
|
191
|
+
config.events.after_item_created ->(event) { log(event.item) }
|
|
192
|
+
config.events.after_fetch_completed do |event|
|
|
193
|
+
StatsTracker.record(event.source, event.status)
|
|
194
|
+
end
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
---
|
|
198
|
+
|
|
199
|
+
## Models (`config.models`)
|
|
200
|
+
|
|
201
|
+
Class: `SourceMonitor::Configuration::Models`
|
|
202
|
+
|
|
203
|
+
Per-model extension points for host apps.
|
|
204
|
+
|
|
205
|
+
| Setting | Type | Default | Description |
|
|
206
|
+
|---|---|---|---|
|
|
207
|
+
| `table_name_prefix` | String | `"sourcemon_"` | Table name prefix for all engine tables |
|
|
208
|
+
|
|
209
|
+
### Model Accessors
|
|
210
|
+
|
|
211
|
+
| Method | Key | Engine Model |
|
|
212
|
+
|---|---|---|
|
|
213
|
+
| `config.models.source` | `:source` | `SourceMonitor::Source` |
|
|
214
|
+
| `config.models.item` | `:item` | `SourceMonitor::Item` |
|
|
215
|
+
| `config.models.fetch_log` | `:fetch_log` | `SourceMonitor::FetchLog` |
|
|
216
|
+
| `config.models.scrape_log` | `:scrape_log` | `SourceMonitor::ScrapeLog` |
|
|
217
|
+
| `config.models.health_check_log` | `:health_check_log` | `SourceMonitor::HealthCheckLog` |
|
|
218
|
+
| `config.models.item_content` | `:item_content` | `SourceMonitor::ItemContent` |
|
|
219
|
+
| `config.models.log_entry` | `:log_entry` | `SourceMonitor::LogEntry` |
|
|
220
|
+
|
|
221
|
+
Each accessor returns a `ModelDefinition` with:
|
|
222
|
+
|
|
223
|
+
| Method | Signature | Description |
|
|
224
|
+
|---|---|---|
|
|
225
|
+
| `include_concern` | `(concern=nil, &block)` | Include a concern module |
|
|
226
|
+
| `validate` | `(handler=nil, **options, &block)` | Register a validation |
|
|
227
|
+
|
|
228
|
+
```ruby
|
|
229
|
+
config.models.table_name_prefix = "sm_"
|
|
230
|
+
config.models.source.include_concern "MyApp::SourceTagging"
|
|
231
|
+
config.models.item.validate :check_content_length
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
---
|
|
235
|
+
|
|
236
|
+
## Realtime Settings (`config.realtime`)
|
|
237
|
+
|
|
238
|
+
Class: `SourceMonitor::Configuration::RealtimeSettings`
|
|
239
|
+
|
|
240
|
+
| Setting | Type | Default | Description |
|
|
241
|
+
|---|---|---|---|
|
|
242
|
+
| `adapter` | Symbol | `:solid_cable` | One of `:solid_cable`, `:redis`, `:async` |
|
|
243
|
+
| `redis_url` | String/nil | `nil` | Redis URL when using `:redis` adapter |
|
|
244
|
+
|
|
245
|
+
### Solid Cable Options (`config.realtime.solid_cable`)
|
|
246
|
+
|
|
247
|
+
| Setting | Type | Default | Description |
|
|
248
|
+
|---|---|---|---|
|
|
249
|
+
| `polling_interval` | String | `"0.1.seconds"` | Polling frequency |
|
|
250
|
+
| `message_retention` | String | `"1.day"` | How long to retain messages |
|
|
251
|
+
| `autotrim` | Boolean | `true` | Auto-trim old messages |
|
|
252
|
+
| `silence_polling` | Boolean | `true` | Suppress polling log noise |
|
|
253
|
+
| `use_skip_locked` | Boolean | `true` | Use PostgreSQL SKIP LOCKED |
|
|
254
|
+
| `trim_batch_size` | Integer/nil | `nil` | Batch size for trim operations |
|
|
255
|
+
| `connects_to` | Hash/nil | `nil` | Multi-database routing |
|
|
256
|
+
|
|
257
|
+
### Methods
|
|
258
|
+
|
|
259
|
+
| Method | Returns | Description |
|
|
260
|
+
|---|---|---|
|
|
261
|
+
| `action_cable_config` | Hash | Full configuration hash for cable.yml |
|
|
262
|
+
|
|
263
|
+
```ruby
|
|
264
|
+
config.realtime.adapter = :redis
|
|
265
|
+
config.realtime.redis_url = "redis://localhost:6379/1"
|
|
266
|
+
|
|
267
|
+
# Or with Solid Cable tuning:
|
|
268
|
+
config.realtime.adapter = :solid_cable
|
|
269
|
+
config.realtime.solid_cable.polling_interval = "0.05.seconds"
|
|
270
|
+
config.realtime.solid_cable.connects_to = { database: { writing: :cable } }
|
|
271
|
+
```
|
|
272
|
+
|
|
273
|
+
---
|
|
274
|
+
|
|
275
|
+
## Authentication Settings (`config.authentication`)
|
|
276
|
+
|
|
277
|
+
Class: `SourceMonitor::Configuration::AuthenticationSettings`
|
|
278
|
+
|
|
279
|
+
| Setting | Type | Default | Description |
|
|
280
|
+
|---|---|---|---|
|
|
281
|
+
| `current_user_method` | Symbol/nil | `nil` | Controller method to get current user |
|
|
282
|
+
| `user_signed_in_method` | Symbol/nil | `nil` | Controller method to check sign-in status |
|
|
283
|
+
|
|
284
|
+
### Methods
|
|
285
|
+
|
|
286
|
+
| Method | Signature | Description |
|
|
287
|
+
|---|---|---|
|
|
288
|
+
| `authenticate_with` | `(handler=nil, &block)` | Set authentication handler |
|
|
289
|
+
| `authorize_with` | `(handler=nil, &block)` | Set authorization handler |
|
|
290
|
+
|
|
291
|
+
Handlers can be:
|
|
292
|
+
- **Symbol**: Invoked as `controller.public_send(handler)`
|
|
293
|
+
- **Callable**: Called with `callable.call(controller)` (or `instance_exec` if arity is 0)
|
|
294
|
+
|
|
295
|
+
```ruby
|
|
296
|
+
# Symbol handler (Devise)
|
|
297
|
+
config.authentication.authenticate_with :authenticate_user!
|
|
298
|
+
|
|
299
|
+
# Callable handler
|
|
300
|
+
config.authentication.authorize_with ->(controller) {
|
|
301
|
+
controller.current_user&.feature_enabled?(:source_monitor)
|
|
302
|
+
}
|
|
303
|
+
|
|
304
|
+
# Block handler (instance_exec on controller)
|
|
305
|
+
config.authentication.authorize_with do
|
|
306
|
+
redirect_to root_path unless current_user&.admin?
|
|
307
|
+
end
|
|
308
|
+
```
|
|
309
|
+
|
|
310
|
+
---
|
|
311
|
+
|
|
312
|
+
## Environment Variables
|
|
313
|
+
|
|
314
|
+
| Variable | Purpose |
|
|
315
|
+
|---|---|
|
|
316
|
+
| `SOLID_QUEUE_SKIP_RECURRING` | Skip loading `config/recurring.yml` |
|
|
317
|
+
| `SOLID_QUEUE_RECURRING_SCHEDULE_FILE` | Alternative schedule file path |
|
|
318
|
+
| `SOFT_DELETE` | Override retention strategy in rake tasks |
|
|
319
|
+
| `SOURCE_IDS` / `SOURCE_ID` | Scope cleanup rake tasks to specific sources |
|
|
320
|
+
| `FETCH_LOG_DAYS` / `SCRAPE_LOG_DAYS` | Retention windows for log cleanup |
|
|
321
|
+
| `SOURCE_MONITOR_SETUP_TELEMETRY` | Enable setup verification telemetry logging |
|
|
@@ -0,0 +1,344 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: sm-dashboard-widget
|
|
3
|
+
description: Creates dashboard widgets with queries, presenters, and Turbo broadcasts for Source Monitor. Use when building dashboard metrics, stats panels, or real-time monitoring displays.
|
|
4
|
+
allowed-tools: Read, Write, Edit, Bash, Glob, Grep
|
|
5
|
+
disable-model-invocation: true
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# SourceMonitor Dashboard Widget
|
|
9
|
+
|
|
10
|
+
## Overview
|
|
11
|
+
|
|
12
|
+
The SourceMonitor dashboard lives at `DashboardController#index` and is composed of widgets: stat cards, recent activity feeds, job queue metrics, fetch schedules, and quick actions. Each widget follows a query-presenter-view pattern with optional Turbo Stream broadcasting for real-time updates.
|
|
13
|
+
|
|
14
|
+
## Architecture
|
|
15
|
+
|
|
16
|
+
```
|
|
17
|
+
Controller (app/controllers/source_monitor/dashboard_controller.rb)
|
|
18
|
+
|
|
|
19
|
+
v
|
|
20
|
+
Queries (lib/source_monitor/dashboard/queries.rb) -- facade
|
|
21
|
+
|-- StatsQuery -- single SQL for source counts + item/fetch totals
|
|
22
|
+
|-- RecentActivityQuery -- UNION ALL across fetch_logs, scrape_logs, items
|
|
23
|
+
|-- UpcomingFetchSchedule -- groups active sources into time-window buckets
|
|
24
|
+
|-- job_metrics -- delegates to Jobs::SolidQueueMetrics
|
|
25
|
+
|-- quick_actions -- static QuickAction structs
|
|
26
|
+
|
|
|
27
|
+
v
|
|
28
|
+
Presenters
|
|
29
|
+
|-- RecentActivityPresenter -- maps Event structs to view-model hashes
|
|
30
|
+
|-- QuickActionsPresenter -- maps QuickAction structs to path-resolved hashes
|
|
31
|
+
|
|
|
32
|
+
v
|
|
33
|
+
Views (app/views/source_monitor/dashboard/)
|
|
34
|
+
|-- index.html.erb -- layout shell with turbo_stream_from
|
|
35
|
+
|-- _stats.html.erb -- grid of stat cards
|
|
36
|
+
|-- _stat_card.html.erb -- single stat card partial (collection render)
|
|
37
|
+
|-- _recent_activity.html.erb
|
|
38
|
+
|-- _fetch_schedule.html.erb
|
|
39
|
+
|-- _job_metrics.html.erb
|
|
40
|
+
|
|
|
41
|
+
v
|
|
42
|
+
TurboBroadcaster (lib/source_monitor/dashboard/turbo_broadcaster.rb)
|
|
43
|
+
|-- registers event callbacks (after_fetch_completed, after_item_created)
|
|
44
|
+
|-- broadcasts replace_to for stats, recent_activity, fetch_schedule
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
## Creating a New Dashboard Widget
|
|
48
|
+
|
|
49
|
+
### Step 1: Define the Query
|
|
50
|
+
|
|
51
|
+
Create a query class under `lib/source_monitor/dashboard/queries/`. Follow the existing pattern:
|
|
52
|
+
|
|
53
|
+
```ruby
|
|
54
|
+
# frozen_string_literal: true
|
|
55
|
+
|
|
56
|
+
module SourceMonitor
|
|
57
|
+
module Dashboard
|
|
58
|
+
class Queries
|
|
59
|
+
class MyWidgetQuery
|
|
60
|
+
def initialize(reference_time:)
|
|
61
|
+
@reference_time = reference_time
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
def call
|
|
65
|
+
# Return a hash, array of structs, or value object
|
|
66
|
+
{
|
|
67
|
+
metric_a: compute_metric_a,
|
|
68
|
+
metric_b: compute_metric_b
|
|
69
|
+
}
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
private
|
|
73
|
+
|
|
74
|
+
attr_reader :reference_time
|
|
75
|
+
|
|
76
|
+
def compute_metric_a
|
|
77
|
+
# Use quoted_table_name for SQL safety
|
|
78
|
+
SourceMonitor::Source.connection.select_value(<<~SQL.squish).to_i
|
|
79
|
+
SELECT COUNT(*)
|
|
80
|
+
FROM #{SourceMonitor::Source.quoted_table_name}
|
|
81
|
+
WHERE some_condition
|
|
82
|
+
SQL
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
def compute_metric_b
|
|
86
|
+
SourceMonitor::Item.where("created_at >= ?", reference_time.beginning_of_day).count
|
|
87
|
+
end
|
|
88
|
+
end
|
|
89
|
+
end
|
|
90
|
+
end
|
|
91
|
+
end
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
Key patterns:
|
|
95
|
+
- Constructor takes `reference_time:` for time-relative queries
|
|
96
|
+
- `call` returns the result (hash, array, or value object)
|
|
97
|
+
- Use `quoted_table_name` for all SQL references
|
|
98
|
+
- Use `connection.exec_query` for multi-column results
|
|
99
|
+
- Use `connection.select_value` for single scalar values
|
|
100
|
+
- Use ActiveRecord scopes when raw SQL is not needed
|
|
101
|
+
|
|
102
|
+
### Step 2: Register in the Queries Facade
|
|
103
|
+
|
|
104
|
+
Add a method in `lib/source_monitor/dashboard/queries.rb`:
|
|
105
|
+
|
|
106
|
+
```ruby
|
|
107
|
+
def my_widget
|
|
108
|
+
cache.fetch(:my_widget) do
|
|
109
|
+
measure(:my_widget) do
|
|
110
|
+
MyWidgetQuery.new(reference_time:).call
|
|
111
|
+
end
|
|
112
|
+
end
|
|
113
|
+
end
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
The `cache` prevents duplicate computation within a single request. The `measure` wrapper:
|
|
117
|
+
1. Records timing via `ActiveSupport::Notifications.instrument`
|
|
118
|
+
2. Publishes gauge metrics via `SourceMonitor::Metrics.gauge`
|
|
119
|
+
|
|
120
|
+
Add a require at the top of the file:
|
|
121
|
+
```ruby
|
|
122
|
+
require "source_monitor/dashboard/queries/my_widget_query"
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
### Step 3: Create a Presenter (if needed)
|
|
126
|
+
|
|
127
|
+
Presenters live in `lib/source_monitor/dashboard/`. They transform query results into view-friendly hashes, resolving route paths using `url_helpers`.
|
|
128
|
+
|
|
129
|
+
```ruby
|
|
130
|
+
# frozen_string_literal: true
|
|
131
|
+
|
|
132
|
+
module SourceMonitor
|
|
133
|
+
module Dashboard
|
|
134
|
+
class MyWidgetPresenter
|
|
135
|
+
def initialize(data, url_helpers:)
|
|
136
|
+
@data = data
|
|
137
|
+
@url_helpers = url_helpers
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
def to_a
|
|
141
|
+
data.map { |item| build_view_model(item) }
|
|
142
|
+
end
|
|
143
|
+
|
|
144
|
+
private
|
|
145
|
+
|
|
146
|
+
attr_reader :data, :url_helpers
|
|
147
|
+
|
|
148
|
+
def build_view_model(item)
|
|
149
|
+
{
|
|
150
|
+
label: item.name,
|
|
151
|
+
value: item.count,
|
|
152
|
+
path: url_helpers.source_path(item.source_id)
|
|
153
|
+
}
|
|
154
|
+
end
|
|
155
|
+
end
|
|
156
|
+
end
|
|
157
|
+
end
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
Presenter conventions:
|
|
161
|
+
- Constructor takes raw data + `url_helpers:` keyword
|
|
162
|
+
- `to_a` returns array of plain hashes for the view
|
|
163
|
+
- `url_helpers` comes from `SourceMonitor::Engine.routes.url_helpers`
|
|
164
|
+
|
|
165
|
+
### Step 4: Wire into the Controller
|
|
166
|
+
|
|
167
|
+
In `app/controllers/source_monitor/dashboard_controller.rb`:
|
|
168
|
+
|
|
169
|
+
```ruby
|
|
170
|
+
def index
|
|
171
|
+
queries = SourceMonitor::Dashboard::Queries.new
|
|
172
|
+
url_helpers = SourceMonitor::Engine.routes.url_helpers
|
|
173
|
+
|
|
174
|
+
# existing assigns...
|
|
175
|
+
@my_widget = MyWidgetPresenter.new(queries.my_widget, url_helpers:).to_a
|
|
176
|
+
end
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
### Step 5: Create the View Partial
|
|
180
|
+
|
|
181
|
+
Create `app/views/source_monitor/dashboard/_my_widget.html.erb`:
|
|
182
|
+
|
|
183
|
+
```erb
|
|
184
|
+
<div id="source_monitor_dashboard_my_widget" class="rounded-lg border border-slate-200 bg-white shadow-sm">
|
|
185
|
+
<div class="border-b border-slate-200 px-5 py-4">
|
|
186
|
+
<h2 class="text-lg font-medium">My Widget</h2>
|
|
187
|
+
<p class="mt-1 text-xs text-slate-500">Description of what this widget shows.</p>
|
|
188
|
+
</div>
|
|
189
|
+
<div class="divide-y divide-slate-100">
|
|
190
|
+
<%% if my_widget.any? %>
|
|
191
|
+
<%% my_widget.each do |item| %>
|
|
192
|
+
<div class="flex items-center justify-between px-5 py-4">
|
|
193
|
+
<div class="text-sm font-medium text-slate-900"><%%= item[:label] %></div>
|
|
194
|
+
<div class="text-sm text-slate-600"><%%= item[:value] %></div>
|
|
195
|
+
</div>
|
|
196
|
+
<%% end %>
|
|
197
|
+
<%% else %>
|
|
198
|
+
<div class="px-5 py-6 text-sm text-slate-500">No data available.</div>
|
|
199
|
+
<%% end %>
|
|
200
|
+
</div>
|
|
201
|
+
</div>
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
Important: The outer div must have an `id` prefixed with `source_monitor_dashboard_` for Turbo Stream targeting.
|
|
205
|
+
|
|
206
|
+
Render in `index.html.erb`:
|
|
207
|
+
```erb
|
|
208
|
+
<%%= render "my_widget", my_widget: @my_widget %>
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
### Step 6: Add Turbo Stream Broadcasting (optional)
|
|
212
|
+
|
|
213
|
+
In `lib/source_monitor/dashboard/turbo_broadcaster.rb`, add a broadcast call inside `broadcast_dashboard_updates`:
|
|
214
|
+
|
|
215
|
+
```ruby
|
|
216
|
+
Turbo::StreamsChannel.broadcast_replace_to(
|
|
217
|
+
STREAM_NAME,
|
|
218
|
+
target: "source_monitor_dashboard_my_widget",
|
|
219
|
+
html: render_partial(
|
|
220
|
+
"source_monitor/dashboard/my_widget",
|
|
221
|
+
my_widget: MyWidgetPresenter.new(
|
|
222
|
+
queries.my_widget,
|
|
223
|
+
url_helpers:
|
|
224
|
+
).to_a
|
|
225
|
+
)
|
|
226
|
+
)
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
The broadcaster is triggered by event callbacks registered in `setup!`. Existing triggers:
|
|
230
|
+
- `after_fetch_completed` -- fires after each feed fetch
|
|
231
|
+
- `after_item_created` -- fires when a new item is persisted
|
|
232
|
+
|
|
233
|
+
To add a new trigger, register another callback:
|
|
234
|
+
```ruby
|
|
235
|
+
register_callback(:after_scrape_completed, scrape_callback)
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
### Step 7: Add Metrics Recording
|
|
239
|
+
|
|
240
|
+
In the `record_metrics` method of `queries.rb`, add a case for your widget:
|
|
241
|
+
|
|
242
|
+
```ruby
|
|
243
|
+
when :my_widget
|
|
244
|
+
SourceMonitor::Metrics.gauge(:dashboard_my_widget_count, result.size)
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
## Existing Widget Reference
|
|
248
|
+
|
|
249
|
+
### Stat Cards
|
|
250
|
+
- Query: `StatsQuery` returns `{ total_sources:, active_sources:, failed_sources:, total_items:, fetches_today: }`
|
|
251
|
+
- View: `_stats.html.erb` renders `_stat_card.html.erb` as a collection
|
|
252
|
+
- Turbo target: `source_monitor_dashboard_stats`
|
|
253
|
+
|
|
254
|
+
### Recent Activity
|
|
255
|
+
- Query: `RecentActivityQuery` uses UNION ALL across fetch_logs, scrape_logs, items
|
|
256
|
+
- Struct: `RecentActivity::Event` (keyword_init Struct)
|
|
257
|
+
- Presenter: `RecentActivityPresenter` maps events to `{ label:, description:, status:, type:, time:, path: }`
|
|
258
|
+
- View: `_recent_activity.html.erb`
|
|
259
|
+
- Turbo target: `source_monitor_dashboard_recent_activity`
|
|
260
|
+
|
|
261
|
+
### Job Metrics
|
|
262
|
+
- Query: `job_metrics` delegates to `Jobs::SolidQueueMetrics.call`
|
|
263
|
+
- Returns array of `{ role:, queue_name:, summary: }` hashes
|
|
264
|
+
- View: `_job_metrics.html.erb`
|
|
265
|
+
- No Turbo broadcasting (refreshed on page load)
|
|
266
|
+
|
|
267
|
+
### Fetch Schedule
|
|
268
|
+
- Query: `UpcomingFetchSchedule` groups sources into time-window buckets
|
|
269
|
+
- Struct: `UpcomingFetchSchedule::Group` with `:key, :label, :sources, :window_start, :window_end`
|
|
270
|
+
- View: `_fetch_schedule.html.erb`
|
|
271
|
+
- Turbo target: `source_monitor_dashboard_fetch_schedule`
|
|
272
|
+
|
|
273
|
+
### Quick Actions
|
|
274
|
+
- Static `QuickAction` structs defined in `Queries::QUICK_ACTIONS`
|
|
275
|
+
- Presenter: `QuickActionsPresenter` resolves route names to paths
|
|
276
|
+
- View: inline in `index.html.erb`
|
|
277
|
+
|
|
278
|
+
## Data Structures
|
|
279
|
+
|
|
280
|
+
### RecentActivity::Event (Struct)
|
|
281
|
+
```ruby
|
|
282
|
+
Struct.new(
|
|
283
|
+
:type, :id, :occurred_at, :success, :items_created,
|
|
284
|
+
:items_updated, :scraper_adapter, :item_title, :item_url,
|
|
285
|
+
:source_name, :source_id, keyword_init: true
|
|
286
|
+
)
|
|
287
|
+
```
|
|
288
|
+
|
|
289
|
+
### UpcomingFetchSchedule::Group (Struct)
|
|
290
|
+
```ruby
|
|
291
|
+
Struct.new(
|
|
292
|
+
:key, :label, :min_minutes, :max_minutes,
|
|
293
|
+
:window_start, :window_end, :include_unscheduled, :sources,
|
|
294
|
+
keyword_init: true
|
|
295
|
+
)
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
### QuickAction (Struct)
|
|
299
|
+
```ruby
|
|
300
|
+
Struct.new(:label, :description, :route_name, keyword_init: true)
|
|
301
|
+
```
|
|
302
|
+
|
|
303
|
+
## Turbo Stream Setup
|
|
304
|
+
|
|
305
|
+
The dashboard view subscribes to the broadcast channel:
|
|
306
|
+
```erb
|
|
307
|
+
<%%= turbo_stream_from SourceMonitor::Dashboard::TurboBroadcaster::STREAM_NAME %>
|
|
308
|
+
```
|
|
309
|
+
|
|
310
|
+
Stream name: `"source_monitor_dashboard"`
|
|
311
|
+
|
|
312
|
+
The broadcaster uses `DashboardController.render(partial:, locals:)` to render partials outside of a request context.
|
|
313
|
+
|
|
314
|
+
## Caching
|
|
315
|
+
|
|
316
|
+
The `Queries` class uses an in-memory `Cache` (simple hash store) scoped to the instance. Each `Queries.new` gets a fresh cache, so data is fresh per request but not duplicated within one request.
|
|
317
|
+
|
|
318
|
+
## File Locations
|
|
319
|
+
|
|
320
|
+
| Component | Path |
|
|
321
|
+
|-----------|------|
|
|
322
|
+
| Controller | `app/controllers/source_monitor/dashboard_controller.rb` |
|
|
323
|
+
| Queries facade | `lib/source_monitor/dashboard/queries.rb` |
|
|
324
|
+
| StatsQuery | `lib/source_monitor/dashboard/queries/stats_query.rb` |
|
|
325
|
+
| RecentActivityQuery | `lib/source_monitor/dashboard/queries/recent_activity_query.rb` |
|
|
326
|
+
| UpcomingFetchSchedule | `lib/source_monitor/dashboard/upcoming_fetch_schedule.rb` |
|
|
327
|
+
| RecentActivity::Event | `lib/source_monitor/dashboard/recent_activity.rb` |
|
|
328
|
+
| RecentActivityPresenter | `lib/source_monitor/dashboard/recent_activity_presenter.rb` |
|
|
329
|
+
| QuickAction | `lib/source_monitor/dashboard/quick_action.rb` |
|
|
330
|
+
| QuickActionsPresenter | `lib/source_monitor/dashboard/quick_actions_presenter.rb` |
|
|
331
|
+
| TurboBroadcaster | `lib/source_monitor/dashboard/turbo_broadcaster.rb` |
|
|
332
|
+
| Views | `app/views/source_monitor/dashboard/` |
|
|
333
|
+
|
|
334
|
+
## Testing
|
|
335
|
+
|
|
336
|
+
Dashboard queries should be tested with integration tests that verify SQL correctness against real records. See `test/lib/source_monitor/dashboard/` for existing test patterns.
|
|
337
|
+
|
|
338
|
+
Presenters can be unit-tested with mock url_helpers:
|
|
339
|
+
```ruby
|
|
340
|
+
class FakeUrlHelpers
|
|
341
|
+
def source_path(id) = "/source_monitor/sources/#{id}"
|
|
342
|
+
def fetch_log_path(id) = "/source_monitor/fetch_logs/#{id}"
|
|
343
|
+
end
|
|
344
|
+
```
|