source_monitor 0.3.1 → 0.3.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.claude/agent-memory/vbw-vbw-dev/MEMORY.md +34 -0
- data/.claude/agent-memory/vbw-vbw-lead/MEMORY.md +49 -0
- data/.claude/commands/release.md +212 -0
- data/.claude/skills/sm-host-setup/SKILL.md +7 -3
- data/.claude/skills/sm-host-setup/reference/setup-checklist.md +3 -0
- data/.claude/skills/sm-job/SKILL.md +10 -9
- data/.gitignore +4 -0
- data/AGENTS.md +20 -57
- data/CHANGELOG.md +22 -0
- data/CLAUDE.md +1 -1
- data/CONTRIBUTING.md +5 -5
- data/Gemfile.lock +1 -1
- data/README.md +18 -5
- data/VERSION +1 -0
- data/docs/deployment.md +1 -1
- data/docs/setup.md +8 -8
- data/lib/generators/source_monitor/install/install_generator.rb +100 -0
- data/lib/source_monitor/setup/skills_installer.rb +94 -0
- data/lib/source_monitor/version.rb +1 -1
- metadata +6 -3
- data/.vbw-planning/.notification-log.jsonl +0 -246
- data/.vbw-planning/.session-log.jsonl +0 -992
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: ae2bf8e389d7c36d4f7061a13f7f4bb213c8365408f9d0c1fb818ea25b385ff7
|
|
4
|
+
data.tar.gz: a3b8b537ea6bb06251edbc53a203d9c9e9c20a1d555679a722f00ff62eef9967
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 418df72b1757e7a941e645e171d00f012072c5f2974352851015cc3501adf6a546703a0b3cc38c35de7f1d6034b05c4bf13ce162815640a609612d395a40ac3d
|
|
7
|
+
data.tar.gz: bef273caaf974979aa4a9e5dce7c4b0043efa47149a0b3f2bcba80874f7b3ff303e5f4672e90ab176d79306a4085ff6ed692f883709a37490e7df279595803e7
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# VBW Dev Agent Memory
|
|
2
|
+
|
|
3
|
+
## Project: source_monitor
|
|
4
|
+
- Rails engine gem for RSS/feed monitoring
|
|
5
|
+
- Ruby 3.4.4, Rails 8.x
|
|
6
|
+
- Test suite: 473 tests via `bin/rails test` (takes ~76 seconds)
|
|
7
|
+
- RuboCop uses `rubocop-rails-omakase` base config
|
|
8
|
+
|
|
9
|
+
## Key Learnings
|
|
10
|
+
|
|
11
|
+
### Shell/Bash in zsh
|
|
12
|
+
- `!` in zsh inline scripts causes `command not found` errors. Use `case` statements instead of `if ! ...` patterns.
|
|
13
|
+
- Use `IFS= read -r` when reading filenames from pipes to handle edge cases.
|
|
14
|
+
|
|
15
|
+
### RuboCop scope vs git scope
|
|
16
|
+
- RuboCop inspects ALL Ruby-like files (Gemfile, Rakefile, .gemspec, .rake, bin/*, config.ru), not just `.rb` files.
|
|
17
|
+
- `git ls-files -- '*.rb'` only matches `.rb` extensions. Plans scoped to `.rb` files may miss RuboCop violations in non-`.rb` Ruby files.
|
|
18
|
+
- `test/tmp/` contains untracked generated Rails app templates. These are NOT git-tracked but RuboCop scans them unless excluded.
|
|
19
|
+
|
|
20
|
+
### File structure
|
|
21
|
+
- `test/lib/tmp/install_generator/` contains test fixtures (1 tracked file: `config/initializers/source_monitor.rb`)
|
|
22
|
+
- `test/tmp/host_app_template_*` directories are generated test artifacts, not git-tracked
|
|
23
|
+
- `.rubocop.yml` is at project root, inherits from `rubocop-rails-omakase`
|
|
24
|
+
|
|
25
|
+
### Frozen string literal pragma
|
|
26
|
+
- Completed in commit `5f02db8` -- 113 files modified
|
|
27
|
+
- Added `test/tmp/**/*` to RuboCop exclude list
|
|
28
|
+
|
|
29
|
+
### RuboCop omakase ruleset details
|
|
30
|
+
- Only 45 cops enabled (out of 775 available)
|
|
31
|
+
- ALL Metrics cops disabled (ClassLength, MethodLength, BlockLength, etc.)
|
|
32
|
+
- After frozen_string_literal fix, codebase had zero remaining violations
|
|
33
|
+
- No `.rubocop.yml` exclusions needed for large files since Metrics cops are off
|
|
34
|
+
- Plan 02 completed with no code changes required
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Lead Agent Memory -- SourceMonitor
|
|
2
|
+
|
|
3
|
+
## Project Quick Facts
|
|
4
|
+
- Rails 8 engine, Ruby 3.4+, PostgreSQL-only
|
|
5
|
+
- 325 git-tracked Ruby files, 130 test files
|
|
6
|
+
- Coverage baseline: 2328 uncovered lines in config/coverage_baseline.json
|
|
7
|
+
- RuboCop: rubocop-rails-omakase, config at .rubocop.yml
|
|
8
|
+
- CI: .github/workflows/ci.yml (lint, security, test, release-verify, nightly profiling)
|
|
9
|
+
- Large files: FeedFetcher (627), Configuration (655), ImportSessionsController (792)
|
|
10
|
+
|
|
11
|
+
## Phase 1 Findings
|
|
12
|
+
- 98 git-tracked .rb files missing frozen_string_literal (out of 325)
|
|
13
|
+
- Breakdown: app/(5), lib/(24), test/(43 non-dummy), test/dummy/(22), config/(1), db/migrate/(4)
|
|
14
|
+
- test/tmp/ is NOT git-tracked (generated by install_generator tests) -- exclude from changes
|
|
15
|
+
- test/lib/tmp/install_generator/config/initializers/source_monitor.rb IS tracked and already has pragma
|
|
16
|
+
- bin/rubocop wrapper forces --config .rubocop.yml
|
|
17
|
+
- Phase 1 criterion #3 (10% coverage shrink) not addressed by plans 01/02 -- noted for future
|
|
18
|
+
|
|
19
|
+
## Phase 2 Planning Insights
|
|
20
|
+
- Coverage baseline.json has 2117 uncovered lines across 105 files (not 2328 -- that was line count)
|
|
21
|
+
- Top 7 files account for ~743 uncovered lines (35% of total)
|
|
22
|
+
- FeedFetcher: 245 uncovered, 12 existing tests, uses VCR+WebMock
|
|
23
|
+
- ItemCreator: 228 uncovered, 8 existing tests, needs mock entries for Atom/JSON branches
|
|
24
|
+
- Configuration: 94 uncovered, 5 existing tests, ~12 nested settings classes
|
|
25
|
+
- Dashboard::Queries: 66 uncovered, 7 existing tests, uses raw SQL + Cache
|
|
26
|
+
- BulkSourceScraper: 66 uncovered, 6 existing tests, ActiveJob test adapter
|
|
27
|
+
- Broadcaster: 48 uncovered, NO existing test file -- needs creation
|
|
28
|
+
- SourcesIndexMetrics: 34 uncovered, 3 existing tests
|
|
29
|
+
- All 5 Phase 2 plans are Wave 1 (no inter-plan dependencies, no file conflicts)
|
|
30
|
+
- Testing main files will indirectly cover supporting files (retry_policy, fetch_error, state, enqueuer, etc.)
|
|
31
|
+
- Broadcaster tests need Turbo::StreamsChannel stubs -- no full Action Cable required
|
|
32
|
+
- 50% coverage target requires ~1059 lines covered; direct plans target ~630, rest from indirect
|
|
33
|
+
|
|
34
|
+
## Phase 4 Planning Insights
|
|
35
|
+
- 3 plans: conventions-audit (wave 1), item-creator-extraction (wave 1), final-verification (wave 2)
|
|
36
|
+
- SourcesController has dead `fetch`/`retry` methods (leftover from Phase 3 CRUD extraction)
|
|
37
|
+
- ImportSessionsController `new` and `create` are byte-for-byte identical
|
|
38
|
+
- 4 RuboCop violations in db/migrate/20260210204022_add_composite_index_to_log_entries.rb
|
|
39
|
+
- Duplicate test: test/controllers/concerns/ vs test/controllers/source_monitor/concerns/
|
|
40
|
+
- ItemCreator at 601 lines is the last file over 300 lines -- extract to entry_parser + content_extractor
|
|
41
|
+
- Coverage baseline still at 2117 uncovered (not regenerated since Phase 1) -- must regenerate
|
|
42
|
+
- 60% reduction target: at most 847 uncovered lines
|
|
43
|
+
- dashboard/queries.rb (356) and application_helper.rb (346) are near 300 but acceptable as view/query code
|
|
44
|
+
- `bin/update-coverage-baseline` requires `COVERAGE=1 bin/rails test` first
|
|
45
|
+
|
|
46
|
+
## Bash/Shell Notes
|
|
47
|
+
- zsh on macOS -- `!` in shell conditionals causes `command not found` errors
|
|
48
|
+
- Use `case` statements instead of `if ! grep` patterns in zsh
|
|
49
|
+
- git ls-files piped to while loops works reliably for file enumeration
|
|
@@ -0,0 +1,212 @@
|
|
|
1
|
+
# Release: PR, CI, Merge, and Gem Build
|
|
2
|
+
|
|
3
|
+
Orchestrate a full release cycle for the source_monitor gem. This command handles changelog generation, version bumping, PR creation, CI monitoring, auto-merge on success, release tagging, and gem build with push instructions.
|
|
4
|
+
|
|
5
|
+
## Inputs
|
|
6
|
+
|
|
7
|
+
- `$ARGUMENTS` -- Optional: version bump description or release notes summary. If empty, derive from commits since last tag.
|
|
8
|
+
|
|
9
|
+
## Step 1: Git Hygiene
|
|
10
|
+
|
|
11
|
+
Run these checks. If ANY fail, STOP and report the issue to the user.
|
|
12
|
+
|
|
13
|
+
1. **Working tree clean**: `git status --porcelain` must be empty. If not, list the dirty files and ask the user whether to commit, stash, or abort.
|
|
14
|
+
2. **On main branch**: `git branch --show-current` must be `main`. If not, ask the user if they want to continue from the current branch or switch.
|
|
15
|
+
3. **Fetch latest**: `git fetch origin main`
|
|
16
|
+
4. **Up to date with remote**: Compare `git rev-parse HEAD` with `git rev-parse origin/main`. If behind, ask the user whether to pull.
|
|
17
|
+
5. **No unpushed commits**: Compare local HEAD with `origin/main`. If ahead, note how many commits are unpushed -- these will be included in the PR.
|
|
18
|
+
|
|
19
|
+
Report a summary:
|
|
20
|
+
```
|
|
21
|
+
Git Status:
|
|
22
|
+
Branch: main
|
|
23
|
+
Clean: yes
|
|
24
|
+
Synced with origin: yes
|
|
25
|
+
Unpushed commits: N
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## Step 2: Version Check
|
|
29
|
+
|
|
30
|
+
1. Read `lib/source_monitor/version.rb` to get the current VERSION.
|
|
31
|
+
2. Read the latest git tag with `git tag --sort=-v:refname | head -1`.
|
|
32
|
+
3. If the VERSION matches the latest tag (e.g., both are `0.3.2`), the version hasn't been bumped. Ask the user:
|
|
33
|
+
- "Current version is X.Y.Z which already has a tag. What should the new version be?"
|
|
34
|
+
- Offer options: patch (X.Y.Z+1), minor (X.Y+1.0), major (X+1.0.0), or custom.
|
|
35
|
+
- Update `lib/source_monitor/version.rb` with the new version.
|
|
36
|
+
4. If VERSION is ahead of the latest tag, proceed with the current version.
|
|
37
|
+
|
|
38
|
+
Store the release version for later steps. Do NOT commit yet -- that happens after the changelog is updated.
|
|
39
|
+
|
|
40
|
+
## Step 3: Update CHANGELOG.md
|
|
41
|
+
|
|
42
|
+
The changelog follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) format. The file is at `CHANGELOG.md` in the project root.
|
|
43
|
+
|
|
44
|
+
1. **Gather commit history** since the last tag:
|
|
45
|
+
```
|
|
46
|
+
git log vPREVIOUS..HEAD --oneline --no-merges
|
|
47
|
+
```
|
|
48
|
+
2. **Categorize commits** into Keep a Changelog sections based on commit prefixes and content:
|
|
49
|
+
- `feat:`, `add:` --> **Added**
|
|
50
|
+
- `fix:`, `bugfix:` --> **Fixed**
|
|
51
|
+
- `chore:`, `refactor:`, `perf:` --> **Changed**
|
|
52
|
+
- `docs:` --> **Documentation** (only include if substantive)
|
|
53
|
+
- `BREAKING:` or `!:` --> **Breaking Changes** (at the top)
|
|
54
|
+
- `remove:`, `deprecate:` --> **Removed**
|
|
55
|
+
- Skip merge commits, version bumps, and CI-only changes.
|
|
56
|
+
- If `$ARGUMENTS` was provided, use it to inform/supplement the categorization.
|
|
57
|
+
|
|
58
|
+
3. **Draft the changelog entry** and present it to the user for review:
|
|
59
|
+
```
|
|
60
|
+
## [X.Y.Z] - YYYY-MM-DD
|
|
61
|
+
|
|
62
|
+
### Added
|
|
63
|
+
- <items>
|
|
64
|
+
|
|
65
|
+
### Fixed
|
|
66
|
+
- <items>
|
|
67
|
+
|
|
68
|
+
### Changed
|
|
69
|
+
- <items>
|
|
70
|
+
```
|
|
71
|
+
Each bullet should be a concise, user-facing description (not a raw commit message). Consolidate related commits into single bullets where it makes sense.
|
|
72
|
+
|
|
73
|
+
4. **Ask the user to approve** the changelog entry. Offer to edit if they want changes.
|
|
74
|
+
|
|
75
|
+
5. **Write the entry** into `CHANGELOG.md`:
|
|
76
|
+
- Replace the `## [Unreleased]` section contents with `- No unreleased changes yet.`
|
|
77
|
+
- Insert the new versioned entry immediately after the `## [Unreleased]` block and before the previous release entry.
|
|
78
|
+
- Preserve all existing entries below.
|
|
79
|
+
|
|
80
|
+
## Step 4: Commit Version Bump + Changelog
|
|
81
|
+
|
|
82
|
+
Once both `lib/source_monitor/version.rb` (if changed) and `CHANGELOG.md` are updated:
|
|
83
|
+
|
|
84
|
+
1. Stage the changed files:
|
|
85
|
+
```
|
|
86
|
+
git add lib/source_monitor/version.rb CHANGELOG.md
|
|
87
|
+
```
|
|
88
|
+
2. Create a single commit:
|
|
89
|
+
```
|
|
90
|
+
chore: bump version to X.Y.Z and update changelog
|
|
91
|
+
```
|
|
92
|
+
If only the changelog changed (version was already bumped), use:
|
|
93
|
+
```
|
|
94
|
+
chore: update changelog for vX.Y.Z release
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## Step 5: Create Release Branch and PR
|
|
98
|
+
|
|
99
|
+
1. Create a release branch: `git checkout -b release/vX.Y.Z`
|
|
100
|
+
2. Push the branch: `git push -u origin release/vX.Y.Z`
|
|
101
|
+
3. Generate a PR body from the new CHANGELOG.md entry (use the entry written in Step 3, not raw commits).
|
|
102
|
+
4. Create the PR using `gh pr create`:
|
|
103
|
+
- Title: `Release vX.Y.Z`
|
|
104
|
+
- Body format:
|
|
105
|
+
```
|
|
106
|
+
## Release vX.Y.Z
|
|
107
|
+
|
|
108
|
+
<paste the CHANGELOG.md entry content here>
|
|
109
|
+
|
|
110
|
+
### Release Checklist
|
|
111
|
+
- [x] Version bumped in `lib/source_monitor/version.rb`
|
|
112
|
+
- [x] CHANGELOG.md updated
|
|
113
|
+
- [ ] CI passes (lint, security, test, release_verification)
|
|
114
|
+
|
|
115
|
+
---
|
|
116
|
+
Auto-generated release PR by `/release` command.
|
|
117
|
+
```
|
|
118
|
+
- Base: `main`
|
|
119
|
+
5. Report the PR URL to the user.
|
|
120
|
+
|
|
121
|
+
## Step 6: Monitor CI Pipeline
|
|
122
|
+
|
|
123
|
+
Poll the CI status using `gh pr checks <PR_NUMBER> --watch` or repeated `gh pr checks <PR_NUMBER>` calls. The CI has 4 required jobs: `lint`, `security`, `test`, `release_verification`.
|
|
124
|
+
|
|
125
|
+
1. Tell the user: "Monitoring CI pipeline for PR #N... This typically takes 3-5 minutes."
|
|
126
|
+
2. Poll with `gh pr checks <PR_NUMBER>` every 30 seconds, up to 15 minutes.
|
|
127
|
+
3. After each poll, report progress if any jobs completed.
|
|
128
|
+
|
|
129
|
+
### If CI PASSES (all checks green):
|
|
130
|
+
|
|
131
|
+
Continue to Step 7.
|
|
132
|
+
|
|
133
|
+
### If CI FAILS:
|
|
134
|
+
|
|
135
|
+
1. Get the failure details: `gh pr checks <PR_NUMBER>` to identify which jobs failed.
|
|
136
|
+
2. For each failed job, fetch the log summary:
|
|
137
|
+
```
|
|
138
|
+
gh run view <RUN_ID> --log-failed | tail -50
|
|
139
|
+
```
|
|
140
|
+
3. Present to the user:
|
|
141
|
+
```
|
|
142
|
+
CI Failed on PR #N
|
|
143
|
+
|
|
144
|
+
Failed Jobs:
|
|
145
|
+
- <job_name>: <brief summary of failure>
|
|
146
|
+
|
|
147
|
+
Log excerpt:
|
|
148
|
+
<relevant log lines>
|
|
149
|
+
```
|
|
150
|
+
4. Ask the user what to do:
|
|
151
|
+
- "Fix the issues and re-push" -- Help fix the issues, commit, push, and restart CI monitoring
|
|
152
|
+
- "Close the PR and abort" -- Close the PR, delete the branch, switch back to main
|
|
153
|
+
- "Investigate manually" -- Stop and let the user handle it
|
|
154
|
+
|
|
155
|
+
## Step 7: Auto-Merge PR
|
|
156
|
+
|
|
157
|
+
Once CI is green:
|
|
158
|
+
|
|
159
|
+
1. Merge the PR: `gh pr merge <PR_NUMBER> --merge --delete-branch`
|
|
160
|
+
2. Switch back to main and pull: `git checkout main && git pull origin main`
|
|
161
|
+
3. Report: "PR #N merged successfully."
|
|
162
|
+
|
|
163
|
+
## Step 8: Tag the Release
|
|
164
|
+
|
|
165
|
+
1. Create an annotated tag:
|
|
166
|
+
```
|
|
167
|
+
git tag -a vX.Y.Z -m "Release vX.Y.Z"
|
|
168
|
+
```
|
|
169
|
+
2. Push the tag: `git push origin vX.Y.Z`
|
|
170
|
+
3. Create a GitHub release from the tag:
|
|
171
|
+
```
|
|
172
|
+
gh release create vX.Y.Z --title "vX.Y.Z" --notes "<changelog entry from Step 3>"
|
|
173
|
+
```
|
|
174
|
+
4. Report the release URL.
|
|
175
|
+
|
|
176
|
+
## Step 9: Build the Gem
|
|
177
|
+
|
|
178
|
+
1. Clean any old gem files: remove any `source_monitor-*.gem` files in the project root.
|
|
179
|
+
2. Build the gem: `gem build source_monitor.gemspec`
|
|
180
|
+
3. Verify the gem was built: check for `source_monitor-X.Y.Z.gem` in the project root.
|
|
181
|
+
4. Show the gem contents summary: `gem spec source_monitor-X.Y.Z.gem | head -30`
|
|
182
|
+
|
|
183
|
+
## Step 10: Gem Push Instructions
|
|
184
|
+
|
|
185
|
+
Present the final instructions to the user:
|
|
186
|
+
|
|
187
|
+
```
|
|
188
|
+
Release vX.Y.Z Complete!
|
|
189
|
+
|
|
190
|
+
Git tag: vX.Y.Z (pushed)
|
|
191
|
+
GitHub: <release URL>
|
|
192
|
+
Gem built: source_monitor-X.Y.Z.gem
|
|
193
|
+
|
|
194
|
+
To publish to RubyGems:
|
|
195
|
+
|
|
196
|
+
gem push source_monitor-X.Y.Z.gem
|
|
197
|
+
|
|
198
|
+
You'll be prompted for your RubyGems OTP code (check your authenticator app).
|
|
199
|
+
The gem has `rubygems_mfa_required` enabled, so the OTP is mandatory.
|
|
200
|
+
|
|
201
|
+
After pushing, verify at:
|
|
202
|
+
https://rubygems.org/gems/source_monitor/versions/X.Y.Z
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
Do NOT run `gem push` automatically -- always let the user handle the OTP-protected push manually.
|
|
206
|
+
|
|
207
|
+
## Error Recovery
|
|
208
|
+
|
|
209
|
+
- If any step fails unexpectedly, report what happened and where things stand.
|
|
210
|
+
- If a release branch already exists, ask the user whether to reuse or recreate it.
|
|
211
|
+
- If the tag already exists, skip tagging and inform the user.
|
|
212
|
+
- Always leave the user on the `main` branch in a clean state when possible.
|
|
@@ -61,7 +61,7 @@ bin/source_monitor install --yes
|
|
|
61
61
|
gem "source_monitor", "~> 0.3.0" # in Gemfile
|
|
62
62
|
bundle install
|
|
63
63
|
|
|
64
|
-
# 2. Run the install generator
|
|
64
|
+
# 2. Run the install generator (mounts engine, creates initializer, configures recurring jobs)
|
|
65
65
|
bin/rails generate source_monitor:install --mount-path=/source_monitor
|
|
66
66
|
|
|
67
67
|
# 3. Copy engine migrations
|
|
@@ -70,7 +70,7 @@ bin/rails railties:install:migrations FROM=source_monitor
|
|
|
70
70
|
# 4. Apply migrations
|
|
71
71
|
bin/rails db:migrate
|
|
72
72
|
|
|
73
|
-
# 5. Start background workers
|
|
73
|
+
# 5. Start background workers (recurring jobs configured automatically in config/recurring.yml)
|
|
74
74
|
bin/rails solid_queue:start
|
|
75
75
|
|
|
76
76
|
# 6. Verify
|
|
@@ -86,7 +86,7 @@ gem "source_monitor", github: "dchuk/source_monitor"
|
|
|
86
86
|
|
|
87
87
|
## What the Install Generator Does
|
|
88
88
|
|
|
89
|
-
The generator (`SourceMonitor::Generators::InstallGenerator`) performs
|
|
89
|
+
The generator (`SourceMonitor::Generators::InstallGenerator`) performs three actions:
|
|
90
90
|
|
|
91
91
|
1. **Mounts the engine** in `config/routes.rb`:
|
|
92
92
|
```ruby
|
|
@@ -97,6 +97,9 @@ The generator (`SourceMonitor::Generators::InstallGenerator`) performs two actio
|
|
|
97
97
|
2. **Creates the initializer** at `config/initializers/source_monitor.rb`:
|
|
98
98
|
Uses the template at `lib/generators/source_monitor/install/templates/source_monitor.rb.tt`. Skips if the file already exists.
|
|
99
99
|
|
|
100
|
+
3. **Configures recurring jobs** in `config/recurring.yml`:
|
|
101
|
+
Adds entries for `ScheduleFetchesJob` (every minute), scrape scheduling (every 2 minutes), `ItemCleanupJob` (2am daily), and `LogCleanupJob` (3am daily). If the file doesn't exist, creates it with `default: &default` and environment sections. If it exists, merges entries without overwriting. Skips if SourceMonitor entries are already present.
|
|
102
|
+
|
|
100
103
|
Re-running the generator is safe and idempotent.
|
|
101
104
|
|
|
102
105
|
## Post-Install Configuration
|
|
@@ -217,6 +220,7 @@ end
|
|
|
217
220
|
- [ ] `bundle install` completed
|
|
218
221
|
- [ ] Install generator ran (`bin/rails generate source_monitor:install`)
|
|
219
222
|
- [ ] Engine migrations copied and applied
|
|
223
|
+
- [ ] Recurring jobs configured in `config/recurring.yml`
|
|
220
224
|
- [ ] Solid Queue workers started
|
|
221
225
|
- [ ] Authentication hooks configured in initializer
|
|
222
226
|
- [ ] `bin/source_monitor verify` passes
|
|
@@ -39,6 +39,7 @@ bin/rails generate source_monitor:install --mount-path=/source_monitor
|
|
|
39
39
|
Verify after running:
|
|
40
40
|
- [ ] `config/routes.rb` contains `mount SourceMonitor::Engine, at: "/source_monitor"`
|
|
41
41
|
- [ ] `config/initializers/source_monitor.rb` exists
|
|
42
|
+
- [ ] `config/recurring.yml` contains SourceMonitor recurring job entries
|
|
42
43
|
|
|
43
44
|
## Phase 4: Database Setup
|
|
44
45
|
|
|
@@ -75,6 +76,8 @@ end
|
|
|
75
76
|
|
|
76
77
|
## Phase 6: Configure Workers
|
|
77
78
|
|
|
79
|
+
The install generator automatically configures recurring jobs in `config/recurring.yml` (fetch scheduling, scrape scheduling, item cleanup, log cleanup). These run automatically with `bin/dev` or `bin/jobs`.
|
|
80
|
+
|
|
78
81
|
Ensure `config/solid_queue.yml` (or equivalent) includes the SourceMonitor queues:
|
|
79
82
|
|
|
80
83
|
```yaml
|
|
@@ -159,15 +159,16 @@ SourceMonitor.queue_name(:fetch) # => "source_monitor_fetch"
|
|
|
159
159
|
|
|
160
160
|
### Recurring Jobs
|
|
161
161
|
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
162
|
+
The install generator (`bin/rails generate source_monitor:install`) automatically configures these recurring jobs in `config/recurring.yml`:
|
|
163
|
+
|
|
164
|
+
| Job | Schedule |
|
|
165
|
+
|-----|----------|
|
|
166
|
+
| `SourceMonitor::ScheduleFetchesJob` | every minute |
|
|
167
|
+
| `SourceMonitor::Scraping::Scheduler.run` | every 2 minutes |
|
|
168
|
+
| `SourceMonitor::ItemCleanupJob` | at 2am every day |
|
|
169
|
+
| `SourceMonitor::LogCleanupJob` | at 3am every day |
|
|
170
|
+
|
|
171
|
+
These run automatically with `bin/dev` or `bin/jobs`. If you need to customize, edit `config/recurring.yml` directly.
|
|
171
172
|
|
|
172
173
|
## Retry Policies
|
|
173
174
|
|
data/.gitignore
CHANGED
data/AGENTS.md
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
# Repository Guidelines
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Refer to `CLAUDE.md` for project conventions and skills catalog.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
Use rbenv for all ruby and bundler/gem commands, not the system ruby.
|
|
6
6
|
|
|
7
7
|
## Contribution Workflow
|
|
8
8
|
|
|
9
9
|
- Treat the engine like any external contributor would: no direct commits to `main`.
|
|
10
|
-
- Before writing code, branch off the latest `origin/main` (use a descriptive `feature/` or `bugfix/` prefix) and open a draft PR on `github.com/dchuk/source_monitor
|
|
10
|
+
- Before writing code, branch off the latest `origin/main` (use a descriptive `feature/` or `bugfix/` prefix) and open a draft PR on `github.com/dchuk/source_monitor`.
|
|
11
11
|
- Push early and often to that branch so history stays visible; keep commits scoped and rebases local to your branch only.
|
|
12
12
|
- Move the PR out of draft once tests pass and the slice is ready for review; request at least one review and wait for all CI jobs (lint, security, Rails tests + diff coverage) to succeed before merging.
|
|
13
13
|
- The `test` job enforces diff coverage via `bin/check-diff-coverage`; if legitimate gaps remain after new code paths, refresh `config/coverage_baseline.json` by running `bin/test-coverage` followed by `bin/update-coverage-baseline` on the updated branch and commit the regenerated baseline.
|
|
@@ -59,7 +59,7 @@ Run `bin/setup` to install gems, prepare the dummy database, and compile Tailwin
|
|
|
59
59
|
|
|
60
60
|
## Coding Style & Naming Conventions
|
|
61
61
|
|
|
62
|
-
Use two-space indentation and Ruby
|
|
62
|
+
Use two-space indentation and Ruby 4.0+ syntax. Keep engine classes under the `SourceMonitor::` namespace; new modules should mirror their directory, e.g., `lib/source_monitor/fetching/pipeline.rb`. Favor service objects ending in `Service`, jobs ending in `Job`, and background channels ending in `Channel`. Rails defaults handle formatting, but run `bundle exec rubocop` (configured via `.rubocop.yml`) before opening a PR. For views, stick with ERB and Tailwind utility classes.
|
|
63
63
|
|
|
64
64
|
## Testing Guidelines
|
|
65
65
|
|
|
@@ -67,66 +67,29 @@ MiniTest drives coverage (`test/models`, `test/controllers`, `test/system`). Nam
|
|
|
67
67
|
|
|
68
68
|
## Commit & Pull Request Guidelines
|
|
69
69
|
|
|
70
|
-
Adopt imperative commit messages in the format `scope: action`, e.g., `sources: enforce URL normalization`. Group unrelated work into separate commits. PRs should describe context, summarise the slice delivered, list validation steps (`bin/rails test`, manual fetch run)
|
|
70
|
+
Adopt imperative commit messages in the format `scope: action`, e.g., `sources: enforce URL normalization`. Group unrelated work into separate commits. PRs should describe context, summarise the slice delivered, and list validation steps (`bin/rails test`, manual fetch run). Include screenshots or console output when altering UI or background jobs. Request at least one review and ensure CI completes before merge.
|
|
71
71
|
|
|
72
72
|
## Security & Configuration Tips
|
|
73
73
|
|
|
74
74
|
Store secrets (API keys, webhook tokens) in `config/credentials/` and never commit plain-text values. When adding HTTP endpoints or webhooks, default to Solid Queue middleware for retries and respect the allowlist in `config/source_monitor.yml`. Document new environment variables in `config/application.yml.sample` and call out any migrations that impact host apps.
|
|
75
75
|
|
|
76
|
-
|
|
76
|
+
## Clean Coding Principles
|
|
77
77
|
|
|
78
|
-
|
|
78
|
+
- **SRP**: Classes and methods should have a single responsibility.
|
|
79
|
+
- **DRY**: Avoid duplication; changes should only need one edit.
|
|
80
|
+
- **Depend on behaviour, not data**: Wrap instance variables in methods (`attr_reader`); use Struct for data structures.
|
|
81
|
+
- **Minimise dependencies**: Use dependency injection, encapsulate external messages, prefer hash arguments.
|
|
82
|
+
- **Depend on things that change less often than you do.**
|
|
79
83
|
|
|
80
|
-
|
|
84
|
+
## Claude Code Skills
|
|
81
85
|
|
|
82
|
-
|
|
86
|
+
SourceMonitor ships 14 engine-specific Claude Code skills (`sm-*` prefix) covering the domain model, configuration DSL, pipeline stages, testing conventions, and more. Skills are distributed with the gem and installed into `.claude/skills/` via rake tasks:
|
|
83
87
|
|
|
84
|
-
|
|
88
|
+
```bash
|
|
89
|
+
bin/rails source_monitor:skills:install # Consumer skills (host app integration)
|
|
90
|
+
bin/rails source_monitor:skills:contributor # Contributor skills (engine development)
|
|
91
|
+
bin/rails source_monitor:skills:all # All skills
|
|
92
|
+
bin/rails source_monitor:skills:remove # Remove all sm-* skills
|
|
93
|
+
```
|
|
85
94
|
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
Hide instance variables by wrapping them in a method, an attr_reader. The method changes a variable from data ( which is referenced all over) to behaviour (which is defined once). This means if you need to adjust the data you only have to make a change in one place.
|
|
89
|
-
|
|
90
|
-
Hide data structures using the Struct class.
|
|
91
|
-
|
|
92
|
-
Minimise Dependencies: An object depends on another object if, when one object changes, the other might be forced to change in turn. An object has a dependency when it knows:
|
|
93
|
-
|
|
94
|
-
The name of another class.
|
|
95
|
-
|
|
96
|
-
The name of a message that it intends to send to someone other than self.
|
|
97
|
-
|
|
98
|
-
The arguments that a message requires.
|
|
99
|
-
|
|
100
|
-
The order of those arguments.
|
|
101
|
-
|
|
102
|
-
Dependency Injection: Addresses the first dependency. It decouples two objects by moving the creation of object_A outside of object_B eg:
|
|
103
|
-
|
|
104
|
-
object_A.new(1,2,object_B.new)
|
|
105
|
-
|
|
106
|
-
Encapsulation: Addresses the second dependency. Create wrapper methods so that external messages are isolated in one place. The wrapper method is called throughout the class, rather than sending messages externally.
|
|
107
|
-
|
|
108
|
-
Use hashes for initialization arguments: this addresses the final dependencies.
|
|
109
|
-
|
|
110
|
-
Depend on things that change less often than you do
|
|
111
|
-
|
|
112
|
-
## Agent Notes (2025-10-08)
|
|
113
|
-
|
|
114
|
-
- Finished roadmap section 05.03 (structured fetch error handling). Added `SourceMonitor::Fetching::FetchError` hierarchy, expanded `FeedFetcher` error handling, and ensured instrumentation payloads include success/error context.
|
|
115
|
-
- Fetch attempts now always create `FetchLog` records and update `Source` failure state; added MiniTest coverage for timeout, HTTP 404, and malformed feed scenarios.
|
|
116
|
-
- Relevant tests: `bundle exec rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb` and full suite via `bundle exec rails test` (both passing).
|
|
117
|
-
- Next roadmap item is 05.04 (not started). Untracked tmp dir `test/lib/tmp` can be safely ignored if it reappears.
|
|
118
|
-
|
|
119
|
-
## Agent Notes (2025-10-09)
|
|
120
|
-
|
|
121
|
-
- Introduced `SourceMonitor::Scrapers::Base`, establishing the scraper adapter contract and Result object. Subclasses merge default, source, and invocation settings (HashWithIndifferentAccess) and must return a Result with status/html/content/metadata. Use `SourceMonitor::Scrapers::Base.call(item:, source:, settings:, http:)` to execute adapters.
|
|
122
|
-
- Added Readability scraper adapter leveraging SourceMonitor::HTTP, Nokolexbor, and ruby-readability. Supports override selectors via `scrape_settings[:selectors]`, records extraction metadata including strategy and inferred status, and returns structured failure results on HTTP errors.
|
|
123
|
-
- Extracted scraper HTTP fetching and parsing into dedicated collaborators (`SourceMonitor::Scrapers::Fetchers::HttpFetcher` and `SourceMonitor::Scrapers::Parsers::ReadabilityParser`). The adapter now orchestrates these services, simplifying future parser additions while keeping failure metadata consistent.
|
|
124
|
-
- Added Readability scraper adapter leveraging SourceMonitor::HTTP, Nokolexbor, and ruby-readability. Supports override selectors via `scrape_settings[:selectors]`, records extraction metadata including strategy and inferred status, and returns structured failure results on HTTP errors.
|
|
125
|
-
- Phase 07.03 complete: introduced `SourceMonitor::ItemContent` table with one-to-one association, virtual accessors on `Item`, and migration that backfills existing scraped data while dropping legacy columns. Clearing both fields now removes the content row. Run `bundle exec rails db:migrate` in host apps after upgrading.
|
|
126
|
-
- Phase 07.04 complete: added `SourceMonitor::Scraping::ItemScraper` service with log recording, manual scrape controls in the item admin, and a dedicated scraping configuration section on the source form. Item show page now highlights content comparisons, status badges, and recent scrape errors. System coverage updated to exercise the manual scraping flow.
|
|
127
|
-
- Phase 08.02 complete: created `SourceMonitor::FetchFeedJob`, introduced `SourceMonitor::Fetching::FetchRunner` with Postgres advisory locking, and added a temporary `ScrapeItemJob` stub so new items from auto-scrape sources are queued with `scrape_status` set to `pending`. Manual fetch UI now reuses the runner, and tests cover job retry behavior plus follow-up scraping enqueue (`bundle exec rails test test/lib/source_monitor/fetching/fetch_runner_test.rb`, `bundle exec rails test test/jobs/source_monitor/fetch_feed_job_test.rb`, and `bundle exec rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb`).
|
|
128
|
-
- Phase 08.03 complete: implemented `SourceMonitor::Scraping::Enqueuer` with deduplication/auto-scrape guards, promoted `SourceMonitor::ScrapeItemJob` to execute `ItemScraper`, and updated the manual item view to enqueue jobs instead of running scrapes inline. Added unit coverage for the enqueuer/job plus refreshed the manual scrape system test (`bundle exec rails test test/lib/source_monitor/scraping/enqueuer_test.rb test/jobs/source_monitor/scrape_item_job_test.rb test/system/items_test.rb test/lib/source_monitor/fetching/fetch_runner_test.rb`).
|
|
129
|
-
- Phase 08.04 complete: manual "Fetch Now" now queues `FetchFeedJob` via `FetchRunner.enqueue`, dashboard shows live queue metrics from `SourceMonitor::Jobs::Visibility`, and optional Mission Control link appears when configured. System tests updated to assert job enqueues (`test/system/sources_test.rb`, `test/system/items_test.rb`, `test/system/dashboard_test.rb`); run `bundle exec rails test test/system` after UI tweaks.
|
|
130
|
-
- Install generator now also creates `config/initializers/source_monitor.rb` with documented defaults for queue settings, metrics, and Mission Control hooks. Update the initializer if future configuration options are added.
|
|
131
|
-
- `SourceMonitor.mission_control_dashboard_path` now validates that the configured path exists before rendering a dashboard link, preventing 404s when Mission Control isn't mounted yet. Template comments outline how to mount the engine when enabling it.
|
|
132
|
-
- `test/dummy/bin/dev` now runs the Rails server with `BUNDLE_GEMFILE` pointing at the dummy app, so dummy-only gems (like mission_control-jobs) load correctly while the Tailwind watcher still uses the engine's Gemfile.
|
|
95
|
+
See `CLAUDE.md` for the full skills catalog and usage details.
|
data/CHANGELOG.md
CHANGED
|
@@ -15,6 +15,28 @@ All notable changes to this project are documented below. The format follows [Ke
|
|
|
15
15
|
|
|
16
16
|
- No unreleased changes yet.
|
|
17
17
|
|
|
18
|
+
## [0.3.3] - 2026-02-11
|
|
19
|
+
|
|
20
|
+
### Fixed
|
|
21
|
+
|
|
22
|
+
- Added missing `recurring.yml` configuration to the install generator so host apps get Solid Queue recurring job config on install.
|
|
23
|
+
- Fixed YAML alias parsing in install generator so merging into existing `recurring.yml` files with `<<: *default` anchors works correctly.
|
|
24
|
+
|
|
25
|
+
### Changed
|
|
26
|
+
|
|
27
|
+
- Updated `sm-host-setup` and `sm-job` skills to reflect latest conventions.
|
|
28
|
+
- Updated setup documentation with current installation steps.
|
|
29
|
+
|
|
30
|
+
## [0.3.2] - 2026-02-10
|
|
31
|
+
|
|
32
|
+
### Fixed
|
|
33
|
+
|
|
34
|
+
- Updated README, AGENTS.md, CONTRIBUTING.md, and docs/ to reflect v0.3.1 changes (Ruby 4.0+, gem version refs, skills system documentation).
|
|
35
|
+
- Replaced stale `.ai/` references with `CLAUDE.md` and `AGENTS.md` across all docs.
|
|
36
|
+
- Corrected `SOURCE_MONITOR_TEST_WORKERS` env var to `PARALLEL_WORKERS` in CONTRIBUTING.md.
|
|
37
|
+
- Removed historical agent development notes from AGENTS.md.
|
|
38
|
+
- Condensed verbose clean coding guidelines into concise bullet summary.
|
|
39
|
+
|
|
18
40
|
## [0.3.1] - 2026-02-10
|
|
19
41
|
|
|
20
42
|
### Added
|
data/CLAUDE.md
CHANGED
|
@@ -45,7 +45,7 @@ Run /vbw:help for all commands.
|
|
|
45
45
|
|
|
46
46
|
| Layer | Technology |
|
|
47
47
|
|-------|------------|
|
|
48
|
-
| Ruby |
|
|
48
|
+
| Ruby | 4.0+ |
|
|
49
49
|
| Rails | 8.x |
|
|
50
50
|
| Testing | Minitest (no fixtures -- uses factory helpers + WebMock/VCR) |
|
|
51
51
|
| Authorization | Host app responsibility (mountable engine) |
|
data/CONTRIBUTING.md
CHANGED
|
@@ -4,7 +4,7 @@ This project ships each roadmap slice behind a fast, reliable test loop. The not
|
|
|
4
4
|
|
|
5
5
|
## Test Suite Expectations
|
|
6
6
|
|
|
7
|
-
- **Fast feedback:** run `bundle exec rake app:test:smoke` for unit, helper, and job coverage before pushing. Use `
|
|
7
|
+
- **Fast feedback:** run `bundle exec rake app:test:smoke` for unit, helper, and job coverage before pushing. Use `PARALLEL_WORKERS=1` locally when profiling failures for determinism.
|
|
8
8
|
- **Full validation:** continue to run `bundle exec rails test` (or `bin/test-coverage` in CI) before marking a slice ready for review.
|
|
9
9
|
- **Background jobs:** the default adapter is `:test`. Switch to inline execution only for the precise block that needs it via `with_inline_jobs { ... }`; never flip the global adapter.
|
|
10
10
|
- **Database usage:** prefer transactional fixtures over manual `delete_all`. Reach for `setup_once` (TestProf’s `before_all`) when immutable data can be shared safely across examples.
|
|
@@ -13,16 +13,16 @@ This project ships each roadmap slice behind a fast, reliable test loop. The not
|
|
|
13
13
|
## Performance Profiling
|
|
14
14
|
|
|
15
15
|
- **Local commands:**
|
|
16
|
-
- `TAG_PROF=type
|
|
17
|
-
- `EVENT_PROF=sql.active_record
|
|
18
|
-
- `TEST_STACK_PROF=1
|
|
16
|
+
- `TAG_PROF=type PARALLEL_WORKERS=1 bundle exec rails test`
|
|
17
|
+
- `EVENT_PROF=sql.active_record PARALLEL_WORKERS=1 bundle exec rails test`
|
|
18
|
+
- `TEST_STACK_PROF=1 PARALLEL_WORKERS=1 bundle exec rails test test/integration`
|
|
19
19
|
- **CI automation:** a scheduled `profiling` workflow runs the commands above nightly, archives `tmp/test_prof`, and applies guardrails via `bin/check-test-prof-metrics` (suite ≤ 80s, integration ≤ 35s, DB time ≤ 5s). Failures block the job so regressions surface quickly.
|
|
20
20
|
|
|
21
21
|
## Code Review Checklist
|
|
22
22
|
|
|
23
23
|
Before approval, verify that the changes:
|
|
24
24
|
|
|
25
|
-
1. Avoid unnecessary database persistence (
|
|
25
|
+
1. Avoid unnecessary database persistence (test doubles over `.create` when assertions allow).
|
|
26
26
|
2. Share heavy setup with `setup_once` or helper memoization instead of per-test rebuilds.
|
|
27
27
|
3. Scope inline job execution to `with_inline_jobs { ... }` blocks—no suite-wide adapter swaps.
|
|
28
28
|
4. Keep new or modified system specs lean, relying on lower-level coverage when UI is not essential.
|