simplecov-mcp 0.3.0 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/LICENSE +21 -0
- data/README.md +173 -356
- data/docs/ADVANCED_USAGE.md +967 -0
- data/docs/ARCHITECTURE.md +79 -0
- data/docs/BRANCH_ONLY_COVERAGE.md +81 -0
- data/docs/CLI_USAGE.md +637 -0
- data/docs/DEVELOPMENT.md +82 -0
- data/docs/ERROR_HANDLING.md +93 -0
- data/docs/EXAMPLES.md +430 -0
- data/docs/INSTALLATION.md +352 -0
- data/docs/LIBRARY_API.md +635 -0
- data/docs/MCP_INTEGRATION.md +488 -0
- data/docs/TROUBLESHOOTING.md +276 -0
- data/docs/arch-decisions/001-x-arch-decision.md +93 -0
- data/docs/arch-decisions/002-x-arch-decision.md +157 -0
- data/docs/arch-decisions/003-x-arch-decision.md +163 -0
- data/docs/arch-decisions/004-x-arch-decision.md +199 -0
- data/docs/arch-decisions/005-x-arch-decision.md +187 -0
- data/docs/arch-decisions/README.md +60 -0
- data/docs/presentations/simplecov-mcp-presentation.md +249 -0
- data/exe/simplecov-mcp +4 -4
- data/lib/simplecov_mcp/app_context.rb +26 -0
- data/lib/simplecov_mcp/base_tool.rb +74 -0
- data/lib/simplecov_mcp/cli.rb +234 -0
- data/lib/simplecov_mcp/cli_config.rb +56 -0
- data/lib/simplecov_mcp/commands/base_command.rb +78 -0
- data/lib/simplecov_mcp/commands/command_factory.rb +39 -0
- data/lib/simplecov_mcp/commands/detailed_command.rb +24 -0
- data/lib/simplecov_mcp/commands/list_command.rb +13 -0
- data/lib/simplecov_mcp/commands/raw_command.rb +22 -0
- data/lib/simplecov_mcp/commands/summary_command.rb +24 -0
- data/lib/simplecov_mcp/commands/uncovered_command.rb +26 -0
- data/lib/simplecov_mcp/commands/version_command.rb +18 -0
- data/lib/simplecov_mcp/constants.rb +22 -0
- data/lib/simplecov_mcp/error_handler.rb +124 -0
- data/lib/simplecov_mcp/error_handler_factory.rb +31 -0
- data/lib/simplecov_mcp/errors.rb +179 -0
- data/lib/simplecov_mcp/formatters/source_formatter.rb +148 -0
- data/lib/simplecov_mcp/mcp_server.rb +40 -0
- data/lib/simplecov_mcp/mode_detector.rb +55 -0
- data/lib/simplecov_mcp/model.rb +300 -0
- data/lib/simplecov_mcp/option_normalizers.rb +92 -0
- data/lib/simplecov_mcp/option_parser_builder.rb +134 -0
- data/lib/simplecov_mcp/option_parsers/env_options_parser.rb +50 -0
- data/lib/simplecov_mcp/option_parsers/error_helper.rb +109 -0
- data/lib/simplecov_mcp/path_relativizer.rb +61 -0
- data/lib/simplecov_mcp/presenters/base_coverage_presenter.rb +44 -0
- data/lib/simplecov_mcp/presenters/coverage_detailed_presenter.rb +16 -0
- data/lib/simplecov_mcp/presenters/coverage_raw_presenter.rb +16 -0
- data/lib/simplecov_mcp/presenters/coverage_summary_presenter.rb +16 -0
- data/lib/simplecov_mcp/presenters/coverage_uncovered_presenter.rb +16 -0
- data/lib/simplecov_mcp/presenters/project_coverage_presenter.rb +52 -0
- data/lib/simplecov_mcp/resolvers/coverage_line_resolver.rb +126 -0
- data/lib/simplecov_mcp/resolvers/resolver_factory.rb +28 -0
- data/lib/simplecov_mcp/resolvers/resultset_path_resolver.rb +78 -0
- data/lib/simplecov_mcp/resultset_loader.rb +136 -0
- data/lib/simplecov_mcp/staleness_checker.rb +243 -0
- data/lib/{simple_cov_mcp → simplecov_mcp}/tools/all_files_coverage_tool.rb +31 -13
- data/lib/{simple_cov_mcp → simplecov_mcp}/tools/coverage_detailed_tool.rb +7 -7
- data/lib/{simple_cov_mcp → simplecov_mcp}/tools/coverage_raw_tool.rb +7 -7
- data/lib/{simple_cov_mcp → simplecov_mcp}/tools/coverage_summary_tool.rb +7 -7
- data/lib/simplecov_mcp/tools/coverage_table_tool.rb +90 -0
- data/lib/{simple_cov_mcp → simplecov_mcp}/tools/help_tool.rb +13 -4
- data/lib/{simple_cov_mcp → simplecov_mcp}/tools/uncovered_lines_tool.rb +7 -7
- data/lib/{simple_cov_mcp → simplecov_mcp}/tools/version_tool.rb +11 -3
- data/lib/simplecov_mcp/util.rb +82 -0
- data/lib/{simple_cov_mcp → simplecov_mcp}/version.rb +1 -1
- data/lib/simplecov_mcp.rb +144 -2
- data/spec/MCP_INTEGRATION_TESTS_README.md +111 -0
- data/spec/TIMESTAMPS.md +48 -0
- data/spec/all_files_coverage_tool_spec.rb +29 -25
- data/spec/base_tool_spec.rb +11 -10
- data/spec/cli/show_default_report_spec.rb +33 -0
- data/spec/cli_config_spec.rb +137 -0
- data/spec/cli_enumerated_options_spec.rb +68 -0
- data/spec/cli_error_spec.rb +105 -47
- data/spec/cli_source_spec.rb +82 -23
- data/spec/cli_spec.rb +140 -5
- data/spec/cli_success_predicate_spec.rb +141 -0
- data/spec/cli_table_spec.rb +1 -1
- data/spec/cli_usage_spec.rb +10 -26
- data/spec/commands/base_command_spec.rb +187 -0
- data/spec/commands/command_factory_spec.rb +72 -0
- data/spec/commands/detailed_command_spec.rb +48 -0
- data/spec/commands/raw_command_spec.rb +46 -0
- data/spec/commands/summary_command_spec.rb +47 -0
- data/spec/commands/uncovered_command_spec.rb +49 -0
- data/spec/constants_spec.rb +61 -0
- data/spec/coverage_table_tool_spec.rb +17 -33
- data/spec/error_handler_spec.rb +22 -13
- data/spec/error_mode_spec.rb +143 -0
- data/spec/errors_edge_cases_spec.rb +239 -0
- data/spec/errors_stale_spec.rb +2 -2
- data/spec/file_based_mcp_tools_spec.rb +99 -0
- data/spec/fixtures/project1/lib/bar.rb +0 -1
- data/spec/fixtures/project1/lib/foo.rb +0 -1
- data/spec/help_tool_spec.rb +11 -17
- data/spec/integration_spec.rb +845 -0
- data/spec/logging_fallback_spec.rb +128 -0
- data/spec/mcp_logging_spec.rb +44 -0
- data/spec/mcp_server_integration_spec.rb +23 -0
- data/spec/mcp_server_spec.rb +15 -4
- data/spec/mode_detector_spec.rb +148 -0
- data/spec/model_error_handling_spec.rb +210 -0
- data/spec/model_staleness_spec.rb +40 -10
- data/spec/option_normalizers_spec.rb +204 -0
- data/spec/option_parsers/env_options_parser_spec.rb +233 -0
- data/spec/option_parsers/error_helper_spec.rb +222 -0
- data/spec/path_relativizer_spec.rb +83 -0
- data/spec/presenters/coverage_detailed_presenter_spec.rb +19 -0
- data/spec/presenters/coverage_raw_presenter_spec.rb +15 -0
- data/spec/presenters/coverage_summary_presenter_spec.rb +15 -0
- data/spec/presenters/coverage_uncovered_presenter_spec.rb +16 -0
- data/spec/presenters/project_coverage_presenter_spec.rb +86 -0
- data/spec/resolvers/coverage_line_resolver_spec.rb +57 -0
- data/spec/resolvers/resolver_factory_spec.rb +61 -0
- data/spec/resolvers/resultset_path_resolver_spec.rb +55 -0
- data/spec/resultset_loader_spec.rb +167 -0
- data/spec/shared_examples/README.md +115 -0
- data/spec/shared_examples/coverage_presenter_examples.rb +66 -0
- data/spec/shared_examples/file_based_mcp_tools.rb +174 -0
- data/spec/shared_examples/mcp_tool_text_json_response.rb +16 -0
- data/spec/simple_cov_mcp_module_spec.rb +16 -0
- data/spec/simplecov_mcp_model_spec.rb +340 -9
- data/spec/simplecov_mcp_opts_spec.rb +182 -0
- data/spec/spec_helper.rb +147 -4
- data/spec/staleness_checker_spec.rb +373 -0
- data/spec/staleness_more_spec.rb +16 -13
- data/spec/support/mcp_runner.rb +64 -0
- data/spec/tools_error_handling_spec.rb +144 -0
- data/spec/util_spec.rb +109 -34
- data/spec/version_spec.rb +117 -9
- data/spec/version_tool_spec.rb +131 -10
- metadata +120 -63
- data/lib/simple_cov/mcp.rb +0 -9
- data/lib/simple_cov_mcp/base_tool.rb +0 -70
- data/lib/simple_cov_mcp/cli.rb +0 -390
- data/lib/simple_cov_mcp/error_handler.rb +0 -131
- data/lib/simple_cov_mcp/error_handler_factory.rb +0 -38
- data/lib/simple_cov_mcp/errors.rb +0 -176
- data/lib/simple_cov_mcp/mcp_server.rb +0 -30
- data/lib/simple_cov_mcp/model.rb +0 -104
- data/lib/simple_cov_mcp/staleness_checker.rb +0 -125
- data/lib/simple_cov_mcp/tools/coverage_table_tool.rb +0 -61
- data/lib/simple_cov_mcp/util.rb +0 -122
- data/lib/simple_cov_mcp.rb +0 -102
- data/spec/coverage_detailed_tool_spec.rb +0 -36
- data/spec/coverage_raw_tool_spec.rb +0 -32
- data/spec/coverage_summary_tool_spec.rb +0 -39
- data/spec/legacy_shim_spec.rb +0 -13
- data/spec/uncovered_lines_tool_spec.rb +0 -33
@@ -0,0 +1,199 @@
|
|
1
|
+
# ADR 004: Ruby `instance_eval` for Success Predicates
|
2
|
+
|
3
|
+
## Status
|
4
|
+
|
5
|
+
Accepted
|
6
|
+
|
7
|
+
## Context
|
8
|
+
|
9
|
+
SimpleCov MCP needed a mechanism for users to define custom coverage policies beyond simple percentage thresholds. Different projects have different requirements:
|
10
|
+
|
11
|
+
- Some want all files above 80%, others allow a few files below threshold
|
12
|
+
- Some need different thresholds for different directories (e.g., 90% for API code, 60% for legacy)
|
13
|
+
- Some want total project coverage minimums
|
14
|
+
- CI/CD pipelines need exit codes based on policy compliance
|
15
|
+
|
16
|
+
We considered several approaches:
|
17
|
+
|
18
|
+
1. **Built-in policy DSL**: Define a limited language for expressing policies (e.g., YAML/JSON config)
|
19
|
+
2. **Plugin architecture**: Define a protocol/interface, require users to create Ruby classes implementing it
|
20
|
+
3. **Ruby file evaluation**: Load and execute arbitrary Ruby code that returns a callable predicate
|
21
|
+
4. **Sandboxed DSL**: Use a restricted Ruby environment (e.g., `$SAFE` levels, isolated VMs)
|
22
|
+
|
23
|
+
### Key Requirements
|
24
|
+
|
25
|
+
- Flexibility: Support arbitrarily complex coverage policies
|
26
|
+
- Simplicity: Easy for users to write and understand
|
27
|
+
- Debuggability: Users can use standard Ruby debugging tools
|
28
|
+
- CI/CD integration: Clear exit codes (0 = pass, 1 = fail, 2 = error)
|
29
|
+
- Access to coverage data: Predicates need access to the full `CoverageModel` API
|
30
|
+
|
31
|
+
### Why Not a Custom DSL?
|
32
|
+
|
33
|
+
A custom DSL would be:
|
34
|
+
- Limited in expressiveness (hard to predict all future use cases)
|
35
|
+
- Harder to debug (users can't use standard Ruby tools)
|
36
|
+
- More maintenance burden (parsing, validation, documentation)
|
37
|
+
- Still vulnerable to injection if it allowed any dynamic computation
|
38
|
+
|
39
|
+
### Why Not Sandboxing?
|
40
|
+
|
41
|
+
Ruby's sandboxing options are limited:
|
42
|
+
- `$SAFE` levels were deprecated and removed in Ruby 2.7+
|
43
|
+
- Full VM isolation (Docker, etc.) is too heavy for a CLI tool
|
44
|
+
- Any Turing-complete sandbox can be escaped given enough effort
|
45
|
+
- True security requires not executing untrusted code at all
|
46
|
+
|
47
|
+
## Decision
|
48
|
+
|
49
|
+
We chose to **evaluate Ruby files using `instance_eval`** with prominent security warnings rather than attempting to create a false sense of security through incomplete sandboxing.
|
50
|
+
|
51
|
+
### Implementation
|
52
|
+
|
53
|
+
The implementation is in `lib/simplecov_mcp/cli.rb:191-214`:
|
54
|
+
|
55
|
+
```ruby
|
56
|
+
def load_success_predicate(path)
|
57
|
+
unless File.exist?(path)
|
58
|
+
raise "Success predicate file not found: #{path}"
|
59
|
+
end
|
60
|
+
|
61
|
+
content = File.read(path)
|
62
|
+
|
63
|
+
# WARNING: The predicate code executes with full Ruby privileges.
|
64
|
+
# It has unrestricted access to the file system, network, and system commands.
|
65
|
+
# Only use predicate files from trusted sources.
|
66
|
+
#
|
67
|
+
# We evaluate in a fresh Object context to prevent accidental access to
|
68
|
+
# CLI internals, but this provides NO security isolation.
|
69
|
+
evaluation_context = Object.new
|
70
|
+
predicate = evaluation_context.instance_eval(content, path, 1)
|
71
|
+
|
72
|
+
unless predicate.respond_to?(:call)
|
73
|
+
raise "Success predicate must be callable (lambda, proc, or object with #call method)"
|
74
|
+
end
|
75
|
+
|
76
|
+
predicate
|
77
|
+
rescue SyntaxError => e
|
78
|
+
raise "Syntax error in success predicate file: #{e.message}"
|
79
|
+
end
|
80
|
+
```
|
81
|
+
|
82
|
+
The predicate is then called with a `CoverageModel` instance:
|
83
|
+
|
84
|
+
```ruby
|
85
|
+
def run_success_predicate
|
86
|
+
predicate = load_success_predicate(config.success_predicate)
|
87
|
+
model = CoverageModel.new(**config.model_options)
|
88
|
+
|
89
|
+
result = predicate.call(model)
|
90
|
+
exit(result ? 0 : 1) # 0 = success, 1 = failure
|
91
|
+
rescue => e
|
92
|
+
warn "Success predicate error: #{e.message}"
|
93
|
+
warn e.backtrace.first(5).join("\n") if config.error_mode == :trace
|
94
|
+
exit 2 # Exit code 2 for predicate errors
|
95
|
+
end
|
96
|
+
```
|
97
|
+
|
98
|
+
### Security Model: Treat as Executable Code
|
99
|
+
|
100
|
+
Rather than pretending to sandbox untrusted code, we treat success predicates **exactly like any other Ruby code in the project**:
|
101
|
+
|
102
|
+
1. **Prominent warnings** in documentation (examples/success_predicates/README.md:5-17):
|
103
|
+
```
|
104
|
+
⚠️ SECURITY WARNING
|
105
|
+
|
106
|
+
Success predicates execute as arbitrary Ruby code with full system privileges.
|
107
|
+
Only use predicate files from trusted sources.
|
108
|
+
- Never use predicates from untrusted or unknown sources
|
109
|
+
- Review predicates before use, especially in CI/CD environments
|
110
|
+
- Store predicates in version control with code review
|
111
|
+
```
|
112
|
+
|
113
|
+
2. **Code review workflow**: Predicates live in version control alongside tests
|
114
|
+
3. **CI/CD best practices**: Same permissions model as running tests themselves
|
115
|
+
4. **Example predicates**: Well-documented examples showing safe patterns
|
116
|
+
|
117
|
+
### Predicate API
|
118
|
+
|
119
|
+
Success predicates must be callable (lambda, proc, or object with `#call` method):
|
120
|
+
|
121
|
+
**Lambda example:**
|
122
|
+
```ruby
|
123
|
+
->(model) do
|
124
|
+
model.all_files.all? { |f| f['percentage'] >= 80 }
|
125
|
+
end
|
126
|
+
```
|
127
|
+
|
128
|
+
**Class example:**
|
129
|
+
```ruby
|
130
|
+
class CoveragePolicy
|
131
|
+
def call(model)
|
132
|
+
api_files = model.all_files.select { |f| f['file'].start_with?('lib/api/') }
|
133
|
+
api_files.all? { |f| f['percentage'] >= 90 }
|
134
|
+
end
|
135
|
+
end
|
136
|
+
|
137
|
+
CoveragePolicy.new
|
138
|
+
```
|
139
|
+
|
140
|
+
The predicate receives a full `CoverageModel` instance with access to:
|
141
|
+
- `all_files(tracked_globs:, sort_order:)` - All file coverage data
|
142
|
+
- `summary_for(path)` - Coverage summary for a specific file
|
143
|
+
- `uncovered_for(path)` - Uncovered lines for a specific file
|
144
|
+
- `detailed_for(path)` - Per-line coverage data
|
145
|
+
|
146
|
+
## Consequences
|
147
|
+
|
148
|
+
### Positive
|
149
|
+
|
150
|
+
1. **Maximum flexibility**: Users can express arbitrarily complex coverage policies using full Ruby
|
151
|
+
2. **Familiar tooling**: Users can debug predicates with standard Ruby tools (pry, byebug, etc.)
|
152
|
+
3. **Simplicity**: No custom DSL to learn, document, or maintain
|
153
|
+
4. **Honesty**: Security model is clear and doesn't provide false confidence
|
154
|
+
5. **Composability**: Users can require other libraries, define helper methods, etc.
|
155
|
+
6. **Excellent examples**: We provide 5+ well-documented example predicates
|
156
|
+
|
157
|
+
### Negative
|
158
|
+
|
159
|
+
1. **Security responsibility**: Users must understand the security implications
|
160
|
+
2. **Potential misuse**: Users might mistakenly trust untrusted predicate files
|
161
|
+
3. **No isolation**: Buggy predicates can access/modify anything in the system
|
162
|
+
4. **Documentation burden**: Must clearly communicate security model
|
163
|
+
|
164
|
+
### Trade-offs
|
165
|
+
|
166
|
+
- **Versus custom DSL**: More powerful and debuggable, but requires user awareness of security
|
167
|
+
- **Versus plugin architecture**: Simpler (no gem dependencies, no protocol to learn), but same security profile
|
168
|
+
- **Versus incomplete sandboxing**: Honest about capabilities rather than security theater
|
169
|
+
|
170
|
+
### Threat Model
|
171
|
+
|
172
|
+
This approach is **appropriate** when:
|
173
|
+
- Predicate files are stored in version control with code review
|
174
|
+
- Users treat predicates like any other code in their project (tests, Rakefile, etc.)
|
175
|
+
- CI/CD environments already execute arbitrary code (tests, build scripts)
|
176
|
+
|
177
|
+
This approach is **inappropriate** when:
|
178
|
+
- Processing untrusted predicate files from unknown sources
|
179
|
+
- Allowing users to upload predicates via web interface
|
180
|
+
- Running in a multi-tenant environment without isolation
|
181
|
+
|
182
|
+
### Future Considerations
|
183
|
+
|
184
|
+
If demand arises for truly untrusted predicate execution, alternatives include:
|
185
|
+
|
186
|
+
1. **JSON-based policy format**: Limited expressiveness but safe
|
187
|
+
2. **WebAssembly sandbox**: Execute policies in an isolated WASM runtime
|
188
|
+
3. **External process**: Run predicates in separate process with restricted permissions
|
189
|
+
|
190
|
+
However, for the primary use case (CI/CD policy enforcement), the current approach is simpler and more flexible than these alternatives.
|
191
|
+
|
192
|
+
## References
|
193
|
+
|
194
|
+
- Implementation: `lib/simplecov_mcp/cli.rb:191-214` (load predicate), `lib/simplecov_mcp/cli.rb:179-189` (execute)
|
195
|
+
- Security warnings: `examples/success_predicates/README.md:5-17`
|
196
|
+
- Example predicates: `examples/success_predicates/*.rb`
|
197
|
+
- CoverageModel API: `lib/simplecov_mcp/model.rb`
|
198
|
+
- CLI config: `lib/simplecov_mcp/cli_config.rb:18` (success_predicate field)
|
199
|
+
- Option parsing: `lib/simplecov_mcp/option_parser_builder.rb` (--success-predicate flag)
|
@@ -0,0 +1,187 @@
|
|
1
|
+
# ADR 005: No SimpleCov Runtime Dependency
|
2
|
+
|
3
|
+
## Status
|
4
|
+
|
5
|
+
Replaced – simplecov-mcp now requires SimpleCov at runtime so that multi-suite resultsets can be merged using SimpleCov’s combine helpers.
|
6
|
+
|
7
|
+
## Context
|
8
|
+
|
9
|
+
SimpleCov MCP provides tooling for inspecting SimpleCov coverage reports. When designing the gem, we had to decide whether to depend on SimpleCov as a runtime dependency.
|
10
|
+
|
11
|
+
### Alternative Approaches
|
12
|
+
|
13
|
+
1. **Runtime dependency on SimpleCov**: Use SimpleCov's API to read and process coverage data
|
14
|
+
2. **Development-only dependency**: Read SimpleCov's `.resultset.json` files directly without requiring SimpleCov at runtime
|
15
|
+
3. **Support multiple coverage formats**: Parse coverage data from multiple tools (SimpleCov, Coverage, etc.)
|
16
|
+
|
17
|
+
### Key Considerations
|
18
|
+
|
19
|
+
**Dependency weight**: SimpleCov itself has dependencies:
|
20
|
+
- `docile` (~> 1.1)
|
21
|
+
- `simplecov-html` (~> 0.11)
|
22
|
+
- `simplecov_json_formatter` (~> 0.1)
|
23
|
+
|
24
|
+
**Use case separation**:
|
25
|
+
- SimpleCov is needed when **running tests** to collect coverage
|
26
|
+
- SimpleCov MCP is needed when **inspecting coverage** after tests complete
|
27
|
+
- These are temporally separated activities
|
28
|
+
|
29
|
+
**Deployment contexts**:
|
30
|
+
- CI/CD: Coverage collection happens in test job, inspection might happen in a separate analysis job
|
31
|
+
- Production: Some teams want to analyze coverage data without installing test dependencies
|
32
|
+
- Developer machines: May want to inspect coverage without full test suite dependencies
|
33
|
+
|
34
|
+
**Format stability**:
|
35
|
+
- SimpleCov's `.resultset.json` format is stable and well-documented
|
36
|
+
- The format is simple JSON with predictable structure
|
37
|
+
- Breaking changes would affect all SimpleCov users, so the format is unlikely to change
|
38
|
+
|
39
|
+
## Decision
|
40
|
+
|
41
|
+
We chose to **make SimpleCov a development dependency only** and read `.resultset.json` files directly using Ruby's standard library JSON parser.
|
42
|
+
|
43
|
+
### Implementation
|
44
|
+
|
45
|
+
The gem only depends on `mcp` at runtime (simplecov-mcp.gemspec:26):
|
46
|
+
```ruby
|
47
|
+
# Runtime deps (stdlib: json, time, pathname)
|
48
|
+
spec.add_runtime_dependency 'mcp', '~> 0.3'
|
49
|
+
spec.add_development_dependency 'simplecov', '~> 0.21'
|
50
|
+
```
|
51
|
+
|
52
|
+
Coverage data is read directly from JSON files (lib/simplecov_mcp/model.rb:34-42):
|
53
|
+
```ruby
|
54
|
+
rs = CovUtil.find_resultset(@root, resultset: resultset)
|
55
|
+
raw = JSON.parse(File.read(rs))
|
56
|
+
# SimpleCov typically writes a single test suite entry to .resultset.json
|
57
|
+
# Find the first entry that has coverage data (skip comment entries)
|
58
|
+
_suite, data = raw.find { |k, v| v.is_a?(Hash) && v.key?('coverage') }
|
59
|
+
raise "No test suite with coverage data found in resultset file: #{rs}" unless data
|
60
|
+
cov = data['coverage'] or raise "No 'coverage' key found in resultset file: #{rs}"
|
61
|
+
@cov = cov.transform_keys { |k| File.absolute_path(k, @root) }
|
62
|
+
@cov_timestamp = (data['timestamp'] || data['created_at'] || 0).to_i
|
63
|
+
```
|
64
|
+
|
65
|
+
Coverage calculations use simple algorithms (lib/simplecov_mcp/util.rb:42-71):
|
66
|
+
```ruby
|
67
|
+
def summary(arr)
|
68
|
+
total = 0
|
69
|
+
covered = 0
|
70
|
+
arr.compact.each do |hits|
|
71
|
+
total += 1
|
72
|
+
covered += 1 if hits.to_i > 0
|
73
|
+
end
|
74
|
+
pct = total.zero? ? 100.0 : ((covered.to_f * 100.0 / total) * 100).round / 100.0
|
75
|
+
{ 'covered' => covered, 'total' => total, 'pct' => pct }
|
76
|
+
end
|
77
|
+
|
78
|
+
def uncovered(arr)
|
79
|
+
out = []
|
80
|
+
arr.each_with_index do |hits, i|
|
81
|
+
next if hits.nil?
|
82
|
+
out << (i + 1) if hits.to_i.zero?
|
83
|
+
end
|
84
|
+
out
|
85
|
+
end
|
86
|
+
|
87
|
+
def detailed(arr)
|
88
|
+
rows = []
|
89
|
+
arr.each_with_index do |hits, i|
|
90
|
+
h = hits&.to_i
|
91
|
+
rows << { 'line' => i + 1, 'hits' => h, 'covered' => h.positive? } if h
|
92
|
+
end
|
93
|
+
rows
|
94
|
+
end
|
95
|
+
```
|
96
|
+
|
97
|
+
### SimpleCov .resultset.json Format
|
98
|
+
|
99
|
+
The format we parse has this structure:
|
100
|
+
```json
|
101
|
+
{
|
102
|
+
"RSpec": {
|
103
|
+
"coverage": {
|
104
|
+
"/absolute/path/to/file.rb": {
|
105
|
+
"lines": [null, 1, 3, 0, null, 5, ...]
|
106
|
+
}
|
107
|
+
},
|
108
|
+
"timestamp": 1633072800
|
109
|
+
}
|
110
|
+
}
|
111
|
+
```
|
112
|
+
|
113
|
+
Where:
|
114
|
+
- Top level keys are test suite names (e.g., "RSpec", "Minitest")
|
115
|
+
- `coverage` contains file paths mapped to coverage data
|
116
|
+
- `lines` is an array where each index represents a line number (0-indexed)
|
117
|
+
- Array values: `null` = not executable, `0` = not covered, `>0` = hit count
|
118
|
+
- `timestamp` is Unix timestamp when coverage was collected
|
119
|
+
|
120
|
+
### Resultset Discovery
|
121
|
+
|
122
|
+
We implement flexible discovery of `.resultset.json` files (lib/simplecov_mcp/util.rb:6-10):
|
123
|
+
```ruby
|
124
|
+
RESULTSET_CANDIDATES = [
|
125
|
+
'.resultset.json',
|
126
|
+
'coverage/.resultset.json',
|
127
|
+
'tmp/.resultset.json'
|
128
|
+
].freeze
|
129
|
+
```
|
130
|
+
|
131
|
+
This supports common SimpleCov configurations without requiring SimpleCov to be loaded.
|
132
|
+
|
133
|
+
## Consequences
|
134
|
+
|
135
|
+
### Positive
|
136
|
+
|
137
|
+
1. **Lightweight installation**: No transitive dependencies beyond `mcp` gem
|
138
|
+
2. **Deployment flexibility**: Can analyze coverage in environments without test dependencies
|
139
|
+
3. **Faster installation**: Fewer gems to download and install
|
140
|
+
4. **Clear separation of concerns**: Coverage collection vs. coverage analysis are independent
|
141
|
+
5. **CI/CD optimization**: Analysis jobs don't need full test suite dependencies
|
142
|
+
6. **Production-safe**: Can be deployed to production environments if needed (e.g., for monitoring)
|
143
|
+
|
144
|
+
### Negative
|
145
|
+
|
146
|
+
1. **Format dependency**: Tightly coupled to SimpleCov's JSON format
|
147
|
+
2. **Breaking changes risk**: If SimpleCov changes `.resultset.json` structure, we must adapt
|
148
|
+
3. **Limited to SimpleCov**: Cannot read coverage data from other Ruby coverage tools
|
149
|
+
4. **Duplicate logic**: Coverage percentage calculations reimplemented (though simple)
|
150
|
+
5. **Maintenance**: Must track SimpleCov format changes manually
|
151
|
+
|
152
|
+
### Trade-offs
|
153
|
+
|
154
|
+
- **Versus runtime dependency**: Lighter weight but less resilient to format changes
|
155
|
+
- **Versus multi-format support**: Simpler implementation but locked to SimpleCov ecosystem
|
156
|
+
- **Versus using SimpleCov API**: More flexible deployment but requires understanding the file format
|
157
|
+
|
158
|
+
### Risk Mitigation
|
159
|
+
|
160
|
+
1. **Format stability**: SimpleCov has maintained `.resultset.json` compatibility for years
|
161
|
+
2. **Simple format**: JSON structure is straightforward and unlikely to change dramatically
|
162
|
+
3. **Development dependency**: We still use SimpleCov in our own tests, so format changes would be detected immediately
|
163
|
+
4. **Documentation**: CLAUDE.md documents the format dependency explicitly
|
164
|
+
5. **Error handling**: Robust error messages when format doesn't match expectations
|
165
|
+
|
166
|
+
### Format Evolution Strategy
|
167
|
+
|
168
|
+
If SimpleCov's format changes:
|
169
|
+
1. **Minor additions** (new keys): Ignore unknown keys, only parse what we need
|
170
|
+
2. **Breaking changes** (structure changes): Version detection logic to support multiple formats
|
171
|
+
3. **Alternative formats**: Could add support for other coverage tools' JSON formats if needed
|
172
|
+
|
173
|
+
### Current Limitations Accepted
|
174
|
+
|
175
|
+
- Only supports SimpleCov (not Coverage gem, other tools)
|
176
|
+
- Assumes standard `.resultset.json` locations
|
177
|
+
- No support for merged coverage from multiple test runs (SimpleCov handles this before writing JSON)
|
178
|
+
- No support for branch coverage (SimpleCov feature not widely used yet)
|
179
|
+
|
180
|
+
## References
|
181
|
+
|
182
|
+
- Gemspec dependencies: `simplecov-mcp.gemspec:26-28`
|
183
|
+
- JSON parsing: `lib/simplecov_mcp/model.rb:34-57`
|
184
|
+
- Coverage calculations: `lib/simplecov_mcp/util.rb:42-71`
|
185
|
+
- Resultset discovery: `lib/simplecov_mcp/util.rb:6-10`, `lib/simplecov_mcp/util.rb:34-36`
|
186
|
+
- SimpleCov format documentation: https://github.com/simplecov-ruby/simplecov
|
187
|
+
- Development usage: Uses SimpleCov in `spec/spec_helper.rb` to test itself
|
@@ -0,0 +1,60 @@
|
|
1
|
+
# Architecture Decision Records
|
2
|
+
|
3
|
+
This directory contains Architecture Decision Records (ADRs) for the SimpleCov MCP project.
|
4
|
+
|
5
|
+
## What is an ADR?
|
6
|
+
|
7
|
+
An Architecture Decision Record (ADR) captures a significant architectural decision made during the development of this project, along with its context and consequences.
|
8
|
+
|
9
|
+
## ADR Format
|
10
|
+
|
11
|
+
Each ADR follows this structure:
|
12
|
+
|
13
|
+
### Title
|
14
|
+
A short phrase describing the decision (e.g., "Dual-Mode Operation: CLI and MCP Server")
|
15
|
+
|
16
|
+
### Status
|
17
|
+
- **Accepted**: The decision has been made and is currently in effect
|
18
|
+
- **Proposed**: Under consideration
|
19
|
+
- **Deprecated**: No longer applicable
|
20
|
+
- **Superseded**: Replaced by a newer decision
|
21
|
+
|
22
|
+
### Context
|
23
|
+
The background, problem statement, and constraints that led to this decision. This section answers "Why did we need to make a decision here?"
|
24
|
+
|
25
|
+
### Decision
|
26
|
+
The architectural decision that was made. This section answers "What did we decide to do?"
|
27
|
+
|
28
|
+
### Consequences
|
29
|
+
The implications of this decision, both positive and negative. This includes:
|
30
|
+
- Benefits gained
|
31
|
+
- Trade-offs accepted
|
32
|
+
- Complexity introduced
|
33
|
+
- Future constraints
|
34
|
+
|
35
|
+
### References
|
36
|
+
Links to related code, issues, documentation, or other ADRs.
|
37
|
+
|
38
|
+
## Index of ADRs
|
39
|
+
|
40
|
+
- [001: Dual-Mode Operation](001-x-arch-decision.md) - CLI vs MCP server mode detection
|
41
|
+
- [002: Context-Aware Error Handling](002-x-arch-decision.md) - Mode-specific error handling strategy
|
42
|
+
- [003: Coverage Staleness Detection](003-x-arch-decision.md) - Three-type staleness system
|
43
|
+
- [004: Ruby Instance Eval for Success Predicates](004-x-arch-decision.md) - Dynamic Ruby evaluation approach
|
44
|
+
- [005: No SimpleCov Runtime Dependency](005-x-arch-decision.md) - Superseded by the multi-suite merge work (runtime SimpleCov dependency)
|
45
|
+
|
46
|
+
## When to Create an ADR
|
47
|
+
|
48
|
+
Create an ADR when:
|
49
|
+
- Making a decision that affects the structure or behavior of the system
|
50
|
+
- Choosing between multiple viable approaches
|
51
|
+
- Accepting significant trade-offs
|
52
|
+
- Making decisions that future maintainers should understand
|
53
|
+
|
54
|
+
## Contributing
|
55
|
+
|
56
|
+
When adding a new ADR:
|
57
|
+
1. Use the next sequential number (e.g., `006-x-arch-decision.md`)
|
58
|
+
2. Follow the format outlined above
|
59
|
+
3. Update this README's index
|
60
|
+
4. Link to relevant code and documentation
|
@@ -0,0 +1,249 @@
|
|
1
|
+
---
|
2
|
+
marp: true
|
3
|
+
theme: default
|
4
|
+
class: lead
|
5
|
+
paginate: true
|
6
|
+
backgroundColor: #fff
|
7
|
+
color: #333
|
8
|
+
---
|
9
|
+
|
10
|
+
# SimpleCovMCP
|
11
|
+
### MCP Server, CLI, and Library for SimpleCov Ruby Test Coverage
|
12
|
+
|
13
|
+
- Keith Bennett
|
14
|
+
- First presented to PhRUG (Philippines Ruby User Group), 2025-10-01
|
15
|
+
|
16
|
+
---
|
17
|
+
|
18
|
+
## What is SimpleCov MCP?
|
19
|
+
|
20
|
+
A **three-in-one** gem that makes SimpleCov coverage data accessible to:
|
21
|
+
|
22
|
+
- 🤖 **AI agents** via Model Context Protocol (MCP)
|
23
|
+
- 💻 **Command line** via its command line interface
|
24
|
+
- 📚 **Ruby scripts and applications** as a library
|
25
|
+
|
26
|
+
**Lazy dependency** on SimpleCov - single-suite resultsets avoid loading it; multi-suite files trigger a merge via SimpleCov’s combine helpers.
|
27
|
+
|
28
|
+
What is it *not*? It is not a replacement for SimpleCov's generated web presentation of the coverage data.
|
29
|
+
|
30
|
+
|
31
|
+
This code base requires a Ruby version >= 3.2.0, because this is required by the mcp gem it uses.
|
32
|
+
|
33
|
+
---
|
34
|
+
|
35
|
+
## High Level Objectives
|
36
|
+
|
37
|
+
- Query coverage programmatically
|
38
|
+
- Integrate with AI tools
|
39
|
+
- Automate coverage analysis
|
40
|
+
- Focus on specific files/patterns
|
41
|
+
|
42
|
+
---
|
43
|
+
|
44
|
+
## Key Features
|
45
|
+
|
46
|
+
- **Lazy SimpleCov dependency** - only loaded when multi-suite resultsets need merging
|
47
|
+
- **Flexible resultset location** - via CLI flags, passed parameter, or env var
|
48
|
+
- **Staleness detection** - warns or optionally errors when files newer than coverage
|
49
|
+
- **JSON output** - perfect for jq, scripts, CI/CD
|
50
|
+
- **Source code integration** - show uncovered lines with or without context
|
51
|
+
- **Colored output** - readable terminal display
|
52
|
+
|
53
|
+
---
|
54
|
+
|
55
|
+
## Demo 1: MCP Server Mode
|
56
|
+
### AI Coverage Assistant
|
57
|
+
|
58
|
+
```bash
|
59
|
+
# Test the MCP server manually
|
60
|
+
echo '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"coverage_summary_tool","arguments":{"path":"lib/simplecov_mcp/model.rb"}}}' | simplecov-mcp
|
61
|
+
```
|
62
|
+
|
63
|
+
**What AI agents can do:**
|
64
|
+
- Analyze coverage gaps
|
65
|
+
- Suggest testing priorities
|
66
|
+
- Generate ad-hoc coverage reports
|
67
|
+
|
68
|
+
---
|
69
|
+
|
70
|
+
## MCP Tools (Functions) Available
|
71
|
+
|
72
|
+
| Tool | Purpose |
|
73
|
+
|---------------------------|---------|
|
74
|
+
| `all_files_coverage_tool` | Project-wide coverage data |
|
75
|
+
| `coverage_detailed_tool` | Per-line hit counts |
|
76
|
+
| `coverage_summary_tool` | Get coverage % for a file |
|
77
|
+
| `coverage_table_tool` | Formatted coverage table |
|
78
|
+
| `uncovered_lines_tool` | Find missing test coverage |
|
79
|
+
|
80
|
+
| Tool | Purpose | Example Command |
|
81
|
+
|-------------------------|----------------------------|-----------------------------------------------------|
|
82
|
+
| all_files_coverage_tool | Project-wide coverage data | simplecov-mcp all-files |
|
83
|
+
| coverage_detailed_tool | Per-line hit counts | simplecov-mcp detailed lib/simplecov_mcp/model.rb |
|
84
|
+
| coverage_raw_tool | Raw SimpleCov lines array | simplecov-mcp raw lib/simplecov_mcp/model.rb |
|
85
|
+
| coverage_summary_tool | Get coverage % for a file | simplecov-mcp summary lib/simplecov_mcp/model.rb |
|
86
|
+
| coverage_table_tool | Formatted coverage table | simplecov-mcp table |
|
87
|
+
| help_tool | Tool usage guidance | simplecov-mcp help |
|
88
|
+
| uncovered_lines_tool | Find missing test coverage | simplecov-mcp uncovered lib/simplecov_mcp/model.rb |
|
89
|
+
| version_tool | Display version info | simplecov-mcp version |
|
90
|
+
|
91
|
+
---
|
92
|
+
|
93
|
+
## Demo 2: CLI Tool
|
94
|
+
###
|
95
|
+
|
96
|
+
```bash
|
97
|
+
# Show all files, worst coverage first
|
98
|
+
simplecov-mcp
|
99
|
+
|
100
|
+
# Focus on a specific file
|
101
|
+
simplecov-mcp summary lib/simplecov_mcp/cli.rb
|
102
|
+
|
103
|
+
# Find untested lines with source context
|
104
|
+
simplecov-mcp uncovered lib/simplecov_mcp/cli.rb --source=uncovered --source-context 3
|
105
|
+
|
106
|
+
# JSON for scripts
|
107
|
+
simplecov-mcp --json | jq '.files[] | select(.percentage < 80)'
|
108
|
+
```
|
109
|
+
|
110
|
+
---
|
111
|
+
|
112
|
+
## Demo 2: CLI Tool (cont'd.)
|
113
|
+
|
114
|
+
```bash
|
115
|
+
# Custom resultset location
|
116
|
+
simplecov-mcp --resultset coverage-all/
|
117
|
+
|
118
|
+
# Sort by highest coverage
|
119
|
+
simplecov-mcp --sort-order d
|
120
|
+
|
121
|
+
# Staleness checking (file newer than coverage?)
|
122
|
+
simplecov-mcp --stale error
|
123
|
+
|
124
|
+
# Track new files missing from coverage
|
125
|
+
simplecov-mcp --tracked-globs "lib/**/tools/*.rb"
|
126
|
+
```
|
127
|
+
|
128
|
+
---
|
129
|
+
|
130
|
+
## Demo 3: Ruby Library
|
131
|
+
### Programmatic Integration
|
132
|
+
|
133
|
+
```ruby
|
134
|
+
require 'simplecov_mcp'
|
135
|
+
|
136
|
+
model = SimpleCovMcp::CoverageModel.new
|
137
|
+
|
138
|
+
# Get project overview
|
139
|
+
files = model.all_files
|
140
|
+
puts "Lowest coverage: #{files.first['percentage']}%"
|
141
|
+
|
142
|
+
# Focus on specific concerns
|
143
|
+
uncovered = model.uncovered_for("lib/wifi-wand/models/ubuntu_model.rb")
|
144
|
+
puts "Uncovered hash's keys: #{uncovered.keys.inspect}"
|
145
|
+
puts "Missing lines: #{uncovered['uncovered'].inspect}"
|
146
|
+
|
147
|
+
# Output:
|
148
|
+
# Lowest coverage: 17.0%
|
149
|
+
# Uncovered hash's keys: ["file", "uncovered", "summary"]
|
150
|
+
# Missing lines: [13, 17, 21,...200, 203]
|
151
|
+
```
|
152
|
+
|
153
|
+
---
|
154
|
+
|
155
|
+
## Custom Threshold Git Pre-Commit Hook
|
156
|
+
|
157
|
+
```ruby
|
158
|
+
require_relative 'lib/simple_cov_mcp'
|
159
|
+
|
160
|
+
files = SimpleCovMcp::CoverageModel.new.all_files
|
161
|
+
critical, other = files.partition { |f| f['file'].include?('/lib/critical/') }
|
162
|
+
|
163
|
+
fails = critical.select { |f| f['percentage'] < 100.0 } +
|
164
|
+
other.select { |f| f['percentage'] < 90.0 }
|
165
|
+
|
166
|
+
if fails.any?
|
167
|
+
puts "❌ Coverage failures:"
|
168
|
+
fails.each { |f| puts " #{f['file']}: #{f['percentage']}%" }
|
169
|
+
exit 1
|
170
|
+
else
|
171
|
+
puts "✅ All thresholds met!"
|
172
|
+
end
|
173
|
+
```
|
174
|
+
|
175
|
+
---
|
176
|
+
|
177
|
+
## Architecture Overview
|
178
|
+
|
179
|
+
```
|
180
|
+
lib/simple_cov_mcp
|
181
|
+
├── base_tool.rb
|
182
|
+
├── cli.rb
|
183
|
+
├── error_handler_factory.rb
|
184
|
+
├── error_handler.rb
|
185
|
+
├── errors.rb
|
186
|
+
├── mcp_server.rb
|
187
|
+
├── model.rb
|
188
|
+
├── path_relativizer.rb
|
189
|
+
├── staleness_checker.rb
|
190
|
+
├── tools
|
191
|
+
│ ├── all_files_coverage_tool.rb
|
192
|
+
│ ├── coverage_detailed_tool.rb
|
193
|
+
│ ├── coverage_raw_tool.rb
|
194
|
+
│ ├── coverage_summary_tool.rb
|
195
|
+
│ ├── coverage_table_tool.rb
|
196
|
+
│ ├── help_tool.rb
|
197
|
+
│ ├── uncovered_lines_tool.rb
|
198
|
+
│ └── version_tool.rb
|
199
|
+
├── util.rb
|
200
|
+
└── version.rb
|
201
|
+
```
|
202
|
+
|
203
|
+
**Clean separation:** CLI ↔ Model ↔ MCP Tools
|
204
|
+
|
205
|
+
---
|
206
|
+
|
207
|
+
## MCP Plumbing - the MCP Gem
|
208
|
+
|
209
|
+
#### BaseTool subclasses the `mcp` gem's Tool class and defines a schema (see base_tool.rb):
|
210
|
+
```ruby
|
211
|
+
class BaseTool < ::MCP::Tool
|
212
|
+
# ...
|
213
|
+
end
|
214
|
+
```
|
215
|
+
|
216
|
+
* [BaseTool source](https://github.com/keithrbennett/simplecov-mcp/blob/main/lib/simplecov_mcp/base_tool.rb)
|
217
|
+
* [BaseTool subclass source](https://github.com/keithrbennett/simplecov-mcp/blob/main/lib/simplecov_mcp/tools/coverage_detailed_tool.rb)
|
218
|
+
|
219
|
+
The MCP tools available to the model subclass BaseTool and implement their respective tasks.
|
220
|
+
|
221
|
+
#### mcp_server.rb creates an instance of the mcp gem's Server class and runs it:
|
222
|
+
|
223
|
+
```ruby
|
224
|
+
server = ::MCP::Server.new(
|
225
|
+
name: 'simplecov-mcp',
|
226
|
+
version: SimpleCovMcp::VERSION,
|
227
|
+
tools: tools
|
228
|
+
)
|
229
|
+
::MCP::Server::Transports::StdioTransport.new(server).open
|
230
|
+
```
|
231
|
+
|
232
|
+
|
233
|
+
----
|
234
|
+
|
235
|
+
## Questions?
|
236
|
+
|
237
|
+
**Demo requests:**
|
238
|
+
- Specific MCP tool usage?
|
239
|
+
- CLI workflow examples?
|
240
|
+
- Library integration patterns?
|
241
|
+
- AI assistant setup?
|
242
|
+
|
243
|
+
**Contact:**
|
244
|
+
- GitHub issues for bugs/features
|
245
|
+
- Ruby community discussions
|
246
|
+
|
247
|
+
**Thank you!** 🙏
|
248
|
+
|
249
|
+
*Making test coverage accessible to humans and AI alike*
|