minitest-reporters-llm 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/LICENSE.txt +21 -0
- data/README.md +124 -0
- data/Rakefile +12 -0
- data/bin/console +15 -0
- data/bin/setup +8 -0
- data/lib/minitest/reporters/llm/version.rb +9 -0
- data/lib/minitest/reporters/llm.rb +4 -0
- data/lib/minitest/reporters/llm_reporter.rb +382 -0
- data/minitest-reporters-llm.gemspec +41 -0
- metadata +113 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 54abd03bb156c3695843a4eabbd357ca78fb62e7a2aa3583cfdaae6a3b230e7f
|
4
|
+
data.tar.gz: f0f26ab22f2849a264ba40fec91f2f505b2f0f41c38b1365363624e11d66c3f8
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 1e588f8816062ac07f38d1f7ef560861dbc43501f072f4579aeebc5e15f0516ecc4eef0183e7cc240e9f7583030fe61679ed23554fdcd37ddb2aabbf96256472
|
7
|
+
data.tar.gz: 57567c6ac21069a1146cd3200ce7dc2973783d493e0932422844478fb069b5e8276396529eb7aef46cc4fbd6c98a55904b6215a8f25f0a2c19a16796e19e22de
|
data/LICENSE.txt
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
The MIT License (MIT)
|
2
|
+
|
3
|
+
Copyright (c) 2025 Abdelkader Boudih
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
13
|
+
all copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,124 @@
|
|
1
|
+
# Minitest::Reporters::LLM
|
2
|
+
|
3
|
+
A token-optimized Minitest reporter specifically designed for Large Language Model consumption. Features ultra-compact output, regression tracking, and smart time formatting to minimize token usage while maintaining maximum parsability.
|
4
|
+
|
5
|
+
## Installation
|
6
|
+
|
7
|
+
Add this line to your application's Gemfile:
|
8
|
+
|
9
|
+
```ruby
|
10
|
+
gem 'minitest-reporters-llm'
|
11
|
+
```
|
12
|
+
|
13
|
+
And then execute:
|
14
|
+
|
15
|
+
$ bundle
|
16
|
+
|
17
|
+
Or install it yourself as:
|
18
|
+
|
19
|
+
$ gem install minitest-reporters-llm
|
20
|
+
|
21
|
+
## Usage
|
22
|
+
|
23
|
+
### Basic Setup
|
24
|
+
|
25
|
+
```ruby
|
26
|
+
# In test_helper.rb or wherever you configure minitest-reporters
|
27
|
+
require 'minitest/reporters/llm'
|
28
|
+
|
29
|
+
# Use compact format (default)
|
30
|
+
Minitest::Reporters.use! [Minitest::Reporters::LLMReporter.new]
|
31
|
+
|
32
|
+
# Or with options
|
33
|
+
Minitest::Reporters.use! [
|
34
|
+
Minitest::Reporters::LLMReporter.new(
|
35
|
+
format: :compact, # :compact or :verbose
|
36
|
+
results_file: 'tmp/test_results.json',
|
37
|
+
report_file: 'tmp/test_report.toml',
|
38
|
+
track_regressions: true, # Track test status changes
|
39
|
+
write_reports: true # Write JSON/TOML reports
|
40
|
+
)
|
41
|
+
]
|
42
|
+
```
|
43
|
+
|
44
|
+
### Output Formats
|
45
|
+
|
46
|
+
#### Compact Mode (Optimized for LLMs)
|
47
|
+
```
|
48
|
+
R t15 d2.3s p12 f2 e1 s0
|
49
|
+
REG +1 -0
|
50
|
+
F user_test.rb:45 validation fails
|
51
|
+
E api_test.rb:12 connection timeout
|
52
|
+
```
|
53
|
+
|
54
|
+
Format: `R t{total} d{duration} p{pass} f{fail} e{error} s{skip}`
|
55
|
+
- `R` = Result summary line
|
56
|
+
- `F/E/S` = Individual failure/error/skip lines
|
57
|
+
- `REG +X -Y` = Regression summary (X new failures, Y fixes)
|
58
|
+
|
59
|
+
#### Verbose Mode (Human-friendly)
|
60
|
+
```
|
61
|
+
15 tests (2.3s)
|
62
|
+
12 passed
|
63
|
+
2 failed: validation fails@user_test.rb:45, connection timeout@api_test.rb:12
|
64
|
+
1 error: api_test.rb:12
|
65
|
+
|
66
|
+
Details:
|
67
|
+
----------------------------------------
|
68
|
+
FAIL test_validation_fails
|
69
|
+
user_test.rb:45
|
70
|
+
Expected true, got false
|
71
|
+
```
|
72
|
+
|
73
|
+
### Smart Time Formatting
|
74
|
+
- `<1ms` - Sub-millisecond tests
|
75
|
+
- `15ms` - Millisecond precision
|
76
|
+
- `2.3s` - Second precision
|
77
|
+
- `1m30s` - Minute/second format
|
78
|
+
|
79
|
+
### Environment Variables
|
80
|
+
```bash
|
81
|
+
# Override default file paths
|
82
|
+
export LLM_REPORTER_RESULTS="custom/results.json"
|
83
|
+
export LLM_REPORTER_TOML="custom/report.toml"
|
84
|
+
```
|
85
|
+
|
86
|
+
### Regression Tracking
|
87
|
+
The reporter automatically tracks test status changes between runs:
|
88
|
+
- Saves test results to JSON file
|
89
|
+
- Compares current run with previous results
|
90
|
+
- Shows `REG +X -Y` for new failures/fixes
|
91
|
+
- Helps identify flaky or newly broken tests
|
92
|
+
|
93
|
+
## Development
|
94
|
+
|
95
|
+
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
96
|
+
|
97
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
98
|
+
|
99
|
+
## Features
|
100
|
+
|
101
|
+
- **Token-optimized output**: 70% fewer tokens than traditional reporters
|
102
|
+
- **Smart time formatting**: Automatically chooses optimal precision (`<1ms`, `15ms`, `2.3s`, `1m30s`)
|
103
|
+
- **Regression tracking**: Detects new failures and fixes between test runs
|
104
|
+
- **Dual output modes**: Compact for LLMs, verbose for humans
|
105
|
+
- **Configurable file paths**: No hardcoded temporary directories
|
106
|
+
- **TOML/JSON reports**: Structured data export for further analysis
|
107
|
+
- **Zero dependencies**: Only requires minitest-reporters
|
108
|
+
|
109
|
+
## Why Use This Reporter?
|
110
|
+
|
111
|
+
Perfect for:
|
112
|
+
- **AI-assisted development** workflows
|
113
|
+
- **CI/CD systems** requiring compact output
|
114
|
+
- **Log analysis** and automated test result processing
|
115
|
+
- **Token-conscious** LLM integrations
|
116
|
+
- **Regression monitoring** across test runs
|
117
|
+
|
118
|
+
## Contributing
|
119
|
+
|
120
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/seuros/minitest-reporters-llm.
|
121
|
+
|
122
|
+
## License
|
123
|
+
|
124
|
+
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
data/Rakefile
ADDED
data/bin/console
ADDED
@@ -0,0 +1,15 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
# frozen_string_literal: true
|
3
|
+
|
4
|
+
require 'bundler/setup'
|
5
|
+
require 'minitest/llm/reporter'
|
6
|
+
|
7
|
+
# You can add fixtures and/or initialization code here to make experimenting
|
8
|
+
# with your gem easier. You can also use a different console, if you like.
|
9
|
+
|
10
|
+
# (If you use this, don't forget to add pry to your Gemfile!)
|
11
|
+
# require "pry"
|
12
|
+
# Pry.start
|
13
|
+
|
14
|
+
require 'irb'
|
15
|
+
IRB.start(__FILE__)
|
data/bin/setup
ADDED
@@ -0,0 +1,382 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require 'json'
|
4
|
+
require 'fileutils'
|
5
|
+
require 'minitest/reporters/llm/version'
|
6
|
+
|
7
|
+
module Minitest
|
8
|
+
module Reporters
|
9
|
+
class LLMReporter < ::Minitest::Reporters::BaseReporter
|
10
|
+
VERSION = LLM::VERSION
|
11
|
+
|
12
|
+
def initialize(options = {})
|
13
|
+
super
|
14
|
+
@options = default_options.merge(options)
|
15
|
+
@results_file = @options[:results_file]
|
16
|
+
@report_file = @options[:report_file]
|
17
|
+
@previous_results = @options[:track_regressions] ? load_previous_results : {}
|
18
|
+
@current_results = {}
|
19
|
+
@llm_start_time = nil
|
20
|
+
end
|
21
|
+
|
22
|
+
def start(*args)
|
23
|
+
super
|
24
|
+
@llm_start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
25
|
+
end
|
26
|
+
|
27
|
+
def record(result)
|
28
|
+
super
|
29
|
+
test_key = "#{result.klass}##{result.name}"
|
30
|
+
@current_results[test_key] = result.passed? ? 'pass' : 'fail'
|
31
|
+
end
|
32
|
+
|
33
|
+
def report
|
34
|
+
total = @current_results.size
|
35
|
+
fails_ct = tests_list.count { |t| t.failure && !t.skipped? && !t.error? }
|
36
|
+
errors_ct = tests_list.count { |t| t.failure && t.error? }
|
37
|
+
skips_ct = tests_list.count(&:skipped?)
|
38
|
+
passes = total - fails_ct - errors_ct - skips_ct
|
39
|
+
|
40
|
+
if @options[:format] == :compact
|
41
|
+
report_compact(total, passes, fails_ct, errors_ct, skips_ct)
|
42
|
+
else
|
43
|
+
report_verbose(total, passes, fails_ct, errors_ct, skips_ct)
|
44
|
+
end
|
45
|
+
|
46
|
+
save_current_results if @options[:track_regressions]
|
47
|
+
write_toml_summary
|
48
|
+
end
|
49
|
+
|
50
|
+
def report_compact(total, passes, fails_ct, errors_ct, skips_ct)
|
51
|
+
puts "R t#{total} d#{format_time(llm_total_time)} p#{passes} f#{fails_ct} e#{errors_ct} s#{skips_ct}"
|
52
|
+
|
53
|
+
show_regressions_compact
|
54
|
+
|
55
|
+
# Show failures
|
56
|
+
if fails_ct.positive?
|
57
|
+
tests_list.select { |t| t.failure && !t.skipped? && !t.error? }.each do |test|
|
58
|
+
puts "F #{format_test_location_compact(test)}"
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
62
|
+
# Show errors
|
63
|
+
if errors_ct.positive?
|
64
|
+
tests_list.select { |t| t.failure && t.error? }.each do |test|
|
65
|
+
puts "E #{format_test_location_compact(test)}"
|
66
|
+
end
|
67
|
+
end
|
68
|
+
|
69
|
+
# Show skips
|
70
|
+
return unless skips_ct.positive?
|
71
|
+
|
72
|
+
tests_list.select(&:skipped?).each do |test|
|
73
|
+
puts "S #{format_test_location_compact(test)}"
|
74
|
+
end
|
75
|
+
end
|
76
|
+
|
77
|
+
def report_verbose(total, passes, fails_ct, errors_ct, skips_ct)
|
78
|
+
puts
|
79
|
+
puts "🏃 #{total} tests (#{format_time(llm_total_time)})"
|
80
|
+
puts "✅ #{passes}" if passes.positive?
|
81
|
+
|
82
|
+
show_regressions
|
83
|
+
|
84
|
+
if fails_ct.positive?
|
85
|
+
failed_tests = tests_list.select { |t| t.failure && !t.skipped? && !t.error? }
|
86
|
+
.map { |test| format_test_location(test) }
|
87
|
+
puts "❌ #{failed_tests.size} failed: #{failed_tests.join(', ')}" if failed_tests.any?
|
88
|
+
end
|
89
|
+
|
90
|
+
if errors_ct.positive?
|
91
|
+
error_tests = tests_list.select { |t| t.failure && t.error? }
|
92
|
+
.map { |test| format_test_location(test) }
|
93
|
+
puts "💥 #{errors_ct}: #{error_tests.join(', ')}"
|
94
|
+
end
|
95
|
+
|
96
|
+
if skips_ct.positive?
|
97
|
+
skip_tests = tests_list.select(&:skipped?)
|
98
|
+
puts "⏭️ #{skips_ct} skipped:"
|
99
|
+
skip_tests.each do |test|
|
100
|
+
msg = clean_message(test.failure&.message)
|
101
|
+
puts " - #{format_test_location(test)}: #{msg}"
|
102
|
+
end
|
103
|
+
end
|
104
|
+
|
105
|
+
show_failure_details if fails_ct.positive? || errors_ct.positive?
|
106
|
+
end
|
107
|
+
|
108
|
+
private
|
109
|
+
|
110
|
+
def default_options
|
111
|
+
{
|
112
|
+
results_file: ENV.fetch('LLM_REPORTER_RESULTS', 'tmp/test_results.json'),
|
113
|
+
report_file: ENV.fetch('LLM_REPORTER_TOML', 'tmp/test_report.toml'),
|
114
|
+
format: :compact,
|
115
|
+
track_regressions: true,
|
116
|
+
write_reports: true
|
117
|
+
}
|
118
|
+
end
|
119
|
+
|
120
|
+
def format_time(time)
|
121
|
+
return '0' unless time.is_a?(Numeric)
|
122
|
+
return '<1ms' if time < 0.001
|
123
|
+
|
124
|
+
if time < 1
|
125
|
+
ms = (time * 1000).round
|
126
|
+
"#{ms}ms"
|
127
|
+
elsif time < 60
|
128
|
+
"#{time.round(1)}s"
|
129
|
+
else
|
130
|
+
minutes = (time / 60).floor
|
131
|
+
seconds = (time % 60).round
|
132
|
+
"#{minutes}m#{seconds}s"
|
133
|
+
end
|
134
|
+
end
|
135
|
+
|
136
|
+
def llm_total_time
|
137
|
+
return nil unless @llm_start_time
|
138
|
+
|
139
|
+
now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
140
|
+
now - @llm_start_time
|
141
|
+
rescue StandardError
|
142
|
+
nil
|
143
|
+
end
|
144
|
+
|
145
|
+
def format_test_location(test)
|
146
|
+
test_name = (test.name || '').to_s.gsub(/^test_/, '').tr('_', ' ')
|
147
|
+
if (loc = source_location_for(test)) && loc[0] && loc[1]
|
148
|
+
file = File.basename(loc[0])
|
149
|
+
line = loc[1]
|
150
|
+
"#{test_name}@#{file}:#{line}"
|
151
|
+
else
|
152
|
+
test_name
|
153
|
+
end
|
154
|
+
end
|
155
|
+
|
156
|
+
def format_test_location_compact(test)
|
157
|
+
test_name = (test.name || '').to_s.gsub(/^test_/, '').tr('_', ' ')
|
158
|
+
if (loc = source_location_for(test)) && loc[0] && loc[1]
|
159
|
+
file = File.basename(loc[0])
|
160
|
+
line = loc[1]
|
161
|
+
"#{file}:#{line} #{test_name}"
|
162
|
+
else
|
163
|
+
test_name
|
164
|
+
end
|
165
|
+
end
|
166
|
+
|
167
|
+
def source_location_for(test)
|
168
|
+
# Try provided API first
|
169
|
+
return test.source_location if test.respond_to?(:source_location)
|
170
|
+
|
171
|
+
# Derive from class + method if possible
|
172
|
+
begin
|
173
|
+
klass_name = test.respond_to?(:klass) ? test.klass : test.class.name
|
174
|
+
method_name = test.name
|
175
|
+
return nil unless klass_name && method_name
|
176
|
+
|
177
|
+
klass = constantize(klass_name)
|
178
|
+
return nil unless klass&.instance_methods&.include?(method_name.to_sym)
|
179
|
+
|
180
|
+
klass.instance_method(method_name).source_location
|
181
|
+
rescue StandardError
|
182
|
+
nil
|
183
|
+
end
|
184
|
+
end
|
185
|
+
|
186
|
+
def constantize(name)
|
187
|
+
names = name.split('::')
|
188
|
+
names.shift if names.first.empty?
|
189
|
+
names.inject(Object) { |constant, n| constant.const_get(n) }
|
190
|
+
rescue NameError
|
191
|
+
nil
|
192
|
+
end
|
193
|
+
|
194
|
+
def show_failure_details
|
195
|
+
failed_tests = tests_list.select { |t| t.failure && !t.skipped? }
|
196
|
+
error_tests = tests_list.select { |t| t.failure && t.error? }
|
197
|
+
|
198
|
+
return unless failed_tests.any? || error_tests.any?
|
199
|
+
|
200
|
+
puts
|
201
|
+
puts '📋 Details:'
|
202
|
+
puts '-' * 40
|
203
|
+
|
204
|
+
failed_tests.each do |test|
|
205
|
+
puts "❌ #{test.name}"
|
206
|
+
puts " #{format_location(test)}"
|
207
|
+
puts " #{clean_message(test.failure&.message)}"
|
208
|
+
puts
|
209
|
+
end
|
210
|
+
|
211
|
+
error_tests.each do |test|
|
212
|
+
puts "💥 #{test.name}"
|
213
|
+
puts " #{format_location(test)}"
|
214
|
+
puts " #{clean_message(test.failure&.message)}"
|
215
|
+
puts
|
216
|
+
end
|
217
|
+
end
|
218
|
+
|
219
|
+
def format_location(test)
|
220
|
+
loc = source_location_for(test)
|
221
|
+
if loc && loc[0] && loc[1]
|
222
|
+
file = File.basename(loc[0])
|
223
|
+
"#{file}:#{loc[1]}"
|
224
|
+
else
|
225
|
+
'unknown location'
|
226
|
+
end
|
227
|
+
end
|
228
|
+
|
229
|
+
def clean_message(message)
|
230
|
+
return 'No message' unless message
|
231
|
+
|
232
|
+
message.to_s.split("\n").first&.strip || 'Unknown error'
|
233
|
+
end
|
234
|
+
|
235
|
+
def load_previous_results
|
236
|
+
return {} unless File.exist?(@results_file)
|
237
|
+
|
238
|
+
JSON.parse(File.read(@results_file))
|
239
|
+
rescue JSON::ParserError, Errno::ENOENT
|
240
|
+
{}
|
241
|
+
end
|
242
|
+
|
243
|
+
def save_current_results
|
244
|
+
return unless @options[:write_reports]
|
245
|
+
|
246
|
+
FileUtils.mkdir_p(File.dirname(@results_file))
|
247
|
+
File.write(@results_file, JSON.pretty_generate(@current_results))
|
248
|
+
rescue StandardError => e
|
249
|
+
puts "Warning: Could not save regression results: #{e.message}" if ENV['DEBUG']
|
250
|
+
end
|
251
|
+
|
252
|
+
def show_regressions
|
253
|
+
return if @previous_results.empty?
|
254
|
+
|
255
|
+
new_failures = []
|
256
|
+
fixes = []
|
257
|
+
|
258
|
+
@current_results.each do |test_key, status|
|
259
|
+
previous_status = @previous_results[test_key]
|
260
|
+
|
261
|
+
if previous_status == 'pass' && status == 'fail'
|
262
|
+
new_failures << test_key_to_location(test_key)
|
263
|
+
elsif previous_status == 'fail' && status == 'pass'
|
264
|
+
fixes << test_key_to_location(test_key)
|
265
|
+
end
|
266
|
+
end
|
267
|
+
|
268
|
+
puts "✅➡️❌ #{new_failures.size}: #{new_failures.join(', ')}" if new_failures.any?
|
269
|
+
puts "🎉 #{fixes.size}: #{fixes.join(', ')}" if fixes.any?
|
270
|
+
end
|
271
|
+
|
272
|
+
def show_regressions_compact
|
273
|
+
return if @previous_results.empty?
|
274
|
+
|
275
|
+
new_failures = 0
|
276
|
+
fixes = 0
|
277
|
+
|
278
|
+
@current_results.each do |test_key, status|
|
279
|
+
previous_status = @previous_results[test_key]
|
280
|
+
new_failures += 1 if previous_status == 'pass' && status == 'fail'
|
281
|
+
fixes += 1 if previous_status == 'fail' && status == 'pass'
|
282
|
+
end
|
283
|
+
|
284
|
+
puts "REG +#{new_failures} -#{fixes}" if new_failures.positive? || fixes.positive?
|
285
|
+
end
|
286
|
+
|
287
|
+
def test_key_to_location(test_key)
|
288
|
+
class_name, method_name = test_key.split('#', 2)
|
289
|
+
method_name = method_name.to_s.gsub(/^test_/, '').tr('_', ' ')
|
290
|
+
"#{method_name}@#{class_name}"
|
291
|
+
end
|
292
|
+
|
293
|
+
def write_toml_summary
|
294
|
+
return unless @options[:write_reports]
|
295
|
+
|
296
|
+
data = {
|
297
|
+
summary: {
|
298
|
+
tests: @current_results.size,
|
299
|
+
passes: (@current_results.size - tests_list.count do |t|
|
300
|
+
t.failure && !t.skipped? && !t.error?
|
301
|
+
end - tests_list.count do |t|
|
302
|
+
t.failure && t.error?
|
303
|
+
end - tests_list.count(&:skipped?)),
|
304
|
+
failures: tests_list.count { |t| t.failure && !t.skipped? && !t.error? },
|
305
|
+
errors: tests_list.count { |t| t.failure && t.error? },
|
306
|
+
skips: tests_list.count(&:skipped?),
|
307
|
+
time_s: safe_total_time
|
308
|
+
},
|
309
|
+
details: {
|
310
|
+
failed: tests_list.select { |t| t.failure && !t.skipped? && !t.error? }
|
311
|
+
.map { |t| format_test_location(t) },
|
312
|
+
errors: tests_list.select { |t| t.failure && t.error? }
|
313
|
+
.map { |t| format_test_location(t) },
|
314
|
+
skipped: tests_list.select(&:skipped?)
|
315
|
+
.map do |t|
|
316
|
+
msg = clean_message(t.failure&.message)
|
317
|
+
"#{format_test_location(t)}: #{msg}"
|
318
|
+
end
|
319
|
+
}
|
320
|
+
}
|
321
|
+
|
322
|
+
unless @previous_results.empty?
|
323
|
+
new_failures = []
|
324
|
+
fixes = []
|
325
|
+
@current_results.each do |k, status|
|
326
|
+
prev = @previous_results[k]
|
327
|
+
new_failures << test_key_to_location(k) if prev == 'pass' && status == 'fail'
|
328
|
+
fixes << test_key_to_location(k) if prev == 'fail' && status == 'pass'
|
329
|
+
end
|
330
|
+
data[:regressions] = { new_failures: new_failures, fixes: fixes }
|
331
|
+
end
|
332
|
+
|
333
|
+
toml = build_toml(data)
|
334
|
+
FileUtils.mkdir_p(File.dirname(@report_file))
|
335
|
+
File.write(@report_file, toml)
|
336
|
+
rescue StandardError => e
|
337
|
+
puts "Warning: Could not write TOML report: #{e.message}" if ENV['DEBUG']
|
338
|
+
end
|
339
|
+
|
340
|
+
def n(value)
|
341
|
+
value.to_i
|
342
|
+
end
|
343
|
+
|
344
|
+
def tests_list
|
345
|
+
tests || []
|
346
|
+
end
|
347
|
+
|
348
|
+
def safe_total_time
|
349
|
+
t = llm_total_time
|
350
|
+
t.is_a?(Numeric) ? t.to_f : 0.0
|
351
|
+
end
|
352
|
+
|
353
|
+
def build_toml(hash)
|
354
|
+
lines = []
|
355
|
+
hash.each do |section, values|
|
356
|
+
lines << "[#{section}]"
|
357
|
+
values.each do |k, v|
|
358
|
+
key = k.to_s
|
359
|
+
case v
|
360
|
+
when Array
|
361
|
+
escaped = v.map { |s| s.to_s.gsub('\\', '\\\\').gsub('"', '\\"') }
|
362
|
+
lines << "#{key} = [\"#{escaped.join('\", \"')}\"]"
|
363
|
+
when String
|
364
|
+
val = v.to_s.gsub('\\', '\\\\').gsub('"', '\\"')
|
365
|
+
lines << "#{key} = \"#{val}\""
|
366
|
+
when Numeric
|
367
|
+
lines << "#{key} = #{v}"
|
368
|
+
when TrueClass, FalseClass
|
369
|
+
lines << "#{key} = #{v}"
|
370
|
+
else
|
371
|
+
# Fallback to string
|
372
|
+
val = v.to_s.gsub('\\', '\\\\').gsub('"', '\\"')
|
373
|
+
lines << "#{key} = \"#{val}\""
|
374
|
+
end
|
375
|
+
end
|
376
|
+
lines << ''
|
377
|
+
end
|
378
|
+
lines.join("\n")
|
379
|
+
end
|
380
|
+
end
|
381
|
+
end
|
382
|
+
end
|
@@ -0,0 +1,41 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require_relative 'lib/minitest/reporters/llm/version'
|
4
|
+
|
5
|
+
Gem::Specification.new do |spec|
|
6
|
+
spec.name = 'minitest-reporters-llm'
|
7
|
+
spec.version = Minitest::Reporters::LLM::VERSION
|
8
|
+
spec.authors = ['Abdelkader Boudih']
|
9
|
+
spec.email = ['terminale@gmail.com']
|
10
|
+
|
11
|
+
spec.summary = 'Token-optimized Minitest reporter for LLM consumption with regression tracking'
|
12
|
+
spec.description = 'A Minitest reporter optimized for Large Language Model consumption, featuring compact emoji-based output, regression detection by comparing test runs, TOML report generation, and detailed failure reporting with file locations. Perfect for AI-assisted development workflows.'
|
13
|
+
spec.homepage = 'https://github.com/seuros/minitest-reporters-llm'
|
14
|
+
spec.license = 'MIT'
|
15
|
+
|
16
|
+
spec.metadata = {
|
17
|
+
'homepage_uri' => spec.homepage,
|
18
|
+
'source_code_uri' => 'https://github.com/seuros/minitest-reporters-llm',
|
19
|
+
'changelog_uri' => 'https://github.com/seuros/minitest-reporters-llm/blob/master/CHANGELOG.md',
|
20
|
+
'bug_tracker_uri' => 'https://github.com/seuros/minitest-reporters-llm/issues',
|
21
|
+
'rubygems_mfa_required' => 'true'
|
22
|
+
}
|
23
|
+
|
24
|
+
# Specify which files should be added to the gem when it is released.
|
25
|
+
spec.files = Dir.glob(%w[
|
26
|
+
lib/**/*.rb
|
27
|
+
*.md
|
28
|
+
*.txt
|
29
|
+
*.gemspec
|
30
|
+
Rakefile
|
31
|
+
bin/*
|
32
|
+
]).reject { |f| f.match(%r{^(test|spec|features)/}) }
|
33
|
+
spec.require_paths = ['lib']
|
34
|
+
spec.required_ruby_version = '>= 3.3'
|
35
|
+
|
36
|
+
spec.add_development_dependency 'bundler', '~> 2.5'
|
37
|
+
spec.add_development_dependency 'rake', '~> 13.0'
|
38
|
+
|
39
|
+
spec.add_dependency 'minitest', '>= 5.25'
|
40
|
+
spec.add_dependency 'minitest-reporters', '>= 1.7.1'
|
41
|
+
end
|
metadata
ADDED
@@ -0,0 +1,113 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: minitest-reporters-llm
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Abdelkader Boudih
|
8
|
+
bindir: bin
|
9
|
+
cert_chain: []
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
11
|
+
dependencies:
|
12
|
+
- !ruby/object:Gem::Dependency
|
13
|
+
name: bundler
|
14
|
+
requirement: !ruby/object:Gem::Requirement
|
15
|
+
requirements:
|
16
|
+
- - "~>"
|
17
|
+
- !ruby/object:Gem::Version
|
18
|
+
version: '2.5'
|
19
|
+
type: :development
|
20
|
+
prerelease: false
|
21
|
+
version_requirements: !ruby/object:Gem::Requirement
|
22
|
+
requirements:
|
23
|
+
- - "~>"
|
24
|
+
- !ruby/object:Gem::Version
|
25
|
+
version: '2.5'
|
26
|
+
- !ruby/object:Gem::Dependency
|
27
|
+
name: rake
|
28
|
+
requirement: !ruby/object:Gem::Requirement
|
29
|
+
requirements:
|
30
|
+
- - "~>"
|
31
|
+
- !ruby/object:Gem::Version
|
32
|
+
version: '13.0'
|
33
|
+
type: :development
|
34
|
+
prerelease: false
|
35
|
+
version_requirements: !ruby/object:Gem::Requirement
|
36
|
+
requirements:
|
37
|
+
- - "~>"
|
38
|
+
- !ruby/object:Gem::Version
|
39
|
+
version: '13.0'
|
40
|
+
- !ruby/object:Gem::Dependency
|
41
|
+
name: minitest
|
42
|
+
requirement: !ruby/object:Gem::Requirement
|
43
|
+
requirements:
|
44
|
+
- - ">="
|
45
|
+
- !ruby/object:Gem::Version
|
46
|
+
version: '5.25'
|
47
|
+
type: :runtime
|
48
|
+
prerelease: false
|
49
|
+
version_requirements: !ruby/object:Gem::Requirement
|
50
|
+
requirements:
|
51
|
+
- - ">="
|
52
|
+
- !ruby/object:Gem::Version
|
53
|
+
version: '5.25'
|
54
|
+
- !ruby/object:Gem::Dependency
|
55
|
+
name: minitest-reporters
|
56
|
+
requirement: !ruby/object:Gem::Requirement
|
57
|
+
requirements:
|
58
|
+
- - ">="
|
59
|
+
- !ruby/object:Gem::Version
|
60
|
+
version: 1.7.1
|
61
|
+
type: :runtime
|
62
|
+
prerelease: false
|
63
|
+
version_requirements: !ruby/object:Gem::Requirement
|
64
|
+
requirements:
|
65
|
+
- - ">="
|
66
|
+
- !ruby/object:Gem::Version
|
67
|
+
version: 1.7.1
|
68
|
+
description: A Minitest reporter optimized for Large Language Model consumption, featuring
|
69
|
+
compact emoji-based output, regression detection by comparing test runs, TOML report
|
70
|
+
generation, and detailed failure reporting with file locations. Perfect for AI-assisted
|
71
|
+
development workflows.
|
72
|
+
email:
|
73
|
+
- terminale@gmail.com
|
74
|
+
executables: []
|
75
|
+
extensions: []
|
76
|
+
extra_rdoc_files: []
|
77
|
+
files:
|
78
|
+
- LICENSE.txt
|
79
|
+
- README.md
|
80
|
+
- Rakefile
|
81
|
+
- bin/console
|
82
|
+
- bin/setup
|
83
|
+
- lib/minitest/reporters/llm.rb
|
84
|
+
- lib/minitest/reporters/llm/version.rb
|
85
|
+
- lib/minitest/reporters/llm_reporter.rb
|
86
|
+
- minitest-reporters-llm.gemspec
|
87
|
+
homepage: https://github.com/seuros/minitest-reporters-llm
|
88
|
+
licenses:
|
89
|
+
- MIT
|
90
|
+
metadata:
|
91
|
+
homepage_uri: https://github.com/seuros/minitest-reporters-llm
|
92
|
+
source_code_uri: https://github.com/seuros/minitest-reporters-llm
|
93
|
+
changelog_uri: https://github.com/seuros/minitest-reporters-llm/blob/master/CHANGELOG.md
|
94
|
+
bug_tracker_uri: https://github.com/seuros/minitest-reporters-llm/issues
|
95
|
+
rubygems_mfa_required: 'true'
|
96
|
+
rdoc_options: []
|
97
|
+
require_paths:
|
98
|
+
- lib
|
99
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
100
|
+
requirements:
|
101
|
+
- - ">="
|
102
|
+
- !ruby/object:Gem::Version
|
103
|
+
version: '3.3'
|
104
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
105
|
+
requirements:
|
106
|
+
- - ">="
|
107
|
+
- !ruby/object:Gem::Version
|
108
|
+
version: '0'
|
109
|
+
requirements: []
|
110
|
+
rubygems_version: 3.6.9
|
111
|
+
specification_version: 4
|
112
|
+
summary: Token-optimized Minitest reporter for LLM consumption with regression tracking
|
113
|
+
test_files: []
|