featureparity 0.0.16 → 0.0.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/lib/fp/cli.rb +2 -0
- data/lib/fp/client.rb +43 -3
- data/lib/fp/commands/changed_tests.rb +126 -0
- data/lib/fp/commands/ci_report.rb +133 -28
- data/lib/fp/commands/comment.rb +99 -0
- data/lib/fp/commands/help.rb +33 -0
- data/lib/fp/commands/init.rb +52 -12
- data/lib/fp/commands/propose_change.rb +46 -1
- data/lib/fp/commands/report.rb +63 -6
- data/lib/fp/commands.rb +2 -0
- data/lib/fp/junit.rb +50 -5
- data/lib/fp/source_fingerprint.rb +193 -0
- data/lib/fp/version.rb +1 -1
- data/lib/fp.rb +1 -0
- data/skills/fp/SKILL.md +31 -0
- data/skills/fp/references/cli.md +61 -1
- metadata +5 -2
data/lib/fp/commands/init.rb
CHANGED
|
@@ -180,17 +180,19 @@ module Fp
|
|
|
180
180
|
end
|
|
181
181
|
end
|
|
182
182
|
|
|
183
|
-
# The CI gate
|
|
184
|
-
#
|
|
185
|
-
#
|
|
183
|
+
# The CI gate and its adaptable reporting recipe are separate concerns: the gate
|
|
184
|
+
# blocks pull requests, while reporting needs a trusted context with FP_API_KEY and
|
|
185
|
+
# should cover both pull requests and default-branch pushes.
|
|
186
186
|
def workflow_body
|
|
187
187
|
<<~YAML
|
|
188
188
|
# FeatureParity — parity enforcement.
|
|
189
189
|
#
|
|
190
|
-
# Generated by `fp init`.
|
|
190
|
+
# Generated by `fp init`. The gate is ready to use; adapt and uncomment the
|
|
191
|
+
# report recipe once this repository emits JUnit:
|
|
191
192
|
#
|
|
192
|
-
# report On
|
|
193
|
-
#
|
|
193
|
+
# report On trusted pull requests, report candidate test revisions before
|
|
194
|
+
# merge. On default-branch pushes, record merged pass/fail and promote
|
|
195
|
+
# an approved exact test revision into the accepted baseline.
|
|
194
196
|
# gate On pull requests, fail the build when the project violates its
|
|
195
197
|
# enforcement level. Make this a required status check in branch
|
|
196
198
|
# protection to actually block merges.
|
|
@@ -261,30 +263,44 @@ module Fp
|
|
|
261
263
|
run: fp check
|
|
262
264
|
|
|
263
265
|
# Uncomment and adapt once your suite emits JUnit XML. `fp ci-report` writes
|
|
264
|
-
# real passing/failing evidence
|
|
265
|
-
#
|
|
266
|
+
# real passing/failing evidence and one source fingerprint per bound testcase.
|
|
267
|
+
# Run it on trusted pull requests too: `--pr` lets FeatureParity open a changed-test
|
|
268
|
+
# finding before merge without replacing the accepted default-branch baseline.
|
|
269
|
+
# The later default-branch report records merged truth and promotes an approved exact
|
|
270
|
+
# fingerprint. Without reporting, the matrix only has agent-filed `present` markers
|
|
271
|
+
# and changed bound tests cannot be detected.
|
|
266
272
|
#
|
|
267
273
|
# It reads the fp:<slug> markers out of the checkout, so run it in the repo
|
|
268
274
|
# and pass --base-dir when the report's test-file paths are relative to a
|
|
269
275
|
# subdirectory (vitest, for instance, reports them relative to its package).
|
|
276
|
+
# Keep `if: always()` so red tests are reported. Fork pull requests cannot read
|
|
277
|
+
# FP_API_KEY and are skipped by the HAVE_FP_KEY guard.
|
|
270
278
|
#
|
|
271
279
|
# report:
|
|
272
280
|
# name: Report evidence
|
|
273
|
-
# if: github.event_name == 'push'
|
|
281
|
+
# if: github.event_name == 'push' || github.event_name == 'pull_request'
|
|
274
282
|
# runs-on: ubuntu-latest
|
|
283
|
+
# env:
|
|
284
|
+
# HAVE_FP_KEY: ${{ secrets.FP_API_KEY != '' }}
|
|
275
285
|
# steps:
|
|
276
286
|
# - uses: actions/checkout@v4
|
|
277
287
|
# - name: Run tests (emit JUnit)
|
|
278
288
|
# run: echo "replace with your test command emitting junit.xml"
|
|
279
289
|
# - name: Report to FeatureParity
|
|
280
|
-
# if: always()
|
|
290
|
+
# if: always() && env.HAVE_FP_KEY == 'true'
|
|
281
291
|
# env:
|
|
282
292
|
# FP_API_KEY: ${{ secrets.FP_API_KEY }}
|
|
283
293
|
# run: |
|
|
284
294
|
# gem install featureparity
|
|
285
|
-
#
|
|
286
|
-
# --
|
|
295
|
+
# args=(
|
|
296
|
+
# --junit junit.xml
|
|
297
|
+
# --surface <surface-key>
|
|
287
298
|
# --ci-url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
|
|
299
|
+
# )
|
|
300
|
+
# if [ "${{ github.event_name }}" = "pull_request" ]; then
|
|
301
|
+
# args+=(--pr "${{ github.event.pull_request.html_url }}")
|
|
302
|
+
# fi
|
|
303
|
+
# fp ci-report "${args[@]}"
|
|
288
304
|
YAML
|
|
289
305
|
end
|
|
290
306
|
|
|
@@ -361,6 +377,30 @@ module Fp
|
|
|
361
377
|
`fp propose-change` (or `--removal`) with the source and reason to put that decision before
|
|
362
378
|
a human. If the intent is unclear, stop and ask rather than making the catalog say both things.
|
|
363
379
|
|
|
380
|
+
**Treat existing tests as protected product evidence.** Before editing an existing test,
|
|
381
|
+
determine what expectation it already records and whether CI has passing evidence for it.
|
|
382
|
+
|
|
383
|
+
- If the test backs passing evidence, do not rewrite it to make the current implementation
|
|
384
|
+
or a new requirement fit. Explicitly flag the intended test edit in the work item and pull
|
|
385
|
+
request. If its expected behaviour would change, propose the requirement change and wait
|
|
386
|
+
for explicit human approval before editing the test; a newly green run must not hide that
|
|
387
|
+
the proof itself changed.
|
|
388
|
+
- When FeatureParity reports a changed bound test, run `fp changed-tests` (or add `--json`)
|
|
389
|
+
and compare the accepted and candidate source revisions with the requirement. Record the
|
|
390
|
+
exact candidate through `fp changed-tests assess <id> --candidate-fingerprint <sha256>`
|
|
391
|
+
with a plain-language explanation and one classification: `still_proves`,
|
|
392
|
+
`no_longer_proves`, or `uncertain`. The first resolves only that fingerprint; the others
|
|
393
|
+
remain open for a human product decision. A later fingerprint needs a new assessment.
|
|
394
|
+
- A new requirement does not retroactively own or supersede a pre-existing failing test —
|
|
395
|
+
whether the requirement is draft or active. Do not change that test, or adopt it as the new
|
|
396
|
+
requirement's proof, merely to make the new work pass. Surface the conflict and wait for
|
|
397
|
+
explicit human approval to change the existing expectation.
|
|
398
|
+
|
|
399
|
+
A changed-test finding is an audit signal alongside the testcase's independent CI outcome.
|
|
400
|
+
It is not a failed test and is separate from `suspect`, which means requirement wording
|
|
401
|
+
changed and its evidence needs re-verification. Neither permits rewriting a test before
|
|
402
|
+
product intent is decided.
|
|
403
|
+
|
|
364
404
|
If no requirement covers what you're building, propose one or a parent/child tree. Agents
|
|
365
405
|
always create **drafts**; a human activates them:
|
|
366
406
|
|
|
@@ -55,7 +55,11 @@ module Fp
|
|
|
55
55
|
|
|
56
56
|
result = client.propose_requirement_change(requirement['id'], params)
|
|
57
57
|
unless result[:ok]
|
|
58
|
-
|
|
58
|
+
if result[:status] == 409 && result[:details].is_a?(Hash)
|
|
59
|
+
print_conflict(result, slug: slug)
|
|
60
|
+
else
|
|
61
|
+
output.error(result[:error], status: result[:status])
|
|
62
|
+
end
|
|
59
63
|
exit 1
|
|
60
64
|
end
|
|
61
65
|
|
|
@@ -64,6 +68,47 @@ module Fp
|
|
|
64
68
|
|
|
65
69
|
private
|
|
66
70
|
|
|
71
|
+
# A 409 means the requirement already has an open proposal — the one-at-a-time rule
|
|
72
|
+
# (two "what it was" snapshots would make a rollback ambiguous). Rather than a dead
|
|
73
|
+
# end, show what the open proposal is and point at `fp comment` so the agent can
|
|
74
|
+
# raise its concern on the proposal a human is about to review.
|
|
75
|
+
def print_conflict(result, slug:)
|
|
76
|
+
pending = result[:details]['pending_change'] || {}
|
|
77
|
+
|
|
78
|
+
if output.json?
|
|
79
|
+
output.error(result[:error], status: result[:status])
|
|
80
|
+
return
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
warn "Error: #{result[:error]}"
|
|
84
|
+
warn ''
|
|
85
|
+
warn 'The open proposal:'
|
|
86
|
+
warn " Change: #{pending['id']}"
|
|
87
|
+
warn " Kind: #{pending['kind']}"
|
|
88
|
+
warn " Proposed: #{pending['actor']} (#{pending['actor_kind']})"
|
|
89
|
+
warn " Reason: #{pending['reason']}" if pending['reason']
|
|
90
|
+
print_conflict_source(pending['source'])
|
|
91
|
+
print_conflict_diff(pending['changes'])
|
|
92
|
+
warn ''
|
|
93
|
+
warn 'You cannot open a rival proposal, but you can comment on this one:'
|
|
94
|
+
warn " fp comment #{slug} --change #{pending['id']} --body \"...\" --project <project>"
|
|
95
|
+
warn 'Or wait for a human to review it. Run `fp history ' \
|
|
96
|
+
"#{slug} --project <project>` for the full diff."
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
def print_conflict_source(source)
|
|
100
|
+
return unless source.is_a?(Hash) && !source.empty?
|
|
101
|
+
|
|
102
|
+
warn " Source: #{source['type']}" if source['type']
|
|
103
|
+
warn " URL: #{source['url']}" if source['url']
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
def print_conflict_diff(changes)
|
|
107
|
+
return unless changes.is_a?(Hash) && !changes.empty?
|
|
108
|
+
|
|
109
|
+
warn ' Fields: ' + changes.keys.join(', ')
|
|
110
|
+
end
|
|
111
|
+
|
|
67
112
|
def validate_enums!(opts)
|
|
68
113
|
if opts[:category] && !VALID_CATEGORIES.include?(opts[:category])
|
|
69
114
|
output.error("Invalid category '#{opts[:category]}'. Must be one of: #{VALID_CATEGORIES.join(', ')}")
|
data/lib/fp/commands/report.rb
CHANGED
|
@@ -115,9 +115,26 @@ module Fp
|
|
|
115
115
|
exit 1
|
|
116
116
|
end
|
|
117
117
|
|
|
118
|
+
# The server echoes back the stored record, whose status/source may differ
|
|
119
|
+
# from what we asked for: CI evidence is authoritative, so an agent's
|
|
120
|
+
# present/stub never downgrades a cell CI already verified. Report the truth
|
|
121
|
+
# the server kept, not the state we requested, so the caller knows whether
|
|
122
|
+
# their report actually landed or was superseded by CI.
|
|
123
|
+
stored = result[:data].is_a?(Hash) ? (result[:data]['evidence'] || {}) : {}
|
|
124
|
+
stored_status = stored['status']
|
|
125
|
+
stored_source = stored['source']
|
|
126
|
+
superseded = ci_superseded?(state, stored_status, stored_source)
|
|
127
|
+
|
|
118
128
|
output.success(result[:data]) do
|
|
119
|
-
|
|
120
|
-
|
|
129
|
+
if superseded
|
|
130
|
+
puts "CI already verified '#{slug}' on surface '#{opts[:surface]}' (status: #{stored_status})."
|
|
131
|
+
puts "Your '#{state}' report was recorded as metadata but did NOT change the status —"
|
|
132
|
+
puts 'CI evidence is authoritative and is never downgraded by an agent report.'
|
|
133
|
+
puts ' Stored: ' + "#{stored_status} (source: #{stored_source})"
|
|
134
|
+
else
|
|
135
|
+
puts "Evidence reported for '#{slug}' on surface '#{opts[:surface]}'."
|
|
136
|
+
puts " State: #{stored_status || state}"
|
|
137
|
+
end
|
|
121
138
|
puts " Type: #{opts[:test_type]}" if opts[:test_type]
|
|
122
139
|
puts " File: #{opts[:file]}#{opts[:line] ? ":#{opts[:line]}" : ''}"
|
|
123
140
|
if opts[:pr]
|
|
@@ -140,6 +157,20 @@ module Fp
|
|
|
140
157
|
|
|
141
158
|
private
|
|
142
159
|
|
|
160
|
+
# CI statuses an agent can never set. Mirrors Evidence::CI_STATUSES on the
|
|
161
|
+
# server — kept here so the CLI can recognise, from the echoed record, when its
|
|
162
|
+
# present/stub report was superseded by an authoritative CI verdict.
|
|
163
|
+
CI_STATUSES = %w[passing pr_passing failing].freeze
|
|
164
|
+
|
|
165
|
+
# True when the server kept a CI verdict instead of applying our report: the
|
|
166
|
+
# stored evidence is CI-sourced with a CI status, yet we asked for present/stub.
|
|
167
|
+
def ci_superseded?(requested_state, stored_status, stored_source)
|
|
168
|
+
return false if stored_status.nil?
|
|
169
|
+
return false unless VALID_STATES.include?(requested_state)
|
|
170
|
+
|
|
171
|
+
stored_source == 'ci' && CI_STATUSES.include?(stored_status) && stored_status != requested_state
|
|
172
|
+
end
|
|
173
|
+
|
|
143
174
|
# Batch-upload evidence from a JUnit XML report.
|
|
144
175
|
def run_junit(opts)
|
|
145
176
|
require_project!(opts)
|
|
@@ -244,16 +275,34 @@ module Fp
|
|
|
244
275
|
exit 1
|
|
245
276
|
end
|
|
246
277
|
|
|
247
|
-
reported << { binding: binding, state: state, recorded: true
|
|
278
|
+
reported << { binding: binding, state: state, recorded: true,
|
|
279
|
+
stored_status: stored_status_from(report_result),
|
|
280
|
+
stored_source: stored_source_from(report_result) }
|
|
248
281
|
end
|
|
249
282
|
|
|
250
283
|
emit_junit_summary(opts, reported, skipped_unknown, unmatched, api_unavailable)
|
|
251
284
|
end
|
|
252
285
|
|
|
286
|
+
# Pull the stored status/source out of the server's echoed evidence record so the
|
|
287
|
+
# batch summary can flag cells where CI superseded the agent's present/stub.
|
|
288
|
+
def stored_status_from(report_result)
|
|
289
|
+
data = report_result[:data]
|
|
290
|
+
return nil unless data.is_a?(Hash)
|
|
291
|
+
|
|
292
|
+
(data['evidence'] || {})['status']
|
|
293
|
+
end
|
|
294
|
+
|
|
295
|
+
def stored_source_from(report_result)
|
|
296
|
+
data = report_result[:data]
|
|
297
|
+
return nil unless data.is_a?(Hash)
|
|
298
|
+
|
|
299
|
+
(data['evidence'] || {})['source']
|
|
300
|
+
end
|
|
301
|
+
|
|
253
302
|
def emit_junit_summary(opts, reported, skipped_unknown, unmatched, api_unavailable)
|
|
254
303
|
data = {
|
|
255
304
|
repo: opts[:repo],
|
|
256
|
-
reported: reported.map { |r| { slug: r[:binding].slug, surface: r[:binding].surface, test_type: r[:binding].test_type, file: r[:binding].file, line: r[:binding].line, state: r[:state], recorded: r[:recorded] } },
|
|
305
|
+
reported: reported.map { |r| { slug: r[:binding].slug, surface: r[:binding].surface, test_type: r[:binding].test_type, file: r[:binding].file, line: r[:binding].line, state: r[:state], recorded: r[:recorded], stored_status: r[:stored_status], superseded_by_ci: ci_superseded?(r[:state], r[:stored_status], r[:stored_source]) } },
|
|
257
306
|
skipped_unknown: skipped_unknown.map { |b| { slug: b.slug, surface: b.surface, file: b.file } },
|
|
258
307
|
unmatched_testcases: unmatched.map { |tc| { name: tc.name, file: tc.file } }
|
|
259
308
|
}
|
|
@@ -261,10 +310,18 @@ module Fp
|
|
|
261
310
|
output.success(data) do
|
|
262
311
|
puts "Reported evidence from #{opts[:junit]}:"
|
|
263
312
|
reported.each do |r|
|
|
264
|
-
|
|
313
|
+
superseded = ci_superseded?(r[:state], r[:stored_status], r[:stored_source])
|
|
314
|
+
marker = if !r[:recorded]
|
|
315
|
+
'•'
|
|
316
|
+
elsif superseded
|
|
317
|
+
'⊘'
|
|
318
|
+
else
|
|
319
|
+
'✓'
|
|
320
|
+
end
|
|
265
321
|
loc = r[:binding].line ? "#{r[:binding].file}:#{r[:binding].line}" : r[:binding].file
|
|
266
322
|
type = r[:binding].test_type ? " [#{r[:binding].test_type}]" : ''
|
|
267
|
-
|
|
323
|
+
shown_state = superseded ? "#{r[:stored_status]}, kept CI — #{r[:state]} ignored" : r[:state]
|
|
324
|
+
puts " #{marker} #{r[:binding].slug} → #{r[:binding].surface}#{type} (#{shown_state}) #{loc}"
|
|
268
325
|
end
|
|
269
326
|
|
|
270
327
|
unless skipped_unknown.empty?
|
data/lib/fp/commands.rb
CHANGED
|
@@ -7,10 +7,12 @@ require_relative 'commands/list'
|
|
|
7
7
|
require_relative 'commands/show'
|
|
8
8
|
require_relative 'commands/propose'
|
|
9
9
|
require_relative 'commands/propose_change'
|
|
10
|
+
require_relative 'commands/comment'
|
|
10
11
|
require_relative 'commands/history'
|
|
11
12
|
require_relative 'commands/changes'
|
|
12
13
|
require_relative 'commands/report'
|
|
13
14
|
require_relative 'commands/ci_report'
|
|
15
|
+
require_relative 'commands/changed_tests'
|
|
14
16
|
require_relative 'commands/check'
|
|
15
17
|
require_relative 'commands/init'
|
|
16
18
|
require_relative 'commands/matrix'
|
data/lib/fp/junit.rb
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require 'rexml/document'
|
|
4
|
+
require_relative 'source_fingerprint'
|
|
4
5
|
|
|
5
6
|
module Fp
|
|
6
7
|
# Parses JUnit XML reports and binds testcases to fp:<slug> markers found in
|
|
@@ -40,6 +41,15 @@ module Fp
|
|
|
40
41
|
CiResult = Struct.new(:slug, :surface, :test_type, :status, :file, :line, :title, :test_count,
|
|
41
42
|
:failure_count, keyword_init: true)
|
|
42
43
|
|
|
44
|
+
# One exact testcase source receipt. Unlike CiResult, these are never aggregated:
|
|
45
|
+
# two tests may back the same requirement cell and each can change independently.
|
|
46
|
+
TestRevision = Struct.new(
|
|
47
|
+
:slug, :surface, :test_type, :testcase_key, :testcase_name, :classname, :outcome,
|
|
48
|
+
:repo, :file, :marker_line, :start_line, :end_line, :commit_sha,
|
|
49
|
+
:fingerprint, :fingerprint_scope,
|
|
50
|
+
keyword_init: true
|
|
51
|
+
)
|
|
52
|
+
|
|
43
53
|
# A failing testcase that carries no fp:<slug> marker at all — a "loose end". Not a
|
|
44
54
|
# cell on the parity grid (nothing binds it to a requirement) and not an orphan
|
|
45
55
|
# marker (there is no marker); it's a red test nobody is tracking against a
|
|
@@ -58,7 +68,7 @@ module Fp
|
|
|
58
68
|
# directory the runner was invoked from (vitest reports `src/pages/Foo.test.jsx` from
|
|
59
69
|
# inside `frontend/`), so reporting it verbatim produces a link to a file that doesn't
|
|
60
70
|
# exist at that path in the repo.
|
|
61
|
-
Match = Struct.new(:testcase, :marker, :precise, :repo_file, keyword_init: true)
|
|
71
|
+
Match = Struct.new(:testcase, :marker, :precise, :repo_file, :source_path, keyword_init: true)
|
|
62
72
|
|
|
63
73
|
# Evidence discovered by binding a testcase to a marker.
|
|
64
74
|
# `line` is the 1-based line number of the test in its source file: the
|
|
@@ -234,9 +244,13 @@ module Fp
|
|
|
234
244
|
# A slug is `failing` if any test claiming it went red, `stub` if every test claiming
|
|
235
245
|
# it was skipped (a skipped test is not proof of anything), and `passing` otherwise.
|
|
236
246
|
#
|
|
237
|
-
# Returns { results: [CiResult, ...],
|
|
238
|
-
|
|
247
|
+
# Returns { results: [CiResult, ...], test_revisions: [TestRevision, ...],
|
|
248
|
+
# unmatched: [TestCase], missing_surface: [...] }.
|
|
249
|
+
def ci_results(testcases, base_dir: Dir.pwd, default_surface: nil, default_test_type: nil,
|
|
250
|
+
commit_sha: nil, repo: nil)
|
|
239
251
|
aggregates = {}
|
|
252
|
+
test_revisions = []
|
|
253
|
+
fingerprint_cache = {}
|
|
240
254
|
matched = {}
|
|
241
255
|
missing_surface = []
|
|
242
256
|
|
|
@@ -251,6 +265,8 @@ module Fp
|
|
|
251
265
|
# The marker's own type pin wins; otherwise the --test-type default (nil → the
|
|
252
266
|
# server files it as `unspecified`). One CI verdict per (slug, surface, type).
|
|
253
267
|
test_type = marker[:test_type] || default_test_type
|
|
268
|
+
fingerprint = fingerprint_cache[[match.source_path, marker[:line]]] ||=
|
|
269
|
+
SourceFingerprint.build(match.source_path, marker)
|
|
254
270
|
|
|
255
271
|
surfaces.each do |surface|
|
|
256
272
|
if surface.nil? || surface.empty?
|
|
@@ -276,6 +292,18 @@ module Fp
|
|
|
276
292
|
agg[:title] ||= title
|
|
277
293
|
agg[:failing_title] ||= title if tc.failed?
|
|
278
294
|
end
|
|
295
|
+
|
|
296
|
+
if fingerprint && commit_sha
|
|
297
|
+
outcome = if tc.stub? then 'stub' elsif tc.failed? then 'failing' else 'passing' end
|
|
298
|
+
test_revisions << TestRevision.new(
|
|
299
|
+
slug: marker[:slug], surface: surface, test_type: test_type,
|
|
300
|
+
testcase_key: testcase_key(match.repo_file, tc), testcase_name: tc.name,
|
|
301
|
+
classname: tc.classname, outcome: outcome, repo: repo, file: match.repo_file,
|
|
302
|
+
marker_line: marker[:line], start_line: fingerprint.start_line,
|
|
303
|
+
end_line: fingerprint.end_line, commit_sha: commit_sha,
|
|
304
|
+
fingerprint: fingerprint.fingerprint, fingerprint_scope: fingerprint.scope
|
|
305
|
+
)
|
|
306
|
+
end
|
|
279
307
|
end
|
|
280
308
|
end
|
|
281
309
|
|
|
@@ -293,7 +321,12 @@ module Fp
|
|
|
293
321
|
end
|
|
294
322
|
|
|
295
323
|
unmatched = testcases.reject { |tc| matched[tc.object_id] }
|
|
296
|
-
{
|
|
324
|
+
{
|
|
325
|
+
results: results,
|
|
326
|
+
test_revisions: test_revisions,
|
|
327
|
+
unmatched: unmatched,
|
|
328
|
+
missing_surface: missing_surface.uniq
|
|
329
|
+
}
|
|
297
330
|
end
|
|
298
331
|
|
|
299
332
|
# The loose ends in a set of unmatched testcases: the ones that *failed* but carry no
|
|
@@ -326,11 +359,23 @@ module Fp
|
|
|
326
359
|
marker, precise = marker_for(tc, cases, markers)
|
|
327
360
|
next unless marker
|
|
328
361
|
|
|
329
|
-
yield Match.new(
|
|
362
|
+
yield Match.new(
|
|
363
|
+
testcase: tc, marker: marker, precise: precise, repo_file: repo_file,
|
|
364
|
+
source_path: File.expand_path(file, base_dir)
|
|
365
|
+
)
|
|
330
366
|
end
|
|
331
367
|
end
|
|
332
368
|
end
|
|
333
369
|
|
|
370
|
+
# Stable across body edits and line movement. A rename or move is intentionally a
|
|
371
|
+
# new identity; marker syntax has no durable per-test id from which to infer that the
|
|
372
|
+
# renamed testcase is the old one.
|
|
373
|
+
def testcase_key(repo_file, testcase)
|
|
374
|
+
Digest::SHA256.hexdigest(
|
|
375
|
+
[repo_file, testcase.classname, testcase.name].map(&:to_s).join("\0")
|
|
376
|
+
)
|
|
377
|
+
end
|
|
378
|
+
|
|
334
379
|
# The repository root containing `dir`, or nil when `dir` isn't inside one. #
|
|
335
380
|
# Found by walking up for a `.git` entry rather than shelling out to git, so it also
|
|
336
381
|
# works in a worktree (where `.git` is a file) and costs nothing.
|
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'digest'
|
|
4
|
+
|
|
5
|
+
module Fp
|
|
6
|
+
# Produces a receipt for the exact test source CI executed. Isolation is deliberately
|
|
7
|
+
# conservative: when a complete test body cannot be identified confidently, the whole
|
|
8
|
+
# file is hashed and the scope says so instead of pretending a noisy file hash is an
|
|
9
|
+
# isolated test receipt.
|
|
10
|
+
module SourceFingerprint
|
|
11
|
+
Result = Struct.new(:fingerprint, :scope, :start_line, :end_line, keyword_init: true)
|
|
12
|
+
|
|
13
|
+
BRACE_EXTENSIONS = %w[.js .jsx .mjs .cjs .ts .tsx .kt .kts .swift .java .go .cs .php .rs .scala].freeze
|
|
14
|
+
|
|
15
|
+
module_function
|
|
16
|
+
|
|
17
|
+
def build(path, marker)
|
|
18
|
+
contents = File.binread(path)
|
|
19
|
+
lines = contents.lines
|
|
20
|
+
range = isolated_range(lines, marker, File.extname(path).downcase)
|
|
21
|
+
|
|
22
|
+
if range
|
|
23
|
+
first, last = range
|
|
24
|
+
source = lines[first..last].join
|
|
25
|
+
Result.new(
|
|
26
|
+
fingerprint: Digest::SHA256.hexdigest(source),
|
|
27
|
+
scope: 'isolated_test',
|
|
28
|
+
start_line: first + 1,
|
|
29
|
+
end_line: last + 1
|
|
30
|
+
)
|
|
31
|
+
else
|
|
32
|
+
Result.new(
|
|
33
|
+
fingerprint: Digest::SHA256.hexdigest(contents),
|
|
34
|
+
scope: 'whole_file',
|
|
35
|
+
start_line: 1,
|
|
36
|
+
end_line: lines.length
|
|
37
|
+
)
|
|
38
|
+
end
|
|
39
|
+
rescue Errno::ENOENT, Errno::EACCES
|
|
40
|
+
nil
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
def isolated_range(lines, marker, extension)
|
|
44
|
+
return nil unless marker[:title_kind] == :test
|
|
45
|
+
|
|
46
|
+
start_index = declaration_index(lines, marker)
|
|
47
|
+
return nil if start_index.nil? || start_index.negative? || start_index >= lines.length
|
|
48
|
+
|
|
49
|
+
case extension
|
|
50
|
+
when '.rb'
|
|
51
|
+
ruby_range(lines, start_index)
|
|
52
|
+
when '.py'
|
|
53
|
+
python_range(lines, start_index)
|
|
54
|
+
when *BRACE_EXTENSIONS
|
|
55
|
+
brace_range(lines, start_index)
|
|
56
|
+
end
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
def declaration_index(lines, marker)
|
|
60
|
+
marker_index = marker[:line].to_i - 1
|
|
61
|
+
((marker_index + 1)...[marker_index + 41, lines.length].min).each do |index|
|
|
62
|
+
line = lines[index]
|
|
63
|
+
stripped = line.strip
|
|
64
|
+
next if stripped.empty? || stripped.match?(%r{\A(?:#|//|--|/\*|\*)})
|
|
65
|
+
|
|
66
|
+
return index
|
|
67
|
+
end
|
|
68
|
+
nil
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
def ruby_range(lines, start_index)
|
|
72
|
+
declaration = strip_ruby_literals(lines[start_index])
|
|
73
|
+
return nil if declaration.nil?
|
|
74
|
+
return nil unless declaration.match?(/\bdef\s+test_?\w+/i) ||
|
|
75
|
+
declaration.match?(/\b(?:it|specify|scenario)\b.*\bdo\b/)
|
|
76
|
+
|
|
77
|
+
depth = 0
|
|
78
|
+
(start_index...lines.length).each do |index|
|
|
79
|
+
code = strip_ruby_literals(lines[index])
|
|
80
|
+
return nil if code.nil?
|
|
81
|
+
|
|
82
|
+
depth += code.scan(/\bdo\b/).length
|
|
83
|
+
depth += 1 if code.match?(/^\s*(?:def|class|module|if|unless|case|begin|while|until|for)\b/)
|
|
84
|
+
depth -= code.scan(/\bend\b/).length
|
|
85
|
+
return [start_index, index] if depth.zero? && index >= start_index
|
|
86
|
+
return nil if depth.negative?
|
|
87
|
+
end
|
|
88
|
+
nil
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def strip_ruby_literals(line)
|
|
92
|
+
return nil if line.match?(/<<[-~]?['"`]?\w+/)
|
|
93
|
+
|
|
94
|
+
line.gsub(/'(?:\\.|[^'\\])*'|"(?:\\.|[^"\\])*"/, '').sub(/#.*/, '')
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
def python_range(lines, start_index)
|
|
98
|
+
declaration = lines[start_index]
|
|
99
|
+
return nil unless declaration.match?(/^\s*(?:async\s+)?def\s+test_?\w+.*:\s*(?:#.*)?$/i)
|
|
100
|
+
|
|
101
|
+
indent = declaration[/\A\s*/].length
|
|
102
|
+
last_body = nil
|
|
103
|
+
((start_index + 1)...lines.length).each do |index|
|
|
104
|
+
line = lines[index]
|
|
105
|
+
next if line.strip.empty? || line.lstrip.start_with?('#')
|
|
106
|
+
|
|
107
|
+
current_indent = line[/\A\s*/].length
|
|
108
|
+
break if current_indent <= indent
|
|
109
|
+
|
|
110
|
+
last_body = index
|
|
111
|
+
end
|
|
112
|
+
last_body ? [start_index, last_body] : nil
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
def brace_range(lines, start_index)
|
|
116
|
+
depth = 0
|
|
117
|
+
opened = false
|
|
118
|
+
in_block_comment = false
|
|
119
|
+
|
|
120
|
+
(start_index...lines.length).each do |index|
|
|
121
|
+
sanitized, in_block_comment = strip_brace_literals(lines[index], in_block_comment)
|
|
122
|
+
return nil if sanitized.nil?
|
|
123
|
+
|
|
124
|
+
sanitized.each_char do |char|
|
|
125
|
+
if char == '{'
|
|
126
|
+
depth += 1
|
|
127
|
+
opened = true
|
|
128
|
+
elsif char == '}' && opened
|
|
129
|
+
depth -= 1
|
|
130
|
+
return nil if depth.negative?
|
|
131
|
+
end
|
|
132
|
+
end
|
|
133
|
+
|
|
134
|
+
return [start_index, index] if opened && depth.zero?
|
|
135
|
+
return nil if !opened && index - start_index >= 10
|
|
136
|
+
end
|
|
137
|
+
nil
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
# Remove comments and quoted strings before balancing braces. Template literals are
|
|
141
|
+
# intentionally unsupported because interpolation can contain real code braces; those
|
|
142
|
+
# files use the labelled whole-file fallback rather than risk a false boundary.
|
|
143
|
+
def strip_brace_literals(line, in_block_comment)
|
|
144
|
+
return [nil, in_block_comment] if line.include?('`')
|
|
145
|
+
|
|
146
|
+
output = +''
|
|
147
|
+
quote = nil
|
|
148
|
+
escaped = false
|
|
149
|
+
index = 0
|
|
150
|
+
while index < line.length
|
|
151
|
+
char = line[index]
|
|
152
|
+
pair = line[index, 2]
|
|
153
|
+
|
|
154
|
+
if in_block_comment
|
|
155
|
+
if pair == '*/'
|
|
156
|
+
in_block_comment = false
|
|
157
|
+
index += 2
|
|
158
|
+
else
|
|
159
|
+
index += 1
|
|
160
|
+
end
|
|
161
|
+
next
|
|
162
|
+
end
|
|
163
|
+
|
|
164
|
+
if quote
|
|
165
|
+
if escaped
|
|
166
|
+
escaped = false
|
|
167
|
+
elsif char == '\\'
|
|
168
|
+
escaped = true
|
|
169
|
+
elsif char == quote
|
|
170
|
+
quote = nil
|
|
171
|
+
end
|
|
172
|
+
index += 1
|
|
173
|
+
next
|
|
174
|
+
end
|
|
175
|
+
|
|
176
|
+
break if pair == '//'
|
|
177
|
+
if pair == '/*'
|
|
178
|
+
in_block_comment = true
|
|
179
|
+
index += 2
|
|
180
|
+
next
|
|
181
|
+
end
|
|
182
|
+
if char == '"' || char == "'"
|
|
183
|
+
quote = char
|
|
184
|
+
else
|
|
185
|
+
output << char
|
|
186
|
+
end
|
|
187
|
+
index += 1
|
|
188
|
+
end
|
|
189
|
+
|
|
190
|
+
[output, in_block_comment]
|
|
191
|
+
end
|
|
192
|
+
end
|
|
193
|
+
end
|
data/lib/fp/version.rb
CHANGED
data/lib/fp.rb
CHANGED
data/skills/fp/SKILL.md
CHANGED
|
@@ -86,6 +86,27 @@ requirement around it. Identify which promise matches product intent, then use
|
|
|
86
86
|
`fp propose-change` (or `--removal`) with the source and reason to put that decision before a
|
|
87
87
|
human. If the intent is unclear, stop and ask rather than making the catalog say both things.
|
|
88
88
|
|
|
89
|
+
**Treat existing tests as protected product evidence.** Before editing an existing test,
|
|
90
|
+
determine what expectation it already records and whether CI has passing evidence for it.
|
|
91
|
+
|
|
92
|
+
- If the test backs passing evidence, do not rewrite it to make the current implementation or a
|
|
93
|
+
new requirement fit. Explicitly flag the intended test edit in the work item and pull request.
|
|
94
|
+
If its expected behaviour would change, propose the requirement change and wait for explicit
|
|
95
|
+
human approval before editing the test; a newly green run must not hide that the proof changed.
|
|
96
|
+
- When FeatureParity reports a changed bound test, run `fp changed-tests` (or add `--json`) and
|
|
97
|
+
compare its accepted and candidate revisions with the requirement. Record a plain-language
|
|
98
|
+
assessment for the exact candidate fingerprint with `fp changed-tests assess`: `still_proves`
|
|
99
|
+
resolves only that fingerprint; `no_longer_proves` and `uncertain` remain open for a human
|
|
100
|
+
product decision. A later fingerprint always needs a new assessment.
|
|
101
|
+
- A new requirement does not retroactively own or supersede a pre-existing failing test, whether
|
|
102
|
+
it is draft or active. Do not change that test, or adopt it as the new requirement's proof,
|
|
103
|
+
merely to make new work pass. Surface the conflict and wait for explicit human approval to
|
|
104
|
+
change the existing expectation.
|
|
105
|
+
|
|
106
|
+
A changed-test finding is an audit signal alongside the independent CI outcome, not a failed
|
|
107
|
+
test. It is also separate from `suspect`, which means requirement wording changed and its
|
|
108
|
+
evidence needs re-verification. Neither permits rewriting a test before product intent is decided.
|
|
109
|
+
|
|
89
110
|
### 2. Add the fp:<slug> marker to tests
|
|
90
111
|
|
|
91
112
|
Place a comment immediately before the test to bind it to a requirement:
|
|
@@ -118,6 +139,16 @@ fp report <slug> \
|
|
|
118
139
|
|
|
119
140
|
**Agents report `present` (default) or `stub` only. Never `passing` or `failing`.**
|
|
120
141
|
|
|
142
|
+
**Don't re-report a cell CI already covers.** CI evidence (`passing`, `pr_passing`,
|
|
143
|
+
`failing`) is authoritative. If CI has already run the test for a surface, a `present`
|
|
144
|
+
report adds nothing — it won't upgrade the cell, and the server will keep the CI verdict
|
|
145
|
+
rather than downgrade it to a bare marker. Check first with `fp show <slug> --project X`
|
|
146
|
+
(or `fp matrix`): if the surface already shows a CI status, skip the report. Report
|
|
147
|
+
`present`/`stub` for surfaces CI hasn't reached yet — e.g. a brand-new test before its
|
|
148
|
+
first CI run, or local work where CI isn't wired up. When you do report over a CI-covered
|
|
149
|
+
cell anyway, `fp report` now tells you it was superseded (the status stays CI's, your
|
|
150
|
+
report lands as metadata only) so you can see the no-op instead of assuming it took.
|
|
151
|
+
|
|
121
152
|
### 4. Report evidence from JUnit XML (batch upload without CI)
|
|
122
153
|
|
|
123
154
|
When CI isn't available, run the suite locally and upload the report:
|