jitcatch 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.ja.md +22 -5
- data/README.md +10 -9
- data/lib/jitcatch/assessor.rb +27 -4
- data/lib/jitcatch/changes.rb +86 -8
- data/lib/jitcatch/git_repo.rb +1 -1
- data/lib/jitcatch/readers.rb +7 -0
- data/lib/jitcatch/render.rb +15 -1
- data/lib/jitcatch/runner.rb +6 -2
- data/lib/jitcatch/source_index.rb +85 -6
- data/lib/jitcatch/version.rb +1 -1
- data/skills/jitcatch/SKILL.md +44 -8
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 883fa4fe8954d93bdce7752d6ffa59ef88b01366cb087a7b3a63e43b3a9b06a2
|
|
4
|
+
data.tar.gz: 0fec80cd0350343eb4d8c29189408f282730553717ec2a183aa0dbfe25998798
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 53a71c2e727f5453d3ed21d310e2d8134a4275c00d64817161d8b569cc3bc81f7d4504b015ad65a610c7c39ea0e047f741bbb74dc5569f544a5aa70ae3a0ae29
|
|
7
|
+
data.tar.gz: c7407444b9c4b9918c6c8ca649e8d3259f827580eee016b7ce08434d04a326a79fe15f8a0d817a1ff7548da5935e3d9d24c31d0b1d42e08a9130306945e33b2a
|
data/README.ja.md
CHANGED
|
@@ -75,7 +75,7 @@ jitcatch は次を出す。
|
|
|
75
75
|
<!-- rails:output -->
|
|
76
76
|
```
|
|
77
77
|
jitcatch run
|
|
78
|
-
version 0.
|
|
78
|
+
version 0.2.0
|
|
79
79
|
base e994a60bff5a85b3bcc92960dd251d598b15835a
|
|
80
80
|
head c3ef8997c50fc389a24ec4aa3f79caca2d6437b9
|
|
81
81
|
reruns 2
|
|
@@ -172,7 +172,7 @@ jitcatch は次を出す。
|
|
|
172
172
|
<!-- quickstart:output -->
|
|
173
173
|
```
|
|
174
174
|
jitcatch run
|
|
175
|
-
version 0.
|
|
175
|
+
version 0.2.0
|
|
176
176
|
base 81b05b22f14f2b5a1e580c048856a5e9d9b25b85
|
|
177
177
|
head 7c53dd7485d4c994d97bddd3cb7bd5481a337b61
|
|
178
178
|
reruns 2
|
|
@@ -181,6 +181,7 @@ tests:
|
|
|
181
181
|
t01 fail_on_base ShippingTest#test_empty_order_has_no_fee shipping_test.rb
|
|
182
182
|
base: fail Minitest::Assertion Expected: 0
|
|
183
183
|
Actual: 5
|
|
184
|
+
head: pass
|
|
184
185
|
t02 reported ShippingTest#test_fifty_ships_free shipping_test.rb m01
|
|
185
186
|
t03 no_signal ShippingTest#test_small_order_pays_five shipping_test.rb
|
|
186
187
|
counts generated=3 no_signal=1 fail_on_base=1 candidates=1 flaky=0 discarded=0 reported=1
|
|
@@ -232,8 +233,8 @@ jitcatch はそのテストを両方のリビジョンで走らせ、フレー
|
|
|
232
233
|
Ruby 3.3 以降。
|
|
233
234
|
追加の gem はない。
|
|
234
235
|
`minitest` と `prism` は Ruby に付いている。
|
|
235
|
-
`jitcatch --version`(または `-v`)は `jitcatch 0.
|
|
236
|
-
git から入れた gem のように jitcatch 自身の git のチェックアウトから動くときは、`jitcatch 0.
|
|
236
|
+
`jitcatch --version`(または `-v`)は `jitcatch 0.2.0` を出し、`--help` はコマンドの一覧を出す。
|
|
237
|
+
git から入れた gem のように jitcatch 自身の git のチェックアウトから動くときは、`jitcatch 0.2.0 (1fbcd18)` のようにコミットも出す。
|
|
237
238
|
RSpec の候補には、実行する場所に `rspec-core` が要る。
|
|
238
239
|
プロジェクトの bundle に入れるか、`Gemfile` が無いプロジェクトならその Ruby にインストールしておく。
|
|
239
240
|
`BASE` は親のリビジョン、`HEAD` は変更である。
|
|
@@ -447,7 +448,7 @@ bin/jitcatch assess --json run.json
|
|
|
447
448
|
| 区分 | 意味 |
|
|
448
449
|
| --- | --- |
|
|
449
450
|
| `no_signal` | base でも head でも通った。 |
|
|
450
|
-
| `fail_on_base` | 親がそのテストを満たさなかった。新しい機能だけを書いていることが多い。テキストの報告は、その行の下に base で起きたことを `base: error NoMethodError undefined method 'apply_coupon' for an instance of Shop::Cart`
|
|
451
|
+
| `fail_on_base` | 親がそのテストを満たさなかった。新しい機能だけを書いていることが多い。テキストの報告は、その行の下に base で起きたことを `base: error NoMethodError undefined method 'apply_coupon' for an instance of Shop::Cart` のように出し、次の行に head で起きたことを出す。変更でテストが通るようになったときは `head: pass`、head が別の失敗をしたときはその失敗、同じ失敗のときは `head: fail, the same as base` となる。 |
|
|
451
452
|
| `flaky` | base で通り head で落ちたあと、再実行が同じ落ち方をしなかった。捨てる。 |
|
|
452
453
|
| `discarded` | base で通り head で落ちたあと、ルールが除いた。`assess:` には残る。 |
|
|
453
454
|
| `reported` | base で通り head で落ち、ルールは除かなかった。この行を判断する。 |
|
|
@@ -489,6 +490,11 @@ gem の中から始まったエラーには `touches:changed_method` を付け
|
|
|
489
490
|
テストが変更されていない呼び出し元を通してだけ届くメソッドの中では、テストが名指しする入口から最も多くの呼び出しを下ったメソッドを選ぶ。
|
|
490
491
|
たとえば、リクエストスペックが 1 回下の `LoansController#officer` と 5 回下の `Borrower#approved_cents` に届くときは `Borrower#approved_cents` を選ぶ。
|
|
491
492
|
`--json` の `caller_depths` がその回数を持つ。
|
|
493
|
+
変更されたスクリプトを起動するテストは、そのコードが呼ぶ変更メソッドより先に、スクリプトの `exe/tool <main>` を選ぶ。
|
|
494
|
+
起動するたびにそのコードは動くが、その下のメソッドはそれを選ぶコマンドのときだけ動くからである。
|
|
495
|
+
たとえば、使い方の文言が変わった `quint_driver bogus` のテストは `related guess: exe/quint_driver <main> (m01), ranked first of 3: exe/quint_driver <main> (m01), QuintDriver::Spec#outline (m10), QuintDriver::Targets.report (m14)` となる。
|
|
496
|
+
テストが待つ例外を head が上げなくなった行 (`X expected but nothing was raised.`) では、`raise` の引数だけを変えた編集 (`--json` の `edits_only_raise`) を候補から外す。
|
|
497
|
+
そのメソッドはどの呼び出し元よりも下にあるが、head はもうそこへ届かず、上げなくなった原因はそれを呼ぶ側にあることが多い。
|
|
492
498
|
上位の二つを id でしか並べられないときは、どれも選ばない。
|
|
493
499
|
その行は `related guess: none; the test reaches 7 changed methods equally and no frame of head's failure is in one: Book#apply_catalog_record (m02), LoanRules.for (m03), ...` となり、id の位置に `-` が出て、`method_id` は `null` になる。
|
|
494
500
|
その行は次の一手で終わる。
|
|
@@ -504,6 +510,9 @@ id が最も小さいものだけを見るのではない。
|
|
|
504
510
|
|
|
505
511
|
`context` は 2 つのリビジョンを diff し、ハンクを Prism でメソッドに対応づける。
|
|
506
512
|
変更メソッドは `m01`、`m02` と付く。
|
|
513
|
+
`exe/` か `bin/` にある拡張子のない実行ファイルで、`#!` の行に `ruby` があるものも読む。
|
|
514
|
+
その中の `def`、クラス、モジュールの外のコードは `exe/tool <main>` という一つのメソッドになるので、使い方の文言の変更も出る。
|
|
515
|
+
その `tests:` はそのパスを書いたテストファイルで、それを起動するテストはそのメソッドへの言及になる。
|
|
507
516
|
`Gemfile.lock` が変わったときは、`dependencies:` に gem ごとに `gem 旧 -> 新` を出す。
|
|
508
517
|
変わった定数は `k01` で、その定数を読むだけで本体が変わっていないメソッドは `affected`(`via=k01`)になる。
|
|
509
518
|
あるファイルから消えたメソッドを、head が別のファイル(テストファイルも含む)で同じ名前で定義していれば、それは `moved` になる。
|
|
@@ -557,6 +566,8 @@ Concern の `included do`、`prepended do`、`class_methods do` ブロックの
|
|
|
557
566
|
これは `new` のほかにそれを走らせる呼び出しで、Active Record のモデルの `create`、`create!`、`build` と、そのクラスを作る FactoryBot のファクトリーごとの `build(:loan)` と `create(:loan)` である。
|
|
558
567
|
その `tests:` には、そのどれかか `Owner.new` を呼ぶテストファイルが、呼んでいるもの(`spec/models/loan_spec.rb via create(:loan)`)と一緒に出る。
|
|
559
568
|
テストが `initialize` をその名前で呼ぶことはないので、コンストラクタには grep の結果を出さない。
|
|
569
|
+
`attr_reader` や `Data.define` のメンバー(`Targets::Hit#code`)の名前は短い語が多いので、grep の結果は、そのクラスかそれを囲む名前空間の名前(`Hit` か `Targets`)も書いたファイルに限る。
|
|
570
|
+
`test/fixture/depot/app/` のように、`test/` や `spec/` の下に置いたフィクスチャのプロジェクトの `app/`、`lib/`、`config/`、`db/` にあるコードはテストではないので、`tests:` に出さない。
|
|
560
571
|
ファクトリーは `spec/factories` と `test/factories` から、読み込まずにソースとして読む。
|
|
561
572
|
クラスはまず `class:`(定数、文字列、シンボル)から、次に親(`parent:` か、入れ子の外側のファクトリー)から、最後に名前を camelize したものから決まり、`aliases:` の別名も同じクラスを作る。
|
|
562
573
|
ファクトリーは、その関連が作るクラスにも数える。
|
|
@@ -918,6 +929,7 @@ readers: 1 for 1 changed value
|
|
|
918
929
|
メソッドの `callsites:` にすでにある行と、定数の `read_by:` にすでにあるメソッドの中のただの読み出しは出さない。
|
|
919
930
|
`not seen:` は、見えないものを挙げる。
|
|
920
931
|
レコードのモデルは変数名かメソッド名(`@branch`、`home_branch`、`branches`)で決めるので、`x = Branch.first; link_to x` は出ない。
|
|
932
|
+
`app/`、`config/routes.rb`、`config/application.rb` のどれもない履歴 (gem など) では、モデルもルートもテンプレートもないので、どの Ruby にもある二つだけを挙げる。
|
|
921
933
|
|
|
922
934
|
## インストール
|
|
923
935
|
|
|
@@ -964,6 +976,10 @@ CLI を使わない場合は、`skills/jitcatch` をエージェントの skills
|
|
|
964
976
|
そうした呼び出しは、送る名前が変更メソッドの名前か、その `reachable_from` にあるメソッドの名前であれば、その変更メソッドの `dynamic_callers:` に `Emit.looked_up (render)` のように出る。
|
|
965
977
|
ただし名前が `Controller` で終わるクラスのインスタンスメソッドの名前は除く。
|
|
966
978
|
コントローラーのアクションはルーターが呼び、アプリのコードがその名前を送ることはないので、`ThingsController#update` が変更に届いても `cipher.update` は出ない。
|
|
979
|
+
ブロックだけを持つローカル変数への呼び出しも出ない。
|
|
980
|
+
`&block` の引数と、メソッドの中のすべての代入が `-> {}`、`lambda {}`、`proc {}`、`Proc.new {}` であるローカル変数がそれにあたる。
|
|
981
|
+
その `call` や `.()` はそのブロックを動かすだけで、リポジトリのメソッドを呼ぶことはない。
|
|
982
|
+
`def sync(job)` の中の `job.call` のような、引数への `call` は出る。
|
|
967
983
|
クラスが定義していないメソッドを `self` に呼ぶと、その名前を持つ索引上ただ一つのメソッドへ向かう。
|
|
968
984
|
モジュールやスーパークラスのメソッドのためである。
|
|
969
985
|
インスタンスメソッドからはインスタンスメソッドだけが対象になる。
|
|
@@ -1013,6 +1029,7 @@ CLI を使わない場合は、`skills/jitcatch` をエージェントの skills
|
|
|
1013
1029
|
エラークラスは、メッセージを作るために `to_s` を上書きするからである。
|
|
1014
1030
|
`"exe/tool"` や `"../bin/x"` のような相対パスの形の文字列が head のファイルを指すときは、テストが別のプロセスで起動するスクリプトとみなし、そのファイルの中の名前も言及になる。
|
|
1015
1031
|
パスは、リポジトリの根から、候補のディレクトリから、または候補の隣で `File.expand_path` が読むとおりに読む。
|
|
1032
|
+
`assert_includes error.backtrace.first, "lib/tool/model.rb"` や `include("lib/x.rb")` のように、アサーションやマッチャーが直接比べる文字列は期待するテキストであり、スクリプトを指さない。
|
|
1016
1033
|
別のファイルのヘルパーは読まない。
|
|
1017
1034
|
- `--workers` を渡さなければ、ファイルは 1 つずつ実行する。
|
|
1018
1035
|
1 つのファイルの base 側と head 側は、それぞれの worktree で同時に走る。
|
data/README.md
CHANGED
|
@@ -63,7 +63,7 @@ The last commit asks a senior officer from 10000 cents, where it used to ask onl
|
|
|
63
63
|
<!-- rails:output -->
|
|
64
64
|
```
|
|
65
65
|
jitcatch run
|
|
66
|
-
version 0.
|
|
66
|
+
version 0.2.0
|
|
67
67
|
base e994a60bff5a85b3bcc92960dd251d598b15835a
|
|
68
68
|
head c3ef8997c50fc389a24ec4aa3f79caca2d6437b9
|
|
69
69
|
reruns 2
|
|
@@ -145,7 +145,7 @@ jitcatch prints this. The two commit ids are your own.
|
|
|
145
145
|
<!-- quickstart:output -->
|
|
146
146
|
```
|
|
147
147
|
jitcatch run
|
|
148
|
-
version 0.
|
|
148
|
+
version 0.2.0
|
|
149
149
|
base 81b05b22f14f2b5a1e580c048856a5e9d9b25b85
|
|
150
150
|
head 7c53dd7485d4c994d97bddd3cb7bd5481a337b61
|
|
151
151
|
reruns 2
|
|
@@ -154,6 +154,7 @@ tests:
|
|
|
154
154
|
t01 fail_on_base ShippingTest#test_empty_order_has_no_fee shipping_test.rb
|
|
155
155
|
base: fail Minitest::Assertion Expected: 0
|
|
156
156
|
Actual: 5
|
|
157
|
+
head: pass
|
|
157
158
|
t02 reported ShippingTest#test_fifty_ships_free shipping_test.rb m01
|
|
158
159
|
t03 no_signal ShippingTest#test_small_order_pays_five shipping_test.rb
|
|
159
160
|
counts generated=3 no_signal=1 fail_on_base=1 candidates=1 flaky=0 discarded=0 reported=1
|
|
@@ -192,7 +193,7 @@ A coding agent reviewing a change writes tests for that diff. A test that only d
|
|
|
192
193
|
|
|
193
194
|
The method follows Meta's catching JiT tests ([Harden and Catch](https://arxiv.org/abs/2504.16472); [Just-in-Time Catching Test Generation at Meta](https://arxiv.org/abs/2601.22832)).
|
|
194
195
|
|
|
195
|
-
Ruby 3.3 or newer. No gems. `minitest` and `prism` ship with Ruby. `jitcatch --version` (or `-v`) prints `jitcatch 0.
|
|
196
|
+
Ruby 3.3 or newer. No gems. `minitest` and `prism` ship with Ruby. `jitcatch --version` (or `-v`) prints `jitcatch 0.2.0`, followed by the commit, as in `jitcatch 0.2.0 (1fbcd18)`, when jitcatch runs from its own git checkout, as a gem installed from git does, and `--help` prints the commands. RSpec candidates need `rspec-core` where they run: in the project's bundle, or installed for that Ruby when the project has no `Gemfile`. `BASE` is the parent revision and `HEAD` is the change. Each must resolve to a commit. Uncommitted files are not in the diff.
|
|
196
197
|
|
|
197
198
|
You write the candidate files in a scratch directory. A Minitest file starts with `require "minitest/autorun"`, then requires the library the way the repository loads it. An RSpec file is named `*_spec.rb` and starts with `require "spec_helper"` or `require "rails_helper"`, as the project's own specs do. jitcatch runs those files. You judge each `reported` row against the intent and the diff. The intent in the brief is the commit messages from `BASE` to `HEAD`, without the trailer lines git parses at the end of each message (such as `Co-Authored-By:`). likely-regression means the behavior changed and the intent does not ask for that change. intended means the intent asks for it. unclear means the intent is not enough to tell.
|
|
198
199
|
|
|
@@ -344,18 +345,18 @@ bin/jitcatch assess --json run.json
|
|
|
344
345
|
| Bucket | Meaning |
|
|
345
346
|
| --- | --- |
|
|
346
347
|
| `no_signal` | Passed on base and on head. |
|
|
347
|
-
| `fail_on_base` | The parent did not satisfy the test. Often the test only describes a new feature. The text report prints what base did under the row, as `base: error NoMethodError undefined method 'apply_coupon' for an instance of Shop::Cart`. |
|
|
348
|
+
| `fail_on_base` | The parent did not satisfy the test. Often the test only describes a new feature. The text report prints what base did under the row, as `base: error NoMethodError undefined method 'apply_coupon' for an instance of Shop::Cart`, and what head did on the next line: `head: pass` when the change makes the test pass, head's own failure, or `head: fail, the same as base`. |
|
|
348
349
|
| `flaky` | Passed on base, failed on head, then a rerun did not fail the same way. Dropped. |
|
|
349
350
|
| `discarded` | Passed on base and failed on head, then a rule removed it. Still listed under `assess:`. |
|
|
350
351
|
| `reported` | Passed on base and failed on head, and no rule removed it. Judge this row. |
|
|
351
352
|
|
|
352
353
|
`generated` counts test methods. `candidates` counts stable pass-on-base / fail-on-head tests before the discard rules. `reported` is what remains.
|
|
353
354
|
|
|
354
|
-
A tag is a hint, not the judgment. `touches:changed_method` means the test names a method the diff changed. `Owner.new` names the `initialize` of that owner alone (`Mismatch.new` is not `Spec#initialize`), and a changed constructor is the related method only when the test names no other changed method and goes through no changed method's caller. `touches:changed_constant` means it names a constant the diff changed. `touches:changed_dependency` means the error came from inside a gem whose locked version changed. An error that starts inside any gem does not get `touches:changed_method`. `touches:unchanged_caller` means the test goes through an entry point the diff did not edit, and that entry point reaches a changed method. `discard:signature_change`, `discard:load_error`, `discard:timeout`, `discard:no_database`, and `discard:no_table` name the rule that removed a candidate. `discard:signature_change` applies only when the test itself makes the call that a removed method or a new signature broke, or calls `new` of a `Data` or `Struct` class whose members the diff changed, as `Record.new(...)` failing with `missing keyword: :media`. Such a row is not a finding, and its next line says what changed: `candidate is out of date with head: CatalogService::Record (k07) changed its definition; update the call at ../cand/stale_spec.rb:5 to head's code, or drop the test`. `out_of_date` in `--json` holds its `id`, `name`, `change` (`removed`, `parameters`, or `definition`), and `call`. When unchanged code makes it, such as `class B < A` still calling the `A#helper` that head removed, the row is reported with `caller:unchanged:lib/b.rb:5`, the line of that call (`caller:changed:` for a line the diff wrote or a line in a changed method; `changed_lines` in `--json` lists the head lines each file's hunks wrote). The related id (`m05`, or `-` when there is none) is the downstream changed method when several changed methods sit on the path. When the failure's backtrace runs through changed methods, it is the innermost of them, and that pick is not a guess: the outer frames only called it. A frame's line is read as a line of head, so it never hits the method that held that line on base. `related:guess` means the evidence that picked the id points at more than one changed method, as when the test names only an unchanged caller that reaches several; the row's next line names them all with their ids, as `related guess: Shop::Cart#volume_discount_cents (m05), ranked first of 3: Shop::Cart#coupon_discount_cents (m02), Shop::Cart#taxable_cents (m04), Shop::Cart#volume_discount_cents (m05)`, and `method_ids` holds them in `--json`. The pick prefers a method that existed on base over one the diff added, except over an edit that only adds a call to an added method (`forwards_to_added` in `--json`), as `net - volume` becoming `net - volume - coupon(n)` does: the new value comes from the added method. Among methods the test reaches only through an unchanged caller, the pick is the one the most calls below the entry point the test names, so a request spec that reaches `LoansController#officer` one call down and `Borrower#approved_cents` five calls down picks `Borrower#approved_cents`; `caller_depths` in `--json` holds those counts. When only the id would order the best two, the row names no pick: the line reads `related guess: none; the test reaches 7 changed methods equally and no frame of head's failure is in one: Book#apply_catalog_record (m02), LoanRules.for (m03), ...`, and it ends with a next step: find what makes head's value in those methods, or in a changed file that holds no method, such as a view, from the `git diff --stat BASE HEAD` it prints. The row shows `-` in place of the id, and `method_id` is `null`. Such a row gets `touches:changed_method` when the test names any method of `method_ids`, not only the one with the lowest id. `message_varies` means head failed the same way on each rerun with a message that moved, and base passed on each rerun; compare a field the test controls rather than the whole record. `related:dynamic` means no changed method is related, but the test calls a method from `dynamic_callers:`; the row's next line says so, as in `related: none, Emit.looked_up sends render to a receiver the source does not name`.
|
|
355
|
+
A tag is a hint, not the judgment. `touches:changed_method` means the test names a method the diff changed. `Owner.new` names the `initialize` of that owner alone (`Mismatch.new` is not `Spec#initialize`), and a changed constructor is the related method only when the test names no other changed method and goes through no changed method's caller. `touches:changed_constant` means it names a constant the diff changed. `touches:changed_dependency` means the error came from inside a gem whose locked version changed. An error that starts inside any gem does not get `touches:changed_method`. `touches:unchanged_caller` means the test goes through an entry point the diff did not edit, and that entry point reaches a changed method. `discard:signature_change`, `discard:load_error`, `discard:timeout`, `discard:no_database`, and `discard:no_table` name the rule that removed a candidate. `discard:signature_change` applies only when the test itself makes the call that a removed method or a new signature broke, or calls `new` of a `Data` or `Struct` class whose members the diff changed, as `Record.new(...)` failing with `missing keyword: :media`. Such a row is not a finding, and its next line says what changed: `candidate is out of date with head: CatalogService::Record (k07) changed its definition; update the call at ../cand/stale_spec.rb:5 to head's code, or drop the test`. `out_of_date` in `--json` holds its `id`, `name`, `change` (`removed`, `parameters`, or `definition`), and `call`. When unchanged code makes it, such as `class B < A` still calling the `A#helper` that head removed, the row is reported with `caller:unchanged:lib/b.rb:5`, the line of that call (`caller:changed:` for a line the diff wrote or a line in a changed method; `changed_lines` in `--json` lists the head lines each file's hunks wrote). The related id (`m05`, or `-` when there is none) is the downstream changed method when several changed methods sit on the path. When the failure's backtrace runs through changed methods, it is the innermost of them, and that pick is not a guess: the outer frames only called it. A frame's line is read as a line of head, so it never hits the method that held that line on base. `related:guess` means the evidence that picked the id points at more than one changed method, as when the test names only an unchanged caller that reaches several; the row's next line names them all with their ids, as `related guess: Shop::Cart#volume_discount_cents (m05), ranked first of 3: Shop::Cart#coupon_discount_cents (m02), Shop::Cart#taxable_cents (m04), Shop::Cart#volume_discount_cents (m05)`, and `method_ids` holds them in `--json`. The pick prefers a method that existed on base over one the diff added, except over an edit that only adds a call to an added method (`forwards_to_added` in `--json`), as `net - volume` becoming `net - volume - coupon(n)` does: the new value comes from the added method. Among methods the test reaches only through an unchanged caller, the pick is the one the most calls below the entry point the test names, so a request spec that reaches `LoansController#officer` one call down and `Borrower#approved_cents` five calls down picks `Borrower#approved_cents`; `caller_depths` in `--json` holds those counts. A test that starts a changed script picks the script's `exe/tool <main>` ahead of the changed methods that code calls: every start runs it, and each method below it runs only for the command that picks it, so a test of `quint_driver bogus` whose usage text changed reads `related guess: exe/quint_driver <main> (m01), ranked first of 3: exe/quint_driver <main> (m01), QuintDriver::Spec#outline (m10), QuintDriver::Targets.report (m14)`. On a row where head stopped raising the error the test expects (`X expected but nothing was raised.`), the pool leaves out an edit that only changes the arguments of a `raise` (`edits_only_raise` in `--json`): that method sits below every caller, but head no longer reaches it, and the cause is usually in the code that called it. When only the id would order the best two, the row names no pick: the line reads `related guess: none; the test reaches 7 changed methods equally and no frame of head's failure is in one: Book#apply_catalog_record (m02), LoanRules.for (m03), ...`, and it ends with a next step: find what makes head's value in those methods, or in a changed file that holds no method, such as a view, from the `git diff --stat BASE HEAD` it prints. The row shows `-` in place of the id, and `method_id` is `null`. Such a row gets `touches:changed_method` when the test names any method of `method_ids`, not only the one with the lowest id. `message_varies` means head failed the same way on each rerun with a message that moved, and base passed on each rerun; compare a field the test controls rather than the whole record. `related:dynamic` means no changed method is related, but the test calls a method from `dynamic_callers:`; the row's next line says so, as in `related: none, Emit.looked_up sends render to a receiver the source does not name`.
|
|
355
356
|
|
|
356
357
|
## How it filters
|
|
357
358
|
|
|
358
|
-
`context` diffs the two revisions and maps hunks onto methods with Prism. Changed methods are `m01`, `m02`, and so on. When `Gemfile.lock` changed, a `dependencies:` block lists each gem as `gem old -> new`. A changed constant is `k01`. A class or module that only one side defines anywhere is a constant too, `added` or `removed`, shown by its header line. A method that did not itself change but reads that constant is `affected` (`via=k01`). A method that leaves one file while head defines the same name in another file, test files included, is `moved`, shown at its new place with `moved_from=` naming the old file; when several files define it, one the diff touched comes first. A constant that moves the same way with the same value is `moved` with `moved_from=` too, and a method that reads it is not `affected`. A constant that moved and took a new value is `modified`, with `moved_from=`. A reader or writer from `attr_reader`, `attr_writer`, `attr_accessor`, or a `Struct.new` or `Data.define` member list is a method, with the one-name form (`attr_reader :seed`) as its source, so a name dropped from such a line is `removed` and the names left on it do not change. A list given as `*FIELDS` counts when the same file set `FIELDS` to a list of symbols before it, as in `FIELDS = %i[x y].freeze`; each member then takes the line of its symbol in that list. Three Active Record macros of a class body define methods too. `scope :recent, -> { ... }` (or `scope(:recent) { ... }`) is the class method `Post.recent`, with the whole statement as its source, and the calls in its body are calls from `Post.recent`. `delegate :name, to: :author` is `Post#name` (`Post#author_name` with `prefix: true`), with the one-name form and the options as its source, so a change to `to:` changes every name on the line. `enum :status, { draft: 0, live: 1 }` gives each value `Post#draft?`, `Post#draft!`, `Post.draft`, and `Post.not_draft`, with `prefix:`, `suffix:`, `scopes: false`, and `instance_methods: false` applied, and its source is that one value (`enum :status, draft: 0`), so changing one value changes only its four methods. The `enum status: {...}` form that Rails 8 removed is not read. A Grape endpoint is a method too: in a class whose superclass is `Grape::API` (or a name that ends in `API` or `Api`, as an app's own base class does), `post "/refresh" do ... end` inside `resource :things do` is `Things#POST /things/refresh`. Its path joins the class's `prefix` and the segments of `resource`, `resources`, `namespace`, `group`, `segment`, and `route_param :id` (`:id`). Its source is the block, with the `desc` and `params` statements just before it, and the calls in the block are calls from the endpoint, so a method it calls shows it under `reachable_from`. A request a test sends, as `post "/api/things/7/refresh"`, names an endpoint with the same verb whose segments end the path, with a `:param` matching any one segment and any segments before them taken as a mount point. A path that interpolates a string constant the file assigns, in the API class (`namespace VERSION`, `post "#{ACTION}/now"`) or in the test (`post "#{API_ROOT}/things"`), reads as that string. In a test, any other interpolation matches any one segment (`"/things/#{thing.id}/refresh"`), or any mount point when the path starts with it (`"#{api_root}/things"`). The `desc` and `params` before a `route_param` or another path block belong to the first endpoint inside it. Such a test is listed under `tests:` and `entry_point_tests:`, and a candidate that sends it gets `touches:changed_method`. An endpoint whose path the API class computes any other way keeps that part as its source in braces, as `Books#GET /books/{Settings.legacy}/list`, matches no request, and gets a `warning:` that says so with its file and line. In a Rails app, `context` and `run` read the routes `config/routes.rb` draws (`resources`, `resource`, `namespace`, `scope` with `module:`, `as:`, or `controller:`, `member`, `collection`, `on:`, `root`, and a verb with `to:`, `=>`, or `controller:` and `action:`), so a request a candidate sends to a path helper (`post things_path`, `patch thing_url(thing)`) or to a path names the controller action that serves it, as `ThingsController#create`. Helpers and controllers take the names Rails gives them with Active Support's inflections and the literal `irregular`, `uncountable`, `singular`, `plural`, and `acronym` calls in `config/initializers/inflections.rb`: `resources :people` draws `person_path`, `resources :news` draws `news_index_path` and `news_path`, and `inflect.acronym "API"` makes `resources :api_keys` reach `APIKeysController`. A rule that file builds at run time is not read. A changed controller action that the routes draw lists under `tests:` each test file whose requests reach it, and each controller test that names the controller and calls the action as `post :create`, with what it sends (`spec/requests/things_spec.rb via POST things_path`); a grep of the action's name matched `create(:thing)` and missed `post things_path`. An action no route draws keeps the grep. A helper drawn any other way, as by a `draw` file, a loop, or a concern, is not read: `run` prints `warning: 1 candidate request names a path helper config/routes.rb does not draw (report_path), so it is tied to no controller action.` A `def` or reader inside a conditional of the class body (`attr_reader :x if cond`) counts. The branches read as if each ran in order, so a bare `private` in a branch carries past the conditional. A definition in a Concern's `included do`, `prepended do`, or `class_methods do` block counts too. Each method block has the old and new source, call sites in the library, the entry points that reach it (`reachable_from`), and existing tests that mention it. A changed `initialize` also shows `built_by:`, the calls besides `new` that run it: `create`, `create!`, and `build` on an Active Record model, and `build(:loan)` and `create(:loan)` for each FactoryBot factory that builds the class. Its `tests:` line lists each test file that calls one of those or `Owner.new`, with the calls it makes (`spec/models/loan_spec.rb via create(:loan)`). No test calls `initialize` by that name, so a constructor gets no grep hits. Factories are read from `spec/factories` and `test/factories` without loading them: the class comes from `class:` (a constant, a string, or a symbol), then from the parent (`parent:`, or the factory it nests in), then from the camelized name, and an alias in `aliases:` builds the same class. A factory also counts for each class its associations build, through their own associations too: a bare `borrower` that names a factory, `association :borrower` (or `factory: :user`), or an attribute block that calls `association`, in the factory or one it inherits from, so `create(:loan)` saves a `Borrower` too. An association in a trait counts only for a call that names the trait: a trait `:cosigned` of `:loan` that builds a cosigner adds `create(:loan, :cosigned)` and `build(:loan, :cosigned)`, not `create(:loan)`, and a test that writes `create(:loan, :cosigned, amount_cents: 5)` mentions both `create(:loan)` and `create(:loan, :cosigned)`. A factory takes the traits of its parent and of the factory it nests in. A trait defined outside a factory, and a trait that includes another trait, are not followed. A factory whose class the source does not settle is not followed and is listed under `factories_skipped:` with the reason: a `class:` computed at run time, a parent that is not a factory read there, or a class that no file defines, which an inflection the app configures can cause (`ApiKey` against `APIKey`). A factory that builds a Ruby value rather than a class of the app (`class: Hash`, as for the shape of an API response, or `Array`, `Data`, `Object`, `OpenStruct`, `Set`, `String`, `Struct`) is left out and not listed. A method that an Active Record model's class body names in a callback or a validation (`after_save :count_save`, `validate :check`, or an `if:` or `unless:` of either, as in `validates :amount, presence: true, if: :checked?`) shows `run_by:`, the calls that run it by Rails's rules, followed by the lines that name it. A save callback lists `Loan.create`, `Loan.create!`, `Loan#save`, `Loan#save!`, `Loan#update`, and `Loan#update!` (a create callback leaves out the updates, an update callback the creates), with `create(:loan)` and `build(:loan)` for each factory of the class. A validation adds `Loan#valid?`, `Loan#invalid?`, `Loan#validate`, and `Loan#validate!`. An `on:` on a validation or on a commit or rollback callback keeps only the calls of that context, and the line shows it: `before_validation :mark_pending, on: :create` lists no update, so a method that calls `update!` does not reach it, and `on: :update` lists no create and no factory. A context of the app's own (`on: :publish`) runs only from a call that takes one, so it lists `Loan#save`, `Loan#save!`, and the four validation calls. An `on:` that is not written as symbols narrows nothing. A destroy callback lists `Loan#destroy` and `Loan#destroy!`. A method that such a callback reaches (`reachable_from`) gets the same line. A method outside the tests that makes one of those calls, as the source writes it (`Loan.create!` in `LoanDesk.open`, or `update!` inside a method of `Loan`), joins `reachable_from` with its own callers, so a test that calls `LoanDesk.open` names the callback. A call on a local or an instance variable the source does not type, such as a parameter, is read as a call on the model Rails names it after, so `loan.save!` in `Teller#renew(loan)` reaches a save callback of `Loan`, and `record.save!` reaches none. Its `tests:` lists each test file that makes one of those calls (`spec/models/loan_spec.rb via Loan.create!`), next to the grep hits, and `run` counts the call as a mention of the method. `Loan#save` counts when the test's source settles the record's class (`loan = Loan.new`). `build(:loan)` does not save, so it counts only in a test that also makes one of the record calls the callback lists, as in `build(:loan).save!` or `expect(build(:loan)).to be_valid`, where `be_valid` calls `valid?`. That call's receiver is not checked, since a local the source does not type holds the built record, so a test that saves another record next to `build(:loan)` counts too. A test that only builds gets no `tests:` hit and no mention in `run`. A callback line in a Concern, one a parent class declares, and a call on a subclass are not followed. When no test mentions it, `entry_point_tests` lists the test files that name the class of a public entry point in `reachable_from` and call a method of that entry point's name. `--coverage` uses the line lists in the JSON file instead of the grep. The brief ends with one `entrypoints <Const>:` section for each class or module that holds a changed method. It lists that class's public methods on head (on base when head has none), changed ones first with their change kind, then the rest as `unchanged`, at most 12, with the ones that reach a changed method first and `(+N more)` for the methods left out. A class that defines `initialize` on either side shows `Owner.new` with the role of `Owner#initialize` in its place, unless the class makes `new` private (`private_class_method :new`, or `private :new` inside `class << self`). These are the methods a test can call, and `run` counts their names as mentions.
|
|
359
|
+
`context` diffs the two revisions and maps hunks onto methods with Prism. Changed methods are `m01`, `m02`, and so on. An executable under `exe/` or `bin/` with no suffix and `ruby` on its `#!` line is read too: its code outside a `def`, a class, or a module is one method, `exe/tool <main>`, so a change to its usage text is listed, its `tests:` are the test files that write its path, and a test that starts it mentions it. When `Gemfile.lock` changed, a `dependencies:` block lists each gem as `gem old -> new`. A changed constant is `k01`. A class or module that only one side defines anywhere is a constant too, `added` or `removed`, shown by its header line. A method that did not itself change but reads that constant is `affected` (`via=k01`). A method that leaves one file while head defines the same name in another file, test files included, is `moved`, shown at its new place with `moved_from=` naming the old file; when several files define it, one the diff touched comes first. A constant that moves the same way with the same value is `moved` with `moved_from=` too, and a method that reads it is not `affected`. A constant that moved and took a new value is `modified`, with `moved_from=`. A reader or writer from `attr_reader`, `attr_writer`, `attr_accessor`, or a `Struct.new` or `Data.define` member list is a method, with the one-name form (`attr_reader :seed`) as its source, so a name dropped from such a line is `removed` and the names left on it do not change. A list given as `*FIELDS` counts when the same file set `FIELDS` to a list of symbols before it, as in `FIELDS = %i[x y].freeze`; each member then takes the line of its symbol in that list. Three Active Record macros of a class body define methods too. `scope :recent, -> { ... }` (or `scope(:recent) { ... }`) is the class method `Post.recent`, with the whole statement as its source, and the calls in its body are calls from `Post.recent`. `delegate :name, to: :author` is `Post#name` (`Post#author_name` with `prefix: true`), with the one-name form and the options as its source, so a change to `to:` changes every name on the line. `enum :status, { draft: 0, live: 1 }` gives each value `Post#draft?`, `Post#draft!`, `Post.draft`, and `Post.not_draft`, with `prefix:`, `suffix:`, `scopes: false`, and `instance_methods: false` applied, and its source is that one value (`enum :status, draft: 0`), so changing one value changes only its four methods. The `enum status: {...}` form that Rails 8 removed is not read. A Grape endpoint is a method too: in a class whose superclass is `Grape::API` (or a name that ends in `API` or `Api`, as an app's own base class does), `post "/refresh" do ... end` inside `resource :things do` is `Things#POST /things/refresh`. Its path joins the class's `prefix` and the segments of `resource`, `resources`, `namespace`, `group`, `segment`, and `route_param :id` (`:id`). Its source is the block, with the `desc` and `params` statements just before it, and the calls in the block are calls from the endpoint, so a method it calls shows it under `reachable_from`. A request a test sends, as `post "/api/things/7/refresh"`, names an endpoint with the same verb whose segments end the path, with a `:param` matching any one segment and any segments before them taken as a mount point. A path that interpolates a string constant the file assigns, in the API class (`namespace VERSION`, `post "#{ACTION}/now"`) or in the test (`post "#{API_ROOT}/things"`), reads as that string. In a test, any other interpolation matches any one segment (`"/things/#{thing.id}/refresh"`), or any mount point when the path starts with it (`"#{api_root}/things"`). The `desc` and `params` before a `route_param` or another path block belong to the first endpoint inside it. Such a test is listed under `tests:` and `entry_point_tests:`, and a candidate that sends it gets `touches:changed_method`. An endpoint whose path the API class computes any other way keeps that part as its source in braces, as `Books#GET /books/{Settings.legacy}/list`, matches no request, and gets a `warning:` that says so with its file and line. In a Rails app, `context` and `run` read the routes `config/routes.rb` draws (`resources`, `resource`, `namespace`, `scope` with `module:`, `as:`, or `controller:`, `member`, `collection`, `on:`, `root`, and a verb with `to:`, `=>`, or `controller:` and `action:`), so a request a candidate sends to a path helper (`post things_path`, `patch thing_url(thing)`) or to a path names the controller action that serves it, as `ThingsController#create`. Helpers and controllers take the names Rails gives them with Active Support's inflections and the literal `irregular`, `uncountable`, `singular`, `plural`, and `acronym` calls in `config/initializers/inflections.rb`: `resources :people` draws `person_path`, `resources :news` draws `news_index_path` and `news_path`, and `inflect.acronym "API"` makes `resources :api_keys` reach `APIKeysController`. A rule that file builds at run time is not read. A changed controller action that the routes draw lists under `tests:` each test file whose requests reach it, and each controller test that names the controller and calls the action as `post :create`, with what it sends (`spec/requests/things_spec.rb via POST things_path`); a grep of the action's name matched `create(:thing)` and missed `post things_path`. An action no route draws keeps the grep. A helper drawn any other way, as by a `draw` file, a loop, or a concern, is not read: `run` prints `warning: 1 candidate request names a path helper config/routes.rb does not draw (report_path), so it is tied to no controller action.` A `def` or reader inside a conditional of the class body (`attr_reader :x if cond`) counts. The branches read as if each ran in order, so a bare `private` in a branch carries past the conditional. A definition in a Concern's `included do`, `prepended do`, or `class_methods do` block counts too. Each method block has the old and new source, call sites in the library, the entry points that reach it (`reachable_from`), and existing tests that mention it. A changed `initialize` also shows `built_by:`, the calls besides `new` that run it: `create`, `create!`, and `build` on an Active Record model, and `build(:loan)` and `create(:loan)` for each FactoryBot factory that builds the class. Its `tests:` line lists each test file that calls one of those or `Owner.new`, with the calls it makes (`spec/models/loan_spec.rb via create(:loan)`). No test calls `initialize` by that name, so a constructor gets no grep hits. A reader or member from `attr_reader` or `Data.define` (`Targets::Hit#code`) is most often a short word, so its grep hits are only the files that also name its class or the namespace around it (`Hit` or `Targets`). The code of a fixture project under `test/` or `spec/`, in its `app/`, `lib/`, `config/`, or `db/` (`test/fixture/depot/app/`), tests nothing and is not listed under `tests:`. Factories are read from `spec/factories` and `test/factories` without loading them: the class comes from `class:` (a constant, a string, or a symbol), then from the parent (`parent:`, or the factory it nests in), then from the camelized name, and an alias in `aliases:` builds the same class. A factory also counts for each class its associations build, through their own associations too: a bare `borrower` that names a factory, `association :borrower` (or `factory: :user`), or an attribute block that calls `association`, in the factory or one it inherits from, so `create(:loan)` saves a `Borrower` too. An association in a trait counts only for a call that names the trait: a trait `:cosigned` of `:loan` that builds a cosigner adds `create(:loan, :cosigned)` and `build(:loan, :cosigned)`, not `create(:loan)`, and a test that writes `create(:loan, :cosigned, amount_cents: 5)` mentions both `create(:loan)` and `create(:loan, :cosigned)`. A factory takes the traits of its parent and of the factory it nests in. A trait defined outside a factory, and a trait that includes another trait, are not followed. A factory whose class the source does not settle is not followed and is listed under `factories_skipped:` with the reason: a `class:` computed at run time, a parent that is not a factory read there, or a class that no file defines, which an inflection the app configures can cause (`ApiKey` against `APIKey`). A factory that builds a Ruby value rather than a class of the app (`class: Hash`, as for the shape of an API response, or `Array`, `Data`, `Object`, `OpenStruct`, `Set`, `String`, `Struct`) is left out and not listed. A method that an Active Record model's class body names in a callback or a validation (`after_save :count_save`, `validate :check`, or an `if:` or `unless:` of either, as in `validates :amount, presence: true, if: :checked?`) shows `run_by:`, the calls that run it by Rails's rules, followed by the lines that name it. A save callback lists `Loan.create`, `Loan.create!`, `Loan#save`, `Loan#save!`, `Loan#update`, and `Loan#update!` (a create callback leaves out the updates, an update callback the creates), with `create(:loan)` and `build(:loan)` for each factory of the class. A validation adds `Loan#valid?`, `Loan#invalid?`, `Loan#validate`, and `Loan#validate!`. An `on:` on a validation or on a commit or rollback callback keeps only the calls of that context, and the line shows it: `before_validation :mark_pending, on: :create` lists no update, so a method that calls `update!` does not reach it, and `on: :update` lists no create and no factory. A context of the app's own (`on: :publish`) runs only from a call that takes one, so it lists `Loan#save`, `Loan#save!`, and the four validation calls. An `on:` that is not written as symbols narrows nothing. A destroy callback lists `Loan#destroy` and `Loan#destroy!`. A method that such a callback reaches (`reachable_from`) gets the same line. A method outside the tests that makes one of those calls, as the source writes it (`Loan.create!` in `LoanDesk.open`, or `update!` inside a method of `Loan`), joins `reachable_from` with its own callers, so a test that calls `LoanDesk.open` names the callback. A call on a local or an instance variable the source does not type, such as a parameter, is read as a call on the model Rails names it after, so `loan.save!` in `Teller#renew(loan)` reaches a save callback of `Loan`, and `record.save!` reaches none. Its `tests:` lists each test file that makes one of those calls (`spec/models/loan_spec.rb via Loan.create!`), next to the grep hits, and `run` counts the call as a mention of the method. `Loan#save` counts when the test's source settles the record's class (`loan = Loan.new`). `build(:loan)` does not save, so it counts only in a test that also makes one of the record calls the callback lists, as in `build(:loan).save!` or `expect(build(:loan)).to be_valid`, where `be_valid` calls `valid?`. That call's receiver is not checked, since a local the source does not type holds the built record, so a test that saves another record next to `build(:loan)` counts too. A test that only builds gets no `tests:` hit and no mention in `run`. A callback line in a Concern, one a parent class declares, and a call on a subclass are not followed. When no test mentions it, `entry_point_tests` lists the test files that name the class of a public entry point in `reachable_from` and call a method of that entry point's name. `--coverage` uses the line lists in the JSON file instead of the grep. The brief ends with one `entrypoints <Const>:` section for each class or module that holds a changed method. It lists that class's public methods on head (on base when head has none), changed ones first with their change kind, then the rest as `unchanged`, at most 12, with the ones that reach a changed method first and `(+N more)` for the methods left out. A class that defines `initialize` on either side shows `Owner.new` with the role of `Owner#initialize` in its place, unless the class makes `new` private (`private_class_method :new`, or `private :new` inside `class << self`). These are the methods a test can call, and `run` counts their names as mentions.
|
|
359
360
|
|
|
360
361
|
`run` adds a git worktree for each revision, copies the scratch directory to `.jitcatch/` in that worktree, and runs each file in its own process:
|
|
361
362
|
|
|
@@ -470,7 +471,7 @@ readers: 1 for 1 changed value
|
|
|
470
471
|
not seen: receivers named unlike their model (`x = Branch.first; link_to x`); dispatch names built at run time from data outside the source (a DB column, a request param); routes drawn by `draw`, loops, or concerns; `to_param` reached through a gem's helper; serializers, jbuilder, and other template languages than ERB; callbacks and validations added by concerns or gems; config read through ENV or a YAML file at boot
|
|
471
472
|
```
|
|
472
473
|
|
|
473
|
-
A row is the place (`commit:path:line`, which `git show` takes), how it reads the value (`mechanism`), its score, `rank:` (why it scores what it does), `why:` (what the source says), `also:` (more places of the same kind for the same value), and `test:` (what a test would check). `note:` marks a row on a line the change itself edits. Rows go from the highest score down, at most 10 in all; the first line counts the rest, and `--all-readers` prints every one, in the text and in `--json`. A row that a method's `callsites:` already lists, or a plain read inside a method that a constant's `read_by:` already lists, is left out. `not seen:` names what the reader cannot see: it names a record's model by the name of its variable or method (`@branch`, `home_branch`, `branches`), so `x = Branch.first; link_to x` is not listed.
|
|
474
|
+
A row is the place (`commit:path:line`, which `git show` takes), how it reads the value (`mechanism`), its score, `rank:` (why it scores what it does), `why:` (what the source says), `also:` (more places of the same kind for the same value), and `test:` (what a test would check). `note:` marks a row on a line the change itself edits. Rows go from the highest score down, at most 10 in all; the first line counts the rest, and `--all-readers` prints every one, in the text and in `--json`. A row that a method's `callsites:` already lists, or a plain read inside a method that a constant's `read_by:` already lists, is left out. `not seen:` names what the reader cannot see: it names a record's model by the name of its variable or method (`@branch`, `home_branch`, `branches`), so `x = Branch.first; link_to x` is not listed. A history with no `app/`, `config/routes.rb`, or `config/application.rb`, such as a gem's, has no models, routes, or templates, so there it names only the two that any Ruby hides.
|
|
474
475
|
|
|
475
476
|
## Install
|
|
476
477
|
|
|
@@ -497,7 +498,7 @@ Without the CLI, copy or symlink `skills/jitcatch` into the agent's skills direc
|
|
|
497
498
|
|
|
498
499
|
## Limits
|
|
499
500
|
|
|
500
|
-
- A call site counts only when the source names its receiver's class: a call on `self` (or with no receiver), on a constant (`Cart.build`), on `self.class`, on `new` of one of those, on an instance variable that the class assigns only from one `Const.new`, or on a local variable that every assignment in its method sets from the same `Const.new` (`repo = Repo.new; repo.read`). A value the source does not type goes to the class Rails names it after, when that class defines the method: a parameter or a block parameter (`loan.approve` in `Gate#check(loan)`), a local that no assignment sets from a `Const.new`, and an instance variable that none sets (`@loan = loan`), all as a `Loan`. A reader that `belongs_to :borrower` or `has_one :borrower` declares reads a `Borrower`, or the class its `class_name:` string names, so `borrower.approved_cents` inside `Loan#approve` calls `Borrower#approved_cents`; a polymorphic one reads no class. A name that disagrees with a `Const.new` the method or class assigns to it names no class. A call on what a method of the same class returns, as in `emitter_for(format).render`, goes to each constant that method can return, when it returns only constants and nil (through `case`, `if`, and the ternary). A call on a local assigned two different constants, on a name that names no class, or on any other return value makes no edge, so `reachable_from` can miss a caller. Such a call is listed under `dynamic_callers:` of each changed method whose name, or the name of a method in its `reachable_from`, it sends, as `Emit.looked_up (render)`. An instance method of a class whose name ends in `Controller` is left out of those names, since the router calls a controller action and no code in the app sends its name, so `cipher.update` is not listed when `ThingsController#update` reaches the change. A call on `self` that its class does not define goes to the one indexed method of that name, for a method from a module or a superclass. From an instance method, only an instance method counts. A name the class declares with `attr_reader`, `attr_writer`, or `attr_accessor` goes to that reader or writer. `Const.new`, and `new` called with no receiver in a class method, also goes to that class's `initialize`, or to the nearest one up its superclass chain, so the callers of `Runner.new` reach `Runner#initialize` and what it calls. A superclass is looked up from outside the class it heads, as Ruby does, so `class Error < Error` inside `module Shop` names the outer `Error`. `reachable_from` keeps callers up to four calls away, and the hop from `new` into `initialize` is not one of the four. It keeps the eight nearest. `reached_by` in `--json` holds every caller at any depth, with no cap (`caller_depths` maps each to the fewest calls between them), and `run` reads it for the mentions and the related id, so a test that enters high, such as a request spec through a controller and a service or a CLI test that calls `CLI.start`, still names a changed method five or more calls down. Such a test reaches most changed methods, so its row is usually `related:guess` with that method in the list. `create`, `create!`, and `build` go there too when the class's superclass chain in the repository reaches `ActiveRecord::Base` (`class Post < ApplicationRecord`), since Active Record defines them to call `new`. On any other class they make no edge, since the code that defines them is not in the index. A class method of the same name up the chain takes the call instead.
|
|
501
|
+
- A call site counts only when the source names its receiver's class: a call on `self` (or with no receiver), on a constant (`Cart.build`), on `self.class`, on `new` of one of those, on an instance variable that the class assigns only from one `Const.new`, or on a local variable that every assignment in its method sets from the same `Const.new` (`repo = Repo.new; repo.read`). A value the source does not type goes to the class Rails names it after, when that class defines the method: a parameter or a block parameter (`loan.approve` in `Gate#check(loan)`), a local that no assignment sets from a `Const.new`, and an instance variable that none sets (`@loan = loan`), all as a `Loan`. A reader that `belongs_to :borrower` or `has_one :borrower` declares reads a `Borrower`, or the class its `class_name:` string names, so `borrower.approved_cents` inside `Loan#approve` calls `Borrower#approved_cents`; a polymorphic one reads no class. A name that disagrees with a `Const.new` the method or class assigns to it names no class. A call on what a method of the same class returns, as in `emitter_for(format).render`, goes to each constant that method can return, when it returns only constants and nil (through `case`, `if`, and the ternary). A call on a local assigned two different constants, on a name that names no class, or on any other return value makes no edge, so `reachable_from` can miss a caller. Such a call is listed under `dynamic_callers:` of each changed method whose name, or the name of a method in its `reachable_from`, it sends, as `Emit.looked_up (render)`. An instance method of a class whose name ends in `Controller` is left out of those names, since the router calls a controller action and no code in the app sends its name, so `cipher.update` is not listed when `ThingsController#update` reaches the change. A call on a local that holds only a block is left out too: a `&block` parameter, or a local that every assignment in its method sets to `-> {}`, `lambda {}`, `proc {}`, or `Proc.new {}`. Its `call` or `.()` runs that block, never a method of the repository. A `call` on a parameter, such as `job.call` in `def sync(job)`, is still listed. A call on `self` that its class does not define goes to the one indexed method of that name, for a method from a module or a superclass. From an instance method, only an instance method counts. A name the class declares with `attr_reader`, `attr_writer`, or `attr_accessor` goes to that reader or writer. `Const.new`, and `new` called with no receiver in a class method, also goes to that class's `initialize`, or to the nearest one up its superclass chain, so the callers of `Runner.new` reach `Runner#initialize` and what it calls. A superclass is looked up from outside the class it heads, as Ruby does, so `class Error < Error` inside `module Shop` names the outer `Error`. `reachable_from` keeps callers up to four calls away, and the hop from `new` into `initialize` is not one of the four. It keeps the eight nearest. `reached_by` in `--json` holds every caller at any depth, with no cap (`caller_depths` maps each to the fewest calls between them), and `run` reads it for the mentions and the related id, so a test that enters high, such as a request spec through a controller and a service or a CLI test that calls `CLI.start`, still names a changed method five or more calls down. Such a test reaches most changed methods, so its row is usually `related:guess` with that method in the list. `create`, `create!`, and `build` go there too when the class's superclass chain in the repository reaches `ActiveRecord::Base` (`class Post < ApplicationRecord`), since Active Record defines them to call `new`. On any other class they make no edge, since the code that defines them is not in the index. A class method of the same name up the chain takes the call instead.
|
|
501
502
|
- `define_method` and `class_eval` are not indexed. A definition with a receiver other than `def self.foo` is coarse.
|
|
502
503
|
- A rename is a deletion plus an addition (`git diff --no-renames`).
|
|
503
504
|
- Coverage is not collected. `--coverage` only reads the JSON you pass.
|
|
@@ -505,7 +506,7 @@ Without the CLI, copy or symlink `skills/jitcatch` into the agent's skills direc
|
|
|
505
506
|
- If the candidate tests never state the old behavior, a regression in that gap stays at `reported` 0.
|
|
506
507
|
- A test of a new feature fails on base and is not a candidate.
|
|
507
508
|
- The related id can sit one step away from the line that caused the failure.
|
|
508
|
-
- A mention is a method call, a constant, or a symbol in that test method or in a method of the same file that it calls, directly or through other such methods. `Owner.new` mentions the `initialize` of the class whose name ends in `Owner`, and `mentions` lists it as the test writes it. `A::Owner.new` mentions only a class whose name ends in `A::Owner`, so it does not name `B::Owner#initialize`. A call in the changed constructor's `built_by:` mentions it the same way: `Loan.create!` on an Active Record model, or a FactoryBot call (`create(:loan)`, `build(:loan)`, `create_list(:loan, 2)`, `build_stubbed(:loan)`), which `mentions` lists as `create(:loan)` for a call that saves (`create`, `create_list`, `create_pair`) and as `build(:loan)` for one that does not. A file that defines its own method of that name calls that method, not FactoryBot, and `attributes_for` builds no object. A call sent to a class the test names, or to an instance the test builds with `.new`, is also listed with that class, as `Index.parse` or `Cart#total`. Among changed methods of one name, the related id is then the one on that class (or on a class whose name ends in it), unless none is, as for a method a superclass defines. A method name the test sends only to constants the repository does not define, as in `File.write(path, text)` or `Dir.mktmpdir`, is not a mention, so `File.write` does not name a changed method reached from `Emitter.write`. `emitter.send(:write)` sends `write` to `emitter`, so a test that writes both still mentions `write`. A `send`, `__send__`, or `public_send` with a literal name is read as a call of that name in the code too, so `send(:helper)` in a method reaches `helper`. Words in strings and comments are not mentions, except a method name passed as a plain string to `send`, `public_send`, `method`, or `respond_to?`, as in `send("total")`. A symbol that keys a Hash, as in `snap[:total]`, `snap.fetch(:total)`, or `{ total: 1 }`, is not a mention. A call to `message`, `full_message`, or `detailed_message` also mentions `to_s`, which an error class overrides to build its message. A string shaped like a relative path (`"exe/tool"`, `"../bin/x"`) that names a file of head, read from the repository root, from the candidate's directory, or as `File.expand_path` reads it next to the candidate, is a script the test may start in another process: the names in that file count as mentions too. A helper in another file is not read.
|
|
509
|
+
- A mention is a method call, a constant, or a symbol in that test method or in a method of the same file that it calls, directly or through other such methods. `Owner.new` mentions the `initialize` of the class whose name ends in `Owner`, and `mentions` lists it as the test writes it. `A::Owner.new` mentions only a class whose name ends in `A::Owner`, so it does not name `B::Owner#initialize`. A call in the changed constructor's `built_by:` mentions it the same way: `Loan.create!` on an Active Record model, or a FactoryBot call (`create(:loan)`, `build(:loan)`, `create_list(:loan, 2)`, `build_stubbed(:loan)`), which `mentions` lists as `create(:loan)` for a call that saves (`create`, `create_list`, `create_pair`) and as `build(:loan)` for one that does not. A file that defines its own method of that name calls that method, not FactoryBot, and `attributes_for` builds no object. A call sent to a class the test names, or to an instance the test builds with `.new`, is also listed with that class, as `Index.parse` or `Cart#total`. Among changed methods of one name, the related id is then the one on that class (or on a class whose name ends in it), unless none is, as for a method a superclass defines. A method name the test sends only to constants the repository does not define, as in `File.write(path, text)` or `Dir.mktmpdir`, is not a mention, so `File.write` does not name a changed method reached from `Emitter.write`. `emitter.send(:write)` sends `write` to `emitter`, so a test that writes both still mentions `write`. A `send`, `__send__`, or `public_send` with a literal name is read as a call of that name in the code too, so `send(:helper)` in a method reaches `helper`. Words in strings and comments are not mentions, except a method name passed as a plain string to `send`, `public_send`, `method`, or `respond_to?`, as in `send("total")`. A symbol that keys a Hash, as in `snap[:total]`, `snap.fetch(:total)`, or `{ total: 1 }`, is not a mention. A call to `message`, `full_message`, or `detailed_message` also mentions `to_s`, which an error class overrides to build its message. A string shaped like a relative path (`"exe/tool"`, `"../bin/x"`) that names a file of head, read from the repository root, from the candidate's directory, or as `File.expand_path` reads it next to the candidate, is a script the test may start in another process: the names in that file count as mentions too. A string that an assertion or a matcher compares directly, as `assert_includes error.backtrace.first, "lib/tool/model.rb"` or `include("lib/x.rb")`, is expected text and names no script. A helper in another file is not read.
|
|
509
510
|
- Files run one at a time unless you pass `--workers`. The base run and the head run of one file run at the same time, each in its own worktree, so a test that writes a fixed path outside the worktree or listens on a fixed port can collide with its other side, and with other files under `--workers`. Pass `--serial` for such files. The repository's existing suite is not run. The repository has to be able to add worktrees.
|
|
510
511
|
- An assertion backtrace usually stops in the test, so a library line is found from names the test mentions.
|
|
511
512
|
- An RSpec example's mentions come from its own block, the `let`, `subject`, and hook blocks of the groups around it, those groups' arguments, and the same-file helpers they call. A group described as `"#total"` or `".build"` names that method, and `described_class.new` names `Owner.new`. `described_class.parse` is listed as `Owner.parse`. A `let(:result)` or `subject(:result)` name is the spec's own helper, so neither it nor a bare `result` call mentions a changed method `result`. When its block builds one class, as `let(:cart) { Cart.new }` does, `cart.total` is listed as `Cart#total`. An implicit `subject`, a shared example group, and a helper in `spec/support` are not read; an example from a shared group defined in another file reads the whole candidate file.
|
data/lib/jitcatch/assessor.rb
CHANGED
|
@@ -314,7 +314,8 @@ module JitCatch
|
|
|
314
314
|
# Almost every test builds its object with `.new`, so a changed
|
|
315
315
|
# constructor is the related method only when nothing else points.
|
|
316
316
|
constructors, named = mentioned.partition { |item| mention(item).end_with?(".new") }
|
|
317
|
-
|
|
317
|
+
stopped = stopped_raising?(test)
|
|
318
|
+
return guess(named, {}, stopped) if named.any?
|
|
318
319
|
|
|
319
320
|
via_caller = @methods.select { |item|
|
|
320
321
|
upstream = reached_by(item).map { |name| bare(name) }
|
|
@@ -323,17 +324,32 @@ module JitCatch
|
|
|
323
324
|
sent = via_caller.select { |item| reached_by(item).any? { |name| sent_to?(name, mentions) } }
|
|
324
325
|
via_caller = sent unless sent.empty?
|
|
325
326
|
pool = via_caller.empty? ? constructors : via_caller
|
|
326
|
-
guess(pool, depth_below_entry(pool, mentions))
|
|
327
|
+
guess(pool, depth_below_entry(pool, mentions), stopped)
|
|
327
328
|
end
|
|
328
329
|
|
|
329
330
|
# The pick, the pool, and whether only the id ordered the best two.
|
|
330
331
|
# A tied pick still sets the tags, as it did when the id was shown.
|
|
331
|
-
|
|
332
|
+
#
|
|
333
|
+
# On a row where head stopped raising the error the test expects, the
|
|
334
|
+
# method that raises it sits below every caller, so the downstream rule
|
|
335
|
+
# picks it, but head no longer reaches it. A method whose edit only
|
|
336
|
+
# changes the arguments of a raise did not decide that, so it leaves
|
|
337
|
+
# the pool while another method is in it.
|
|
338
|
+
def guess(items, depths = {}, stopped = false)
|
|
339
|
+
deciders = items.reject { |item| item["edits_only_raise"] }
|
|
340
|
+
items = deciders if stopped && deciders.any?
|
|
332
341
|
ranked = items.sort_by { |item| guess_rank(item, items, depths) }
|
|
333
342
|
tied = ranked.size > 1 && guess_rank(ranked[0], items, depths)[0...-1] == guess_rank(ranked[1], items, depths)[0...-1]
|
|
334
343
|
[ranked.first, items, tied]
|
|
335
344
|
end
|
|
336
345
|
|
|
346
|
+
# Minitest's assert_raises and RSpec's raise_error both end their
|
|
347
|
+
# message this way when the block returned.
|
|
348
|
+
def stopped_raising?(test)
|
|
349
|
+
test.dig("base", "status") == "pass" && test.dig("head", "status") == "fail" &&
|
|
350
|
+
test.dig("head", "message").to_s.match?(/but nothing was raised\.?\z/)
|
|
351
|
+
end
|
|
352
|
+
|
|
337
353
|
# A test that names the class it sends a method to (`Index.parse`,
|
|
338
354
|
# `described_class.parse`) calls that class's method of the name, not
|
|
339
355
|
# another class's. When no changed method of the name is on a class
|
|
@@ -438,11 +454,18 @@ module JitCatch
|
|
|
438
454
|
# a request spec of a Rails app, the method farthest from the entry
|
|
439
455
|
# point the test calls was the one the failure came from in both, and
|
|
440
456
|
# the lowest id in one, so depth ranks before the count below.
|
|
457
|
+
#
|
|
458
|
+
# A script's top-level code (`exe/tool <main>`) is in the pool only
|
|
459
|
+
# when the test starts that script, and it runs on every start, while
|
|
460
|
+
# each method it calls runs only for the command that picks it. So it
|
|
461
|
+
# ranks before the downstream rule: a test of `exe/tool bogus` whose
|
|
462
|
+
# usage text changed reached none of the methods below it.
|
|
441
463
|
def guess_rank(item, items, depths)
|
|
442
464
|
others = items.reject { |other| other["id"] == item["id"] }
|
|
443
465
|
covered = others.count { |other| reached_by(item).include?(other["name"]) }
|
|
444
466
|
below = @methods.count { |other| reached_by(other).include?(item["name"]) }
|
|
445
|
-
|
|
467
|
+
started = item["kind"] == "script" ? 0 : 1
|
|
468
|
+
[preference(item), started, -covered, -depths.fetch(item["id"], 0), -below, item["id"].to_s]
|
|
446
469
|
end
|
|
447
470
|
|
|
448
471
|
# The fewest calls between each item and an entry point the test
|
data/lib/jitcatch/changes.rb
CHANGED
|
@@ -56,6 +56,7 @@ module JitCatch
|
|
|
56
56
|
base_edges = edges_for(base_idx)
|
|
57
57
|
decorate_calls(methods, head_edges, base_edges)
|
|
58
58
|
mark_forwarders(methods, head_edges, hunks)
|
|
59
|
+
mark_raise_edits(methods)
|
|
59
60
|
decorate_dynamic_callers(methods, head_idx, base_idx)
|
|
60
61
|
factories = factories_of(head_idx)
|
|
61
62
|
builders = builders_for(methods, head_idx, factories)
|
|
@@ -168,7 +169,7 @@ module JitCatch
|
|
|
168
169
|
map = {}
|
|
169
170
|
files.each do |file|
|
|
170
171
|
path = file["new_path"] || file["old_path"]
|
|
171
|
-
next unless path&.end_with?(".rb")
|
|
172
|
+
next unless path&.end_with?(".rb") || SourceIndex.script_path?(path.to_s)
|
|
172
173
|
|
|
173
174
|
map[path] = file
|
|
174
175
|
end
|
|
@@ -190,6 +191,9 @@ module JitCatch
|
|
|
190
191
|
template_calls = []
|
|
191
192
|
listed = @git.source_blobs(sha).select { |blob| blob.size <= MAX_BLOB }
|
|
192
193
|
@texts.merge!(@git.read_blobs(listed.map(&:oid).uniq - @texts.keys))
|
|
194
|
+
scripts, listed = listed.partition { |blob| SourceIndex.script_path?(blob.path) }
|
|
195
|
+
# An executable that is not Ruby, such as a shell `bin/setup`, is not read.
|
|
196
|
+
scripts.select! { |blob| SourceIndex.ruby_script?(@texts.fetch(blob.oid)) }
|
|
193
197
|
listed, views = listed.partition { |blob| blob.path.end_with?(".rb") }
|
|
194
198
|
# A template defines no method, so its calls only add call sites, each
|
|
195
199
|
# made from the template itself.
|
|
@@ -198,11 +202,11 @@ module JitCatch
|
|
|
198
202
|
index = (@parsed[[blob.path, blob.oid]] ||= SourceIndex.parse(Template.as_ruby(templates[blob.path]), file: blob.path))
|
|
199
203
|
template_calls.concat(index.calls.map { |call| call.with(enclosing: call.enclosing || blob.path) }) unless index.parse_error
|
|
200
204
|
end
|
|
201
|
-
listed.each do |blob|
|
|
205
|
+
(listed + scripts).sort_by(&:path).each do |blob|
|
|
202
206
|
file = blob.path
|
|
203
207
|
sources[file] = @texts.fetch(blob.oid)
|
|
204
208
|
# A file both sides share is parsed once.
|
|
205
|
-
index = (@parsed[[file, blob.oid]] ||= SourceIndex.parse(sources[file], file: file))
|
|
209
|
+
index = (@parsed[[file, blob.oid]] ||= SourceIndex.parse(sources[file], file: file, script: scripts.include?(blob)))
|
|
206
210
|
indexes[file] = index
|
|
207
211
|
if index.parse_error
|
|
208
212
|
errors << { "ref" => label, "file" => file, "message" => index.parse_error }
|
|
@@ -405,6 +409,36 @@ module JitCatch
|
|
|
405
409
|
end
|
|
406
410
|
end
|
|
407
411
|
|
|
412
|
+
# An edit that changes only the arguments of a `raise` changes what the
|
|
413
|
+
# method raises, not whether it does. A test whose expected error head
|
|
414
|
+
# stopped raising then points at the code that stopped calling it, so
|
|
415
|
+
# the assessor ranks such a method after the others.
|
|
416
|
+
def mark_raise_edits(methods)
|
|
417
|
+
methods.each do |method|
|
|
418
|
+
method["edits_only_raise"] = method["change"] == "modified" &&
|
|
419
|
+
method["old"].source != method["new"].source &&
|
|
420
|
+
without_raises(method["old"].source) == without_raises(method["new"].source)
|
|
421
|
+
end
|
|
422
|
+
end
|
|
423
|
+
|
|
424
|
+
# The source with each receiverless `raise` or `fail` call written as
|
|
425
|
+
# the bare word, so two sources that differ only there compare equal.
|
|
426
|
+
def without_raises(source)
|
|
427
|
+
text = source.b
|
|
428
|
+
pending = [Prism.parse(source).value]
|
|
429
|
+
spans = []
|
|
430
|
+
until pending.empty?
|
|
431
|
+
node = pending.pop
|
|
432
|
+
if node.is_a?(Prism::CallNode) && node.receiver.nil? && %i[raise fail].include?(node.name)
|
|
433
|
+
spans << [node.location.start_offset, node.location.end_offset]
|
|
434
|
+
else
|
|
435
|
+
pending.concat(node.compact_child_nodes)
|
|
436
|
+
end
|
|
437
|
+
end
|
|
438
|
+
spans.sort.reverse_each { |first, last| text = text.byteslice(0, first) + "raise".b + text.byteslice(last..) }
|
|
439
|
+
text
|
|
440
|
+
end
|
|
441
|
+
|
|
408
442
|
def forwards?(node, added, head_edges, hunks)
|
|
409
443
|
lines = []
|
|
410
444
|
removed = 0
|
|
@@ -463,10 +497,14 @@ module JitCatch
|
|
|
463
497
|
merge_hits(call_hits(test_sources.keys, ["#{node.qualifier}.new", *method["builders"]], "builder", base_idx, head_idx), [])
|
|
464
498
|
elsif node.kind == "endpoint"
|
|
465
499
|
request_hits(test_sources, node.bare_name)
|
|
500
|
+
elsif node.kind == "script"
|
|
501
|
+
# A test starts an executable by its path, as `exe/tool` or
|
|
502
|
+
# `../exe/tool`, not by a method name.
|
|
503
|
+
grep_hits(test_sources, node.file)
|
|
466
504
|
elsif controller_action?(node.qualified) && (routes ||= app_routes(head_idx)).drawn?(node.qualified)
|
|
467
505
|
route_hits(test_sources, node.qualified, requests ||= test_requests(test_sources, routes, base_idx, head_idx))
|
|
468
506
|
else
|
|
469
|
-
merge_hits(call_hits(test_sources.keys, method["run_by"], "callback", base_idx, head_idx), grep_hits(test_sources, node.bare_name))
|
|
507
|
+
merge_hits(call_hits(test_sources.keys, method["run_by"], "callback", base_idx, head_idx), grep_hits(test_sources, node.bare_name, owner_pattern(node)))
|
|
470
508
|
end
|
|
471
509
|
end
|
|
472
510
|
method["entry_point_tests"] = if @coverage || !method["existing_tests"].empty?
|
|
@@ -564,10 +602,17 @@ module JitCatch
|
|
|
564
602
|
"edited_by_change" => row.edited, "mechanism" => row.mechanism.to_s, "score" => row.score, "rank" => row.rank,
|
|
565
603
|
"why" => row.why, "also" => row.also, "test" => row.test }
|
|
566
604
|
},
|
|
567
|
-
"not_seen" => Readers::BLIND
|
|
605
|
+
"not_seen" => rails_shaped?(head_idx) ? Readers::BLIND : Readers::BLIND_OUTSIDE_RAILS
|
|
568
606
|
}
|
|
569
607
|
end
|
|
570
608
|
|
|
609
|
+
# Models, routes, and templates live in app/ and config/routes.rb, so a
|
|
610
|
+
# history with neither, as a gem's, hides none of what Rails would.
|
|
611
|
+
def rails_shaped?(idx)
|
|
612
|
+
paths = idx["sources"].keys + idx["templates"].keys
|
|
613
|
+
paths.any? { |path| path.start_with?("app/") || %w[config/application.rb config/routes.rb].include?(path) }
|
|
614
|
+
end
|
|
615
|
+
|
|
571
616
|
# The method or constant entry a value is, by its name and where it is
|
|
572
617
|
# declared. A setting, a callback, or a default has none.
|
|
573
618
|
def listed_item(value, methods, constants)
|
|
@@ -614,17 +659,32 @@ module JitCatch
|
|
|
614
659
|
}
|
|
615
660
|
end
|
|
616
661
|
|
|
617
|
-
|
|
662
|
+
# `owner`, when given, is a pattern the file must match too.
|
|
663
|
+
def grep_hits(test_sources, bare, owner = nil)
|
|
618
664
|
return [] if bare.nil? || bare.length < 3
|
|
619
665
|
|
|
620
666
|
pattern = /\b#{called(bare)}/
|
|
621
667
|
test_sources.filter_map { |file, source|
|
|
622
|
-
next unless source.match?(pattern)
|
|
668
|
+
next unless source.match?(pattern) && (owner.nil? || source.match?(owner))
|
|
623
669
|
|
|
624
670
|
{ "file" => file, "via" => "grep" }
|
|
625
671
|
}.sort_by { |hit| hit["file"] }.first(TEST_CAP)
|
|
626
672
|
end
|
|
627
673
|
|
|
674
|
+
# A reader or a member (`attr_reader :file`, `Data.define(:code, :line)`)
|
|
675
|
+
# is most often a common word, which a grep finds in files that never
|
|
676
|
+
# touch the class. Such a hit counts only in a file that also names the
|
|
677
|
+
# class or the namespace around it, as `Hit` or `Targets` for
|
|
678
|
+
# `Targets::Hit#code`.
|
|
679
|
+
MEMBER_MACRO = /\A(?:attr_reader|attr_writer|Struct\.new|Data\.define)\z/
|
|
680
|
+
|
|
681
|
+
def owner_pattern(node)
|
|
682
|
+
return nil unless MEMBER_MACRO.match?(node.macro.to_s)
|
|
683
|
+
|
|
684
|
+
words = node.qualifier.split("::").last(2)
|
|
685
|
+
words.empty? ? nil : /\b(?:#{words.map { |word| Regexp.escape(word) }.join("|")})\b/
|
|
686
|
+
end
|
|
687
|
+
|
|
628
688
|
# A factory that builds a class no file defines may need an inflection
|
|
629
689
|
# the app configures (`ApiKey` against `APIKey`), so it is not followed.
|
|
630
690
|
# One that builds a Ruby value, as `class: Hash` does for the shape of an
|
|
@@ -767,10 +827,27 @@ module JitCatch
|
|
|
767
827
|
end
|
|
768
828
|
|
|
769
829
|
def test_sources_for(base_idx, head_idx)
|
|
770
|
-
paths = (base_idx["sources"].keys | head_idx["sources"].keys).select { |path| test_file?(path) }.sort
|
|
830
|
+
paths = (base_idx["sources"].keys | head_idx["sources"].keys).select { |path| test_file?(path) && !fixture_code?(path) }.sort
|
|
771
831
|
paths.to_h { |path| [path, head_idx["sources"][path] || base_idx["sources"][path]] }
|
|
772
832
|
end
|
|
773
833
|
|
|
834
|
+
# A project kept as a fixture under test/ or spec/ (a small Rails app in
|
|
835
|
+
# `test/fixture/depot`) holds code of its own in app/, lib/, config/,
|
|
836
|
+
# and db/. That code tests nothing, so it is no test a `tests:` line
|
|
837
|
+
# can name; the fixture's own test and spec files still are.
|
|
838
|
+
FIXTURE_DIR = /\Afixtures?\z/
|
|
839
|
+
FIXTURE_CODE = %w[app lib config db].freeze
|
|
840
|
+
|
|
841
|
+
def fixture_code?(path)
|
|
842
|
+
parts = path.split("/")
|
|
843
|
+
at = parts.index { |part| FIXTURE_DIR.match?(part) }
|
|
844
|
+
return false unless at && parts.first(at).any? { |part| %w[test spec].include?(part) }
|
|
845
|
+
|
|
846
|
+
base = parts.last
|
|
847
|
+
inner = parts[(at + 1)...-1]
|
|
848
|
+
!(base.start_with?("test_") || base.end_with?("_test.rb", "_spec.rb")) && inner.any? { |part| FIXTURE_CODE.include?(part) }
|
|
849
|
+
end
|
|
850
|
+
|
|
774
851
|
def test_file?(path)
|
|
775
852
|
base = File.basename(path)
|
|
776
853
|
return false unless base.end_with?(".rb")
|
|
@@ -1000,6 +1077,7 @@ module JitCatch
|
|
|
1000
1077
|
"entry_point_tests" => item["entry_point_tests"],
|
|
1001
1078
|
"via_constant" => item["via_constant"],
|
|
1002
1079
|
"forwards_to_added" => item["forwards_to_added"],
|
|
1080
|
+
"edits_only_raise" => item["edits_only_raise"],
|
|
1003
1081
|
"moved_from" => item["moved_from"]
|
|
1004
1082
|
}
|
|
1005
1083
|
end
|
data/lib/jitcatch/git_repo.rb
CHANGED
|
@@ -83,7 +83,7 @@ module JitCatch
|
|
|
83
83
|
git("ls-tree", "-r", "-l", "-z", "--end-of-options", ref).split("\0").filter_map { |entry|
|
|
84
84
|
meta, path = entry.split("\t", 2)
|
|
85
85
|
mode, type, oid, size = meta.split
|
|
86
|
-
next unless type == "blob" && mode != SYMLINK_MODE && (path.end_with?(".rb") || Template.template?(path))
|
|
86
|
+
next unless type == "blob" && mode != SYMLINK_MODE && (path.end_with?(".rb") || Template.template?(path) || SourceIndex.script_path?(path))
|
|
87
87
|
|
|
88
88
|
Blob.new(path: path, oid: oid, size: size.to_i)
|
|
89
89
|
}.sort_by(&:path)
|
data/lib/jitcatch/readers.rb
CHANGED
|
@@ -46,6 +46,13 @@ module JitCatch
|
|
|
46
46
|
"config read through ENV or a YAML file at boot"
|
|
47
47
|
].freeze
|
|
48
48
|
|
|
49
|
+
# What a repository with no `config/application.rb` can still hide:
|
|
50
|
+
# models, routes, and templates are Rails's, and a gem has none.
|
|
51
|
+
BLIND_OUTSIDE_RAILS = [
|
|
52
|
+
"dispatch names built at run time from data outside the source (a file, an argument)",
|
|
53
|
+
"config read through ENV or a YAML file at load time"
|
|
54
|
+
].freeze
|
|
55
|
+
|
|
49
56
|
DISPATCH = %i[send public_send __send__ try try! method public_method respond_to? instance_method].freeze
|
|
50
57
|
SQL_CALLS = %i[where order reorder select joins group having pluck find_by_sql exec_query execute from not count_by_sql].freeze
|
|
51
58
|
QUERY_CALLS = %i[all find find_by find_by! where first last take count exists? pluck order includes find_each].freeze
|
data/lib/jitcatch/render.rb
CHANGED
|
@@ -347,7 +347,7 @@ module JitCatch
|
|
|
347
347
|
|
|
348
348
|
first, *rest = message_lines(base)
|
|
349
349
|
outcome = [base["status"], base["error_class"], first].reject { |part| part.to_s.empty? }
|
|
350
|
-
lines = [" base: #{outcome.join(" ")}", *rest.map { |line| " #{line}" }]
|
|
350
|
+
lines = [" base: #{outcome.join(" ")}", *rest.map { |line| " #{line}" }, *head_outcome_lines(test)]
|
|
351
351
|
return lines unless Assessor.no_table?(base)
|
|
352
352
|
|
|
353
353
|
databases = base_databases(doc)
|
|
@@ -355,6 +355,20 @@ module JitCatch
|
|
|
355
355
|
lines << " hint: base found no table, so this row checked nothing; load base's schema into #{into}"
|
|
356
356
|
end
|
|
357
357
|
|
|
358
|
+
# A test written to pin what a change fixes fails on base by design, and
|
|
359
|
+
# head's result is what it checks: `pass`, head's own failure, or that
|
|
360
|
+
# head failed as base did.
|
|
361
|
+
def head_outcome_lines(test)
|
|
362
|
+
head = test["head"]
|
|
363
|
+
return [] unless head
|
|
364
|
+
|
|
365
|
+
same = %w[status error_class message].all? { |key| head[key] == test["base"][key] }
|
|
366
|
+
return [" head: #{head["status"]}, the same as base"] if same && head["status"] != "pass"
|
|
367
|
+
|
|
368
|
+
first, *rest = message_lines(head)
|
|
369
|
+
[" head: #{[head["status"], head["error_class"], first].reject { |part| part.to_s.empty? }.join(" ")}", *rest.map { |line| " #{line}" }]
|
|
370
|
+
end
|
|
371
|
+
|
|
358
372
|
# Only the numbered databases have names the reader can load a schema
|
|
359
373
|
# into; the URL itself can hold a password, so it is never printed.
|
|
360
374
|
def untested_lines(doc)
|
data/lib/jitcatch/runner.rb
CHANGED
|
@@ -783,8 +783,12 @@ module JitCatch
|
|
|
783
783
|
[head_dir, File.dirname(candidate), candidate].map { |dir| File.expand_path(path, dir) }
|
|
784
784
|
.select { |full| full.start_with?(root) && !full.start_with?(own) && File.file?(full) }
|
|
785
785
|
}.uniq.flat_map { |full|
|
|
786
|
-
|
|
787
|
-
script
|
|
786
|
+
relative_path = full.delete_prefix(root)
|
|
787
|
+
script = SourceIndex.parse(read_text(full), file: relative_path)
|
|
788
|
+
# Starting an executable runs its top-level code, which the change
|
|
789
|
+
# document lists as one method when the diff edits it.
|
|
790
|
+
main = SourceIndex.script_path?(relative_path) ? [SourceIndex.script_name(relative_path)] : []
|
|
791
|
+
script.parse_error ? main : main + script.referenced_names("")
|
|
788
792
|
}
|
|
789
793
|
end
|
|
790
794
|
|
|
@@ -19,6 +19,9 @@ module JitCatch
|
|
|
19
19
|
MESSAGE_CALLS = %i[message full_message detailed_message].freeze
|
|
20
20
|
# A relative path with a directory part and no spaces: `exe/tool`.
|
|
21
21
|
PATH_LIKE = %r{\A(?:\.\.?/)*[\w.-]+(?:/[\w.-]+)+\z}
|
|
22
|
+
# Minitest assertions and expectations, RSpec matchers, and the string
|
|
23
|
+
# predicates a test asserts on.
|
|
24
|
+
COMPARING_CALL = /\A(?:assert|refute|must|wont)(?:_\w+)?\z|\A(?:eq|eql|equal|include|match|start_with|end_with|include\?|match\?|start_with\?|end_with\?|==|=~)\z/
|
|
22
25
|
ATTRIBUTE_MACROS = { attr_reader: [""], attr: [""], attr_writer: ["="], attr_accessor: ["", "="] }.freeze
|
|
23
26
|
# The constant a member list is given to, and whether its members get writers.
|
|
24
27
|
MEMBER_BUILDERS = { %i[Struct new] => true, %i[Data define] => false }.freeze
|
|
@@ -62,9 +65,13 @@ module JitCatch
|
|
|
62
65
|
# does. A local variable is an `instance` when every assignment to it in
|
|
63
66
|
# its method builds the same constant, and `named` when none builds
|
|
64
67
|
# one, as for a parameter. A reader that `belongs_to` or `has_one`
|
|
65
|
-
# declares is an `instance` of the model it names.
|
|
68
|
+
# declares is an `instance` of the model it names. A local that every
|
|
69
|
+
# assignment in its method sets to a lambda or a proc literal, or that
|
|
70
|
+
# `&block` binds, is a `proc`: its `call` runs that block, never a
|
|
71
|
+
# method the repository defines.
|
|
66
72
|
Receiver = Data.define(:kind, :const)
|
|
67
73
|
UNKNOWN_RECEIVER = Receiver.new(kind: "unknown", const: nil)
|
|
74
|
+
PROC_RECEIVER = Receiver.new(kind: "proc", const: nil)
|
|
68
75
|
# A block given to an RSpec call: a `group` (describe, context), an
|
|
69
76
|
# `example` (it, specify), or a `setup` (let, subject, before). Code in
|
|
70
77
|
# an example or a setup block is referred to under `key`, and so are a
|
|
@@ -153,8 +160,27 @@ module JitCatch
|
|
|
153
160
|
@new_visibility.filter_map { |qualifier, visibility| qualifier unless visibility == "public" }
|
|
154
161
|
end
|
|
155
162
|
|
|
156
|
-
|
|
157
|
-
|
|
163
|
+
# `script: true` reads the file as an executable, such as `exe/tool`:
|
|
164
|
+
# its top-level code is one method of kind `script`, named by
|
|
165
|
+
# `script_name`, whose calls are that method's.
|
|
166
|
+
def self.parse(source, file:, script: false)
|
|
167
|
+
new(source, file, script).parse
|
|
168
|
+
end
|
|
169
|
+
|
|
170
|
+
# The executables a gem or an app ships, which carry no `.rb` suffix.
|
|
171
|
+
SCRIPT_PATH = %r{\A(?:exe|bin)/[\w-]+\z}
|
|
172
|
+
|
|
173
|
+
def self.script_path?(path)
|
|
174
|
+
SCRIPT_PATH.match?(path)
|
|
175
|
+
end
|
|
176
|
+
|
|
177
|
+
# A file whose first line runs Ruby.
|
|
178
|
+
def self.ruby_script?(text)
|
|
179
|
+
text.to_s.lines.first.to_s.match?(/\A#!.*\bruby\b/)
|
|
180
|
+
end
|
|
181
|
+
|
|
182
|
+
def self.script_name(file)
|
|
183
|
+
"#{file} <main>"
|
|
158
184
|
end
|
|
159
185
|
|
|
160
186
|
# Every require_relative of a source, wherever it is written. Only a
|
|
@@ -180,7 +206,8 @@ module JitCatch
|
|
|
180
206
|
parts.all?(Prism::StringNode) && !parts.empty? ? File.join(*parts.map(&:unescaped)) : nil
|
|
181
207
|
end
|
|
182
208
|
|
|
183
|
-
def initialize(source, file)
|
|
209
|
+
def initialize(source, file, script = false)
|
|
210
|
+
@script = script
|
|
184
211
|
@source = source.to_s.gsub("\r\n", "\n")
|
|
185
212
|
@lines = @source.lines
|
|
186
213
|
@file = file.tr("\\", "/")
|
|
@@ -196,6 +223,7 @@ module JitCatch
|
|
|
196
223
|
@references = Hash.new { |hash, key| hash[key] = [] }
|
|
197
224
|
@ivar_classes = Hash.new { |hash, key| hash[key] = [] }
|
|
198
225
|
@local_classes = Hash.new { |hash, key| hash[key] = [] }
|
|
226
|
+
@local_procs = Hash.new { |hash, key| hash[key] = [] }
|
|
199
227
|
@helper_classes = Hash.new { |hash, key| hash[key] = [] }
|
|
200
228
|
@parse_error = nil
|
|
201
229
|
@spec_blocks = []
|
|
@@ -205,6 +233,7 @@ module JitCatch
|
|
|
205
233
|
@association_classes = {}
|
|
206
234
|
@symbol_lists = {}
|
|
207
235
|
@paths = Hash.new { |hash, key| hash[key] = [] }
|
|
236
|
+
@compared = {}
|
|
208
237
|
@factory_calls = Hash.new { |hash, key| hash[key] = [] }
|
|
209
238
|
@endpoints = {}
|
|
210
239
|
end
|
|
@@ -218,8 +247,9 @@ module JitCatch
|
|
|
218
247
|
|
|
219
248
|
@string_constants = string_constants(result.value)
|
|
220
249
|
scan_body(result.value, "", :public, false)
|
|
250
|
+
main = @script ? record_script(result.value) : nil
|
|
221
251
|
@spec_helpers = spec_helper_names(result.value)
|
|
222
|
-
walk(result.value, "", false,
|
|
252
|
+
walk(result.value, "", false, main)
|
|
223
253
|
type_local_variables
|
|
224
254
|
type_instance_variables
|
|
225
255
|
type_spec_helpers
|
|
@@ -622,6 +652,25 @@ module JitCatch
|
|
|
622
652
|
)
|
|
623
653
|
end
|
|
624
654
|
|
|
655
|
+
# The statements of an executable outside its `def`, class, and module
|
|
656
|
+
# bodies, as one method: a change to its usage text or to the `case`
|
|
657
|
+
# that picks a command is a change to it, and an edit inside a `def`
|
|
658
|
+
# is not. Its source holds only those statements, so `changes` sees
|
|
659
|
+
# an edit to them by the source alone.
|
|
660
|
+
def record_script(program)
|
|
661
|
+
top = statements(program).reject { |stmt| [Prism::DefNode, Prism::ClassNode, Prism::ModuleNode].any? { |kind| stmt.is_a?(kind) } }
|
|
662
|
+
name = self.class.script_name(@file)
|
|
663
|
+
return name if top.empty?
|
|
664
|
+
|
|
665
|
+
@methods << self.class::Method.new(
|
|
666
|
+
file: @file, bare_name: name, qualifier: "", qualified: name, kind: "script", visibility: "public",
|
|
667
|
+
start_line: top.first.location.start_line, end_line: top.last.location.end_line,
|
|
668
|
+
source: top.map { |stmt| extract_lines(stmt.location.start_line, stmt.location.end_line) }.join("\n"),
|
|
669
|
+
params: "", macro: "script"
|
|
670
|
+
)
|
|
671
|
+
name
|
|
672
|
+
end
|
|
673
|
+
|
|
625
674
|
# A string, a lambda, or a block runs code the line does not name, so
|
|
626
675
|
# only symbols count.
|
|
627
676
|
def record_callbacks(stmt, qualifier)
|
|
@@ -984,6 +1033,9 @@ module JitCatch
|
|
|
984
1033
|
@calls.map! do |call|
|
|
985
1034
|
next call unless call.receiver.kind == "named"
|
|
986
1035
|
|
|
1036
|
+
procs = @local_procs.fetch([call.enclosing, call.receiver.const], [])
|
|
1037
|
+
next call.with(receiver: PROC_RECEIVER) if procs.any? && procs.all?
|
|
1038
|
+
|
|
987
1039
|
assigned = @local_classes.fetch([call.enclosing, call.receiver.const], [])
|
|
988
1040
|
next call if assigned.all?(&:empty?)
|
|
989
1041
|
|
|
@@ -1061,6 +1113,18 @@ module JitCatch
|
|
|
1061
1113
|
|
|
1062
1114
|
# The constants a value builds with `Const.new`, through `a || b`,
|
|
1063
1115
|
# `c ? a : b`, and parentheses.
|
|
1116
|
+
# `-> {}`, `lambda {}`, `proc {}`, and `Proc.new {}` make a Proc.
|
|
1117
|
+
def proc_literal?(node)
|
|
1118
|
+
return true if node.is_a?(Prism::LambdaNode)
|
|
1119
|
+
return false unless node.is_a?(Prism::CallNode) && node.block.is_a?(Prism::BlockNode)
|
|
1120
|
+
|
|
1121
|
+
if node.receiver.nil?
|
|
1122
|
+
%i[lambda proc].include?(node.name)
|
|
1123
|
+
else
|
|
1124
|
+
node.name == :new && node.receiver.is_a?(Prism::ConstantReadNode) && node.receiver.name == :Proc
|
|
1125
|
+
end
|
|
1126
|
+
end
|
|
1127
|
+
|
|
1064
1128
|
def built_constants(node)
|
|
1065
1129
|
case node
|
|
1066
1130
|
when Prism::CallNode
|
|
@@ -1167,6 +1231,7 @@ module JitCatch
|
|
|
1167
1231
|
refer(enclosing, request) if request
|
|
1168
1232
|
factory = factory_built(node)
|
|
1169
1233
|
@factory_calls[enclosing] << [node.name.to_s, factory, factory_traits(node)] if factory
|
|
1234
|
+
mark_compared_strings(node)
|
|
1170
1235
|
end
|
|
1171
1236
|
node.compact_child_nodes.each do |child|
|
|
1172
1237
|
children = child.is_a?(Prism::ArgumentsNode) && HASH_KEY_CALLS.include?(node.name) ? child.arguments.grep_v(Prism::SymbolNode) : [child]
|
|
@@ -1186,12 +1251,15 @@ module JitCatch
|
|
|
1186
1251
|
refer(enclosing, node.unescaped)
|
|
1187
1252
|
when Prism::StringNode
|
|
1188
1253
|
text = node.unescaped
|
|
1189
|
-
@paths[enclosing] << text if text.valid_encoding? && text.match?(PATH_LIKE)
|
|
1254
|
+
@paths[enclosing] << text if text.valid_encoding? && text.match?(PATH_LIKE) && !@compared.key?(node.location.start_offset)
|
|
1190
1255
|
when Prism::LocalVariableWriteNode
|
|
1191
1256
|
@local_classes[[enclosing, node.name.to_s]] << built_constants(node.value).uniq
|
|
1257
|
+
@local_procs[[enclosing, node.name.to_s]] << proc_literal?(node.value)
|
|
1192
1258
|
node.compact_child_nodes.each { |child| walk(child, qualifier, singleton_context, enclosing) }
|
|
1193
1259
|
when *LOCAL_BINDINGS
|
|
1194
1260
|
@local_classes[[enclosing, node.name.to_s]] << [] if node.name
|
|
1261
|
+
# `&block` binds the block given, which is a Proc.
|
|
1262
|
+
@local_procs[[enclosing, node.name.to_s]] << node.is_a?(Prism::BlockParameterNode) if node.name
|
|
1195
1263
|
node.compact_child_nodes.each { |child| walk(child, qualifier, singleton_context, enclosing) }
|
|
1196
1264
|
when Prism::InstanceVariableWriteNode, Prism::InstanceVariableOrWriteNode
|
|
1197
1265
|
@ivar_classes[[qualifier.to_s, node.name.to_s]].concat(built_constants(node.value))
|
|
@@ -1201,6 +1269,17 @@ module JitCatch
|
|
|
1201
1269
|
end
|
|
1202
1270
|
end
|
|
1203
1271
|
|
|
1272
|
+
# A string an assertion or a matcher compares with a value, as in
|
|
1273
|
+
# `assert_includes error.backtrace.first, "lib/tool/model.rb"`, is
|
|
1274
|
+
# text the test expects, not the path of a script it starts. Only a
|
|
1275
|
+
# direct argument counts, so `assert_equal "1\n", IO.popen(["ruby",
|
|
1276
|
+
# "exe/tool"], &:read)` still names `exe/tool`.
|
|
1277
|
+
def mark_compared_strings(node)
|
|
1278
|
+
return unless COMPARING_CALL.match?(node.name.to_s) && node.arguments
|
|
1279
|
+
|
|
1280
|
+
node.arguments.arguments.grep(Prism::StringNode).each { |arg| @compared[arg.location.start_offset] = true }
|
|
1281
|
+
end
|
|
1282
|
+
|
|
1204
1283
|
def spec_block?(node)
|
|
1205
1284
|
return false unless node.block.is_a?(Prism::BlockNode) && SPEC_KINDS.key?(node.name)
|
|
1206
1285
|
|
data/lib/jitcatch/version.rb
CHANGED
data/skills/jitcatch/SKILL.md
CHANGED
|
@@ -13,7 +13,7 @@ Do not commit the candidate tests unless the human asks to promote one.
|
|
|
13
13
|
|
|
14
14
|
Ruby 3.3 or newer, and git. No gems. `minitest` and `prism` ship with Ruby. RSpec candidates need `rspec-core`: in the project's bundle when it has a `Gemfile`, otherwise installed for the Ruby that runs jitcatch. The subject must be a git repository that can add worktrees. The tool reads commits only. Uncommitted files are not in the diff.
|
|
15
15
|
|
|
16
|
-
`gem install jitcatch` installs the `jitcatch` command, with no runtime dependency. In an app with a `Gemfile`, such as a Rails app, the gem can sit in the `:development, :test` group instead (`gem "jitcatch", require: false`); run it as `bundle exec jitcatch`. Each side still loads its own bundle, so the line does not need to be in either commit. From a checkout of the jitcatch repository, put its `bin` directory on `PATH`, or run `ruby /path/to/jitcatch/bin/jitcatch`. `jitcatch --help` prints the commands, and `jitcatch --version` prints `jitcatch 0.
|
|
16
|
+
`gem install jitcatch` installs the `jitcatch` command, with no runtime dependency. In an app with a `Gemfile`, such as a Rails app, the gem can sit in the `:development, :test` group instead (`gem "jitcatch", require: false`); run it as `bundle exec jitcatch`. Each side still loads its own bundle, so the line does not need to be in either commit. From a checkout of the jitcatch repository, put its `bin` directory on `PATH`, or run `ruby /path/to/jitcatch/bin/jitcatch`. `jitcatch --help` prints the commands, and `jitcatch --version` prints `jitcatch 0.2.0`, with the commit after it (`jitcatch 0.2.0 (1fbcd18)`) when jitcatch runs from its own git checkout; quote that line with any output you quote. The `version` line of every report and `version` in `--json` hold the number only.
|
|
17
17
|
|
|
18
18
|
Exit status 0 is success, including when catching tests were found. Exit status 1 is a usage error. Exit status 2 is a git error. Exit status 3 means `run` tested nothing: a side failed its setup check before any test ran, no candidate file loaded on either side, or every row timed out, or found no database or no table, on a side (below).
|
|
19
19
|
|
|
@@ -38,13 +38,49 @@ jitcatch context --repo DIR --all-readers BASE HEAD
|
|
|
38
38
|
|
|
39
39
|
Omit `--repo` when the working directory is the repository.
|
|
40
40
|
|
|
41
|
-
Read `intent:` first. Then the changed files, `dependencies:` (each gem whose `Gemfile.lock` version changed, as `gem old -> new`, shown only when one did), constants (`k01`, …), and methods (`m01`, …). Each method has a change kind, signature, visibility, call sites, `reachable_from`, existing tests, and the old and new source.
|
|
41
|
+
Read `intent:` first. Then the changed files, `dependencies:` (each gem whose `Gemfile.lock` version changed, as `gem old -> new`, shown only when one did), constants (`k01`, …), and methods (`m01`, …). Each method has a change kind, signature, visibility, call sites, `reachable_from`, existing tests, and the old and new source.
|
|
42
|
+
|
|
43
|
+
`reachable_from` goes through constructors: a caller of `Runner.new` reaches `Runner#initialize` and the methods it calls, and that hop does not count toward the four calls `reachable_from` follows. `reachable_from` keeps the eight nearest. `run` reads `reached_by` (`--json`), every caller at any depth, so a test that enters high (a request spec through a controller and a service, or a CLI test of `CLI.start`) still reaches a changed method five or more calls down; its row is usually `related:guess` with that method in the list.
|
|
44
|
+
|
|
45
|
+
Change kinds:
|
|
46
|
+
|
|
47
|
+
- `moved`: the method left one file while head defines it in another, test files included. The line shows its new place and `moved_from=` the old file, and a call to it is not a call to a removed method. A constant that moves to another file with the same value is `moved` with `moved_from=` as well, and its readers are not `affected`; with a new value it is `modified`, with `moved_from=`.
|
|
48
|
+
- `affected`: the method did not change its body. It reads a constant that did (`via=k01`).
|
|
49
|
+
- A class or module that only one side defines anywhere is listed as an `added` or `removed` constant, with its header line as the source. A file that reopens an existing class adds nothing.
|
|
50
|
+
|
|
51
|
+
What counts as a method, in any Ruby:
|
|
52
|
+
|
|
53
|
+
- Methods defined in a block given to a constant (`Point = Struct.new(:x) do`, `Class.new do`, `Data.define do`) are listed under that constant (`Point#norm`), and the constant's source is the text before `do`.
|
|
54
|
+
- A reader or writer that `attr_reader`, `attr_writer`, `attr_accessor`, or a `Struct.new` / `Data.define` member list defines is a method too (`Point#x`, `Point#x=`), with the one-name form as its source (`attr_reader :seed`, `Struct.new(:x)`). A name dropped from that line is `removed`, the names left on it are unchanged, and a `NoMethodError` on the dropped name, called from the test, is `discard:signature_change`. `Struct.new(*FIELDS)` and `attr_reader(*FIELDS)` count when the same file set `FIELDS` to a list of symbols earlier; a list built at run time or in another file gives no members.
|
|
55
|
+
- A `def` or a reader inside a conditional of the class body (`attr_reader :x if cond`, `if`, `unless`, `else`) counts, read as if every branch ran in order, so a bare `private` in a branch carries past the conditional.
|
|
56
|
+
|
|
57
|
+
### Rails and Grape definitions
|
|
58
|
+
|
|
59
|
+
- On an Active Record model (a class whose superclass chain in the repository reaches `ActiveRecord::Base`), a caller of `create`, `create!`, or `build` reaches `initialize` too, as a caller of `new` does. On another class those three names reach nothing.
|
|
60
|
+
- `scope :recent, -> { ... }` is the class method `Post.recent` with the whole statement as its source.
|
|
61
|
+
- `delegate :name, to: :author` is `Post#name` (`Post#author_name` with `prefix: true`).
|
|
62
|
+
- `enum :status, { draft: 0 }` gives each value `Post#draft?`, `Post#draft!`, `Post.draft`, and `Post.not_draft`, with that one value as the source.
|
|
63
|
+
- A definition counts too in an `ActiveSupport::Concern` block: `included do` and `prepended do` define instance methods of the module, `class_methods do` its class methods.
|
|
64
|
+
- A request to a path helper (`post things_path`) reaches the controller action `config/routes.rb` draws for it, with the helper and controller names the app's `config/initializers/inflections.rb` gives (`person_path` for `resources :people`, `APIKeysController` with `inflect.acronym "API"`). So a changed action's `tests:` lists the files whose requests reach it (`via POST things_path`) and the controller tests that call it as `post :create`, not a grep of its name. A helper the routes file does not draw gets a `warning:`.
|
|
65
|
+
- A Grape endpoint (`post "/refresh" do` inside `resource :things do`, in a class under `Grape::API`) is `Things#POST /things/refresh`. A request spec reaches it by sending that verb to a path ending in those segments: `:id` matches any one, an interpolation in the test matches any one segment, and a string constant the file assigns reads as its string. An endpoint path the source computes otherwise is named with that part in braces and gets a `warning:`.
|
|
66
|
+
|
|
67
|
+
### Visibility, test files, and executables
|
|
42
68
|
|
|
43
69
|
`visibility=` follows `private`, `protected`, and `public` (bare, with names, or before a `def`), the same inside `class << self`, and `private_class_method` / `public_class_method` (with names or before `def self.x`). A `private` method is not an entry point. Test it through a public caller in `reachable_from`. A method after `module_function`, or named by it, is listed as the public module method `Mod.name`. A method of a class or module that `private_constant` names in the same file, or of anything nested in it, is listed as `private`, because a test cannot name its receiver. Visibility set by `send`, or by `private :name` written above its `def`, is not seen.
|
|
44
70
|
|
|
45
71
|
Methods in test files (under `test/` or `spec/`, or named `test_*.rb` or `*_test.rb`) are not listed, and are not entry points. The changed test files still appear under `files:`.
|
|
46
72
|
|
|
47
|
-
|
|
73
|
+
An executable under `exe/` or `bin/` with no suffix and a `ruby` line after `#!` is read as Ruby. Its code outside a `def`, a class, or a module is one method, `exe/tool <main>` (`kind: "script"` in `--json`), so a change to its usage text or to the `case` that picks a command is listed with its old and new source. Its `tests:` are the test files that write its path, and a candidate that starts it (`IO.popen(["ruby", "exe/tool"])`) mentions it, so a row can name it as its related id.
|
|
74
|
+
|
|
75
|
+
### Tests, entry points, and readers
|
|
76
|
+
|
|
77
|
+
`tests: (none)` means no test file names that method. Then `entry_point_tests:` lists, when it finds any, the test files that call a public `reachable_from` entry point: the file names the entry point's class (`Cart.new`, not only `Cart::LIMIT`) and calls a method of that name (`.total`). Each file shows the entry points it matched. Read those tests before you write yours. They show how the repository builds the object and which inputs already reach the changed method. Like `tests:`, this line is a grep. It misses tests that drive the code through a subprocess, and it can list a file that calls a same-named method on another object. A `tests:` grep hit on a common name such as `call` can be any file that defines or calls that name. For a reader or member from `attr_reader` or `Data.define` (`Targets::Hit#code`), a hit counts only in a file that also names its class or the namespace around it (`Hit` or `Targets`). The code of a fixture project under `test/` or `spec/` (`test/fixture/depot/app/`, `lib/`, `config/`, `db/`) is not a test file there.
|
|
78
|
+
|
|
79
|
+
In a Rails app, a changed `initialize` shows `built_by:` above `tests:`: `create`, `create!`, and `build` on an Active Record model, and `build(:loan)` and `create(:loan)` for each factory under `spec/factories` or `test/factories` that builds the class. Its `tests:` lists only the test files that call `Owner.new` or one of those, as `spec/models/loan_spec.rb via create(:loan)`, and no grep hit for the word `initialize`. `factories_skipped:` lists each factory whose class the source does not settle (a `class:` computed at run time, an unknown parent, or a class no file defines, as an inflection such as `APIKey` causes). A factory that builds a Ruby value (`class: Hash`, `Array`, `Struct`, `OpenStruct`, and the like) is left out and not listed. A test that builds from one of those names no constructor. A factory also counts for the classes its associations build (`borrower`, `association :borrower`, or `association` in an attribute block), so a changed `Borrower` callback lists `create(:loan)` when `:loan` builds a borrower. An association in a trait lists the call with the trait, `create(:loan, :cosigned)`, and a test that writes `create(:loan, :cosigned)` mentions it; a global trait and a trait that includes another are not followed.
|
|
80
|
+
|
|
81
|
+
A method that a model's class body names in a callback or a validation (`after_save :count_save`, or `if: :checked?` on a `validates`) shows `run_by:`: the calls that run it by Rails's rules (`Loan.create!`, `Loan#save`, a factory of the class), with the lines that name it. An `on: :create` validation lists no update, and an `on: :update` one no create. `build(:loan)` does not save, so it counts only in a test that also saves or validates (`.save!`, `be_valid`). Its `tests:` lists the test files that make those calls, next to the grep hits. A method of the app that makes one of those calls (`LoanDesk.open` calling `Loan.create!`) is in `reachable_from`, with its callers, so a test of `LoanDesk.open` names the callback.
|
|
82
|
+
|
|
83
|
+
A save on a parameter, a local, or an instance variable the source does not type counts for the model it is named after: `loan.save!` for `Loan`, but `record.save!` for none. Any other call on such a value, and on a `belongs_to` or `has_one` reader (`borrower.approved_cents` inside `Loan`), is a call edge to that model's method too, so `reachable_from` follows `@loan.approve` in a service up to its controller. A callback in a Concern or a parent class is not followed.
|
|
48
84
|
|
|
49
85
|
The brief ends with one `entrypoints <Const>:` section for each class or module that holds a changed method. Each line is a public method of that class on head (on base when head has none) and its role: the change kind (`modified`, `added`, `affected`, `removed`) first, then `unchanged`, at most 12 lines. An unchanged method in a changed method's `reachable_from` comes before the other unchanged ones, and `(+3 more)` counts the methods the cap left out. The cap only shortens the list: a method left out still counts as a mention through `reachable_from`. A class that defines `initialize` on either side lists `Owner.new` first in its group instead, with the role of `Owner#initialize`: `Mismatch.new removed` means head dropped the constructor and `.new` now runs the inherited one. A class that makes `new` private (`private_class_method :new`) shows no `Owner.new`; build it through the public method that calls `new`. Pick the methods your tests call from this list. `run` counts these names as mentions too, so calling one gives a row its `touches:unchanged_caller` tag when it reaches a changed method.
|
|
50
86
|
|
|
@@ -60,7 +96,7 @@ readers: 1 for 1 changed value
|
|
|
60
96
|
not seen: receivers named unlike their model (`x = Branch.first; link_to x`); ...
|
|
61
97
|
```
|
|
62
98
|
|
|
63
|
-
For each row, from the top: open the place (`git show aac7821:app/views/branches/show.html.erb`), read `why:` against the intent, and write the candidate `test:` describes. It states what the old code satisfied, so it passes on base: build the record so the old and the new value differ (a `code` that is not the `id`), and drive the entry point the row names (render the view or send the request, then follow the URL). A row whose effect the intent asks for still gets a test: its `reported` result is then `intended`. `rank:` says why a row scores what it does; a low row (`bare STATUSES is defined nowhere visible`, `in a file the change edits`) is the first to skip. `also:` lists more places of the same kind for the same value, and one test usually covers them. `note: this line is itself part of BASE..HEAD` marks a row the author edited, so it was in front of them. The 10 highest rows print, and the first line counts the rest; `--all-readers` prints every one. A row finds a record's model by the name of the variable that holds it (`@branch`, `home_branch`, `branches`). `not seen:` names what no row can show: a reader there needs a `git grep` of your own. A constant's `read_by:` lists the changed methods that read it by name, and a plain read inside one of those is not repeated under `readers:`.
|
|
99
|
+
For each row, from the top: open the place (`git show aac7821:app/views/branches/show.html.erb`), read `why:` against the intent, and write the candidate `test:` describes. It states what the old code satisfied, so it passes on base: build the record so the old and the new value differ (a `code` that is not the `id`), and drive the entry point the row names (render the view or send the request, then follow the URL). A row whose effect the intent asks for still gets a test: its `reported` result is then `intended`. `rank:` says why a row scores what it does; a low row (`bare STATUSES is defined nowhere visible`, `in a file the change edits`) is the first to skip. `also:` lists more places of the same kind for the same value, and one test usually covers them. `note: this line is itself part of BASE..HEAD` marks a row the author edited, so it was in front of them. The 10 highest rows print, and the first line counts the rest; `--all-readers` prints every one. A row finds a record's model by the name of the variable that holds it (`@branch`, `home_branch`, `branches`). `not seen:` names what no row can show: a reader there needs a `git grep` of your own. In a history with no `app/`, `config/routes.rb`, or `config/application.rb`, such as a gem's, it names only run-time dispatch names and config read at load time. A constant's `read_by:` lists the changed methods that read it by name, and a plain read inside one of those is not repeated under `readers:`.
|
|
64
100
|
|
|
65
101
|
`--coverage FILE` replaces the grep of existing tests. The file is JSON: `"format": "jitcatch-coverage-v1"` and a `tests` array of `{ "id", "file", "lines" }`, where `lines` maps repo paths to line numbers. Without it, the brief says `existing_tests grep`.
|
|
66
102
|
|
|
@@ -189,7 +225,7 @@ Each candidate row under `assess:` ends with `rerun:`, the command that runs tha
|
|
|
189
225
|
| Bucket | Meaning |
|
|
190
226
|
| --- | --- |
|
|
191
227
|
| `no_signal` | Passed on both sides. Not evidence. |
|
|
192
|
-
| `fail_on_base` | The parent did not satisfy the test. Often a new feature. Not a candidate. The text report prints the base outcome under the row (`base: fail RSpec::Expectations::ExpectationNotMetError expected block to output to stdout, but did not`), with the `got:` or `Actual:` line of a value comparison under it, so a test that does not run on base can be fixed. |
|
|
228
|
+
| `fail_on_base` | The parent did not satisfy the test. Often a new feature. Not a candidate. The text report prints the base outcome under the row (`base: fail RSpec::Expectations::ExpectationNotMetError expected block to output to stdout, but did not`), with the `got:` or `Actual:` line of a value comparison under it, so a test that does not run on base can be fixed. A `head:` line follows: `head: pass` for a test that pins what the change fixes, head's own failure, or `head: fail, the same as base`. |
|
|
193
229
|
| `flaky` | Passed on base, failed on head, then a rerun disagreed. Dropped. The text prints each rerun under the row (`rerun 1: pass`), and the `--json` row keeps them in `reruns`. |
|
|
194
230
|
| `discarded` | Stable pass-on-base / fail-on-head, removed by a rule. Still listed under `assess:`. |
|
|
195
231
|
| `reported` | You must judge this one. |
|
|
@@ -206,11 +242,11 @@ Tags are hints, not a verdict. They are sorted:
|
|
|
206
242
|
- `caller:unchanged:lib/b.rb:5` — the broken call is on that line, which the diff did not change. For example, `class B < A` still calls the private `A#helper` that head removed. Judge it as a likely regression. `caller:changed:FILE:LINE` names a line the diff wrote, or a line inside a changed method, such as an edited lambda in a constant. A row whose error kept no frames keeps `discard:signature_change`.
|
|
207
243
|
- `discard:load_error` — `LoadError` on head. `discard:timeout` — the base run of a head that did not pass timed out, or the head run timed out beside a base that took more than half of `--timeout` (a run file from before base kept its time counts as that). Timeouts are not rerun. `discard:no_database` — head raised Active Record's `NoDatabaseError`, `ConnectionNotEstablished`, or `DatabaseConnectionError`: the database of head's checkout does not exist or cannot be reached (Rails apps, below). `discard:no_table` — head raised an `ActiveRecord::StatementInvalid` for a missing table: its database exists but has no schema loaded.
|
|
208
244
|
- The related id (`m05`, or `-` when none) is the downstream modified method when several changed methods sit on the path. When the failure's backtrace runs through changed methods, it is the innermost of them, by each method's lines on head. Read that method's old and new source before you judge.
|
|
209
|
-
- `related:guess` — the evidence that picked the related id points at several changed methods, most often because the test names only an unchanged caller (`build`, `between`) that reaches all of them. The next line names every one with its id (`method_ids` in `--json`): `related guess: Shop::Cart#volume_discount_cents (m05), ranked first of 3: Shop::Cart#coupon_discount_cents (m02), Shop::Cart#taxable_cents (m04), Shop::Cart#volume_discount_cents (m05)`. Read each of them before you judge, not only the pick. The pick prefers a method that existed on base, except an edit whose every added line only calls an added method (`forwards_to_added: true` in `--json`): then the added method is the pick. Among the methods an unchanged caller reaches, the pick is the one farthest below the entry point the test calls (`caller_depths` in `--json`), since a test that enters high passes the plumbing near its entry point on the way to the code it checks. When only the id would order the best two, there is no pick: the row shows `-` and the line reads `related guess: none; the test reaches 7 changed methods equally and no frame of head's failure is in one: Book#apply_catalog_record (m02), LoanRules.for (m03), ...` (`method_id: null`), then gives the next step: find what makes head's value in them, or in a changed file that holds no method, such as a view, from the `git diff --stat BASE HEAD` it prints, and `touches:changed_method` there means the test names one of `method_ids`. An assertion on error plumbing, such as the frames a backtrace keeps, often lands here, because the depth ranking favors the code a value comes from. A test that names the method itself (through `send` when it is private) gets a firm id, and so does an error whose backtrace runs through changed methods: the innermost of them is the id.
|
|
245
|
+
- `related:guess` — the evidence that picked the related id points at several changed methods, most often because the test names only an unchanged caller (`build`, `between`) that reaches all of them. The next line names every one with its id (`method_ids` in `--json`): `related guess: Shop::Cart#volume_discount_cents (m05), ranked first of 3: Shop::Cart#coupon_discount_cents (m02), Shop::Cart#taxable_cents (m04), Shop::Cart#volume_discount_cents (m05)`. Read each of them before you judge, not only the pick. The pick prefers a method that existed on base, except an edit whose every added line only calls an added method (`forwards_to_added: true` in `--json`): then the added method is the pick. Among the methods an unchanged caller reaches, the pick is the one farthest below the entry point the test calls (`caller_depths` in `--json`), since a test that enters high passes the plumbing near its entry point on the way to the code it checks. A test that starts a changed script picks the script's `exe/tool <main>` ahead of the methods that code calls, since every start runs it and each method below it runs only for the command that picks it: `related guess: exe/quint_driver <main> (m01), ranked first of 3: exe/quint_driver <main> (m01), QuintDriver::Spec#outline (m10), QuintDriver::Targets.report (m14)` for a test of `quint_driver bogus` whose usage text changed. Read the methods after it when the test runs a command that calls them. On a row where head stopped raising the error the test expects (`X expected but nothing was raised.`), the pool leaves out an edit that only changes the arguments of a `raise` (`edits_only_raise: true`): head no longer reaches that method, so look for the cause in the code that called it. When only the id would order the best two, there is no pick: the row shows `-` and the line reads `related guess: none; the test reaches 7 changed methods equally and no frame of head's failure is in one: Book#apply_catalog_record (m02), LoanRules.for (m03), ...` (`method_id: null`), then gives the next step: find what makes head's value in them, or in a changed file that holds no method, such as a view, from the `git diff --stat BASE HEAD` it prints, and `touches:changed_method` there means the test names one of `method_ids`. An assertion on error plumbing, such as the frames a backtrace keeps, often lands here, because the depth ranking favors the code a value comes from. A test that names the method itself (through `send` when it is private) gets a firm id, and so does an error whose backtrace runs through changed methods: the innermost of them is the id.
|
|
210
246
|
- `message_varies` — head failed the same way on every rerun with a message that moved, and base passed on every rerun. The finding holds; the value that moved is noise, so a better test compares a field it controls (`exists?`, one column) rather than a whole printed record.
|
|
211
|
-
- `related:dynamic` — no changed method is related, but the test calls a method that sends a changed method's name, or a caller's name, to a receiver the source does not name (`EMITTERS.fetch(format).render`, or `lines.sum(&:total_cents)`, which sends `total_cents` to each element). The next line names it: `related: none, Emit.looked_up sends render to a receiver the source does not name`. The test may reach the change at run time; read the method to judge. `context` lists these under `dynamic_callers:`, leaving out the names of instance methods of a class whose name ends in `Controller`, which the router calls and no code sends.
|
|
247
|
+
- `related:dynamic` — no changed method is related, but the test calls a method that sends a changed method's name, or a caller's name, to a receiver the source does not name (`EMITTERS.fetch(format).render`, or `lines.sum(&:total_cents)`, which sends `total_cents` to each element). The next line names it: `related: none, Emit.looked_up sends render to a receiver the source does not name`. The test may reach the change at run time; read the method to judge. `context` lists these under `dynamic_callers:`, leaving out the names of instance methods of a class whose name ends in `Controller`, which the router calls and no code sends. It also leaves out a `call` or `.()` on a `&block` parameter or on a local that every assignment in its method sets to `-> {}`, `lambda {}`, `proc {}`, or `Proc.new {}`: that call runs the block. A `call` on a parameter (`job.call`), a member, or a method's result stays listed, since a service object with `def call` can receive it.
|
|
212
248
|
- A call through a name computed at run time is not seen at all: `const_get(kind.capitalize)` for the class, or `send(field)`, `public_send(field)`, and `send("#{field}")` for the method. No edge and no mention reaches the changed method, and the row shows `-` with no `touches:` tag and no `related:dynamic`. For `Shop.read("cart", "total")`, where `read` is `build(kind).public_send(field)` and `build` is `const_get(kind.capitalize).new`, a change to `Cart#total` gave `c01 ReadTest#test_reads_a_field_by_name base=pass head=fail - assertion,report`. When `context` shows `callsites: (none)` for a method that the code reaches by name, grep for `send`, `public_send`, and `const_get`, and judge the row from the method you found. A string literal passed to `send` is still a mention (below).
|
|
213
|
-
- A mention is a method call, a constant, or a symbol in the test code. Words in strings and comments are not mentions, so an expected string such as `"jitcatch run\n"` does not point the related id at a method named `run`. A method name passed as a plain string to `send`, `__send__`, `public_send`, `method`, `public_method`, or `respond_to?` is one (`send("total")`). A symbol that keys a Hash is not: an argument of `[]`, `[]=`, `fetch`, `dig`, `key?`, `has_key?`, `values_at`, or `fetch_values`, or a key in a hash literal or keyword argument. So `snap[:discount_cents]` does not point the related id at a method `discount_cents`, and `send(:discount_cents)` does. A call to `message`, `full_message`, or `detailed_message` also mentions `to_s`, so a test that reads `error.message` relates to a changed `Mismatch#to_s`. A test that starts a script (`IO.popen(["ruby", "-Ilib", "exe/tool"])`, `system("bin/x")`, or `File.expand_path("../exe/tool", __dir__)`) mentions the names in that script when the string is a relative path to a file of head, so the row can reach `touches:unchanged_caller`. A path built at run time is not read.
|
|
249
|
+
- A mention is a method call, a constant, or a symbol in the test code. Words in strings and comments are not mentions, so an expected string such as `"jitcatch run\n"` does not point the related id at a method named `run`. A method name passed as a plain string to `send`, `__send__`, `public_send`, `method`, `public_method`, or `respond_to?` is one (`send("total")`). A symbol that keys a Hash is not: an argument of `[]`, `[]=`, `fetch`, `dig`, `key?`, `has_key?`, `values_at`, or `fetch_values`, or a key in a hash literal or keyword argument. So `snap[:discount_cents]` does not point the related id at a method `discount_cents`, and `send(:discount_cents)` does. A call to `message`, `full_message`, or `detailed_message` also mentions `to_s`, so a test that reads `error.message` relates to a changed `Mismatch#to_s`. A test that starts a script (`IO.popen(["ruby", "-Ilib", "exe/tool"])`, `system("bin/x")`, or `File.expand_path("../exe/tool", __dir__)`) mentions the names in that script when the string is a relative path to a file of head, so the row can reach `touches:unchanged_caller`. A path that an assertion or a matcher compares directly (`assert_includes error.backtrace.first, "lib/tool/model.rb"`) is expected text, not a script. A path built at run time is not read.
|
|
214
250
|
|
|
215
251
|
## Judge
|
|
216
252
|
|