holdfast 0.1.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: f19b183542b1eb5619c796e4ed02859386dcf21cc0b381e3b45b84df9eb20c46
4
- data.tar.gz: f1f2a7d651cc193820c8a8edaa049568c81842e2b5b1af0b3366526e12d61c7a
3
+ metadata.gz: 720fa32bd7e5a7cfe1c5e7cadbde8f4a48e96ecbf1872bf755a15f08b049e9ce
4
+ data.tar.gz: 3346a457162841be319397f673f971f5461e3ff6e2066391827df6d6ecaf79bc
5
5
  SHA512:
6
- metadata.gz: a9071a68c3d371c161db59608ffcc3cfef4462b314a86870747fa9604b89a716c7666777017ca637ef0e8353085a78eabbca80413cc4ba077d122d91548f83ca
7
- data.tar.gz: e9f9728bc94ee0bb2bffd4177d218f6ff64b7d53f5f07446711cdc71deeba7701fac230879b5933cc8725b5d1ac0734bfba43b66be9f87245ac584165837fdbb
6
+ metadata.gz: 7203acda8942792da156a24ee95c2a998ddc5cbf862859cc5f66b2be1337fbcb0e68afaf6060dd46083e4bfc11af9c131b779c098c6f87442e500e73aa1a4518
7
+ data.tar.gz: b349dd3e11a45a967438b5a0092d0682f6ea8fc9a3cdc23e565f73d81680ea74391a98d88b44126379ff18ac86186f322f91b4cc9e70febfe7e996e2e742afc2
data/README.ja.md CHANGED
@@ -12,7 +12,7 @@ holdfast は、コードを読んで見つけたことを報告するコーデ
12
12
 
13
13
  ## Rails アプリでのクイックスタート
14
14
 
15
- Ruby 3.4 以降、git、そして RSpec と FactoryBot でテスト用データベースに対してスペックが走るアプリが必要です。
15
+ Ruby 3.4 以降、git、そして RSpec と FactoryBot で PostgreSQL か SQLite のテスト用データベースに対してスペックが走るアプリが必要です。
16
16
  各ステップはアプリのルートで実行します。
17
17
 
18
18
  ### 1. gem を追加する
@@ -63,9 +63,9 @@ bundle exec holdfast check suspicions.yml
63
63
  ```text
64
64
  # holdfast
65
65
 
66
- check: 2 suspicion(s); no change: checks run on the current tree only (af23e82); app, lib, config, db and the Gemfile match main (a17770f)
66
+ check: 2 suspicion(s); no change: checks run on the current tree only (uncommitted changes on 62b0f46); app, lib, config, db and the Gemfile match main (62b0f46)
67
67
  kept specs: spec/holdfast/main/ (branch main); rows: .holdfast/check/main/check.jsonl
68
- seconds: record 9.6, run 5.5
68
+ seconds: record 4.2, run 2.4
69
69
 
70
70
  confirmed=2 could_not_run=0 refuted=0
71
71
 
@@ -75,8 +75,8 @@ confirmed=2 could_not_run=0 refuted=0
75
75
  - ran: NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]
76
76
  - no PaymentGateway charge fails ([[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]])
77
77
  - spec output: suspicion confirmed: no PaymentGateway charge does not hold (observed [[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]])
78
- - repro: `DATABASE_URL=postgres://postgres@127.0.0.1:56111/hf_clinic_9f0097_c2 TEST_ENV_NUMBER=c2 bundle exec rspec spec/holdfast/main/s1_a_passed_tentative_appointment_is_charged_a_no_s_spec.rb`
79
- - next: keep spec/holdfast/main/s1_a_passed_tentative_appointment_is_charged_a_no_s_spec.rb; it fails until the code is fixed
78
+ - repro: `DATABASE_URL=postgres://postgres@127.0.0.1:5440/hf_clinic_c5da1e_c2 TEST_ENV_NUMBER=c2 bundle exec rspec spec/holdfast/main/a_passed_tentative_appointment_is_charged_a_73c143_spec.rb`
79
+ - next: keep spec/holdfast/main/a_passed_tentative_appointment_is_charged_a_73c143_spec.rb; it fails until the code is fixed
80
80
  - `check-in between the sweep's load and its lock becomes a no-show`: first held at batch_load, second runs (held at Appointment load (1 rows)) ends [[Appointment#1(status: "no_show")], Appointment#1(status: "checked_in"); appointment.status == "checked_in" fails ("no_show")], an end no serial order reaches (first then second ends [[Appointment#1(status: "no_show")], raised CheckIn::Refused; appointment.status == "checked_in" fails ("no_show")]; second then first ends [[], Appointment#1(status: "checked_in"); appointment.status == "checked_in" holds ("checked_in")]); serial order "first then second" also fails expect, so this end is the evidence
81
81
  - pause points tried: batch_load: Appointment load (1 rows)
82
82
  - first held at batch_load, second runs: NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]; CheckIn.new(appointment).call returned Appointment#1(status: "checked_in"); held at Appointment load (1 rows); appointment.status == "checked_in" fails ("no_show")
@@ -84,8 +84,8 @@ confirmed=2 could_not_run=0 refuted=0
84
84
  - second then first: CheckIn.new(appointment).call returned Appointment#1(status: "checked_in"); NoShowSweepJob.perform_now returned []; appointment.status == "checked_in" holds ("checked_in")
85
85
  - second held at batch_load, first runs: CheckIn.new(appointment).call returned Appointment#1(status: "checked_in"); NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]; held at Appointment load (1 rows); the second waited on a lock; appointment.status == "checked_in" fails ("no_show")
86
86
  - spec output: suspicion confirmed: first held at batch_load, second runs (held at Appointment load (1 rows)) ends [[Appointment#1(status: "no_show")], Appointment#1(status: "checked_in"); appointment.status == "checked_in" fails ("no_show")], an end no serial order reaches (first then second ends [[Appointment#1(status: "no_show")], raised CheckIn::Refused; appointment.status == "checked_in" fails ("no_show")]; second then first ends [[], Appointment#1(status: "checked_in"); appointment.status == "checked_in" holds ("checked_in")]); serial order "first then second" also fails expect, so this end is the evidence
87
- - repro: `DATABASE_URL=postgres://postgres@127.0.0.1:56111/hf_clinic_9f0097_c2 TEST_ENV_NUMBER=c2 bundle exec rspec spec/holdfast/main/s2_check_in_between_the_sweep_s_load_and_its_lock_b_spec.rb`
88
- - next: keep spec/holdfast/main/s2_check_in_between_the_sweep_s_load_and_its_lock_b_spec.rb; it fails until the code is fixed
87
+ - repro: `DATABASE_URL=postgres://postgres@127.0.0.1:5440/hf_clinic_c5da1e_c2 TEST_ENV_NUMBER=c2 bundle exec rspec spec/holdfast/main/check_in_between_the_sweep_s_load_and_its_lo_d80794_spec.rb`
88
+ - next: keep spec/holdfast/main/check_in_between_the_sweep_s_load_and_its_lo_d80794_spec.rb; it fails until the code is fixed
89
89
  rows: .holdfast/check/main/check.jsonl; kept specs: spec/holdfast/main/
90
90
  ```
91
91
 
@@ -99,7 +99,7 @@ confirmed の行はそれぞれ `spec/holdfast/<branch>/` の下のスペック
99
99
  ジョブが確定済みの予約(booked)だけを選ぶように直すと、一つ目のスペックは通ります。
100
100
 
101
101
  ```sh
102
- bundle exec rspec spec/holdfast/main/s1_a_passed_tentative_appointment_is_charged_a_no_s_spec.rb
102
+ bundle exec rspec spec/holdfast/main/a_passed_tentative_appointment_is_charged_a_73c143_spec.rb
103
103
  ```
104
104
 
105
105
  ```text
@@ -117,22 +117,22 @@ base のブランチと違うブランチの上では(`--base REF` で別の b
117
117
  ```text
118
118
  # holdfast
119
119
 
120
- check: 2 suspicion(s); change: each runs on base main (a17770f) and on the working tree (a0a7f53); 1 file(s) differ: app/jobs/no_show_sweep_job.rb
120
+ check: 2 suspicion(s); change: each runs on base main (62b0f46) and on the working tree (uncommitted changes on 7c510b0); 1 file(s) differ: app/jobs/no_show_sweep_job.rb
121
121
  kept specs: spec/holdfast/skip-tentative/ (branch skip-tentative); rows: .holdfast/check/skip-tentative/check.jsonl
122
- seconds: record 0.0, run 5.6
122
+ seconds: record 0.0, run 2.6
123
123
 
124
124
  confirmed=1 could_not_run=0 refuted=1
125
125
 
126
126
  ## Confirmed (the kept spec fails)
127
127
 
128
128
  - `check-in between the sweep's load and its lock becomes a no-show`: first held at batch_load, second runs (held at Appointment load (1 rows)) ends [...]
129
- - base a17770f: confirmed: [...]; it was there before the change
129
+ - base 62b0f46: confirmed: [...]; it was there before the change
130
130
  [...]
131
131
 
132
132
  ## Refuted (what was observed)
133
133
 
134
134
  - `a passed tentative appointment is charged a no-show fee`: no PaymentGateway charge holds (observed [])
135
- - base a17770f: confirmed: no PaymentGateway charge does not hold (observed [[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]]); the change removes it
135
+ - base 62b0f46: confirmed: no PaymentGateway charge does not hold (observed [[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]]); the change removes it
136
136
  - ran: NoShowSweepJob.perform_now returned []
137
137
  - no PaymentGateway charge holds ([])
138
138
  rows: .holdfast/check/skip-tentative/check.jsonl; kept specs: spec/holdfast/skip-tentative/
@@ -154,29 +154,25 @@ base にないメソッド、カラム、ルートを呼ぶ疑いは head でだ
154
154
  次に、それらのアクションをランダムな順序でテスト用データベースに対して走らせます。
155
155
  その際、不変条件、別のタイムゾーン、外部呼び出しでの二つのアクションの割り込み、最初のバッチ読み込みの後で止めたジョブ、受け付けられたリクエストをもう一度送ることを試します。
156
156
  失敗は最短のシーケンスまで縮められます(縮める時間は最大 60 秒です)。
157
+ このマシン以外のホストへの接続は拒み、その領域は本物のサービスを呼んだと報告します。
158
+ そのサービスの偽物は、領域のファイルに書きます。
157
159
  `--budget SECONDS` で実行時間の上限を決めます(既定は 600)。
158
160
  クリニックアプリで、上限を 120 秒にした出力です。
159
161
 
160
162
  ```text
161
163
  # holdfast
162
164
 
163
- nothing in the app changed against main (27257f9)
164
- note: hunt: unreached: 1 of 2 actions (refused every time or never drawn); 0 of 1 areas reached under half
165
- seconds: record 10.1, hunt 16.8
165
+ nothing in the app changed against main (b5c13d9)
166
+ seconds: record 4.6, hunt 40.8
166
167
 
167
- unintended_change=0 bug_suspect=1 unchecked=1 intended_change=0 clean=0
168
+ unintended_change=0 bug_suspect=1 unchecked=0 intended_change=0 clean=0
168
169
 
169
170
  ## Suspected bugs
170
171
 
171
- - `sequences: check_ins`: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in"
172
- - failure: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in"
172
+ - `sequences: check_ins`: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in" (shrinking stopped after 30 s)
173
+ - failure: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in" (shrinking stopped after 30 s)
173
174
  - log: .holdfast/runs/sequences/check_ins.log (the shortest sequence and an `it` block that replays it)
174
175
  - next: paste the `it` block from the log into .holdfast/sequences/check_ins.rb, see it fail, fix the code, run again
175
-
176
- ## Not checked
177
-
178
- - `(hunt)`: unreached: 1 of 2 actions (refused every time or never drawn); 0 of 1 areas reached under half
179
- - next: read each area's row below for the refusals, fix the setup or the sign-in in its file, then holdfast run; until then the hunt says little about these areas
180
176
  rows: .holdfast/verdicts.jsonl
181
177
  ```
182
178
 
@@ -204,8 +200,15 @@ holdfast はアプリの `.holdfast/` と `spec/holdfast/` の下に書き込み
204
200
 
205
201
  各アプリは、そのディレクトリから作った自分専用のテスト用データベース名を `DATABASE_URL` で受け取ります。
206
202
  そのため、base と head、また一つのサーバー上の二つのアプリが、同じデータベースを共有しません。
203
+ SQLite では、それぞれの側が `.holdfast/db/` の下に自分専用のファイルを受け取り、アプリ自身のテスト用データベースのファイルには書き込みません。
207
204
  セットアップは `bin/rails db:create db:schema:load` を実行します。
208
205
 
206
+ 競合とバグ探しは PostgreSQL と SQLite で走ります。
207
+ SQLite では、一度に書き込める接続は一つです。
208
+ 競合で止めたアクションが、開いたトランザクションの中ですでに書き込んでいると、もう一方のアクションはそれを待ちます。
209
+ その行には `the second waited on a lock` と出ます。
210
+ MySQL は試していません。
211
+
209
212
  ## 制限
210
213
 
211
214
  - 入力がレコードを一つも作らないピンの下書きは何も確かめませんが、そのピンが変更された行を読み込むと、`holdfast run` はその対象を `clean` と示すことがあります。
data/README.md CHANGED
@@ -6,7 +6,7 @@ It is made for a coding agent that reads code and has to say what it found. The
6
6
 
7
7
  ## Quick start in your Rails app
8
8
 
9
- You need Ruby 3.4 or newer, git, and an app whose specs run with RSpec and FactoryBot against a test database. Run each step from the app's root.
9
+ You need Ruby 3.4 or newer, git, and an app whose specs run with RSpec and FactoryBot against a test database on PostgreSQL or SQLite. Run each step from the app's root.
10
10
 
11
11
  ### 1. Add the gem
12
12
 
@@ -46,9 +46,9 @@ The first run learns how the app runs its tests and makes test databases of its
46
46
  ```text
47
47
  # holdfast
48
48
 
49
- check: 2 suspicion(s); no change: checks run on the current tree only (af23e82); app, lib, config, db and the Gemfile match main (a17770f)
49
+ check: 2 suspicion(s); no change: checks run on the current tree only (uncommitted changes on 62b0f46); app, lib, config, db and the Gemfile match main (62b0f46)
50
50
  kept specs: spec/holdfast/main/ (branch main); rows: .holdfast/check/main/check.jsonl
51
- seconds: record 9.6, run 5.5
51
+ seconds: record 4.2, run 2.4
52
52
 
53
53
  confirmed=2 could_not_run=0 refuted=0
54
54
 
@@ -58,8 +58,8 @@ confirmed=2 could_not_run=0 refuted=0
58
58
  - ran: NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]
59
59
  - no PaymentGateway charge fails ([[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]])
60
60
  - spec output: suspicion confirmed: no PaymentGateway charge does not hold (observed [[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]])
61
- - repro: `DATABASE_URL=postgres://postgres@127.0.0.1:56111/hf_clinic_9f0097_c2 TEST_ENV_NUMBER=c2 bundle exec rspec spec/holdfast/main/s1_a_passed_tentative_appointment_is_charged_a_no_s_spec.rb`
62
- - next: keep spec/holdfast/main/s1_a_passed_tentative_appointment_is_charged_a_no_s_spec.rb; it fails until the code is fixed
61
+ - repro: `DATABASE_URL=postgres://postgres@127.0.0.1:5440/hf_clinic_c5da1e_c2 TEST_ENV_NUMBER=c2 bundle exec rspec spec/holdfast/main/a_passed_tentative_appointment_is_charged_a_73c143_spec.rb`
62
+ - next: keep spec/holdfast/main/a_passed_tentative_appointment_is_charged_a_73c143_spec.rb; it fails until the code is fixed
63
63
  - `check-in between the sweep's load and its lock becomes a no-show`: first held at batch_load, second runs (held at Appointment load (1 rows)) ends [[Appointment#1(status: "no_show")], Appointment#1(status: "checked_in"); appointment.status == "checked_in" fails ("no_show")], an end no serial order reaches (first then second ends [[Appointment#1(status: "no_show")], raised CheckIn::Refused; appointment.status == "checked_in" fails ("no_show")]; second then first ends [[], Appointment#1(status: "checked_in"); appointment.status == "checked_in" holds ("checked_in")]); serial order "first then second" also fails expect, so this end is the evidence
64
64
  - pause points tried: batch_load: Appointment load (1 rows)
65
65
  - first held at batch_load, second runs: NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]; CheckIn.new(appointment).call returned Appointment#1(status: "checked_in"); held at Appointment load (1 rows); appointment.status == "checked_in" fails ("no_show")
@@ -67,8 +67,8 @@ confirmed=2 could_not_run=0 refuted=0
67
67
  - second then first: CheckIn.new(appointment).call returned Appointment#1(status: "checked_in"); NoShowSweepJob.perform_now returned []; appointment.status == "checked_in" holds ("checked_in")
68
68
  - second held at batch_load, first runs: CheckIn.new(appointment).call returned Appointment#1(status: "checked_in"); NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]; held at Appointment load (1 rows); the second waited on a lock; appointment.status == "checked_in" fails ("no_show")
69
69
  - spec output: suspicion confirmed: first held at batch_load, second runs (held at Appointment load (1 rows)) ends [[Appointment#1(status: "no_show")], Appointment#1(status: "checked_in"); appointment.status == "checked_in" fails ("no_show")], an end no serial order reaches (first then second ends [[Appointment#1(status: "no_show")], raised CheckIn::Refused; appointment.status == "checked_in" fails ("no_show")]; second then first ends [[], Appointment#1(status: "checked_in"); appointment.status == "checked_in" holds ("checked_in")]); serial order "first then second" also fails expect, so this end is the evidence
70
- - repro: `DATABASE_URL=postgres://postgres@127.0.0.1:56111/hf_clinic_9f0097_c2 TEST_ENV_NUMBER=c2 bundle exec rspec spec/holdfast/main/s2_check_in_between_the_sweep_s_load_and_its_lock_b_spec.rb`
71
- - next: keep spec/holdfast/main/s2_check_in_between_the_sweep_s_load_and_its_lock_b_spec.rb; it fails until the code is fixed
70
+ - repro: `DATABASE_URL=postgres://postgres@127.0.0.1:5440/hf_clinic_c5da1e_c2 TEST_ENV_NUMBER=c2 bundle exec rspec spec/holdfast/main/check_in_between_the_sweep_s_load_and_its_lo_d80794_spec.rb`
71
+ - next: keep spec/holdfast/main/check_in_between_the_sweep_s_load_and_its_lo_d80794_spec.rb; it fails until the code is fixed
72
72
  rows: .holdfast/check/main/check.jsonl; kept specs: spec/holdfast/main/
73
73
  ```
74
74
 
@@ -79,7 +79,7 @@ The race is confirmed only because the held order ends in a state that no serial
79
79
  Each confirmed row names a spec under `spec/holdfast/<branch>/` that fails until the code is fixed. Fix the code, run the spec, and keep it. After the sweep selects only booked appointments, the first spec passes:
80
80
 
81
81
  ```sh
82
- bundle exec rspec spec/holdfast/main/s1_a_passed_tentative_appointment_is_charged_a_no_s_spec.rb
82
+ bundle exec rspec spec/holdfast/main/a_passed_tentative_appointment_is_charged_a_73c143_spec.rb
83
83
  ```
84
84
 
85
85
  ```text
@@ -95,22 +95,22 @@ On a branch that differs from the base branch (`--base REF` names another), each
95
95
  ```text
96
96
  # holdfast
97
97
 
98
- check: 2 suspicion(s); change: each runs on base main (a17770f) and on the working tree (a0a7f53); 1 file(s) differ: app/jobs/no_show_sweep_job.rb
98
+ check: 2 suspicion(s); change: each runs on base main (62b0f46) and on the working tree (uncommitted changes on 7c510b0); 1 file(s) differ: app/jobs/no_show_sweep_job.rb
99
99
  kept specs: spec/holdfast/skip-tentative/ (branch skip-tentative); rows: .holdfast/check/skip-tentative/check.jsonl
100
- seconds: record 0.0, run 5.6
100
+ seconds: record 0.0, run 2.6
101
101
 
102
102
  confirmed=1 could_not_run=0 refuted=1
103
103
 
104
104
  ## Confirmed (the kept spec fails)
105
105
 
106
106
  - `check-in between the sweep's load and its lock becomes a no-show`: first held at batch_load, second runs (held at Appointment load (1 rows)) ends [...]
107
- - base a17770f: confirmed: [...]; it was there before the change
107
+ - base 62b0f46: confirmed: [...]; it was there before the change
108
108
  [...]
109
109
 
110
110
  ## Refuted (what was observed)
111
111
 
112
112
  - `a passed tentative appointment is charged a no-show fee`: no PaymentGateway charge holds (observed [])
113
- - base a17770f: confirmed: no PaymentGateway charge does not hold (observed [[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]]); the change removes it
113
+ - base 62b0f46: confirmed: no PaymentGateway charge does not hold (observed [[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]]); the change removes it
114
114
  - ran: NoShowSweepJob.perform_now returned []
115
115
  - no PaymentGateway charge holds ([])
116
116
  rows: .holdfast/check/skip-tentative/check.jsonl; kept specs: spec/holdfast/skip-tentative/
@@ -122,28 +122,22 @@ rows: .holdfast/check/skip-tentative/check.jsonl; kept specs: spec/holdfast/skip
122
122
 
123
123
  ## Hunt when nothing changed
124
124
 
125
- `holdfast run` with no change drafts one sequence definition per write area of the app (routes, Grape endpoints, jobs) into `.holdfast/sequences/`. It then runs random orders of their actions against the test database, under invariants, other time zones, two actions interleaved at an outside call, a job held after its first batch load, and an accepted request sent once more. A failure is shrunk to the shortest sequence, for at most 60 seconds. `--budget SECONDS` caps the run (default 600). On the clinic app, with a budget of 120 seconds:
125
+ `holdfast run` with no change drafts one sequence definition per write area of the app (routes, Grape endpoints, jobs) into `.holdfast/sequences/`. It then runs random orders of their actions against the test database, under invariants, other time zones, two actions interleaved at an outside call, a job held after its first batch load, and an accepted request sent once more. A failure is shrunk to the shortest sequence, for at most 60 seconds. A connection to any host but this machine is refused, and the area reports the real service it reached; write a fake of that service in the area's file. `--budget SECONDS` caps the run (default 600). On the clinic app, with a budget of 120 seconds:
126
126
 
127
127
  ```text
128
128
  # holdfast
129
129
 
130
- nothing in the app changed against main (27257f9)
131
- note: hunt: unreached: 1 of 2 actions (refused every time or never drawn); 0 of 1 areas reached under half
132
- seconds: record 10.1, hunt 16.8
130
+ nothing in the app changed against main (b5c13d9)
131
+ seconds: record 4.6, hunt 40.8
133
132
 
134
- unintended_change=0 bug_suspect=1 unchecked=1 intended_change=0 clean=0
133
+ unintended_change=0 bug_suspect=1 unchecked=0 intended_change=0 clean=0
135
134
 
136
135
  ## Suspected bugs
137
136
 
138
- - `sequences: check_ins`: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in"
139
- - failure: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in"
137
+ - `sequences: check_ins`: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in" (shrinking stopped after 30 s)
138
+ - failure: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in" (shrinking stopped after 30 s)
140
139
  - log: .holdfast/runs/sequences/check_ins.log (the shortest sequence and an `it` block that replays it)
141
140
  - next: paste the `it` block from the log into .holdfast/sequences/check_ins.rb, see it fail, fix the code, run again
142
-
143
- ## Not checked
144
-
145
- - `(hunt)`: unreached: 1 of 2 actions (refused every time or never drawn); 0 of 1 areas reached under half
146
- - next: read each area's row below for the refusals, fix the setup or the sign-in in its file, then holdfast run; until then the hunt says little about these areas
147
141
  rows: .holdfast/verdicts.jsonl
148
142
  ```
149
143
 
@@ -164,7 +158,9 @@ holdfast show
164
158
 
165
159
  holdfast writes under `.holdfast/` and `spec/holdfast/` in the app. Add `.holdfast/` to the app's `.gitignore`. Commit the specs you keep.
166
160
 
167
- Each app gets its own test database names through `DATABASE_URL`, made from its directory, so base and head, and two apps on one server, do not share one. The setup runs `bin/rails db:create db:schema:load`.
161
+ Each app gets its own test database names through `DATABASE_URL`, made from its directory, so base and head, and two apps on one server, do not share one. On SQLite, each side gets a file of its own under `.holdfast/db/`, and the app's own test database file is never written. The setup runs `bin/rails db:create db:schema:load`.
162
+
163
+ Races and the hunt run on PostgreSQL and on SQLite. SQLite lets one connection write at a time: when a race holds an action that has already written in its open transaction, the other action waits for it, and the row says `the second waited on a lock`. MySQL is not tested.
168
164
 
169
165
  ## Limits
170
166
 
@@ -48,10 +48,18 @@ module Holdfast
48
48
 
49
49
  def changed? = mode == "change"
50
50
  def short(sha) = sha[0, 7]
51
+
52
+ # A dirty tree's snapshot is a commit no `git log` shows; the reader
53
+ # gets the commit it sits on instead.
54
+ def tree_label
55
+ committed = Change.git(root, "rev-parse", "HEAD")
56
+ committed == head ? short(head) : "uncommitted changes on #{short(committed)}"
57
+ end
58
+
51
59
  def header
52
60
  case mode
53
- when "change" then "change: #{base_label} (#{short(base)}) .. working tree (#{short(head)})"
54
- when "before" then "before a change: pinning #{named.join(", ")} at #{short(head)}"
61
+ when "change" then "change: #{base_label} (#{short(base)}) .. working tree (#{tree_label})"
62
+ when "before" then "before a change: pinning #{named.join(", ")} at #{tree_label}"
55
63
  else "#{Change.git(root, "diff", "--quiet", base, head).then { "" } rescue "only tests, factories or docs differ from #{base_label}; " }nothing in the app changed against #{base_label} (#{short(base)})"
56
64
  end
57
65
  end
@@ -55,11 +55,11 @@ module Holdfast
55
55
  # which files made it a change, and one that did not says so.
56
56
  def self.scope_line(change, here)
57
57
  base = "#{change.base_label} (#{change.short(change.base)})"
58
- return "no change: checks run on the current tree only (#{change.short(change.head)}); app, lib, config, db and the Gemfile match #{base}" unless change.changed?
59
- return "--here: checks run on the current tree only (#{change.short(change.head)}); the change against #{base} is not compared" if here
58
+ return "no change: checks run on the current tree only (#{change.tree_label}); app, lib, config, db and the Gemfile match #{base}" unless change.changed?
59
+ return "--here: checks run on the current tree only (#{change.tree_label}); the change against #{base} is not compared" if here
60
60
 
61
61
  files = Change.git(change.root, "diff", "--name-only", change.base, change.head, "--", *Holdfast::CODE).lines.map(&:strip)
62
- "change: each runs on base #{base} and on the working tree (#{change.short(change.head)}); #{files.size} file(s) differ: #{files.first(5).join(", ")}#{" and #{files.size - 5} more" if files.size > 5}"
62
+ "change: each runs on base #{base} and on the working tree (#{change.tree_label}); #{files.size} file(s) differ: #{files.first(5).join(", ")}#{" and #{files.size - 5} more" if files.size > 5}"
63
63
  end
64
64
 
65
65
  # A file holds a list of suspicions (or `suspicions:` with one); each
@@ -79,7 +79,11 @@ module Holdfast
79
79
  raw += inline.map { [YAML.safe_load(quote_ruby(_1)), "--suspect"] }
80
80
  raise Holdfast::Error, "no suspicion given; pass a YAML file or --suspect" if raw.empty?
81
81
 
82
- raw.each_with_index.map { |(h, from), i| suspicion(h, i, from) }
82
+ list = raw.each_with_index.map { |(h, from), i| suspicion(h, i, from) }
83
+ twice = list.group_by(&:id).select { _2.size > 1 }.values.map { _1.first.says }
84
+ raise Holdfast::Error, "two suspicions say #{twice.first.inspect}; give each its own `says`" if twice.any?
85
+
86
+ list
83
87
  end
84
88
 
85
89
  # `do: x.call(a: 1)` is Ruby, not a YAML mapping: a plain value of says,
@@ -155,7 +159,10 @@ module Holdfast
155
159
  set = acts.flat_map { ivars(_1, :write) }
156
160
  unset = expects.flat_map { ivars(_1[:ruby], :read) }.uniq - set
157
161
  raise Holdfast::Error, "#{label}: `expect` reads #{unset.join(", ")}, which no `do` sets; set it in `do`, or read the record by its `given` name" if unset.any?
158
- Suspicion.new(id: "s#{i + 1}_#{Holdfast.slug(says.empty? ? acts.first : says).tr("-", "_")[0, 48].sub(/_\z/, "")}",
162
+ # Named by its text, not its place in the file: a kept spec from an
163
+ # earlier run is the same suspicion only when it says the same thing.
164
+ text = says.empty? ? acts.first : says
165
+ Suspicion.new(id: "#{Holdfast.slug(text).tr("-", "_")[0, 44].sub(/_\z/, "")}_#{Digest::SHA1.hexdigest(text)[0, 6]}",
159
166
  says:, given: Hash(h["given"]), acts:, pause: h["pause"]&.to_s,
160
167
  expects:, calls:, errors:, source: from, shared:)
161
168
  end
@@ -403,7 +410,10 @@ module Holdfast
403
410
  msg = run["log"].to_s.lines.grep(/Error|error/).first.to_s.strip if msg.empty?
404
411
  frame = Array(ex&.dig("exception", "backtrace")).find { _1.include?("spec/holdfast/") }
405
412
  msg = "rspec printed nothing for it; read #{run["out"]}/rspec.log" if msg.empty?
406
- { "result" => "could_not_run", "phase" => "spec", "error" => "the spec did not finish: #{ex&.dig("exception", "class")} #{msg[0, 400]}#{" at #{frame[%r{spec/holdfast/\S+}]}" if frame}".squeeze(" ") }
413
+ # Raised first in holdfast's own support file, not in the suspicion's
414
+ # lines or the app: nothing in the suspicion can fix it.
415
+ own = Array(ex&.dig("exception", "backtrace")).find { !_1.include?("/gems/") }.to_s.include?("/holdfast_check.rb:")
416
+ { "result" => "could_not_run", "phase" => own ? "holdfast" : "spec", "error" => "the spec did not finish: #{ex&.dig("exception", "class")} #{msg[0, 400]}#{" at #{frame[%r{spec/holdfast/\S+}]}" if frame}".squeeze(" ") }
407
417
  end
408
418
 
409
419
  def self.describe(r)
@@ -464,6 +474,9 @@ module Holdfast
464
474
  when "given" then "fix the named part of `given` (an attribute, a factory: or a traits: key), then holdfast check again"
465
475
  when "do" then "the action raised before `expect` ran; if that error is the bug, write `expect: no error`, else fix `do`"
466
476
  when "pause" then "name a method the first action passes through as `pause: Class#method`"
477
+ when "holdfast"
478
+ "holdfast's own support code failed on the #{record.data.dig("database", "adapter") || "unknown"} adapter, not the suspicion or the app; " \
479
+ "report it to holdfast with this row (races and the hunt run on PostgreSQL and SQLite)"
467
480
  else "read the error, fix the suspicion, then holdfast check again"
468
481
  end
469
482
  Verdict.of(target: s.says, stage: "check", kind: "could_not_run", reason: "#{head["phase"] || "spec"}: #{head["error"]}", evidence:, scope: spec_path, repro:, next_step: step)
@@ -75,7 +75,7 @@ module Holdfast
75
75
  ran = pins.observed.count { !_1["input_error"] }
76
76
  kind = ran.zero? ? "unchecked" : "clean"
77
77
  rows << Verdict.of(target: pins.target.name, stage: "fill", kind:,
78
- reason: ran.zero? ? "no input ran" : "pinned #{ran} input(s) at #{change.short(change.head)} by what each returns, raises and writes",
78
+ reason: ran.zero? ? "no input ran" : "pinned #{ran} input(s) at #{change.tree_label} by what each returns, raises and writes",
79
79
  evidence: details + pins.target.tests.first(3).map { "selected test: #{_1}" },
80
80
  scope: pins.dir.delete_prefix("#{change.root}/"),
81
81
  next_step: ran.zero? ? "fix the inputs the evidence names, then holdfast run" : "make the change, then holdfast run; these pins are compared on both sides")
@@ -23,7 +23,7 @@ module Holdfast
23
23
  def self.learn(root)
24
24
  facts = read_facts(root)
25
25
  steps = []
26
- if facts["rails"] && facts["database"]["server"]
26
+ if facts["rails"] && facts["database"]["url"]
27
27
  SIDES.each do |side, number|
28
28
  steps << run_step(root, "create the #{side} database (#{database_name(facts, number) || "TEST_ENV_NUMBER=#{number}"})", facts["side_setup"],
29
29
  side_env(facts, number))
@@ -46,15 +46,18 @@ module Holdfast
46
46
  test_section = db_yml[/^test:.*\z/m].to_s
47
47
  adapter = test_section[/adapter:\s*(\w+)/, 1] || db_yml[/adapter:\s*(\w+)/, 1]
48
48
  server = %w[postgresql mysql2 trilogy].include?(adapter)
49
+ url = if server then database_url(root, adapter)
50
+ elsif adapter == "sqlite3" then sqlite_url(root)
51
+ end
49
52
  {
50
53
  "framework" => framework,
51
54
  "bundle" => File.file?(File.join(root, "Gemfile")) ? "bundle exec " : "",
52
55
  "rails" => File.file?(File.join(root, "config/environment.rb")),
53
56
  "helper" => framework == "rspec" ? (File.file?(File.join(root, "spec/rails_helper.rb")) ? "rails_helper" : "spec_helper") : "test_helper",
54
57
  "database" => { "adapter" => adapter, "server" => server, "numbered" => test_section.include?("TEST_ENV_NUMBER"),
55
- "url" => server ? database_url(root, adapter) : nil }.compact,
58
+ "url" => url }.compact,
56
59
  "env" => db_yml.scan(/ENV(?:\.fetch\(|\[)\s*["'](\w+)["']/).flatten.uniq - %w[TEST_ENV_NUMBER RAILS_ENV],
57
- "side_setup" => server ? "bin/rails db:create db:schema:load" : nil,
60
+ "side_setup" => url ? "bin/rails db:create db:schema:load" : nil,
58
61
  "base_branch" => base_branch(root),
59
62
  "coverage" => COVERAGE.find { File.file?(File.join(root, _1)) }
60
63
  }
@@ -76,6 +79,12 @@ module Holdfast
76
79
  nil
77
80
  end
78
81
 
82
+ # A SQLite database.yml names one file, and TEST_ENV_NUMBER changes it
83
+ # only where the file name reads it: base, head and each hunt worker
84
+ # wrote the app's own storage/test.sqlite3, and emptied it after a race.
85
+ # Each gets a file of its own under .holdfast/db/ instead.
86
+ def self.sqlite_url(root) = "sqlite3:#{File.join(File.expand_path(Holdfast.dir(root)), "db", "test")}"
87
+
79
88
  def self.side_env(facts, number)
80
89
  url = facts.dig("database", "url")
81
90
  { "RAILS_ENV" => "test", "TEST_ENV_NUMBER" => number }.merge(url ? { "DATABASE_URL" => "#{url}_#{number}" } : {})
@@ -24,7 +24,7 @@ module HoldfastCheck
24
24
  # first time the held action passes it; `Class#method` is the fifth kind.
25
25
  POINTS = {
26
26
  "batch_load" => "its first query that returns rows",
27
- "outside_call" => "its first call to an outside-service client",
27
+ "outside_call" => "its first call to an outside-service client, or to a module method the suspicion stubs",
28
28
  "row_lock" => "its first with_lock or lock!, before the lock is taken",
29
29
  "first_write" => "its first INSERT, UPDATE or DELETE"
30
30
  }.freeze
@@ -125,6 +125,29 @@ module HoldfastCheck
125
125
  end
126
126
  OutsideGate = Module.new
127
127
 
128
+ # A service the suspicion stubs on a module (`allow(LockerApi).to
129
+ # receive(:register)`) is an outside call too, and `pause:
130
+ # LockerApi.register` names it. RSpec's stub sits in front of any hook
131
+ # prepended to the module, so the hold is on the stub's own invocation.
132
+ module StubGate
133
+ private
134
+
135
+ def invoke_incrementing_actual_calls_by(*args, &block)
136
+ if orig_object.is_a?(Module) && orig_object.name
137
+ gate = HoldfastCheck.gate
138
+ HoldfastCheck.hold(gate, "outside_call", "#{orig_object.name}.#{message}")
139
+ HoldfastCheck.hold(gate, "#{orig_object.name}.#{message}", "#{orig_object.name}.#{message}")
140
+ end
141
+ super
142
+ end
143
+ end
144
+ # rspec-mocks loads when the first example runs, after this file.
145
+ def self.watch_stubs
146
+ return unless defined?(RSpec::Mocks::MessageExpectation)
147
+
148
+ RSpec::Mocks::MessageExpectation.prepend(StubGate) unless RSpec::Mocks::MessageExpectation <= StubGate
149
+ end
150
+
128
151
  def self.method_gate(point)
129
152
  @method_gates ||= {}
130
153
  return if @method_gates[point]
@@ -338,6 +361,7 @@ module HoldfastCheck
338
361
  def holdfast_race(given, acts, expects, calls, pause, shared)
339
362
  Rails.application.eager_load!
340
363
  HoldfastCheck.watch_writes
364
+ HoldfastCheck.watch_stubs
341
365
  named = pause && pause != "batch_load"
342
366
  HoldfastCheck.method_gate(pause) if named && !POINTS.key?(pause)
343
367
  points = named ? [pause] : ["batch_load", *FALLBACKS]
@@ -531,15 +555,18 @@ module HoldfastCheck
531
555
  def holdfast_step(records, act, errors:)
532
556
  scope = Scope.new(self, records)
533
557
  result = scope.instance_exec(&act.block)
534
- { result:, scope:, shown: "#{act.text} returned #{holdfast_show(result)}" }
558
+ { result:, scope:, shown: "#{HoldfastCheck.one_line(act.text)} returned #{holdfast_show(result)}" }
535
559
  rescue StandardError => e
536
560
  app = HoldfastCheck.app_error?(e)
537
561
  absent = HoldfastCheck.absent(e) unless app
538
562
  raise CouldNotRun.new("do", "`#{act.text}` raised #{e.class}: #{e.message.lines.first.to_s.strip}", absent:) if absent || !(errors || app)
539
563
 
540
- { error: e, scope:, shown: "#{act.text.strip.gsub(/\s*\n\s*/, "; ")} raised #{e.class}: #{e.message.lines.first.to_s.strip[0, 300]}" }
564
+ { error: e, scope:, shown: "#{HoldfastCheck.one_line(act.text)} raised #{e.class}: #{e.message.lines.first.to_s.strip[0, 300]}" }
541
565
  end
542
566
 
567
+ # A `do` of several lines is one item of the Markdown list it is shown in.
568
+ def self.one_line(text) = text.strip.gsub(/\s*\n\s*/, "; ")
569
+
543
570
  CALL_SHAPE = /wrong number of arguments|missing keyword|unknown keyword/
544
571
 
545
572
  def self.app_error?(error)
@@ -653,10 +680,16 @@ module HoldfastCheck
653
680
  end
654
681
  end
655
682
 
683
+ # SQLite has no TRUNCATE; DELETE with the foreign keys off empties the
684
+ # tables in any order, and sqlite_sequence holds the ids to restart.
656
685
  def holdfast_truncate
657
686
  conn = ActiveRecord::Base.connection
658
687
  tables = conn.tables - %w[schema_migrations ar_internal_metadata]
659
- conn.execute("TRUNCATE #{tables.map { conn.quote_table_name(_1) }.join(", ")} RESTART IDENTITY CASCADE") if tables.any?
688
+ return if tables.empty?
689
+ return conn.execute("TRUNCATE #{tables.map { conn.quote_table_name(_1) }.join(", ")} RESTART IDENTITY CASCADE") if conn.adapter_name.match?(/postg/i)
690
+
691
+ conn.disable_referential_integrity { tables.each { conn.execute("DELETE FROM #{conn.quote_table_name(_1)}") } }
692
+ conn.execute("DELETE FROM sqlite_sequence") if conn.adapter_name.match?(/sqlite/i) && conn.select_value("SELECT 1 FROM sqlite_master WHERE name = 'sqlite_sequence'")
660
693
  end
661
694
 
662
695
  def holdfast_write(report)
@@ -595,6 +595,28 @@ module Propose
595
595
  end.uniq
596
596
  end
597
597
 
598
+ # App modules that talk HTTP themselves (`Net::HTTP.start` inside
599
+ # `LockerApi.register`) rather than through a `.client` a spec replaces:
600
+ # the hunt refuses their real calls, so a person writes the fake.
601
+ def http_services
602
+ @http_services ||= (Rails.root.glob("app/**/*.rb") + Rails.root.glob("lib/**/*.rb")).filter_map do |file|
603
+ text = File.read(file)
604
+ next unless text.match?(/\bNet::HTTP\b|\bFaraday\b|\bHTTParty\b|\bHTTP\.(get|post|put|patch|delete)\b/)
605
+
606
+ text[/^\s*(?:module|class)\s+([A-Z][\w:]*)/, 1]
607
+ end.uniq
608
+ end
609
+
610
+ def network_note(selected)
611
+ names = selected.map { it.name.to_s }
612
+ seen = Sequences.network_calls.select { |_, actions| (actions & names).any? }.keys
613
+ services = http_services.select { |service| selected.any? { static_constants(it.file).include?(service) } }
614
+ return [] if seen.empty? && services.empty?
615
+
616
+ ["TODO: #{services.any? ? services.join(", ") : "an action here"} calls a real host#{" (#{seen.join(", ")} in the probe)" if seen.any?}; the hunt refuses real network calls.",
617
+ "Replace its methods with a fake before setup, list the fake in outside_calls, and arm its failures (see the skill)."]
618
+ end
619
+
598
620
  # Methods a fake has and the real client does not, named like a failure:
599
621
  # each becomes a `failure`. One with a required argument needs a person
600
622
  # to say which record it fails for.
@@ -1147,6 +1169,7 @@ end
1147
1169
  armed = failures.select { |f| f.code.start_with?("failure") && selected.any? { reached.(it).include?(f.service) } }
1148
1170
  out = +"Sequences.define(#{name.inspect}) do\n"
1149
1171
  out << " # #{note}\n" if note
1172
+ network_note(selected).each { out << " # #{it}\n" }
1150
1173
  out << " setup do\n"
1151
1174
  fakes.each { |service, fake| out << " #{service}.client = #{fake}.new\n" }
1152
1175
  (api_auth[:envs].map { it[1] } | controller_envs).uniq.each { |env| out << " ENV[#{env.inspect}] = \"sequences-#{env.downcase.tr("_", "-")}\"\n" }
@@ -112,6 +112,12 @@ def verdict(r, limit)
112
112
  return ["unchecked", "-", "unchecked: #{r[:status] == :timeout ? "timed out after #{limit} s" : "no report (#{tail[0, 100]})"}"]
113
113
  end
114
114
  done, total, missing = scope(rep)
115
+ # A failure or a refusal that came from a real service says nothing about
116
+ # the app; the area checks nothing until the service is faked.
117
+ unless rep["network"].to_h.empty?
118
+ calls = rep["network"].map { |where, actions| "#{where}#{" (from #{actions.join(", ")})" unless actions.empty?}" }
119
+ return ["unchecked", "#{done}/#{total}", "unchecked: real network call to #{calls.join(", ")}; the hunt refuses it, so fake that service in the area's setup"]
120
+ end
115
121
  # The time limit stopped the area between cases with no failure found:
116
122
  # what ran still counts, and the row says it was cut short.
117
123
  if rep["verdict"].nil?
@@ -1,5 +1,5 @@
1
1
  # Runs generated sequences of an app's entry points
2
- # against real PostgreSQL: random orders (shrunk by Hegel), two actions
2
+ # against the app's real test database (PostgreSQL or SQLite): random orders (shrunk by Hegel), two actions
3
3
  # interleaved at a named pause point, and the same sequence replayed under
4
4
  # other process time zones. The hunt stage loads it into the app's test
5
5
  # process through SEQUENCES_LIB, so it needs hegeltest in that bundle.
@@ -36,6 +36,43 @@ module Sequences
36
36
  # A replayed step whose record is missing: the run went another way.
37
37
  class Diverged < StandardError; end
38
38
 
39
+ # An action opened a TCP connection to another host: a service the area
40
+ # does not fake. Its answer (or a DNS failure) is the environment, not the
41
+ # app, and a write through it would reach a real system.
42
+ class RealNetwork < StandardError; end
43
+
44
+ LOOPBACK = /\A(localhost|127\.\d+\.\d+\.\d+|::1|0\.0\.0\.0)\z/
45
+
46
+ def self.network_calls = (@network_calls ||= {})
47
+
48
+ def self.real_network!(host, port)
49
+ return if host.nil? || LOOPBACK.match?(host.to_s)
50
+
51
+ where = "#{host}:#{port}"
52
+ (network_calls[where] ||= []) << Thread.current[:sequences_action].to_s
53
+ raise RealNetwork, "real network call to #{where}"
54
+ end
55
+
56
+ module NetworkGate
57
+ def initialize(host, port, *rest, **kw)
58
+ Sequences.real_network!(host, port)
59
+ super
60
+ end
61
+ end
62
+
63
+ module SocketGate
64
+ def tcp(host, port, *rest, **kw, &blk)
65
+ Sequences.real_network!(host, port)
66
+ super
67
+ end
68
+ end
69
+
70
+ def self.refuse_network!
71
+ require "socket"
72
+ TCPSocket.prepend(NetworkGate) unless TCPSocket <= NetworkGate
73
+ Socket.singleton_class.prepend(SocketGate) unless Socket.singleton_class <= SocketGate
74
+ end
75
+
39
76
  # A pause point is a method on a fake (or any class) where an action can be
40
77
  # held while another action runs.
41
78
  module Pause
@@ -512,8 +549,18 @@ end
512
549
 
513
550
  # Steps the app accepted, which a double click or a retried request
514
551
  # would send once more.
552
+ # A step whose record an action since destroyed (a withdrawal of the
553
+ # one parcel) has nothing left to send again: its pool index now points
554
+ # past the end.
515
555
  def accepted_steps
516
- @log.select { |e| @definition.actions.key?(e[:step]) && e[:outcome] && !Sequences.kind(e[:outcome]).match?(/refused|HTTP 4/) }
556
+ @log.select do |e|
557
+ @definition.actions.key?(e[:step]) && e[:outcome] && !Sequences.kind(e[:outcome]).match?(/refused|HTTP 4/) &&
558
+ begin
559
+ resolve(@definition.actions.fetch(e[:step]), e[:args])
560
+ rescue Diverged
561
+ false
562
+ end
563
+ end
517
564
  end
518
565
 
519
566
  # Dates and times the app stored (due dates, deadlines, pauses), where a
@@ -522,9 +569,11 @@ end
522
569
  # took a quarter of the run.
523
570
  def stored_dates
524
571
  conn = ActiveRecord::Base.connection
572
+ # SQLite reads CAST(... AS date) as a number (the year); date() is its day.
573
+ day = conn.adapter_name.match?(/sqlite/i) ? ->(col) { "date(#{col})" } : ->(col) { "CAST(#{col} AS date)" }
525
574
  @dates_sql ||= (conn.tables - %w[schema_migrations ar_internal_metadata]).flat_map do |table|
526
575
  conn.columns(table).select { %i[date datetime].include?(it.type) && !@definition.ignore_columns.include?(it.name) }.map do |column|
527
- "SELECT DISTINCT CAST(#{conn.quote_column_name(column.name)} AS date) AS d FROM #{conn.quote_table_name(table)} WHERE #{conn.quote_column_name(column.name)} IS NOT NULL"
576
+ "SELECT DISTINCT #{day.(conn.quote_column_name(column.name))} AS d FROM #{conn.quote_table_name(table)} WHERE #{conn.quote_column_name(column.name)} IS NOT NULL"
528
577
  end
529
578
  end.join(" UNION ")
530
579
  return [] if @dates_sql.empty?
@@ -876,6 +925,7 @@ end
876
925
  @first_failure = nil
877
926
 
878
927
  Rails.application.eager_load!
928
+ refuse_network!
879
929
  quiet_errors
880
930
  seeds = ENV["SEQ_SEEDS"]&.split(",")&.map { Integer(it) } || [seed || Random.new_seed % 2**31]
881
931
  report = { area: definition.name, cases: test_cases, seeds: [], skipped: definition.skipped, actions: checks(definition),
@@ -918,6 +968,7 @@ end
918
968
  tallies = @tallies&.transform_values(&:dup) || {}
919
969
  $stderr.puts coverage_report
920
970
  report[:reached] = tallies.transform_keys(&:to_s)
971
+ report[:network] = network_calls.transform_values { it.uniq.reject(&:empty?) }
921
972
  report[:refused_with] = (@examples || {}).transform_values { |seen| seen.sort_by { -it[1] }.map(&:first) }
922
973
  @examples = nil
923
974
  # A refusal quotes a response body, which Rack hands over as binary; a
@@ -938,6 +989,7 @@ end
938
989
 
939
990
  def self.observe(definition, out:)
940
991
  Rails.application.eager_load!
992
+ refuse_network!
941
993
  quiet_errors
942
994
  length = Integer(ENV.fetch("SEQ_OBSERVE_LENGTH", "3"))
943
995
  cap = Integer(ENV.fetch("SEQ_OBSERVE_CAP", "240"))
@@ -1150,6 +1202,7 @@ end
1150
1202
  # groups actions into areas by it.
1151
1203
  def self.probe(definition, walks: 60, steps: 20, seed: 1)
1152
1204
  Rails.application.eager_load!
1205
+ refuse_network!
1153
1206
  quiet_errors
1154
1207
  random = Random.new(seed)
1155
1208
  stats = Hash.new { |h, k| h[k] = { writes: Hash.new(0), reads: Hash.new(0), outcomes: Hash.new(0), errors: Hash.new(0), refusals: [], runs: 0 } }
@@ -1221,7 +1274,7 @@ end
1221
1274
  # The fakes are plain objects that remember charges and failures armed; a
1222
1275
  # trial's rollback must undo those too.
1223
1276
  def self.save_outside_state
1224
- seqs = ActiveRecord::Base.connection.select_rows(sequence_state_sql)
1277
+ seqs = sequence_state_sql.empty? ? [] : ActiveRecord::Base.connection.select_rows(sequence_state_sql)
1225
1278
  fakes = outside_classes_declared.flat_map { |klass| ObjectSpace.each_object(klass).to_a }.map do |fake|
1226
1279
  [fake, fake.instance_variables.to_h { [it, Marshal.load(Marshal.dump(fake.instance_variable_get(it)))] }]
1227
1280
  rescue TypeError
@@ -1236,18 +1289,24 @@ end
1236
1289
  fakes.each { |fake, ivars| ivars.each { fake.instance_variable_set(_1, _2) } }
1237
1290
  end
1238
1291
 
1292
+ def self.postgres? = ActiveRecord::Base.connection.adapter_name.match?(/postg/i)
1293
+
1294
+ # Only PostgreSQL keeps id counters outside the transaction a trial rolls
1295
+ # back; SQLite's sqlite_sequence is a table, rolled back with the rows.
1239
1296
  def self.sequence_state_sql
1240
- @sequence_state_sql ||= begin
1241
- conn = ActiveRecord::Base.connection
1242
- names = conn.select_values("SELECT sequencename FROM pg_sequences WHERE schemaname = current_schema()")
1243
- names.map { "SELECT #{conn.quote(it)}, last_value, is_called FROM #{conn.quote_table_name(it)}" }.join(" UNION ALL ")
1244
- end
1297
+ @sequence_state_sql ||= if postgres?
1298
+ conn = ActiveRecord::Base.connection
1299
+ names = conn.select_values("SELECT sequencename FROM pg_sequences WHERE schemaname = current_schema()")
1300
+ names.map { "SELECT #{conn.quote(it)}, last_value, is_called FROM #{conn.quote_table_name(it)}" }.join(" UNION ALL ")
1301
+ else ""
1302
+ end
1245
1303
  end
1246
1304
 
1247
1305
  def self.outside_classes_declared = (@outside_classes_declared ||= Set.new)
1248
1306
 
1249
1307
  def self.replay(definition, entries)
1250
1308
  Rails.application.eager_load!
1309
+ refuse_network!
1251
1310
  runner = Runner.new(definition)
1252
1311
  runner.reset!(sizes: world_sizes(entries))
1253
1312
  runner.replay(entries)
@@ -1265,9 +1324,13 @@ end
1265
1324
  # Empties every table and restarts its ids, so two runs of the same steps
1266
1325
  # get the same ids. DELETE in foreign-key order (children first) took 1 ms
1267
1326
  # on this app's few rows where TRUNCATE ... RESTART IDENTITY took 72 ms.
1268
- # A schema whose keys form a cycle falls back to TRUNCATE.
1327
+ # A schema whose keys form a cycle falls back to TRUNCATE. SQLite has no
1328
+ # TRUNCATE and no sequences: DELETE with the foreign keys off, then
1329
+ # sqlite_sequence, which holds the ids to restart.
1269
1330
  def self.empty_tables
1270
1331
  conn = ActiveRecord::Base.connection
1332
+ return empty_tables_without_truncate(conn) unless postgres?
1333
+
1271
1334
  @empty_sql ||= begin
1272
1335
  tables = conn.tables - %w[schema_migrations ar_internal_metadata]
1273
1336
  parents = tables.to_h { |t| [t, conn.foreign_keys(t).map(&:to_table) - [t]] }
@@ -1290,6 +1353,15 @@ end
1290
1353
  conn.execute(@empty_sql)
1291
1354
  end
1292
1355
 
1356
+ def self.empty_tables_without_truncate(conn)
1357
+ @empty_statements ||= begin
1358
+ tables = conn.tables - %w[schema_migrations ar_internal_metadata]
1359
+ sqlite = conn.adapter_name.match?(/sqlite/i) && conn.select_value("SELECT 1 FROM sqlite_master WHERE name = 'sqlite_sequence'")
1360
+ tables.map { "DELETE FROM #{conn.quote_table_name(it)}" } + (sqlite ? ["DELETE FROM sqlite_sequence"] : [])
1361
+ end
1362
+ conn.disable_referential_integrity { @empty_statements.each { conn.execute(it) } }
1363
+ end
1364
+
1293
1365
 
1294
1366
  def self.with_zone(zone)
1295
1367
  old = ENV["TZ"]
@@ -34,8 +34,10 @@ module Holdfast
34
34
 
35
35
  per_area = [budget * 4 / [areas.size, 4].max, 60].max
36
36
  cases = budget >= 900 ? 300 : 60
37
- # Shrinking a failure takes minutes; a short hunt reports it as found.
38
- env["SEQ_NO_SHRINK"] = "1" if budget < 900
37
+ # Shrinking a failure can take minutes, and an unshrunk sequence of a
38
+ # dozen steps hides the one that matters: every hunt shrinks, for a
39
+ # quarter of an area's time limit, between 15 and 60 s.
40
+ env["SEQ_SHRINK_SECONDS"] ||= ENV.fetch("SEQ_SHRINK_SECONDS", (per_area / 4).clamp(15, 60).to_s)
39
41
  json = File.join(work, "table.json")
40
42
  FileUtils.rm_f(json)
41
43
  out, = Open3.capture2e(env, RbConfig.ruby, File.join(SEQUENCES, "run.rb"), "-j", "4", "--cases", cases.to_s, "--seeds", "1",
@@ -70,7 +72,7 @@ module Holdfast
70
72
  worst = thin.first&.first
71
73
  step = worst ? "start with #{worst["area"]}: make the records or the sign-in its unreached actions need in #{worst["file"].to_s.delete_prefix("#{change.root}/")}, then holdfast run" :
72
74
  "read each area's row below for the refusals, fix the setup or the sign-in in its file, then holdfast run"
73
- row("(hunt)", "unchecked", "unreached: #{total - reached} of #{total} actions (refused every time or never drawn); #{thin.size} of #{counts.size} areas reached under half",
75
+ row("(hunt)", "unchecked", "unreached: #{total - reached} of #{total} actions (refused every time or never drawn); #{thin.size} of #{counts.size} areas reached under half#{" (#{table.size - counts.size} area(s) that failed are not counted)" if table.size > counts.size}",
74
76
  evidence, "#{step}; until then the hunt says little about these areas")
75
77
  end
76
78
 
@@ -102,12 +104,21 @@ module Holdfast
102
104
  _, status = Open3.capture2e(probe_env, "/bin/sh", "-c", record.side_setup, chdir: change.root)
103
105
  return row("(sequences)", "unchecked", "the probe database did not set up", [], "holdfast run --learn") unless status.success?
104
106
  end
105
- return nil if Dir.glob(File.join(dir, "*.rb")).size > 1
107
+ # An area a person kept, merged or wrote is never drafted over.
108
+ return nil if Dir.glob(File.join(dir, "*.rb")).any? { File.basename(_1) != "invariants.rb" }
106
109
 
107
110
  out, status = Open3.capture2e(probe_env, "bundle", "exec", "ruby", File.join(SEQUENCES, "propose.rb"), "--all", chdir: change.root)
108
111
  return nil if status.success?
109
112
 
110
- row("(sequences)", "unchecked", "drafting the areas failed: #{out.lines.grep(/Error|error/).first.to_s.strip}", [],
113
+ error = out.lines.grep(/Error|error/).first.to_s.strip
114
+ # The first frame outside the gems says whose code raised: holdfast's
115
+ # own drafting is nothing the app or its factories can fix.
116
+ frame = out.lines.map(&:strip).find { _1.match?(%r{\A(?:from )?/\S+\.rb:\d+}) && (_1.include?(SEQUENCES) || !_1.include?("/gems/")) }.to_s
117
+ if frame.include?(SEQUENCES)
118
+ return row("(sequences)", "unchecked", "holdfast failed while drafting the areas (#{record.data.dig("database", "adapter")} adapter): #{error}", ["at: #{frame.delete_prefix("from ")}"],
119
+ "report it to holdfast with this row; the app and its factories are not the cause")
120
+ end
121
+ row("(sequences)", "unchecked", "drafting the areas failed: #{error}", [],
111
122
  "fix the error it names in the app or its factories, then holdfast run")
112
123
  end
113
124
 
@@ -164,7 +175,12 @@ module Holdfast
164
175
 
165
176
  row(area, "clean", "every action, offset replay and door pair reached its target (#{scope})", [], "nothing; the claim covers only these seeds and cases", scope:)
166
177
  else
167
- row(area, "unchecked", lines.first.to_s.delete_prefix("unchecked: "), ["log: #{log}"], "read the log, fix the setup it names, then holdfast run")
178
+ reason = lines.first.to_s.delete_prefix("unchecked: ")
179
+ if reason.start_with?("real network call")
180
+ return row(area, "unchecked", "environment: #{reason}", ["log: #{log}"],
181
+ "replace the service's methods with a fake in #{r["file"].to_s.delete_prefix("#{change.root}/")} and declare it with outside_calls (the skill shows how), then holdfast run")
182
+ end
183
+ row(area, "unchecked", reason, ["log: #{log}"], "read the log, fix the setup it names, then holdfast run")
168
184
  end
169
185
  end
170
186
 
@@ -38,7 +38,7 @@ module Holdfast
38
38
  unless change.changed?
39
39
  missing = change.named.reject { |n| kept.any? { named_match?(n, _1) } }
40
40
  verdicts += missing.map do |n|
41
- Verdict.of(target: n, stage: "select", kind: "unchecked", reason: "no method or file named #{n} at #{change.short(change.head)}",
41
+ Verdict.of(target: n, stage: "select", kind: "unchecked", reason: "no method or file named #{n} at #{change.tree_label}",
42
42
  next_step: "name it as Const#method, Const.method, or a path under app/, then run holdfast run --target again")
43
43
  end
44
44
  end
@@ -1,5 +1,5 @@
1
1
  module Holdfast
2
- VERSION = "0.1.0"
2
+ VERSION = "0.2.0"
3
3
  # The hegeltest the gem depends on, and the one the hunt adds to an app bundle that lacks it.
4
4
  HEGELTEST_VERSION = "0.5.1"
5
5
  end
@@ -7,13 +7,13 @@ description: Run what you suspect instead of leaving it "read only". In a Ruby o
7
7
 
8
8
  ## Setup
9
9
 
10
- Add `gem "holdfast", require: false` to the app's `:test` group and run `bundle install`; run every command below as `bundle exec holdfast ...` from the app root. jitcatch and branchout, which `holdfast run` uses on a change, come with it. The app needs RSpec, FactoryBot and a test database.
10
+ Add `gem "holdfast", require: false` to the app's `:test` group and run `bundle install`; run every command below as `bundle exec holdfast ...` from the app root. jitcatch and branchout, which `holdfast run` uses on a change, come with it. The app needs RSpec, FactoryBot and a test database on PostgreSQL or SQLite; races and the hunt run on both (MySQL is untested). SQLite lets one connection write at a time, so a race that holds an action after it wrote in its open transaction makes the other wait, and the row says `the second waited on a lock`.
11
11
 
12
12
  ## The loop: read, suspect, check, keep
13
13
 
14
14
  1. Read the code. Write each thing you suspect in a file, one line of `says` and its `given`, `do` (or `race`) and `expect`. A race or a cross-area effect costs the same few lines; do not leave one "read, not run".
15
15
  2. `holdfast check suspicions.yml` (or `--suspect '<one YAML mapping>'`, repeatable). Run it from the app root.
16
- 3. Keep each confirmed spec, `spec/holdfast/<branch>/<id>_spec.rb` (the header names the folder): it fails until the code is fixed. Report confirmed rows with their spec and refuted rows with what was observed. Fix each could_not_run suspicion and run again.
16
+ 3. Keep each confirmed spec, `spec/holdfast/<branch>/<id>_spec.rb` (the header names the folder; the id is the `says` text and a digest of it, so checking the same suspicion again replaces its spec, and two suspicions in one run must say different things): it fails until the code is fixed. Report confirmed rows with their spec and refuted rows with what was observed. Fix each could_not_run suspicion and run again.
17
17
 
18
18
  ```yaml
19
19
  - says: a passed tentative appointment is charged a no-show fee
@@ -36,7 +36,7 @@ Add `gem "holdfast", require: false` to the app's `:test` group and run `bundle
36
36
 
37
37
  - `given`: one record per key, from the factory of that name (`factory:`, `traits:`). A value is a literal, a relative time (`-1h`, `+2d`, `-20m`), an earlier key, or `{ruby: "..."}`. A string, or `{ruby: "..."}`, in place of the attributes is Ruby that returns the record or any value. An earlier record fills a required `belongs_to` you leave out, and so does what it belongs to.
38
38
  - `do`: Ruby, with each record by its name. A local or `@name` that `do` sets is what `expect` reads by that name (a record read again). Request helpers work: `post "/api/v1/..."`, then `response`. A 404 the app answers is an outcome; a path no route matches is could_not_run.
39
- - `race`: two actions. The first is held at `pause` while the second runs, then the reverse, and both serial orders. Confirmed only when a held order fails `expect` and ends in a way no serial order ends (ids of records the actions made and times do not count). When a serial order fails `expect` too, the row says so and is refuted as a race, unless the held order's end is new; then the row names that end and each serial end. `pause` is `batch_load` (the first query that returns rows), `outside_call` (the first call to a client such as `PaymentGateway.client`), `row_lock` (before the first `with_lock` or `lock!`), `first_write`, or `Class#method`. With no `pause`, or a `batch_load` nobody reaches, each is tried in that order; the row says which one held.
39
+ - `race`: two actions. The first is held at `pause` while the second runs, then the reverse, and both serial orders. Confirmed only when a held order fails `expect` and ends in a way no serial order ends (ids of records the actions made and times do not count). When a serial order fails `expect` too, the row says so and is refuted as a race, unless the held order's end is new; then the row names that end and each serial end. `pause` is `batch_load` (the first query that returns rows), `outside_call` (the first call to a client such as `PaymentGateway.client`, or to a module method `given` stubs, such as `allow(LockerApi).to receive(:register)`), `row_lock` (before the first `with_lock` or `lock!`), `first_write`, or `Class#method`. With no `pause`, or a `batch_load` nobody reaches, each is tried in that order; the row says which one held.
40
40
  - `expect`: Ruby that is true afterwards (records read again; `result`, `error`; in a race `results[0]`, `results[1]`, `errors[i]` by the order `race` lists them; `calls(X, :verb)`, the arguments of each call to `X.client`), or `no X verb`, `X verb [once|N times]`, `raises X`, `no error`; these combine with `&&` and `||`. A list must all hold. When `do` raised, a nil `response` or `result` in `expect` is that raise: the line fails and names it. A failed `a && b` names the part that failed and its value.
41
41
 
42
42
  What to suspect for a race: a lock taken on a different row than the one written (two members booking one room each lock their own member row); a read before the lock (a status or a total read, then `with_lock`); an absolute write after a slow call (`attempts = notice.attempts + 1`, a courier call, then `update!(attempts:)`); check-then-act with no unique index under it (`exists?` then `create!`). Write the two actions and the end no serial order allows; leave `pause` out.
@@ -49,7 +49,7 @@ In a change (the tree differs from the base branch, or `--base REF`), each suspi
49
49
  | `refuted` | `expect` held; each observed value is listed | Nothing to fix for this suspicion. |
50
50
  | `could_not_run` | Nothing was checked: the row names the `given` key and its error, the `do` line that raised, a request with no route or one turned away (401, 403), or that no pause point was reached (with the app methods the action called, to name as `pause`) | Fix the suspicion, run again. |
51
51
 
52
- An error that app code raised is how the action ended, and `expect` is checked against it. An error from your own line (a misspelt name, a wrong argument list) is could_not_run. Each branch keeps its own specs and rows: `.holdfast/check/<branch>/check.md` is the summary, `check.jsonl` beside it has every row.
52
+ An error that app code raised is how the action ended, and `expect` is checked against it. An error from your own line (a misspelt name, a wrong argument list) is could_not_run. A could_not_run row that starts `holdfast:` failed in holdfast's own support code, and its `next` names the database adapter: neither the suspicion nor the app is the cause. Each branch keeps its own specs and rows: `.holdfast/check/<branch>/check.md` is the summary, `check.jsonl` beside it has every row.
53
53
 
54
54
  ## Check a change: holdfast run
55
55
 
@@ -89,8 +89,35 @@ Sequences.everywhere do
89
89
  end
90
90
  ```
91
91
 
92
- A `bug_suspect` row names the broken rule, and its log holds the shortest sequence as a numbered narrative and an `it` block that replays it. Shrinking stops 60 s after the first failure (`SEQ_SHRINK_SECONDS`), and the row then says `shrinking stopped`; when the area's time limit cut it first, the row says `not shrunk`. Either way the log holds the shortest sequence found so far. A twins pair the two doors cannot feed the same inputs (one picks a record the other takes from the sign-in, or one makes a new unique value) is listed as not reached; drop the `twins` line or give both doors the same params. A partly checked area lists each action that was only refused (with up to three refusals, most common first, and the param to change) or never drawn: make the record in the area's `setup`, narrow a `pool`, or change the draws, then run again. An area the time limit stopped between cases says `cut by the time limit after N of M cases`; what ran still counts.
92
+ The hunt refuses a TCP connection to any host but this machine: an action that reaches a real service makes its area `unchecked` with `environment: real network call to HOST:PORT`, never `bug_suspect`. The drafted area fakes a service the specs install as `X.client = FakeX.new`, and names in a TODO a module that calls HTTP itself. Fake that one in the area file, list the fake in `outside_calls` (each of its methods becomes a pause point for interleavings and its state is rolled back with each trial), and arm its failures:
93
+
94
+ ```ruby
95
+ class FakeCourier # the courier as the app sees it: what it booked, and one armed outage
96
+ attr_reader :booked
97
+ def initialize = (@booked = []) && (@down = false)
98
+ def go_down! = (@down = true)
99
+ def book(order_id)
100
+ raise Courier::Unavailable, "fake outage" if @down && !(@down = false)
101
+
102
+ @booked << order_id
103
+ end
104
+ end
105
+ Courier.define_singleton_method(:book) { |order_id| $courier.book(order_id) }
106
+
107
+ Sequences.define("desk") do
108
+ outside_calls FakeCourier
109
+ setup { $courier = FakeCourier.new; create_list(:order, world_size(:orders, 1..3)) }
110
+ pool(:orders) { Order.order(:id) }
111
+ failure(:courier_down) { $courier.go_down! } # an action that lists it also runs as its own step with the fake armed
112
+ action(:ship, order: :orders, failures: [:courier_down]) { |order:| http.post "/orders/#{order.id}/shipment" }
113
+ action(:cancel, order: :orders) { |order:| http.post "/orders/#{order.id}/cancellation" }
114
+ end
115
+ ```
116
+
117
+ Each drafted area holds the actions that write the same tables, so a bug that needs two areas (a withdrawal, then the intake job) is out of reach. Merge them: move the pools, actions, failures and invariants into one `Sequences.define` and delete the other files. A hunt drafts again only when `.holdfast/sequences/` holds no area, so a merged area stays.
118
+
119
+ A `bug_suspect` row names the broken rule, and its log holds the shortest sequence as a numbered narrative and an `it` block that replays it. Every hunt shrinks: shrinking stops after a quarter of the area's time limit, between 15 and 60 s (`SEQ_SHRINK_SECONDS` sets it), and the row then says `shrinking stopped`; when the area's time limit cut it first, the row says `not shrunk`. Either way the log holds the shortest sequence found so far. A twins pair the two doors cannot feed the same inputs (one picks a record the other takes from the sign-in, or one makes a new unique value) is listed as not reached; drop the `twins` line or give both doors the same params. A partly checked area lists each action that was only refused (with up to three refusals, most common first, and the param to change) or never drawn: make the record in the area's `setup`, narrow a `pool`, or change the draws, then run again. An area the time limit stopped between cases says `cut by the time limit after N of M cases`; what ran still counts.
93
120
 
94
121
  ## When setup fails
95
122
 
96
- The first run learns `.holdfast/record.json` (test framework, databases: each app gets its own names through `DATABASE_URL`). A failed step is kept with its command and output, and every row says so. Fix the environment, then `holdfast run --learn`.
123
+ The first run learns `.holdfast/record.json` (test framework, databases: each side gets its own through `DATABASE_URL`, a name of this app's own on a server and a file under `.holdfast/db/` on SQLite; the app's own test database is never written). A failed step is kept with its command and output, and every row says so. Fix the environment, then `holdfast run --learn`.
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: holdfast
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - meganemura