holdfast 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.ja.md +26 -23
- data/README.md +21 -25
- data/lib/holdfast/change.rb +10 -2
- data/lib/holdfast/check.rb +19 -6
- data/lib/holdfast/harness.rb +1 -1
- data/lib/holdfast/record.rb +12 -3
- data/lib/holdfast/runtime/check_support.rb +37 -4
- data/lib/holdfast/sequences/propose.rb +23 -0
- data/lib/holdfast/sequences/run.rb +6 -0
- data/lib/holdfast/sequences/sequences.rb +82 -10
- data/lib/holdfast/stages/hunt.rb +22 -6
- data/lib/holdfast/stages/select.rb +1 -1
- data/lib/holdfast/version.rb +1 -1
- data/skills/holdfast/SKILL.md +33 -6
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 720fa32bd7e5a7cfe1c5e7cadbde8f4a48e96ecbf1872bf755a15f08b049e9ce
|
|
4
|
+
data.tar.gz: 3346a457162841be319397f673f971f5461e3ff6e2066391827df6d6ecaf79bc
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 7203acda8942792da156a24ee95c2a998ddc5cbf862859cc5f66b2be1337fbcb0e68afaf6060dd46083e4bfc11af9c131b779c098c6f87442e500e73aa1a4518
|
|
7
|
+
data.tar.gz: b349dd3e11a45a967438b5a0092d0682f6ea8fc9a3cdc23e565f73d81680ea74391a98d88b44126379ff18ac86186f322f91b4cc9e70febfe7e996e2e742afc2
|
data/README.ja.md
CHANGED
|
@@ -12,7 +12,7 @@ holdfast は、コードを読んで見つけたことを報告するコーデ
|
|
|
12
12
|
|
|
13
13
|
## Rails アプリでのクイックスタート
|
|
14
14
|
|
|
15
|
-
Ruby 3.4 以降、git、そして RSpec と FactoryBot
|
|
15
|
+
Ruby 3.4 以降、git、そして RSpec と FactoryBot で PostgreSQL か SQLite のテスト用データベースに対してスペックが走るアプリが必要です。
|
|
16
16
|
各ステップはアプリのルートで実行します。
|
|
17
17
|
|
|
18
18
|
### 1. gem を追加する
|
|
@@ -63,9 +63,9 @@ bundle exec holdfast check suspicions.yml
|
|
|
63
63
|
```text
|
|
64
64
|
# holdfast
|
|
65
65
|
|
|
66
|
-
check: 2 suspicion(s); no change: checks run on the current tree only (
|
|
66
|
+
check: 2 suspicion(s); no change: checks run on the current tree only (uncommitted changes on 62b0f46); app, lib, config, db and the Gemfile match main (62b0f46)
|
|
67
67
|
kept specs: spec/holdfast/main/ (branch main); rows: .holdfast/check/main/check.jsonl
|
|
68
|
-
seconds: record
|
|
68
|
+
seconds: record 4.2, run 2.4
|
|
69
69
|
|
|
70
70
|
confirmed=2 could_not_run=0 refuted=0
|
|
71
71
|
|
|
@@ -75,8 +75,8 @@ confirmed=2 could_not_run=0 refuted=0
|
|
|
75
75
|
- ran: NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]
|
|
76
76
|
- no PaymentGateway charge fails ([[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]])
|
|
77
77
|
- spec output: suspicion confirmed: no PaymentGateway charge does not hold (observed [[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]])
|
|
78
|
-
- repro: `DATABASE_URL=postgres://postgres@127.0.0.1:
|
|
79
|
-
- next: keep spec/holdfast/main/
|
|
78
|
+
- repro: `DATABASE_URL=postgres://postgres@127.0.0.1:5440/hf_clinic_c5da1e_c2 TEST_ENV_NUMBER=c2 bundle exec rspec spec/holdfast/main/a_passed_tentative_appointment_is_charged_a_73c143_spec.rb`
|
|
79
|
+
- next: keep spec/holdfast/main/a_passed_tentative_appointment_is_charged_a_73c143_spec.rb; it fails until the code is fixed
|
|
80
80
|
- `check-in between the sweep's load and its lock becomes a no-show`: first held at batch_load, second runs (held at Appointment load (1 rows)) ends [[Appointment#1(status: "no_show")], Appointment#1(status: "checked_in"); appointment.status == "checked_in" fails ("no_show")], an end no serial order reaches (first then second ends [[Appointment#1(status: "no_show")], raised CheckIn::Refused; appointment.status == "checked_in" fails ("no_show")]; second then first ends [[], Appointment#1(status: "checked_in"); appointment.status == "checked_in" holds ("checked_in")]); serial order "first then second" also fails expect, so this end is the evidence
|
|
81
81
|
- pause points tried: batch_load: Appointment load (1 rows)
|
|
82
82
|
- first held at batch_load, second runs: NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]; CheckIn.new(appointment).call returned Appointment#1(status: "checked_in"); held at Appointment load (1 rows); appointment.status == "checked_in" fails ("no_show")
|
|
@@ -84,8 +84,8 @@ confirmed=2 could_not_run=0 refuted=0
|
|
|
84
84
|
- second then first: CheckIn.new(appointment).call returned Appointment#1(status: "checked_in"); NoShowSweepJob.perform_now returned []; appointment.status == "checked_in" holds ("checked_in")
|
|
85
85
|
- second held at batch_load, first runs: CheckIn.new(appointment).call returned Appointment#1(status: "checked_in"); NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]; held at Appointment load (1 rows); the second waited on a lock; appointment.status == "checked_in" fails ("no_show")
|
|
86
86
|
- spec output: suspicion confirmed: first held at batch_load, second runs (held at Appointment load (1 rows)) ends [[Appointment#1(status: "no_show")], Appointment#1(status: "checked_in"); appointment.status == "checked_in" fails ("no_show")], an end no serial order reaches (first then second ends [[Appointment#1(status: "no_show")], raised CheckIn::Refused; appointment.status == "checked_in" fails ("no_show")]; second then first ends [[], Appointment#1(status: "checked_in"); appointment.status == "checked_in" holds ("checked_in")]); serial order "first then second" also fails expect, so this end is the evidence
|
|
87
|
-
- repro: `DATABASE_URL=postgres://postgres@127.0.0.1:
|
|
88
|
-
- next: keep spec/holdfast/main/
|
|
87
|
+
- repro: `DATABASE_URL=postgres://postgres@127.0.0.1:5440/hf_clinic_c5da1e_c2 TEST_ENV_NUMBER=c2 bundle exec rspec spec/holdfast/main/check_in_between_the_sweep_s_load_and_its_lo_d80794_spec.rb`
|
|
88
|
+
- next: keep spec/holdfast/main/check_in_between_the_sweep_s_load_and_its_lo_d80794_spec.rb; it fails until the code is fixed
|
|
89
89
|
rows: .holdfast/check/main/check.jsonl; kept specs: spec/holdfast/main/
|
|
90
90
|
```
|
|
91
91
|
|
|
@@ -99,7 +99,7 @@ confirmed の行はそれぞれ `spec/holdfast/<branch>/` の下のスペック
|
|
|
99
99
|
ジョブが確定済みの予約(booked)だけを選ぶように直すと、一つ目のスペックは通ります。
|
|
100
100
|
|
|
101
101
|
```sh
|
|
102
|
-
bundle exec rspec spec/holdfast/main/
|
|
102
|
+
bundle exec rspec spec/holdfast/main/a_passed_tentative_appointment_is_charged_a_73c143_spec.rb
|
|
103
103
|
```
|
|
104
104
|
|
|
105
105
|
```text
|
|
@@ -117,22 +117,22 @@ base のブランチと違うブランチの上では(`--base REF` で別の b
|
|
|
117
117
|
```text
|
|
118
118
|
# holdfast
|
|
119
119
|
|
|
120
|
-
check: 2 suspicion(s); change: each runs on base main (
|
|
120
|
+
check: 2 suspicion(s); change: each runs on base main (62b0f46) and on the working tree (uncommitted changes on 7c510b0); 1 file(s) differ: app/jobs/no_show_sweep_job.rb
|
|
121
121
|
kept specs: spec/holdfast/skip-tentative/ (branch skip-tentative); rows: .holdfast/check/skip-tentative/check.jsonl
|
|
122
|
-
seconds: record 0.0, run
|
|
122
|
+
seconds: record 0.0, run 2.6
|
|
123
123
|
|
|
124
124
|
confirmed=1 could_not_run=0 refuted=1
|
|
125
125
|
|
|
126
126
|
## Confirmed (the kept spec fails)
|
|
127
127
|
|
|
128
128
|
- `check-in between the sweep's load and its lock becomes a no-show`: first held at batch_load, second runs (held at Appointment load (1 rows)) ends [...]
|
|
129
|
-
- base
|
|
129
|
+
- base 62b0f46: confirmed: [...]; it was there before the change
|
|
130
130
|
[...]
|
|
131
131
|
|
|
132
132
|
## Refuted (what was observed)
|
|
133
133
|
|
|
134
134
|
- `a passed tentative appointment is charged a no-show fee`: no PaymentGateway charge holds (observed [])
|
|
135
|
-
- base
|
|
135
|
+
- base 62b0f46: confirmed: no PaymentGateway charge does not hold (observed [[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]]); the change removes it
|
|
136
136
|
- ran: NoShowSweepJob.perform_now returned []
|
|
137
137
|
- no PaymentGateway charge holds ([])
|
|
138
138
|
rows: .holdfast/check/skip-tentative/check.jsonl; kept specs: spec/holdfast/skip-tentative/
|
|
@@ -154,29 +154,25 @@ base にないメソッド、カラム、ルートを呼ぶ疑いは head でだ
|
|
|
154
154
|
次に、それらのアクションをランダムな順序でテスト用データベースに対して走らせます。
|
|
155
155
|
その際、不変条件、別のタイムゾーン、外部呼び出しでの二つのアクションの割り込み、最初のバッチ読み込みの後で止めたジョブ、受け付けられたリクエストをもう一度送ることを試します。
|
|
156
156
|
失敗は最短のシーケンスまで縮められます(縮める時間は最大 60 秒です)。
|
|
157
|
+
このマシン以外のホストへの接続は拒み、その領域は本物のサービスを呼んだと報告します。
|
|
158
|
+
そのサービスの偽物は、領域のファイルに書きます。
|
|
157
159
|
`--budget SECONDS` で実行時間の上限を決めます(既定は 600)。
|
|
158
160
|
クリニックアプリで、上限を 120 秒にした出力です。
|
|
159
161
|
|
|
160
162
|
```text
|
|
161
163
|
# holdfast
|
|
162
164
|
|
|
163
|
-
nothing in the app changed against main (
|
|
164
|
-
|
|
165
|
-
seconds: record 10.1, hunt 16.8
|
|
165
|
+
nothing in the app changed against main (b5c13d9)
|
|
166
|
+
seconds: record 4.6, hunt 40.8
|
|
166
167
|
|
|
167
|
-
unintended_change=0 bug_suspect=1 unchecked=
|
|
168
|
+
unintended_change=0 bug_suspect=1 unchecked=0 intended_change=0 clean=0
|
|
168
169
|
|
|
169
170
|
## Suspected bugs
|
|
170
171
|
|
|
171
|
-
- `sequences: check_ins`: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in"
|
|
172
|
-
- failure: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in"
|
|
172
|
+
- `sequences: check_ins`: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in" (shrinking stopped after 30 s)
|
|
173
|
+
- failure: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in" (shrinking stopped after 30 s)
|
|
173
174
|
- log: .holdfast/runs/sequences/check_ins.log (the shortest sequence and an `it` block that replays it)
|
|
174
175
|
- next: paste the `it` block from the log into .holdfast/sequences/check_ins.rb, see it fail, fix the code, run again
|
|
175
|
-
|
|
176
|
-
## Not checked
|
|
177
|
-
|
|
178
|
-
- `(hunt)`: unreached: 1 of 2 actions (refused every time or never drawn); 0 of 1 areas reached under half
|
|
179
|
-
- next: read each area's row below for the refusals, fix the setup or the sign-in in its file, then holdfast run; until then the hunt says little about these areas
|
|
180
176
|
rows: .holdfast/verdicts.jsonl
|
|
181
177
|
```
|
|
182
178
|
|
|
@@ -204,8 +200,15 @@ holdfast はアプリの `.holdfast/` と `spec/holdfast/` の下に書き込み
|
|
|
204
200
|
|
|
205
201
|
各アプリは、そのディレクトリから作った自分専用のテスト用データベース名を `DATABASE_URL` で受け取ります。
|
|
206
202
|
そのため、base と head、また一つのサーバー上の二つのアプリが、同じデータベースを共有しません。
|
|
203
|
+
SQLite では、それぞれの側が `.holdfast/db/` の下に自分専用のファイルを受け取り、アプリ自身のテスト用データベースのファイルには書き込みません。
|
|
207
204
|
セットアップは `bin/rails db:create db:schema:load` を実行します。
|
|
208
205
|
|
|
206
|
+
競合とバグ探しは PostgreSQL と SQLite で走ります。
|
|
207
|
+
SQLite では、一度に書き込める接続は一つです。
|
|
208
|
+
競合で止めたアクションが、開いたトランザクションの中ですでに書き込んでいると、もう一方のアクションはそれを待ちます。
|
|
209
|
+
その行には `the second waited on a lock` と出ます。
|
|
210
|
+
MySQL は試していません。
|
|
211
|
+
|
|
209
212
|
## 制限
|
|
210
213
|
|
|
211
214
|
- 入力がレコードを一つも作らないピンの下書きは何も確かめませんが、そのピンが変更された行を読み込むと、`holdfast run` はその対象を `clean` と示すことがあります。
|
data/README.md
CHANGED
|
@@ -6,7 +6,7 @@ It is made for a coding agent that reads code and has to say what it found. The
|
|
|
6
6
|
|
|
7
7
|
## Quick start in your Rails app
|
|
8
8
|
|
|
9
|
-
You need Ruby 3.4 or newer, git, and an app whose specs run with RSpec and FactoryBot against a test database. Run each step from the app's root.
|
|
9
|
+
You need Ruby 3.4 or newer, git, and an app whose specs run with RSpec and FactoryBot against a test database on PostgreSQL or SQLite. Run each step from the app's root.
|
|
10
10
|
|
|
11
11
|
### 1. Add the gem
|
|
12
12
|
|
|
@@ -46,9 +46,9 @@ The first run learns how the app runs its tests and makes test databases of its
|
|
|
46
46
|
```text
|
|
47
47
|
# holdfast
|
|
48
48
|
|
|
49
|
-
check: 2 suspicion(s); no change: checks run on the current tree only (
|
|
49
|
+
check: 2 suspicion(s); no change: checks run on the current tree only (uncommitted changes on 62b0f46); app, lib, config, db and the Gemfile match main (62b0f46)
|
|
50
50
|
kept specs: spec/holdfast/main/ (branch main); rows: .holdfast/check/main/check.jsonl
|
|
51
|
-
seconds: record
|
|
51
|
+
seconds: record 4.2, run 2.4
|
|
52
52
|
|
|
53
53
|
confirmed=2 could_not_run=0 refuted=0
|
|
54
54
|
|
|
@@ -58,8 +58,8 @@ confirmed=2 could_not_run=0 refuted=0
|
|
|
58
58
|
- ran: NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]
|
|
59
59
|
- no PaymentGateway charge fails ([[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]])
|
|
60
60
|
- spec output: suspicion confirmed: no PaymentGateway charge does not hold (observed [[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]])
|
|
61
|
-
- repro: `DATABASE_URL=postgres://postgres@127.0.0.1:
|
|
62
|
-
- next: keep spec/holdfast/main/
|
|
61
|
+
- repro: `DATABASE_URL=postgres://postgres@127.0.0.1:5440/hf_clinic_c5da1e_c2 TEST_ENV_NUMBER=c2 bundle exec rspec spec/holdfast/main/a_passed_tentative_appointment_is_charged_a_73c143_spec.rb`
|
|
62
|
+
- next: keep spec/holdfast/main/a_passed_tentative_appointment_is_charged_a_73c143_spec.rb; it fails until the code is fixed
|
|
63
63
|
- `check-in between the sweep's load and its lock becomes a no-show`: first held at batch_load, second runs (held at Appointment load (1 rows)) ends [[Appointment#1(status: "no_show")], Appointment#1(status: "checked_in"); appointment.status == "checked_in" fails ("no_show")], an end no serial order reaches (first then second ends [[Appointment#1(status: "no_show")], raised CheckIn::Refused; appointment.status == "checked_in" fails ("no_show")]; second then first ends [[], Appointment#1(status: "checked_in"); appointment.status == "checked_in" holds ("checked_in")]); serial order "first then second" also fails expect, so this end is the evidence
|
|
64
64
|
- pause points tried: batch_load: Appointment load (1 rows)
|
|
65
65
|
- first held at batch_load, second runs: NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]; CheckIn.new(appointment).call returned Appointment#1(status: "checked_in"); held at Appointment load (1 rows); appointment.status == "checked_in" fails ("no_show")
|
|
@@ -67,8 +67,8 @@ confirmed=2 could_not_run=0 refuted=0
|
|
|
67
67
|
- second then first: CheckIn.new(appointment).call returned Appointment#1(status: "checked_in"); NoShowSweepJob.perform_now returned []; appointment.status == "checked_in" holds ("checked_in")
|
|
68
68
|
- second held at batch_load, first runs: CheckIn.new(appointment).call returned Appointment#1(status: "checked_in"); NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]; held at Appointment load (1 rows); the second waited on a lock; appointment.status == "checked_in" fails ("no_show")
|
|
69
69
|
- spec output: suspicion confirmed: first held at batch_load, second runs (held at Appointment load (1 rows)) ends [[Appointment#1(status: "no_show")], Appointment#1(status: "checked_in"); appointment.status == "checked_in" fails ("no_show")], an end no serial order reaches (first then second ends [[Appointment#1(status: "no_show")], raised CheckIn::Refused; appointment.status == "checked_in" fails ("no_show")]; second then first ends [[], Appointment#1(status: "checked_in"); appointment.status == "checked_in" holds ("checked_in")]); serial order "first then second" also fails expect, so this end is the evidence
|
|
70
|
-
- repro: `DATABASE_URL=postgres://postgres@127.0.0.1:
|
|
71
|
-
- next: keep spec/holdfast/main/
|
|
70
|
+
- repro: `DATABASE_URL=postgres://postgres@127.0.0.1:5440/hf_clinic_c5da1e_c2 TEST_ENV_NUMBER=c2 bundle exec rspec spec/holdfast/main/check_in_between_the_sweep_s_load_and_its_lo_d80794_spec.rb`
|
|
71
|
+
- next: keep spec/holdfast/main/check_in_between_the_sweep_s_load_and_its_lo_d80794_spec.rb; it fails until the code is fixed
|
|
72
72
|
rows: .holdfast/check/main/check.jsonl; kept specs: spec/holdfast/main/
|
|
73
73
|
```
|
|
74
74
|
|
|
@@ -79,7 +79,7 @@ The race is confirmed only because the held order ends in a state that no serial
|
|
|
79
79
|
Each confirmed row names a spec under `spec/holdfast/<branch>/` that fails until the code is fixed. Fix the code, run the spec, and keep it. After the sweep selects only booked appointments, the first spec passes:
|
|
80
80
|
|
|
81
81
|
```sh
|
|
82
|
-
bundle exec rspec spec/holdfast/main/
|
|
82
|
+
bundle exec rspec spec/holdfast/main/a_passed_tentative_appointment_is_charged_a_73c143_spec.rb
|
|
83
83
|
```
|
|
84
84
|
|
|
85
85
|
```text
|
|
@@ -95,22 +95,22 @@ On a branch that differs from the base branch (`--base REF` names another), each
|
|
|
95
95
|
```text
|
|
96
96
|
# holdfast
|
|
97
97
|
|
|
98
|
-
check: 2 suspicion(s); change: each runs on base main (
|
|
98
|
+
check: 2 suspicion(s); change: each runs on base main (62b0f46) and on the working tree (uncommitted changes on 7c510b0); 1 file(s) differ: app/jobs/no_show_sweep_job.rb
|
|
99
99
|
kept specs: spec/holdfast/skip-tentative/ (branch skip-tentative); rows: .holdfast/check/skip-tentative/check.jsonl
|
|
100
|
-
seconds: record 0.0, run
|
|
100
|
+
seconds: record 0.0, run 2.6
|
|
101
101
|
|
|
102
102
|
confirmed=1 could_not_run=0 refuted=1
|
|
103
103
|
|
|
104
104
|
## Confirmed (the kept spec fails)
|
|
105
105
|
|
|
106
106
|
- `check-in between the sweep's load and its lock becomes a no-show`: first held at batch_load, second runs (held at Appointment load (1 rows)) ends [...]
|
|
107
|
-
- base
|
|
107
|
+
- base 62b0f46: confirmed: [...]; it was there before the change
|
|
108
108
|
[...]
|
|
109
109
|
|
|
110
110
|
## Refuted (what was observed)
|
|
111
111
|
|
|
112
112
|
- `a passed tentative appointment is charged a no-show fee`: no PaymentGateway charge holds (observed [])
|
|
113
|
-
- base
|
|
113
|
+
- base 62b0f46: confirmed: no PaymentGateway charge does not hold (observed [[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]]); the change removes it
|
|
114
114
|
- ran: NoShowSweepJob.perform_now returned []
|
|
115
115
|
- no PaymentGateway charge holds ([])
|
|
116
116
|
rows: .holdfast/check/skip-tentative/check.jsonl; kept specs: spec/holdfast/skip-tentative/
|
|
@@ -122,28 +122,22 @@ rows: .holdfast/check/skip-tentative/check.jsonl; kept specs: spec/holdfast/skip
|
|
|
122
122
|
|
|
123
123
|
## Hunt when nothing changed
|
|
124
124
|
|
|
125
|
-
`holdfast run` with no change drafts one sequence definition per write area of the app (routes, Grape endpoints, jobs) into `.holdfast/sequences/`. It then runs random orders of their actions against the test database, under invariants, other time zones, two actions interleaved at an outside call, a job held after its first batch load, and an accepted request sent once more. A failure is shrunk to the shortest sequence, for at most 60 seconds. `--budget SECONDS` caps the run (default 600). On the clinic app, with a budget of 120 seconds:
|
|
125
|
+
`holdfast run` with no change drafts one sequence definition per write area of the app (routes, Grape endpoints, jobs) into `.holdfast/sequences/`. It then runs random orders of their actions against the test database, under invariants, other time zones, two actions interleaved at an outside call, a job held after its first batch load, and an accepted request sent once more. A failure is shrunk to the shortest sequence, for at most 60 seconds. A connection to any host but this machine is refused, and the area reports the real service it reached; write a fake of that service in the area's file. `--budget SECONDS` caps the run (default 600). On the clinic app, with a budget of 120 seconds:
|
|
126
126
|
|
|
127
127
|
```text
|
|
128
128
|
# holdfast
|
|
129
129
|
|
|
130
|
-
nothing in the app changed against main (
|
|
131
|
-
|
|
132
|
-
seconds: record 10.1, hunt 16.8
|
|
130
|
+
nothing in the app changed against main (b5c13d9)
|
|
131
|
+
seconds: record 4.6, hunt 40.8
|
|
133
132
|
|
|
134
|
-
unintended_change=0 bug_suspect=1 unchecked=
|
|
133
|
+
unintended_change=0 bug_suspect=1 unchecked=0 intended_change=0 clean=0
|
|
135
134
|
|
|
136
135
|
## Suspected bugs
|
|
137
136
|
|
|
138
|
-
- `sequences: check_ins`: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in"
|
|
139
|
-
- failure: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in"
|
|
137
|
+
- `sequences: check_ins`: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in" (shrinking stopped after 30 s)
|
|
138
|
+
- failure: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in" (shrinking stopped after 30 s)
|
|
140
139
|
- log: .holdfast/runs/sequences/check_ins.log (the shortest sequence and an `it` block that replays it)
|
|
141
140
|
- next: paste the `it` block from the log into .holdfast/sequences/check_ins.rb, see it fail, fix the code, run again
|
|
142
|
-
|
|
143
|
-
## Not checked
|
|
144
|
-
|
|
145
|
-
- `(hunt)`: unreached: 1 of 2 actions (refused every time or never drawn); 0 of 1 areas reached under half
|
|
146
|
-
- next: read each area's row below for the refusals, fix the setup or the sign-in in its file, then holdfast run; until then the hunt says little about these areas
|
|
147
141
|
rows: .holdfast/verdicts.jsonl
|
|
148
142
|
```
|
|
149
143
|
|
|
@@ -164,7 +158,9 @@ holdfast show
|
|
|
164
158
|
|
|
165
159
|
holdfast writes under `.holdfast/` and `spec/holdfast/` in the app. Add `.holdfast/` to the app's `.gitignore`. Commit the specs you keep.
|
|
166
160
|
|
|
167
|
-
Each app gets its own test database names through `DATABASE_URL`, made from its directory, so base and head, and two apps on one server, do not share one. The setup runs `bin/rails db:create db:schema:load`.
|
|
161
|
+
Each app gets its own test database names through `DATABASE_URL`, made from its directory, so base and head, and two apps on one server, do not share one. On SQLite, each side gets a file of its own under `.holdfast/db/`, and the app's own test database file is never written. The setup runs `bin/rails db:create db:schema:load`.
|
|
162
|
+
|
|
163
|
+
Races and the hunt run on PostgreSQL and on SQLite. SQLite lets one connection write at a time: when a race holds an action that has already written in its open transaction, the other action waits for it, and the row says `the second waited on a lock`. MySQL is not tested.
|
|
168
164
|
|
|
169
165
|
## Limits
|
|
170
166
|
|
data/lib/holdfast/change.rb
CHANGED
|
@@ -48,10 +48,18 @@ module Holdfast
|
|
|
48
48
|
|
|
49
49
|
def changed? = mode == "change"
|
|
50
50
|
def short(sha) = sha[0, 7]
|
|
51
|
+
|
|
52
|
+
# A dirty tree's snapshot is a commit no `git log` shows; the reader
|
|
53
|
+
# gets the commit it sits on instead.
|
|
54
|
+
def tree_label
|
|
55
|
+
committed = Change.git(root, "rev-parse", "HEAD")
|
|
56
|
+
committed == head ? short(head) : "uncommitted changes on #{short(committed)}"
|
|
57
|
+
end
|
|
58
|
+
|
|
51
59
|
def header
|
|
52
60
|
case mode
|
|
53
|
-
when "change" then "change: #{base_label} (#{short(base)}) .. working tree (#{
|
|
54
|
-
when "before" then "before a change: pinning #{named.join(", ")} at #{
|
|
61
|
+
when "change" then "change: #{base_label} (#{short(base)}) .. working tree (#{tree_label})"
|
|
62
|
+
when "before" then "before a change: pinning #{named.join(", ")} at #{tree_label}"
|
|
55
63
|
else "#{Change.git(root, "diff", "--quiet", base, head).then { "" } rescue "only tests, factories or docs differ from #{base_label}; " }nothing in the app changed against #{base_label} (#{short(base)})"
|
|
56
64
|
end
|
|
57
65
|
end
|
data/lib/holdfast/check.rb
CHANGED
|
@@ -55,11 +55,11 @@ module Holdfast
|
|
|
55
55
|
# which files made it a change, and one that did not says so.
|
|
56
56
|
def self.scope_line(change, here)
|
|
57
57
|
base = "#{change.base_label} (#{change.short(change.base)})"
|
|
58
|
-
return "no change: checks run on the current tree only (#{change.
|
|
59
|
-
return "--here: checks run on the current tree only (#{change.
|
|
58
|
+
return "no change: checks run on the current tree only (#{change.tree_label}); app, lib, config, db and the Gemfile match #{base}" unless change.changed?
|
|
59
|
+
return "--here: checks run on the current tree only (#{change.tree_label}); the change against #{base} is not compared" if here
|
|
60
60
|
|
|
61
61
|
files = Change.git(change.root, "diff", "--name-only", change.base, change.head, "--", *Holdfast::CODE).lines.map(&:strip)
|
|
62
|
-
"change: each runs on base #{base} and on the working tree (#{change.
|
|
62
|
+
"change: each runs on base #{base} and on the working tree (#{change.tree_label}); #{files.size} file(s) differ: #{files.first(5).join(", ")}#{" and #{files.size - 5} more" if files.size > 5}"
|
|
63
63
|
end
|
|
64
64
|
|
|
65
65
|
# A file holds a list of suspicions (or `suspicions:` with one); each
|
|
@@ -79,7 +79,11 @@ module Holdfast
|
|
|
79
79
|
raw += inline.map { [YAML.safe_load(quote_ruby(_1)), "--suspect"] }
|
|
80
80
|
raise Holdfast::Error, "no suspicion given; pass a YAML file or --suspect" if raw.empty?
|
|
81
81
|
|
|
82
|
-
raw.each_with_index.map { |(h, from), i| suspicion(h, i, from) }
|
|
82
|
+
list = raw.each_with_index.map { |(h, from), i| suspicion(h, i, from) }
|
|
83
|
+
twice = list.group_by(&:id).select { _2.size > 1 }.values.map { _1.first.says }
|
|
84
|
+
raise Holdfast::Error, "two suspicions say #{twice.first.inspect}; give each its own `says`" if twice.any?
|
|
85
|
+
|
|
86
|
+
list
|
|
83
87
|
end
|
|
84
88
|
|
|
85
89
|
# `do: x.call(a: 1)` is Ruby, not a YAML mapping: a plain value of says,
|
|
@@ -155,7 +159,10 @@ module Holdfast
|
|
|
155
159
|
set = acts.flat_map { ivars(_1, :write) }
|
|
156
160
|
unset = expects.flat_map { ivars(_1[:ruby], :read) }.uniq - set
|
|
157
161
|
raise Holdfast::Error, "#{label}: `expect` reads #{unset.join(", ")}, which no `do` sets; set it in `do`, or read the record by its `given` name" if unset.any?
|
|
158
|
-
|
|
162
|
+
# Named by its text, not its place in the file: a kept spec from an
|
|
163
|
+
# earlier run is the same suspicion only when it says the same thing.
|
|
164
|
+
text = says.empty? ? acts.first : says
|
|
165
|
+
Suspicion.new(id: "#{Holdfast.slug(text).tr("-", "_")[0, 44].sub(/_\z/, "")}_#{Digest::SHA1.hexdigest(text)[0, 6]}",
|
|
159
166
|
says:, given: Hash(h["given"]), acts:, pause: h["pause"]&.to_s,
|
|
160
167
|
expects:, calls:, errors:, source: from, shared:)
|
|
161
168
|
end
|
|
@@ -403,7 +410,10 @@ module Holdfast
|
|
|
403
410
|
msg = run["log"].to_s.lines.grep(/Error|error/).first.to_s.strip if msg.empty?
|
|
404
411
|
frame = Array(ex&.dig("exception", "backtrace")).find { _1.include?("spec/holdfast/") }
|
|
405
412
|
msg = "rspec printed nothing for it; read #{run["out"]}/rspec.log" if msg.empty?
|
|
406
|
-
|
|
413
|
+
# Raised first in holdfast's own support file, not in the suspicion's
|
|
414
|
+
# lines or the app: nothing in the suspicion can fix it.
|
|
415
|
+
own = Array(ex&.dig("exception", "backtrace")).find { !_1.include?("/gems/") }.to_s.include?("/holdfast_check.rb:")
|
|
416
|
+
{ "result" => "could_not_run", "phase" => own ? "holdfast" : "spec", "error" => "the spec did not finish: #{ex&.dig("exception", "class")} #{msg[0, 400]}#{" at #{frame[%r{spec/holdfast/\S+}]}" if frame}".squeeze(" ") }
|
|
407
417
|
end
|
|
408
418
|
|
|
409
419
|
def self.describe(r)
|
|
@@ -464,6 +474,9 @@ module Holdfast
|
|
|
464
474
|
when "given" then "fix the named part of `given` (an attribute, a factory: or a traits: key), then holdfast check again"
|
|
465
475
|
when "do" then "the action raised before `expect` ran; if that error is the bug, write `expect: no error`, else fix `do`"
|
|
466
476
|
when "pause" then "name a method the first action passes through as `pause: Class#method`"
|
|
477
|
+
when "holdfast"
|
|
478
|
+
"holdfast's own support code failed on the #{record.data.dig("database", "adapter") || "unknown"} adapter, not the suspicion or the app; " \
|
|
479
|
+
"report it to holdfast with this row (races and the hunt run on PostgreSQL and SQLite)"
|
|
467
480
|
else "read the error, fix the suspicion, then holdfast check again"
|
|
468
481
|
end
|
|
469
482
|
Verdict.of(target: s.says, stage: "check", kind: "could_not_run", reason: "#{head["phase"] || "spec"}: #{head["error"]}", evidence:, scope: spec_path, repro:, next_step: step)
|
data/lib/holdfast/harness.rb
CHANGED
|
@@ -75,7 +75,7 @@ module Holdfast
|
|
|
75
75
|
ran = pins.observed.count { !_1["input_error"] }
|
|
76
76
|
kind = ran.zero? ? "unchecked" : "clean"
|
|
77
77
|
rows << Verdict.of(target: pins.target.name, stage: "fill", kind:,
|
|
78
|
-
reason: ran.zero? ? "no input ran" : "pinned #{ran} input(s) at #{change.
|
|
78
|
+
reason: ran.zero? ? "no input ran" : "pinned #{ran} input(s) at #{change.tree_label} by what each returns, raises and writes",
|
|
79
79
|
evidence: details + pins.target.tests.first(3).map { "selected test: #{_1}" },
|
|
80
80
|
scope: pins.dir.delete_prefix("#{change.root}/"),
|
|
81
81
|
next_step: ran.zero? ? "fix the inputs the evidence names, then holdfast run" : "make the change, then holdfast run; these pins are compared on both sides")
|
data/lib/holdfast/record.rb
CHANGED
|
@@ -23,7 +23,7 @@ module Holdfast
|
|
|
23
23
|
def self.learn(root)
|
|
24
24
|
facts = read_facts(root)
|
|
25
25
|
steps = []
|
|
26
|
-
if facts["rails"] && facts["database"]["
|
|
26
|
+
if facts["rails"] && facts["database"]["url"]
|
|
27
27
|
SIDES.each do |side, number|
|
|
28
28
|
steps << run_step(root, "create the #{side} database (#{database_name(facts, number) || "TEST_ENV_NUMBER=#{number}"})", facts["side_setup"],
|
|
29
29
|
side_env(facts, number))
|
|
@@ -46,15 +46,18 @@ module Holdfast
|
|
|
46
46
|
test_section = db_yml[/^test:.*\z/m].to_s
|
|
47
47
|
adapter = test_section[/adapter:\s*(\w+)/, 1] || db_yml[/adapter:\s*(\w+)/, 1]
|
|
48
48
|
server = %w[postgresql mysql2 trilogy].include?(adapter)
|
|
49
|
+
url = if server then database_url(root, adapter)
|
|
50
|
+
elsif adapter == "sqlite3" then sqlite_url(root)
|
|
51
|
+
end
|
|
49
52
|
{
|
|
50
53
|
"framework" => framework,
|
|
51
54
|
"bundle" => File.file?(File.join(root, "Gemfile")) ? "bundle exec " : "",
|
|
52
55
|
"rails" => File.file?(File.join(root, "config/environment.rb")),
|
|
53
56
|
"helper" => framework == "rspec" ? (File.file?(File.join(root, "spec/rails_helper.rb")) ? "rails_helper" : "spec_helper") : "test_helper",
|
|
54
57
|
"database" => { "adapter" => adapter, "server" => server, "numbered" => test_section.include?("TEST_ENV_NUMBER"),
|
|
55
|
-
"url" =>
|
|
58
|
+
"url" => url }.compact,
|
|
56
59
|
"env" => db_yml.scan(/ENV(?:\.fetch\(|\[)\s*["'](\w+)["']/).flatten.uniq - %w[TEST_ENV_NUMBER RAILS_ENV],
|
|
57
|
-
"side_setup" =>
|
|
60
|
+
"side_setup" => url ? "bin/rails db:create db:schema:load" : nil,
|
|
58
61
|
"base_branch" => base_branch(root),
|
|
59
62
|
"coverage" => COVERAGE.find { File.file?(File.join(root, _1)) }
|
|
60
63
|
}
|
|
@@ -76,6 +79,12 @@ module Holdfast
|
|
|
76
79
|
nil
|
|
77
80
|
end
|
|
78
81
|
|
|
82
|
+
# A SQLite database.yml names one file, and TEST_ENV_NUMBER changes it
|
|
83
|
+
# only where the file name reads it: base, head and each hunt worker
|
|
84
|
+
# wrote the app's own storage/test.sqlite3, and emptied it after a race.
|
|
85
|
+
# Each gets a file of its own under .holdfast/db/ instead.
|
|
86
|
+
def self.sqlite_url(root) = "sqlite3:#{File.join(File.expand_path(Holdfast.dir(root)), "db", "test")}"
|
|
87
|
+
|
|
79
88
|
def self.side_env(facts, number)
|
|
80
89
|
url = facts.dig("database", "url")
|
|
81
90
|
{ "RAILS_ENV" => "test", "TEST_ENV_NUMBER" => number }.merge(url ? { "DATABASE_URL" => "#{url}_#{number}" } : {})
|
|
@@ -24,7 +24,7 @@ module HoldfastCheck
|
|
|
24
24
|
# first time the held action passes it; `Class#method` is the fifth kind.
|
|
25
25
|
POINTS = {
|
|
26
26
|
"batch_load" => "its first query that returns rows",
|
|
27
|
-
"outside_call" => "its first call to an outside-service client",
|
|
27
|
+
"outside_call" => "its first call to an outside-service client, or to a module method the suspicion stubs",
|
|
28
28
|
"row_lock" => "its first with_lock or lock!, before the lock is taken",
|
|
29
29
|
"first_write" => "its first INSERT, UPDATE or DELETE"
|
|
30
30
|
}.freeze
|
|
@@ -125,6 +125,29 @@ module HoldfastCheck
|
|
|
125
125
|
end
|
|
126
126
|
OutsideGate = Module.new
|
|
127
127
|
|
|
128
|
+
# A service the suspicion stubs on a module (`allow(LockerApi).to
|
|
129
|
+
# receive(:register)`) is an outside call too, and `pause:
|
|
130
|
+
# LockerApi.register` names it. RSpec's stub sits in front of any hook
|
|
131
|
+
# prepended to the module, so the hold is on the stub's own invocation.
|
|
132
|
+
module StubGate
|
|
133
|
+
private
|
|
134
|
+
|
|
135
|
+
def invoke_incrementing_actual_calls_by(*args, &block)
|
|
136
|
+
if orig_object.is_a?(Module) && orig_object.name
|
|
137
|
+
gate = HoldfastCheck.gate
|
|
138
|
+
HoldfastCheck.hold(gate, "outside_call", "#{orig_object.name}.#{message}")
|
|
139
|
+
HoldfastCheck.hold(gate, "#{orig_object.name}.#{message}", "#{orig_object.name}.#{message}")
|
|
140
|
+
end
|
|
141
|
+
super
|
|
142
|
+
end
|
|
143
|
+
end
|
|
144
|
+
# rspec-mocks loads when the first example runs, after this file.
|
|
145
|
+
def self.watch_stubs
|
|
146
|
+
return unless defined?(RSpec::Mocks::MessageExpectation)
|
|
147
|
+
|
|
148
|
+
RSpec::Mocks::MessageExpectation.prepend(StubGate) unless RSpec::Mocks::MessageExpectation <= StubGate
|
|
149
|
+
end
|
|
150
|
+
|
|
128
151
|
def self.method_gate(point)
|
|
129
152
|
@method_gates ||= {}
|
|
130
153
|
return if @method_gates[point]
|
|
@@ -338,6 +361,7 @@ module HoldfastCheck
|
|
|
338
361
|
def holdfast_race(given, acts, expects, calls, pause, shared)
|
|
339
362
|
Rails.application.eager_load!
|
|
340
363
|
HoldfastCheck.watch_writes
|
|
364
|
+
HoldfastCheck.watch_stubs
|
|
341
365
|
named = pause && pause != "batch_load"
|
|
342
366
|
HoldfastCheck.method_gate(pause) if named && !POINTS.key?(pause)
|
|
343
367
|
points = named ? [pause] : ["batch_load", *FALLBACKS]
|
|
@@ -531,15 +555,18 @@ module HoldfastCheck
|
|
|
531
555
|
def holdfast_step(records, act, errors:)
|
|
532
556
|
scope = Scope.new(self, records)
|
|
533
557
|
result = scope.instance_exec(&act.block)
|
|
534
|
-
{ result:, scope:, shown: "#{act.text} returned #{holdfast_show(result)}" }
|
|
558
|
+
{ result:, scope:, shown: "#{HoldfastCheck.one_line(act.text)} returned #{holdfast_show(result)}" }
|
|
535
559
|
rescue StandardError => e
|
|
536
560
|
app = HoldfastCheck.app_error?(e)
|
|
537
561
|
absent = HoldfastCheck.absent(e) unless app
|
|
538
562
|
raise CouldNotRun.new("do", "`#{act.text}` raised #{e.class}: #{e.message.lines.first.to_s.strip}", absent:) if absent || !(errors || app)
|
|
539
563
|
|
|
540
|
-
{ error: e, scope:, shown: "#{act.text
|
|
564
|
+
{ error: e, scope:, shown: "#{HoldfastCheck.one_line(act.text)} raised #{e.class}: #{e.message.lines.first.to_s.strip[0, 300]}" }
|
|
541
565
|
end
|
|
542
566
|
|
|
567
|
+
# A `do` of several lines is one item of the Markdown list it is shown in.
|
|
568
|
+
def self.one_line(text) = text.strip.gsub(/\s*\n\s*/, "; ")
|
|
569
|
+
|
|
543
570
|
CALL_SHAPE = /wrong number of arguments|missing keyword|unknown keyword/
|
|
544
571
|
|
|
545
572
|
def self.app_error?(error)
|
|
@@ -653,10 +680,16 @@ module HoldfastCheck
|
|
|
653
680
|
end
|
|
654
681
|
end
|
|
655
682
|
|
|
683
|
+
# SQLite has no TRUNCATE; DELETE with the foreign keys off empties the
|
|
684
|
+
# tables in any order, and sqlite_sequence holds the ids to restart.
|
|
656
685
|
def holdfast_truncate
|
|
657
686
|
conn = ActiveRecord::Base.connection
|
|
658
687
|
tables = conn.tables - %w[schema_migrations ar_internal_metadata]
|
|
659
|
-
|
|
688
|
+
return if tables.empty?
|
|
689
|
+
return conn.execute("TRUNCATE #{tables.map { conn.quote_table_name(_1) }.join(", ")} RESTART IDENTITY CASCADE") if conn.adapter_name.match?(/postg/i)
|
|
690
|
+
|
|
691
|
+
conn.disable_referential_integrity { tables.each { conn.execute("DELETE FROM #{conn.quote_table_name(_1)}") } }
|
|
692
|
+
conn.execute("DELETE FROM sqlite_sequence") if conn.adapter_name.match?(/sqlite/i) && conn.select_value("SELECT 1 FROM sqlite_master WHERE name = 'sqlite_sequence'")
|
|
660
693
|
end
|
|
661
694
|
|
|
662
695
|
def holdfast_write(report)
|
|
@@ -595,6 +595,28 @@ module Propose
|
|
|
595
595
|
end.uniq
|
|
596
596
|
end
|
|
597
597
|
|
|
598
|
+
# App modules that talk HTTP themselves (`Net::HTTP.start` inside
|
|
599
|
+
# `LockerApi.register`) rather than through a `.client` a spec replaces:
|
|
600
|
+
# the hunt refuses their real calls, so a person writes the fake.
|
|
601
|
+
def http_services
|
|
602
|
+
@http_services ||= (Rails.root.glob("app/**/*.rb") + Rails.root.glob("lib/**/*.rb")).filter_map do |file|
|
|
603
|
+
text = File.read(file)
|
|
604
|
+
next unless text.match?(/\bNet::HTTP\b|\bFaraday\b|\bHTTParty\b|\bHTTP\.(get|post|put|patch|delete)\b/)
|
|
605
|
+
|
|
606
|
+
text[/^\s*(?:module|class)\s+([A-Z][\w:]*)/, 1]
|
|
607
|
+
end.uniq
|
|
608
|
+
end
|
|
609
|
+
|
|
610
|
+
def network_note(selected)
|
|
611
|
+
names = selected.map { it.name.to_s }
|
|
612
|
+
seen = Sequences.network_calls.select { |_, actions| (actions & names).any? }.keys
|
|
613
|
+
services = http_services.select { |service| selected.any? { static_constants(it.file).include?(service) } }
|
|
614
|
+
return [] if seen.empty? && services.empty?
|
|
615
|
+
|
|
616
|
+
["TODO: #{services.any? ? services.join(", ") : "an action here"} calls a real host#{" (#{seen.join(", ")} in the probe)" if seen.any?}; the hunt refuses real network calls.",
|
|
617
|
+
"Replace its methods with a fake before setup, list the fake in outside_calls, and arm its failures (see the skill)."]
|
|
618
|
+
end
|
|
619
|
+
|
|
598
620
|
# Methods a fake has and the real client does not, named like a failure:
|
|
599
621
|
# each becomes a `failure`. One with a required argument needs a person
|
|
600
622
|
# to say which record it fails for.
|
|
@@ -1147,6 +1169,7 @@ end
|
|
|
1147
1169
|
armed = failures.select { |f| f.code.start_with?("failure") && selected.any? { reached.(it).include?(f.service) } }
|
|
1148
1170
|
out = +"Sequences.define(#{name.inspect}) do\n"
|
|
1149
1171
|
out << " # #{note}\n" if note
|
|
1172
|
+
network_note(selected).each { out << " # #{it}\n" }
|
|
1150
1173
|
out << " setup do\n"
|
|
1151
1174
|
fakes.each { |service, fake| out << " #{service}.client = #{fake}.new\n" }
|
|
1152
1175
|
(api_auth[:envs].map { it[1] } | controller_envs).uniq.each { |env| out << " ENV[#{env.inspect}] = \"sequences-#{env.downcase.tr("_", "-")}\"\n" }
|
|
@@ -112,6 +112,12 @@ def verdict(r, limit)
|
|
|
112
112
|
return ["unchecked", "-", "unchecked: #{r[:status] == :timeout ? "timed out after #{limit} s" : "no report (#{tail[0, 100]})"}"]
|
|
113
113
|
end
|
|
114
114
|
done, total, missing = scope(rep)
|
|
115
|
+
# A failure or a refusal that came from a real service says nothing about
|
|
116
|
+
# the app; the area checks nothing until the service is faked.
|
|
117
|
+
unless rep["network"].to_h.empty?
|
|
118
|
+
calls = rep["network"].map { |where, actions| "#{where}#{" (from #{actions.join(", ")})" unless actions.empty?}" }
|
|
119
|
+
return ["unchecked", "#{done}/#{total}", "unchecked: real network call to #{calls.join(", ")}; the hunt refuses it, so fake that service in the area's setup"]
|
|
120
|
+
end
|
|
115
121
|
# The time limit stopped the area between cases with no failure found:
|
|
116
122
|
# what ran still counts, and the row says it was cut short.
|
|
117
123
|
if rep["verdict"].nil?
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# Runs generated sequences of an app's entry points
|
|
2
|
-
# against real PostgreSQL: random orders (shrunk by Hegel), two actions
|
|
2
|
+
# against the app's real test database (PostgreSQL or SQLite): random orders (shrunk by Hegel), two actions
|
|
3
3
|
# interleaved at a named pause point, and the same sequence replayed under
|
|
4
4
|
# other process time zones. The hunt stage loads it into the app's test
|
|
5
5
|
# process through SEQUENCES_LIB, so it needs hegeltest in that bundle.
|
|
@@ -36,6 +36,43 @@ module Sequences
|
|
|
36
36
|
# A replayed step whose record is missing: the run went another way.
|
|
37
37
|
class Diverged < StandardError; end
|
|
38
38
|
|
|
39
|
+
# An action opened a TCP connection to another host: a service the area
|
|
40
|
+
# does not fake. Its answer (or a DNS failure) is the environment, not the
|
|
41
|
+
# app, and a write through it would reach a real system.
|
|
42
|
+
class RealNetwork < StandardError; end
|
|
43
|
+
|
|
44
|
+
LOOPBACK = /\A(localhost|127\.\d+\.\d+\.\d+|::1|0\.0\.0\.0)\z/
|
|
45
|
+
|
|
46
|
+
def self.network_calls = (@network_calls ||= {})
|
|
47
|
+
|
|
48
|
+
def self.real_network!(host, port)
|
|
49
|
+
return if host.nil? || LOOPBACK.match?(host.to_s)
|
|
50
|
+
|
|
51
|
+
where = "#{host}:#{port}"
|
|
52
|
+
(network_calls[where] ||= []) << Thread.current[:sequences_action].to_s
|
|
53
|
+
raise RealNetwork, "real network call to #{where}"
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
module NetworkGate
|
|
57
|
+
def initialize(host, port, *rest, **kw)
|
|
58
|
+
Sequences.real_network!(host, port)
|
|
59
|
+
super
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
module SocketGate
|
|
64
|
+
def tcp(host, port, *rest, **kw, &blk)
|
|
65
|
+
Sequences.real_network!(host, port)
|
|
66
|
+
super
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
def self.refuse_network!
|
|
71
|
+
require "socket"
|
|
72
|
+
TCPSocket.prepend(NetworkGate) unless TCPSocket <= NetworkGate
|
|
73
|
+
Socket.singleton_class.prepend(SocketGate) unless Socket.singleton_class <= SocketGate
|
|
74
|
+
end
|
|
75
|
+
|
|
39
76
|
# A pause point is a method on a fake (or any class) where an action can be
|
|
40
77
|
# held while another action runs.
|
|
41
78
|
module Pause
|
|
@@ -512,8 +549,18 @@ end
|
|
|
512
549
|
|
|
513
550
|
# Steps the app accepted, which a double click or a retried request
|
|
514
551
|
# would send once more.
|
|
552
|
+
# A step whose record an action since destroyed (a withdrawal of the
|
|
553
|
+
# one parcel) has nothing left to send again: its pool index now points
|
|
554
|
+
# past the end.
|
|
515
555
|
def accepted_steps
|
|
516
|
-
@log.select
|
|
556
|
+
@log.select do |e|
|
|
557
|
+
@definition.actions.key?(e[:step]) && e[:outcome] && !Sequences.kind(e[:outcome]).match?(/refused|HTTP 4/) &&
|
|
558
|
+
begin
|
|
559
|
+
resolve(@definition.actions.fetch(e[:step]), e[:args])
|
|
560
|
+
rescue Diverged
|
|
561
|
+
false
|
|
562
|
+
end
|
|
563
|
+
end
|
|
517
564
|
end
|
|
518
565
|
|
|
519
566
|
# Dates and times the app stored (due dates, deadlines, pauses), where a
|
|
@@ -522,9 +569,11 @@ end
|
|
|
522
569
|
# took a quarter of the run.
|
|
523
570
|
def stored_dates
|
|
524
571
|
conn = ActiveRecord::Base.connection
|
|
572
|
+
# SQLite reads CAST(... AS date) as a number (the year); date() is its day.
|
|
573
|
+
day = conn.adapter_name.match?(/sqlite/i) ? ->(col) { "date(#{col})" } : ->(col) { "CAST(#{col} AS date)" }
|
|
525
574
|
@dates_sql ||= (conn.tables - %w[schema_migrations ar_internal_metadata]).flat_map do |table|
|
|
526
575
|
conn.columns(table).select { %i[date datetime].include?(it.type) && !@definition.ignore_columns.include?(it.name) }.map do |column|
|
|
527
|
-
"SELECT DISTINCT
|
|
576
|
+
"SELECT DISTINCT #{day.(conn.quote_column_name(column.name))} AS d FROM #{conn.quote_table_name(table)} WHERE #{conn.quote_column_name(column.name)} IS NOT NULL"
|
|
528
577
|
end
|
|
529
578
|
end.join(" UNION ")
|
|
530
579
|
return [] if @dates_sql.empty?
|
|
@@ -876,6 +925,7 @@ end
|
|
|
876
925
|
@first_failure = nil
|
|
877
926
|
|
|
878
927
|
Rails.application.eager_load!
|
|
928
|
+
refuse_network!
|
|
879
929
|
quiet_errors
|
|
880
930
|
seeds = ENV["SEQ_SEEDS"]&.split(",")&.map { Integer(it) } || [seed || Random.new_seed % 2**31]
|
|
881
931
|
report = { area: definition.name, cases: test_cases, seeds: [], skipped: definition.skipped, actions: checks(definition),
|
|
@@ -918,6 +968,7 @@ end
|
|
|
918
968
|
tallies = @tallies&.transform_values(&:dup) || {}
|
|
919
969
|
$stderr.puts coverage_report
|
|
920
970
|
report[:reached] = tallies.transform_keys(&:to_s)
|
|
971
|
+
report[:network] = network_calls.transform_values { it.uniq.reject(&:empty?) }
|
|
921
972
|
report[:refused_with] = (@examples || {}).transform_values { |seen| seen.sort_by { -it[1] }.map(&:first) }
|
|
922
973
|
@examples = nil
|
|
923
974
|
# A refusal quotes a response body, which Rack hands over as binary; a
|
|
@@ -938,6 +989,7 @@ end
|
|
|
938
989
|
|
|
939
990
|
def self.observe(definition, out:)
|
|
940
991
|
Rails.application.eager_load!
|
|
992
|
+
refuse_network!
|
|
941
993
|
quiet_errors
|
|
942
994
|
length = Integer(ENV.fetch("SEQ_OBSERVE_LENGTH", "3"))
|
|
943
995
|
cap = Integer(ENV.fetch("SEQ_OBSERVE_CAP", "240"))
|
|
@@ -1150,6 +1202,7 @@ end
|
|
|
1150
1202
|
# groups actions into areas by it.
|
|
1151
1203
|
def self.probe(definition, walks: 60, steps: 20, seed: 1)
|
|
1152
1204
|
Rails.application.eager_load!
|
|
1205
|
+
refuse_network!
|
|
1153
1206
|
quiet_errors
|
|
1154
1207
|
random = Random.new(seed)
|
|
1155
1208
|
stats = Hash.new { |h, k| h[k] = { writes: Hash.new(0), reads: Hash.new(0), outcomes: Hash.new(0), errors: Hash.new(0), refusals: [], runs: 0 } }
|
|
@@ -1221,7 +1274,7 @@ end
|
|
|
1221
1274
|
# The fakes are plain objects that remember charges and failures armed; a
|
|
1222
1275
|
# trial's rollback must undo those too.
|
|
1223
1276
|
def self.save_outside_state
|
|
1224
|
-
seqs = ActiveRecord::Base.connection.select_rows(sequence_state_sql)
|
|
1277
|
+
seqs = sequence_state_sql.empty? ? [] : ActiveRecord::Base.connection.select_rows(sequence_state_sql)
|
|
1225
1278
|
fakes = outside_classes_declared.flat_map { |klass| ObjectSpace.each_object(klass).to_a }.map do |fake|
|
|
1226
1279
|
[fake, fake.instance_variables.to_h { [it, Marshal.load(Marshal.dump(fake.instance_variable_get(it)))] }]
|
|
1227
1280
|
rescue TypeError
|
|
@@ -1236,18 +1289,24 @@ end
|
|
|
1236
1289
|
fakes.each { |fake, ivars| ivars.each { fake.instance_variable_set(_1, _2) } }
|
|
1237
1290
|
end
|
|
1238
1291
|
|
|
1292
|
+
def self.postgres? = ActiveRecord::Base.connection.adapter_name.match?(/postg/i)
|
|
1293
|
+
|
|
1294
|
+
# Only PostgreSQL keeps id counters outside the transaction a trial rolls
|
|
1295
|
+
# back; SQLite's sqlite_sequence is a table, rolled back with the rows.
|
|
1239
1296
|
def self.sequence_state_sql
|
|
1240
|
-
@sequence_state_sql ||=
|
|
1241
|
-
|
|
1242
|
-
|
|
1243
|
-
|
|
1244
|
-
|
|
1297
|
+
@sequence_state_sql ||= if postgres?
|
|
1298
|
+
conn = ActiveRecord::Base.connection
|
|
1299
|
+
names = conn.select_values("SELECT sequencename FROM pg_sequences WHERE schemaname = current_schema()")
|
|
1300
|
+
names.map { "SELECT #{conn.quote(it)}, last_value, is_called FROM #{conn.quote_table_name(it)}" }.join(" UNION ALL ")
|
|
1301
|
+
else ""
|
|
1302
|
+
end
|
|
1245
1303
|
end
|
|
1246
1304
|
|
|
1247
1305
|
def self.outside_classes_declared = (@outside_classes_declared ||= Set.new)
|
|
1248
1306
|
|
|
1249
1307
|
def self.replay(definition, entries)
|
|
1250
1308
|
Rails.application.eager_load!
|
|
1309
|
+
refuse_network!
|
|
1251
1310
|
runner = Runner.new(definition)
|
|
1252
1311
|
runner.reset!(sizes: world_sizes(entries))
|
|
1253
1312
|
runner.replay(entries)
|
|
@@ -1265,9 +1324,13 @@ end
|
|
|
1265
1324
|
# Empties every table and restarts its ids, so two runs of the same steps
|
|
1266
1325
|
# get the same ids. DELETE in foreign-key order (children first) took 1 ms
|
|
1267
1326
|
# on this app's few rows where TRUNCATE ... RESTART IDENTITY took 72 ms.
|
|
1268
|
-
# A schema whose keys form a cycle falls back to TRUNCATE.
|
|
1327
|
+
# A schema whose keys form a cycle falls back to TRUNCATE. SQLite has no
|
|
1328
|
+
# TRUNCATE and no sequences: DELETE with the foreign keys off, then
|
|
1329
|
+
# sqlite_sequence, which holds the ids to restart.
|
|
1269
1330
|
def self.empty_tables
|
|
1270
1331
|
conn = ActiveRecord::Base.connection
|
|
1332
|
+
return empty_tables_without_truncate(conn) unless postgres?
|
|
1333
|
+
|
|
1271
1334
|
@empty_sql ||= begin
|
|
1272
1335
|
tables = conn.tables - %w[schema_migrations ar_internal_metadata]
|
|
1273
1336
|
parents = tables.to_h { |t| [t, conn.foreign_keys(t).map(&:to_table) - [t]] }
|
|
@@ -1290,6 +1353,15 @@ end
|
|
|
1290
1353
|
conn.execute(@empty_sql)
|
|
1291
1354
|
end
|
|
1292
1355
|
|
|
1356
|
+
def self.empty_tables_without_truncate(conn)
|
|
1357
|
+
@empty_statements ||= begin
|
|
1358
|
+
tables = conn.tables - %w[schema_migrations ar_internal_metadata]
|
|
1359
|
+
sqlite = conn.adapter_name.match?(/sqlite/i) && conn.select_value("SELECT 1 FROM sqlite_master WHERE name = 'sqlite_sequence'")
|
|
1360
|
+
tables.map { "DELETE FROM #{conn.quote_table_name(it)}" } + (sqlite ? ["DELETE FROM sqlite_sequence"] : [])
|
|
1361
|
+
end
|
|
1362
|
+
conn.disable_referential_integrity { @empty_statements.each { conn.execute(it) } }
|
|
1363
|
+
end
|
|
1364
|
+
|
|
1293
1365
|
|
|
1294
1366
|
def self.with_zone(zone)
|
|
1295
1367
|
old = ENV["TZ"]
|
data/lib/holdfast/stages/hunt.rb
CHANGED
|
@@ -34,8 +34,10 @@ module Holdfast
|
|
|
34
34
|
|
|
35
35
|
per_area = [budget * 4 / [areas.size, 4].max, 60].max
|
|
36
36
|
cases = budget >= 900 ? 300 : 60
|
|
37
|
-
# Shrinking a failure
|
|
38
|
-
|
|
37
|
+
# Shrinking a failure can take minutes, and an unshrunk sequence of a
|
|
38
|
+
# dozen steps hides the one that matters: every hunt shrinks, for a
|
|
39
|
+
# quarter of an area's time limit, between 15 and 60 s.
|
|
40
|
+
env["SEQ_SHRINK_SECONDS"] ||= ENV.fetch("SEQ_SHRINK_SECONDS", (per_area / 4).clamp(15, 60).to_s)
|
|
39
41
|
json = File.join(work, "table.json")
|
|
40
42
|
FileUtils.rm_f(json)
|
|
41
43
|
out, = Open3.capture2e(env, RbConfig.ruby, File.join(SEQUENCES, "run.rb"), "-j", "4", "--cases", cases.to_s, "--seeds", "1",
|
|
@@ -70,7 +72,7 @@ module Holdfast
|
|
|
70
72
|
worst = thin.first&.first
|
|
71
73
|
step = worst ? "start with #{worst["area"]}: make the records or the sign-in its unreached actions need in #{worst["file"].to_s.delete_prefix("#{change.root}/")}, then holdfast run" :
|
|
72
74
|
"read each area's row below for the refusals, fix the setup or the sign-in in its file, then holdfast run"
|
|
73
|
-
row("(hunt)", "unchecked", "unreached: #{total - reached} of #{total} actions (refused every time or never drawn); #{thin.size} of #{counts.size} areas reached under half",
|
|
75
|
+
row("(hunt)", "unchecked", "unreached: #{total - reached} of #{total} actions (refused every time or never drawn); #{thin.size} of #{counts.size} areas reached under half#{" (#{table.size - counts.size} area(s) that failed are not counted)" if table.size > counts.size}",
|
|
74
76
|
evidence, "#{step}; until then the hunt says little about these areas")
|
|
75
77
|
end
|
|
76
78
|
|
|
@@ -102,12 +104,21 @@ module Holdfast
|
|
|
102
104
|
_, status = Open3.capture2e(probe_env, "/bin/sh", "-c", record.side_setup, chdir: change.root)
|
|
103
105
|
return row("(sequences)", "unchecked", "the probe database did not set up", [], "holdfast run --learn") unless status.success?
|
|
104
106
|
end
|
|
105
|
-
|
|
107
|
+
# An area a person kept, merged or wrote is never drafted over.
|
|
108
|
+
return nil if Dir.glob(File.join(dir, "*.rb")).any? { File.basename(_1) != "invariants.rb" }
|
|
106
109
|
|
|
107
110
|
out, status = Open3.capture2e(probe_env, "bundle", "exec", "ruby", File.join(SEQUENCES, "propose.rb"), "--all", chdir: change.root)
|
|
108
111
|
return nil if status.success?
|
|
109
112
|
|
|
110
|
-
|
|
113
|
+
error = out.lines.grep(/Error|error/).first.to_s.strip
|
|
114
|
+
# The first frame outside the gems says whose code raised: holdfast's
|
|
115
|
+
# own drafting is nothing the app or its factories can fix.
|
|
116
|
+
frame = out.lines.map(&:strip).find { _1.match?(%r{\A(?:from )?/\S+\.rb:\d+}) && (_1.include?(SEQUENCES) || !_1.include?("/gems/")) }.to_s
|
|
117
|
+
if frame.include?(SEQUENCES)
|
|
118
|
+
return row("(sequences)", "unchecked", "holdfast failed while drafting the areas (#{record.data.dig("database", "adapter")} adapter): #{error}", ["at: #{frame.delete_prefix("from ")}"],
|
|
119
|
+
"report it to holdfast with this row; the app and its factories are not the cause")
|
|
120
|
+
end
|
|
121
|
+
row("(sequences)", "unchecked", "drafting the areas failed: #{error}", [],
|
|
111
122
|
"fix the error it names in the app or its factories, then holdfast run")
|
|
112
123
|
end
|
|
113
124
|
|
|
@@ -164,7 +175,12 @@ module Holdfast
|
|
|
164
175
|
|
|
165
176
|
row(area, "clean", "every action, offset replay and door pair reached its target (#{scope})", [], "nothing; the claim covers only these seeds and cases", scope:)
|
|
166
177
|
else
|
|
167
|
-
|
|
178
|
+
reason = lines.first.to_s.delete_prefix("unchecked: ")
|
|
179
|
+
if reason.start_with?("real network call")
|
|
180
|
+
return row(area, "unchecked", "environment: #{reason}", ["log: #{log}"],
|
|
181
|
+
"replace the service's methods with a fake in #{r["file"].to_s.delete_prefix("#{change.root}/")} and declare it with outside_calls (the skill shows how), then holdfast run")
|
|
182
|
+
end
|
|
183
|
+
row(area, "unchecked", reason, ["log: #{log}"], "read the log, fix the setup it names, then holdfast run")
|
|
168
184
|
end
|
|
169
185
|
end
|
|
170
186
|
|
|
@@ -38,7 +38,7 @@ module Holdfast
|
|
|
38
38
|
unless change.changed?
|
|
39
39
|
missing = change.named.reject { |n| kept.any? { named_match?(n, _1) } }
|
|
40
40
|
verdicts += missing.map do |n|
|
|
41
|
-
Verdict.of(target: n, stage: "select", kind: "unchecked", reason: "no method or file named #{n} at #{change.
|
|
41
|
+
Verdict.of(target: n, stage: "select", kind: "unchecked", reason: "no method or file named #{n} at #{change.tree_label}",
|
|
42
42
|
next_step: "name it as Const#method, Const.method, or a path under app/, then run holdfast run --target again")
|
|
43
43
|
end
|
|
44
44
|
end
|
data/lib/holdfast/version.rb
CHANGED
data/skills/holdfast/SKILL.md
CHANGED
|
@@ -7,13 +7,13 @@ description: Run what you suspect instead of leaving it "read only". In a Ruby o
|
|
|
7
7
|
|
|
8
8
|
## Setup
|
|
9
9
|
|
|
10
|
-
Add `gem "holdfast", require: false` to the app's `:test` group and run `bundle install`; run every command below as `bundle exec holdfast ...` from the app root. jitcatch and branchout, which `holdfast run` uses on a change, come with it. The app needs RSpec, FactoryBot and a test database.
|
|
10
|
+
Add `gem "holdfast", require: false` to the app's `:test` group and run `bundle install`; run every command below as `bundle exec holdfast ...` from the app root. jitcatch and branchout, which `holdfast run` uses on a change, come with it. The app needs RSpec, FactoryBot and a test database on PostgreSQL or SQLite; races and the hunt run on both (MySQL is untested). SQLite lets one connection write at a time, so a race that holds an action after it wrote in its open transaction makes the other wait, and the row says `the second waited on a lock`.
|
|
11
11
|
|
|
12
12
|
## The loop: read, suspect, check, keep
|
|
13
13
|
|
|
14
14
|
1. Read the code. Write each thing you suspect in a file, one line of `says` and its `given`, `do` (or `race`) and `expect`. A race or a cross-area effect costs the same few lines; do not leave one "read, not run".
|
|
15
15
|
2. `holdfast check suspicions.yml` (or `--suspect '<one YAML mapping>'`, repeatable). Run it from the app root.
|
|
16
|
-
3. Keep each confirmed spec, `spec/holdfast/<branch>/<id>_spec.rb` (the header names the folder): it fails until the code is fixed. Report confirmed rows with their spec and refuted rows with what was observed. Fix each could_not_run suspicion and run again.
|
|
16
|
+
3. Keep each confirmed spec, `spec/holdfast/<branch>/<id>_spec.rb` (the header names the folder; the id is the `says` text and a digest of it, so checking the same suspicion again replaces its spec, and two suspicions in one run must say different things): it fails until the code is fixed. Report confirmed rows with their spec and refuted rows with what was observed. Fix each could_not_run suspicion and run again.
|
|
17
17
|
|
|
18
18
|
```yaml
|
|
19
19
|
- says: a passed tentative appointment is charged a no-show fee
|
|
@@ -36,7 +36,7 @@ Add `gem "holdfast", require: false` to the app's `:test` group and run `bundle
|
|
|
36
36
|
|
|
37
37
|
- `given`: one record per key, from the factory of that name (`factory:`, `traits:`). A value is a literal, a relative time (`-1h`, `+2d`, `-20m`), an earlier key, or `{ruby: "..."}`. A string, or `{ruby: "..."}`, in place of the attributes is Ruby that returns the record or any value. An earlier record fills a required `belongs_to` you leave out, and so does what it belongs to.
|
|
38
38
|
- `do`: Ruby, with each record by its name. A local or `@name` that `do` sets is what `expect` reads by that name (a record read again). Request helpers work: `post "/api/v1/..."`, then `response`. A 404 the app answers is an outcome; a path no route matches is could_not_run.
|
|
39
|
-
- `race`: two actions. The first is held at `pause` while the second runs, then the reverse, and both serial orders. Confirmed only when a held order fails `expect` and ends in a way no serial order ends (ids of records the actions made and times do not count). When a serial order fails `expect` too, the row says so and is refuted as a race, unless the held order's end is new; then the row names that end and each serial end. `pause` is `batch_load` (the first query that returns rows), `outside_call` (the first call to a client such as `PaymentGateway.client`), `row_lock` (before the first `with_lock` or `lock!`), `first_write`, or `Class#method`. With no `pause`, or a `batch_load` nobody reaches, each is tried in that order; the row says which one held.
|
|
39
|
+
- `race`: two actions. The first is held at `pause` while the second runs, then the reverse, and both serial orders. Confirmed only when a held order fails `expect` and ends in a way no serial order ends (ids of records the actions made and times do not count). When a serial order fails `expect` too, the row says so and is refuted as a race, unless the held order's end is new; then the row names that end and each serial end. `pause` is `batch_load` (the first query that returns rows), `outside_call` (the first call to a client such as `PaymentGateway.client`, or to a module method `given` stubs, such as `allow(LockerApi).to receive(:register)`), `row_lock` (before the first `with_lock` or `lock!`), `first_write`, or `Class#method`. With no `pause`, or a `batch_load` nobody reaches, each is tried in that order; the row says which one held.
|
|
40
40
|
- `expect`: Ruby that is true afterwards (records read again; `result`, `error`; in a race `results[0]`, `results[1]`, `errors[i]` by the order `race` lists them; `calls(X, :verb)`, the arguments of each call to `X.client`), or `no X verb`, `X verb [once|N times]`, `raises X`, `no error`; these combine with `&&` and `||`. A list must all hold. When `do` raised, a nil `response` or `result` in `expect` is that raise: the line fails and names it. A failed `a && b` names the part that failed and its value.
|
|
41
41
|
|
|
42
42
|
What to suspect for a race: a lock taken on a different row than the one written (two members booking one room each lock their own member row); a read before the lock (a status or a total read, then `with_lock`); an absolute write after a slow call (`attempts = notice.attempts + 1`, a courier call, then `update!(attempts:)`); check-then-act with no unique index under it (`exists?` then `create!`). Write the two actions and the end no serial order allows; leave `pause` out.
|
|
@@ -49,7 +49,7 @@ In a change (the tree differs from the base branch, or `--base REF`), each suspi
|
|
|
49
49
|
| `refuted` | `expect` held; each observed value is listed | Nothing to fix for this suspicion. |
|
|
50
50
|
| `could_not_run` | Nothing was checked: the row names the `given` key and its error, the `do` line that raised, a request with no route or one turned away (401, 403), or that no pause point was reached (with the app methods the action called, to name as `pause`) | Fix the suspicion, run again. |
|
|
51
51
|
|
|
52
|
-
An error that app code raised is how the action ended, and `expect` is checked against it. An error from your own line (a misspelt name, a wrong argument list) is could_not_run. Each branch keeps its own specs and rows: `.holdfast/check/<branch>/check.md` is the summary, `check.jsonl` beside it has every row.
|
|
52
|
+
An error that app code raised is how the action ended, and `expect` is checked against it. An error from your own line (a misspelt name, a wrong argument list) is could_not_run. A could_not_run row that starts `holdfast:` failed in holdfast's own support code, and its `next` names the database adapter: neither the suspicion nor the app is the cause. Each branch keeps its own specs and rows: `.holdfast/check/<branch>/check.md` is the summary, `check.jsonl` beside it has every row.
|
|
53
53
|
|
|
54
54
|
## Check a change: holdfast run
|
|
55
55
|
|
|
@@ -89,8 +89,35 @@ Sequences.everywhere do
|
|
|
89
89
|
end
|
|
90
90
|
```
|
|
91
91
|
|
|
92
|
-
|
|
92
|
+
The hunt refuses a TCP connection to any host but this machine: an action that reaches a real service makes its area `unchecked` with `environment: real network call to HOST:PORT`, never `bug_suspect`. The drafted area fakes a service the specs install as `X.client = FakeX.new`, and names in a TODO a module that calls HTTP itself. Fake that one in the area file, list the fake in `outside_calls` (each of its methods becomes a pause point for interleavings and its state is rolled back with each trial), and arm its failures:
|
|
93
|
+
|
|
94
|
+
```ruby
|
|
95
|
+
class FakeCourier # the courier as the app sees it: what it booked, and one armed outage
|
|
96
|
+
attr_reader :booked
|
|
97
|
+
def initialize = (@booked = []) && (@down = false)
|
|
98
|
+
def go_down! = (@down = true)
|
|
99
|
+
def book(order_id)
|
|
100
|
+
raise Courier::Unavailable, "fake outage" if @down && !(@down = false)
|
|
101
|
+
|
|
102
|
+
@booked << order_id
|
|
103
|
+
end
|
|
104
|
+
end
|
|
105
|
+
Courier.define_singleton_method(:book) { |order_id| $courier.book(order_id) }
|
|
106
|
+
|
|
107
|
+
Sequences.define("desk") do
|
|
108
|
+
outside_calls FakeCourier
|
|
109
|
+
setup { $courier = FakeCourier.new; create_list(:order, world_size(:orders, 1..3)) }
|
|
110
|
+
pool(:orders) { Order.order(:id) }
|
|
111
|
+
failure(:courier_down) { $courier.go_down! } # an action that lists it also runs as its own step with the fake armed
|
|
112
|
+
action(:ship, order: :orders, failures: [:courier_down]) { |order:| http.post "/orders/#{order.id}/shipment" }
|
|
113
|
+
action(:cancel, order: :orders) { |order:| http.post "/orders/#{order.id}/cancellation" }
|
|
114
|
+
end
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
Each drafted area holds the actions that write the same tables, so a bug that needs two areas (a withdrawal, then the intake job) is out of reach. Merge them: move the pools, actions, failures and invariants into one `Sequences.define` and delete the other files. A hunt drafts again only when `.holdfast/sequences/` holds no area, so a merged area stays.
|
|
118
|
+
|
|
119
|
+
A `bug_suspect` row names the broken rule, and its log holds the shortest sequence as a numbered narrative and an `it` block that replays it. Every hunt shrinks: shrinking stops after a quarter of the area's time limit, between 15 and 60 s (`SEQ_SHRINK_SECONDS` sets it), and the row then says `shrinking stopped`; when the area's time limit cut it first, the row says `not shrunk`. Either way the log holds the shortest sequence found so far. A twins pair the two doors cannot feed the same inputs (one picks a record the other takes from the sign-in, or one makes a new unique value) is listed as not reached; drop the `twins` line or give both doors the same params. A partly checked area lists each action that was only refused (with up to three refusals, most common first, and the param to change) or never drawn: make the record in the area's `setup`, narrow a `pool`, or change the draws, then run again. An area the time limit stopped between cases says `cut by the time limit after N of M cases`; what ran still counts.
|
|
93
120
|
|
|
94
121
|
## When setup fails
|
|
95
122
|
|
|
96
|
-
The first run learns `.holdfast/record.json` (test framework, databases: each
|
|
123
|
+
The first run learns `.holdfast/record.json` (test framework, databases: each side gets its own through `DATABASE_URL`, a name of this app's own on a server and a file under `.holdfast/db/` on SQLite; the app's own test database is never written). A failed step is kept with its command and output, and every row says so. Fix the environment, then `holdfast run --learn`.
|