@suffolklitlab/alkiln 5.13.4 → 5.15.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (44) hide show
  1. package/.github/workflows/github_server.yml +1 -1
  2. package/.github/workflows/playground.yml +2 -1
  3. package/.github/workflows/unit_tests.yml +1 -1
  4. package/CHANGELOG.md +40 -0
  5. package/action.yml +26 -10
  6. package/action_for_github_server/action.yml +31 -27
  7. package/docs/4_environments.md +2 -2
  8. package/docs/artifacts_folder_name.md +121 -71
  9. package/docs/constrained_random_tests.md +220 -289
  10. package/docs/decisions/move_runtime_config_to_sources_folder.md +3 -1
  11. package/docs/prioritize_constrained_random_tests.md +35 -76
  12. package/lib/docassemble/server_install.js +0 -2
  13. package/lib/docassemble/setup.js +1 -3
  14. package/lib/docassemble/takedown.js +2 -2
  15. package/lib/globals.js +10 -0
  16. package/lib/run_cucumber.js +40 -53
  17. package/lib/scope.js +44 -1
  18. package/lib/setup/TestGenerator.js +1757 -0
  19. package/lib/setup/artifacts.js +20 -0
  20. package/lib/setup/generate.js +42 -0
  21. package/lib/setup/get_Gherkin_AST.js +26 -0
  22. package/lib/setup/validate.js +89 -0
  23. package/lib/steps.js +101 -42
  24. package/lib/utils/Log.js +317 -165
  25. package/lib/utils/_alkiln-constrained_random_tests_old.js +1776 -0
  26. package/lib/utils/get_error_stacks.js +16 -0
  27. package/lib/utils/reports.js +44 -22
  28. package/lib/utils/session_vars.js +77 -57
  29. package/lib/utils/set_sources_paths.js +98 -0
  30. package/package.json +18 -7
  31. package/tests/log_codes/check_codes.sh +17 -13
  32. package/tests/log_codes/log_code_expected_instances.txt +58 -0
  33. package/tests/unit_tests/constrained_random_answers/invalid.fixtures.js +178 -0
  34. package/tests/unit_tests/constrained_random_answers/invalid.test.js +251 -0
  35. package/tests/unit_tests/constrained_random_answers/valid.fixtures.js +957 -0
  36. package/tests/unit_tests/constrained_random_answers/valid.test.js +563 -0
  37. package/tests/unit_tests/example_of_chai_test_lifecycle.js +78 -0
  38. package/tests/unit_tests/log.test.js +17 -41
  39. package/docs/seprate_file_generation_GitHub_action.md +0 -11
  40. package/lib/parse.js +0 -36
  41. package/lib/utils/get_sources_paths.js +0 -162
  42. package/lib/utils/parse.js +0 -1
  43. package/lib/utils/proccess_command_line_args.js +0 -267
  44. /package/docs/{expected_inernal_results.md → expected_internal_results.md} +0 -0
package/docs/constrained_random_tests.md CHANGED
@@ -1,347 +1,278 @@
1
- # Constrained random input tests architecture and decisions
1
+ # Constrained random answers tests architecture and decisions
2
2
 
3
+ (stub)
3
4
 
4
- ## What should throw an error in the generator parsing stage?
5
+ Authors are eager to try random answers tests. Development time for testing is hard to find and almost no test suite is exhaustive. They hope to trigger errors they may not have thought to look for.
5
6
 
6
- Nothing? Should it all get reported in the ALKiln tests?
7
+ ## Pros and cons of random tests in general
7
8
 
8
- Want to run as many Scenarios as possible? If so,
9
- - Don't throw a generator validity error at parse time.
10
- - Also catch cucumber parse errors? But then it won't be reported with other parse errors.
9
+ Pros:
11
10
 
12
- All of these happen quickly, so there's not much time wasted:
13
- - cucumber parse error
14
- - ALKiln generator parse error (which is a cucumber error we can catch)
15
- - ALKiln invalid generator error
11
+ - Less work for the test author.
12
+ - Can catch errors authors didn't think to test.
16
13
 
17
- - Test using wrong variable (out of date)
18
- - Form code using wrong variable
14
+ Cons:
19
15
 
20
- Confusing to try to show multiple errors at the same time:
16
+ - The tests are less deliberate and thought-through.
17
+ - The tests may give authors a false sense of security.
21
18
 
22
- ```gherkin
23
- Scenario: blah
24
- Given ALKiln shows an error for the generator test:
25
- """
26
- ALK0000 Error message
27
- """
28
- And I start the interview at "foo.yml"
29
- And I generate 10 random tests...
30
- And I set "name" to "Ethan"
31
- ```
19
+ ## Why set a maximum number of tests per Scenario?
20
+
21
+ Because someone might intend to write 10 and add an extra 0 accidentally. These tests can take a long time to run.
22
+
23
+ ## Concerns with the current flow
24
+
25
+ - The usual process for troubleshooting failing GitHub tests is to re-run the tests. Authors may not realize that they get all new randomized tests with each run _and_ rerun. If the random tests fail during the first run, but pass during the second run, they may mistakenly believe the first failure was a fluke.
26
+ - How easy/hard will it be for ALKilnInThePlayground users to come back to previous runs to get their old generated tests? We will have to see how users find it.
27
+
28
+ ## Fast failure
32
29
 
33
- Potentially less confusing:
30
+ Based on author feedback, constrained random answers test generation should fail early and fast and avoid being too permissive, though some mistakes can get away with warnings.
31
+
32
+ ## The generating process
33
+
34
+ The main test generation logic copies the existing generator Scenario strings and transforms them. This doc gives an example of the transformation. This doesn't show the order of what happens in the code. It just shows how ALKiln thinks about the transformation.
35
+
36
+ A generator file can only have generator Scenarios in it, though it can have any number of generator Scenarios. For each generator file ALKiln gets, ALKiln makes one file with the right generated tests.
37
+
38
+ ### Part 1 - ALKiln gets this generator file
34
39
 
35
40
  ```gherkin
36
- Scenario: blah
37
- Given ALKiln shows an error for the generator test:
38
- """
39
- ALK0000 Error message
40
-
41
- Your Scenario:
42
-
43
- Scenario: blah
44
- And I start the interview at "foo.yml"
45
- And I generate 10 random tests...
46
- And I set "name" to "Ethan"
47
- """
41
+ @go_places_maybe
42
+ Feature: I go places
43
+
44
+ I go to places like the store, the park, the zoo, etc.
45
+
46
+ Scenario: I might go to the store
47
+ Given I start the interview at "store.yml"
48
+ And I make 2 constrained random tests that get to any of ["kickout", "success"] with:
49
+ | var | possible_values |
50
+ | user_goes | True;; False |
51
+ Then I wait .2 seconds
52
+
53
+ # Check with Jaime - is the wording correct?
54
+ Scenario: I visit the zoo
55
+ Given I start the interview at "zoo.yml"
56
+ And I generate 5 constrained random tests that get to any of ["end", "restart"] with:
57
+ | var | possible_values |
58
+ #...etc
48
59
  ```
49
60
 
61
+ By the end of the process there should be 1 file with 7 unique Scenarios.
50
62
 
51
- ### If ALKiln throws all its errors
63
+ ### Part 2 - ALKiln breaks the file up into sections
52
64
 
53
- **Just cucumber throws file parsing errors at parse stage:**
65
+ Section 1 - the top of the file till the first generator Scenario:
66
+
67
+ ```gherkin
68
+ @go_places_maybe
69
+ Feature: I go places
70
+
71
+ I go to places like the store, the park, the zoo, etc.
54
72
 
55
- ```
56
- GitHub starts stuff, lots of text
57
- Cucumber expected #Tag, #Feature, or #Example (more text)
58
- ** Process exits **
59
73
  ```
60
74
 
61
- **Just ALKiln throws cucumber parsing errors from at parse stage:**
75
+ Section 2 - the text of generator Scenario 1 until its constraints Step:
62
76
 
77
+ ```gherkin
78
+ @store @obvious
79
+ Scenario: I might go to the store
80
+ Given I start the interview at "store.yml"
63
81
  ```
64
- GitHub starts stuff, lots of text
65
- Cucumber expected #Tag, #Feature, or #Example (more text) (cucumber parsing error in ALKiln generator file)
66
- ** Process exits **
82
+
83
+ Section 3 - the constrained random answers Step of generator Scenario 1:
84
+
85
+ ```gherkin
86
+ And I generate 2 random tests that get to any of ["kickout", "success"] when I pick from these constraints:
87
+ | var | possible_values |
88
+ | user_goes | True;; False |
67
89
  ```
68
90
 
69
- **Just ALKiln throws generator parsing errors at parse stage:**
91
+ Section 4 - the text of generator Scenario 1 after the constraints Step until the next generator Scenario:
92
+ ```gherkin
93
+ Then I wait .2 seconds
70
94
 
95
+ # Check with Jaime - is the wording correct?
71
96
  ```
72
- GitHub starts stuff, lots of text
73
- ALK0000 Error: generator 1 (file and Scenario with text of Scenario) test was bad
74
- ALK0000 Error: generator 2 (file and Scenario with text of Scenario) test was bad
75
- ** Process exits **
97
+
98
+ Section 5 - top of generator Scenario 2 until its constraints Step:
99
+
100
+ ```gherkin
101
+ Scenario: I visit the zoo
102
+ Given I sign in at ""
103
+ And I start the interview at "zoo.yml"
76
104
  ```
77
105
 
78
- **All 3:**
106
+ TODO:
79
107
 
80
108
  ```
81
- GitHub starts stuff, lots of text
82
- ALK0000 Error: generator 1 (file and Scenario with text of Scenario) test was bad
83
- ALK0000 Error: generator 2 (file and Scenario with text of Scenario) test was bad
84
- Cucumber expected #Tag, #Feature, or #Example (more text) (cucumber parsing error in ALKiln generator file)
85
- ** Process exits **
109
+ And I generate 5 constrained random tests that get to any of ["end", "restart"] with:
110
+ | var | possible_values |
111
+ #...etc
86
112
  ```
87
113
 
88
- Once we throw the errors, cucumber isn't going to get a chance to throw its own parsing errors
114
+ And so on down the file.
115
+
116
+ ### Part 3 - ALKiln creates multiple Scenarios with Story Table Steps
89
117
 
90
- ## Tangents
118
+ 1st Story Table Step for generator Scenario 1:
91
119
 
92
- TODO: Make issue - log reports of failed Scenarios right under their progress line?
93
- TODO: Make issue - log Scenario cucumber error output right under each Scenario's ALKiln output?
120
+ ```gherkin
121
+ And I get to any of ["kickout", "success"] with this data:
122
+ | var | value |
123
+ | user_goes | False |
124
+ ```
125
+
126
+ 2nd Story Table Step for generator Scenario 1:
94
127
 
95
- ## Alternative ways to run generated tests
128
+ ```gherkin
129
+ And I get to any of ["kickout", "success"] with this data:
130
+ | var | value |
131
+ | user_goes | True |
132
+ ```
96
133
 
97
- - In their own action
98
- - At the very start before any of the other test stuff. Before we start any server or install any projects. Then we wouldn't have warnings, only errors, and we'd throw them all before doing anything else. These could fail really fast. We could fail these as soon as the action started so that problems with them could be seen right away. (we might want to parse the cucumber files and throw those errors right away regardless of what we do with the generators.)
134
+ 1st Story Table Step for generator Scenario 2:
99
135
 
100
- ## Where errors/warnings?
136
+ ```gherkin
137
+ And I get to any of ["end", "restart"] with this data:
138
+ | var | value |
139
+ #...etc
140
+ ```
101
141
 
102
- Can see them together
103
- Can see them as soon as possible
104
- Can see them in all places (duplication)
105
- Put in report at parse time
106
- Exclude from report at run time?
107
- Show in console at run time in both places
108
- Maybe put at the top of the file and console so they can be in the same place in both the report and the console output
109
- Can we make sure generated tests are run first? Or maybe last? Two separate runs? Then splits up their location again. Want to find another way.
142
+ And so on for the other 4 Story Table Steps for generator Scenario 2.
110
143
 
111
- ### If ALKiln logs all its errors and warnings in at cucumber parse time
144
+ ### Part 4 - Build whole Scenarios
112
145
 
113
- **ALKiln logs generator and cucumber parsing errors at parse stage:**
146
+ Combine the Story Table Steps with the start and end of their corresponding Scenarios.
114
147
 
115
- (console output)
148
+ 1st generated Scenario based on generator Scenario 1:
116
149
 
117
- ```
118
- GitHub starts stuff, lots of text
119
-
120
- ALK0000 ERROR: generator 1 file Scenario 1 (file and Scenario with text of Scenario) test was bad
121
- ALK0000 ERROR: generator 1 file Scenario 2 (file and Scenario with text of Scenario) test was bad
122
- Cucumber expected #Tag, #Feature, or #Example (more text) (cucumber parsing error in ALKiln generator file)
123
-
124
- ALK0000 Warning: generator 3 file Scenario 6 wasn't great
125
- ALK0000 Warning: generator 4 file Scenario 1 wasn't great
126
-
127
- {
128
- cucumber config...
129
- }
130
-
131
- Tests started successfully
132
-
133
- ## (report starts here)
134
- Scenario: 8 I do something ...***..>..**..F
135
- Scenario: 7 ..**v**>***...
136
- Scenario: generator 3 Scenario 6 ..
137
- Scenario: generator 4 Scenario 1 ..
138
-
139
- ====
140
- Failed
141
- ===
142
-
143
- Scenario: 8 I do something
144
- Step 1
145
- Step 2
146
-
147
- ALK0000 ERROR: Took too long to continue
148
-
149
-
150
- ===
151
- Success
152
- ===
153
-
154
- Scenario: 7
155
- Step 1
156
- Step 2
157
- Step 3
158
-
159
- Scenario: generator 3 Scenario 6
160
- Step 1
161
- Step 2
162
- Step 3
163
- Step 4
164
-
165
- Scenario: generator 4 Scenario 1
166
- Step 1
167
- Step 2
168
-
169
-
170
- Failures:
171
-
172
- 1) Scenario: Sign in to server successfully (attempt 2) # docassemble/ALKilnTests/data/sources/reports.feature:591
173
- ✔ Before # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:115
174
- ✔ Given the Scenario report should include: # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1161
175
- """
176
- signed in
177
- """
178
- ✔ Given I sign in with the email "USER1_EMAIL" and the password "USER1_PASSWORD" # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:284
179
- ✖ After # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1266
180
- ReferenceError: errors is not defined
181
- at World.<anonymous> (/opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1516:131)
182
-
183
- Warnings:
184
-
185
- 1) Scenario: Sign in to server successfully (attempt 1, retried) # docassemble/ALKilnTests/data/sources/reports.feature:591
186
- ✔ Before # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:115
187
- ✔ Given the Scenario report should include: # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1161
188
- """
189
- signed in
190
- """
191
- ✔ Given I sign in with the email "USER1_EMAIL" and the password "USER1_PASSWORD" # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:284
192
- ✖ After # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1266
193
- ReferenceError: errors is not defined
194
- at World.<anonymous> (/opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1516:131)
195
-
196
- 103 scenarios (1 failed, 102 passed)
197
- 582 steps (582 passed)
198
- 8m18.453s (executing steps: 8m18.014s)
150
+ ```gherkin
151
+ @store @obvious
152
+ Scenario: I might go to the store
153
+ Given I start the interview at "store.yml"
154
+ And I get to any of ["kickout", "success"] with this data:
155
+ | var | value |
156
+ | user_goes | False |
157
+ Then I wait .2 seconds
158
+
159
+ # Check with Jaime - is the wording correct?
199
160
  ```
200
161
 
162
+ 2nd generated Scenario based on generator Scenario 1:
201
163
 
164
+ ```gherkin
165
+ @store @obvious
166
+ Scenario: I might go to the store
167
+ Given I start the interview at "store.yml"
168
+ And I get to any of ["kickout", "success"] with this data:
169
+ | var | value |
170
+ | user_goes | True |
171
+ Then I wait .2 seconds
172
+
173
+ # Check with Jaime - is the wording correct?
174
+ ```
202
175
 
203
- ### ALK logs at test run time
176
+ 1st generated Scenario based on generator Scenario 2:
204
177
 
205
- ```
206
- GitHub starts stuff, lots of text
207
-
208
- {
209
- cucumber config...
210
- }
211
-
212
- Tests started successfully
213
-
214
- ## (report starts here)
215
- Scenario: 8 I do something ...***..>..**..F
216
- Scenario: 7 ..**v**>***...
217
- Scenario: generator 1 Scenario 1 ..F
218
- Scenario: generator 1 Scenario 2 ..F
219
- Scenario: generator 3 Scenario 6 ..
220
- Scenario: generator 4 Scenario 1 ..
221
-
222
- ====
223
- Failed
224
- ===
225
-
226
- Scenario: 8 I do something
227
- Step 1
228
- Step 2
229
-
230
- ALK0000 ERROR: Took too long to continue
231
-
232
- Scenario: generator 1 Scenario 1 ..F
233
- ALK0000 ERROR: Generator 1 Scenario 1 was bad
234
-
235
- Scenario: generator 1 Scenario 2 ..F
236
- ALK0000 ERROR: Generator 1 Scenario 2 was bad
237
-
238
-
239
- ===
240
- Success
241
- ===
242
-
243
- Scenario: 7
244
- Step 1
245
- Step 2
246
- Step 3
247
-
248
- Scenario: generator 3 Scenario 6
249
- Step 1
250
- Step 2
251
- Step 3
252
- Step 4
253
- ALK0000 WARNING: generator 3 Scenario 6 wasn't great
254
-
255
- Scenario: generator 4 Scenario 1
256
- Step 1
257
- Step 2
258
- ALK0000 WARNING: generator 4 Scenario 1 wasn't great
259
-
260
-
261
- Failures:
262
-
263
- 1) Scenario: generator 1 Scenario 1 (attempt 2) # docassemble/ALKilnTests/data/sources/reports.feature:591
264
- ✔ Before # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:115
265
- ✔ Given the generator Scenario was invalid with this error: # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1161
266
- """
267
- ALK0000 Error: No rows in this generator Scenario
268
-
269
- Scenario: generator 1 Scenario 1
270
- Given I start the interview at "file.yml"
271
- And I generate 10 tests:
272
- | var | possible values |
273
- And I take a screenshot
274
- """
275
- ✔ Given I sign in with the email "USER1_EMAIL" and the password "USER1_PASSWORD" # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:284
276
- ✖ After # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1266
277
- ReferenceError: errors is not defined
278
- at World.<anonymous> (/opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1516:131)
279
-
280
- 2) Scenario: generator 1 Scenario 2 (attempt 2) # docassemble/ALKilnTests/data/sources/reports.feature:591
281
- ✔ Before # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:115
282
- ✔ Given the generator Scenario was invalid with this error: # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1161
283
- """
284
- ALK0000 Error: No rows in this generator Scenario
285
-
286
- Scenario: generator 1 Scenario 2
287
- Given I start the interview at "file.yml"
288
- And I generate 10 tests:
289
- | var | possible values |
290
- And I take a screenshot
291
- """
292
- ✔ Given I sign in with the email "USER1_EMAIL" and the password "USER1_PASSWORD" # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:284
293
- ✖ After # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1266
294
- ReferenceError: errors is not defined
295
- at World.<anonymous> (/opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1516:131)
296
-
297
- 3) Scenario: 8 I do something (attempt 2) # docassemble/ALKilnTests/data/sources/reports.feature:591
298
- ✔ Before # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:115
299
- ✔ Given the Scenario report should include: # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1161
300
- """
301
- signed in
302
- """
303
- ✔ Given I sign in with the email "USER1_EMAIL" and the password "USER1_PASSWORD" # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:284
304
- ✖ After # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1266
305
- ReferenceError: errors is not defined
306
- at World.<anonymous> (/opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1516:131)
307
-
308
- Warnings:
309
-
310
- 1) Scenario: Sign in to server successfully (attempt 1, retried) # docassemble/ALKilnTests/data/sources/reports.feature:591
311
- ✔ Before # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:115
312
- ✔ Given the Scenario report should include: # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1161
313
- """
314
- signed in
315
- """
316
- ✔ Given I sign in with the email "USER1_EMAIL" and the password "USER1_PASSWORD" # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:284
317
- ✖ After # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1266
318
- ReferenceError: errors is not defined
319
- at World.<anonymous> (/opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1516:131)
320
-
321
- 103 scenarios (1 failed, 102 passed)
322
- 582 steps (582 passed)
323
- 8m18.453s (executing steps: 8m18.014s)
178
+ ```gherkin
179
+ Scenario: I visit the zoo
180
+ Given I start the interview at "zoo.yml"
181
+ And I get to any of ["end", "restart"] with this data:
182
+ | var | value |
183
+ #...etc
324
184
  ```
325
185
 
186
+ ### Part 5 - Combine everything and add generation information
326
187
 
188
+ We want to make sure authors know these tests were generated by ALKiln and we want to make each Scenario unique. We need unique text that is visible during test output and we need unique tags authors can use to test individual tests.
327
189
 
328
- ## Author wrote multiple generator Steps in one Scenario
190
+ ```gherkin
191
+ @go_places_maybe @alkiln @generated @randomized @constraints
192
+ Feature: I go places with ALKiln randomization
193
+
194
+ I go to places like the store, the park, the zoo, etc.
195
+
196
+ @store @obvious @generator1_scenario1_of_2
197
+ Scenario: I might go to the store (1 of 2)
198
+ Given I start the interview at "store.yml"
199
+ And I get to any of ["kickout", "success"] with this data:
200
+ | var | value |
201
+ | user_goes | False |
202
+ Then I wait .2 seconds
203
+
204
+ # Check with Jaime - is the wording correct?
205
+ @store @obvious @generator1_scenario2_of_2
206
+ Scenario: I might go to the store (2 of 2)
207
+ Given I start the interview at "store.yml"
208
+ And I get to any of ["kickout", "success"] with this data:
209
+ | var | value |
210
+ | user_goes | True |
211
+ Then I wait .2 seconds
212
+
213
+ # Check with Jaime - is the wording correct?
214
+ @generator2_scenario1_of_5
215
+ Scenario: I visit the zoo (1 of 5)
216
+ Given I start the interview at "zoo.yml"
217
+ And I get to any of ["end", "restart"] with this data:
218
+ | var | value |
219
+ #...etc
220
+ ```
329
221
 
330
- Discuss: Warning or error
222
+ See unit tests for more examples.
331
223
 
332
- We think the easiest and clearest reaction is to refuse to generate any test and give a warning or error.
333
224
 
334
- Other options:
225
+ ```
226
+ @go_places_maybe
227
+ Feature: I go places
228
+
229
+ I go to places like the store, the park, the zoo, etc.
230
+
231
+ Scenario: I might go to the store
232
+ Given I will make 10 random constrained tests
233
+ Given I start the interview at "store.yml"
234
+ And constrained random tests get to any of ["kickout", "success"] with:
235
+ | var | possible_values |
236
+ | user_goes | True;; False |
237
+ Then I wait .2 seconds
238
+ And constrained random tests get to any of ["download"] with:
239
+ | var | possible_values |
240
+ | user_goes | True;; False |
241
+ And I see the phrase "Goodbye"
242
+
243
+ # Check with Jaime - is the wording correct?
244
+ Scenario: I visit the zoo
245
+ Given I start the interview at "zoo.yml"
246
+ And I generate 5 constrained random tests that get to any of ["end", "restart"] with:
247
+ | var | possible_values |
248
+ #...etc
249
+ ```
335
250
 
336
- - Use the last Step and delete the others. This may be hard to do. We'd need to mutate the Scenario's file text lines, maybe as we were looping through them. It would take some research
337
- - Use the first Step and leave the others as they are. This would generate a broken test, but the test would at least work until it gets to the next generator Step line. We could add a Step in the core code for the generator Step and use it to give a meaningful error instead of just letting cucumber set a status of `undefined`. Maybe that would at least be a bit more clear, but I think not as clear as just refusing to generate and producing a warning or error.
251
+ ## Why not just create a new AST?
338
252
 
339
- Complications:
253
+ The AST doesn't have a list of lines. For examples, Scenarios and comments are in separate lists. The AST also stores line numbers, so we'd have to keep updating line numbers for every object, which adds complexity. We've currently decided that the more structured model isn't worth that complexity.
340
254
 
341
- - What if there are other Steps between and after the generator Steps? Do we keep those and put them all at the bottom? Do we only keep the ones after the chosen generator Step?
342
255
 
343
- If we do decide to pick a generator Step, maybe we can make sure to pick one that is otherwise valid. That is, it has a dataTable that has rows, is otherwise correctly formatted, etc.
256
+ ## Ranges may not be that useful
344
257
 
345
- ## Grammar
258
+ Focused tests with actual thought out options are so useful. Take these examples:
346
259
 
347
- Post MVP, discuss: should generator Steps without a number, but with the word "tests" generate 2 Scenarios?
260
+ ```gherkin
261
+ # ❌ Too broad: May miss important scenarios
262
+ Then ALKiln will stop at "end" with any combination of:
263
+ | var | possible values |
264
+ | income | 0..1000000 |
265
+ | assets | 0..1000000 |
266
+
267
+ # ✅ Better: Focused ranges with critical points
268
+ Then ALKiln will stop at "end" with any combination of:
269
+ | var | possible values |
270
+ | income | 0;; 30000;; 50000;; 75000;; 100000 |
271
+ | assets | 0;; 50000;; 100000;; 250000 |
272
+ ```
273
+
274
+ The first one will make a lot of useless tests. Is there a way to have a cake and eat it too? I'm not sure. Example:
275
+
276
+ ```
277
+ | income | 0..100;; 900..1000 |
278
+ ```
package/docs/decisions/move_runtime_config_to_sources_folder.md CHANGED
@@ -1,3 +1,5 @@
1
+ For ALKilnInThePlayground tests
2
+
1
3
  Moving the runtime config to the sources folder would make it very easy for each Project to have its own runtime config which would make ALKilnInThePlayground work more smoothly. On the other hand, it could be confusing to authors (though it's fine if they check it in now, it may not be fine later). On the other other hand, we could delete it after every test suite run and and "temp" and "alkiln" to the name. If there are multiple sources folders with config files then...? We'll just have to pick one?
2
4
 
3
- Maybe this wouldn't be useful? Check if runtime_config file might be used after further tests are run and will only point to the most recent test (and thus be unable to get past tests). If so, that's what's going on now too.
5
+ Maybe this wouldn't be useful? Check if runtime_config file might be used after further tests are run and will only point to the most recent test (and thus be unable to get previous tests). If so, that's what's going on now too.