@suffolklitlab/alkiln 5.13.2 → 5.13.4

package/docs/constrained_random_tests.md

@@ -0,0 +1,347 @@
+# Constrained random input tests architecture and decisions
+
+
+## What should throw an error in the generator parsing stage?
+
+Nothing? Should it all get reported in the ALKiln tests?
+
+Want to run as many Scenarios as possible? If so,
+- Don't throw a generator validity error at parse time.
+- Also catch cucumber parse errors? But then it won't be reported with other parse errors.
+
+All of these happen quickly, so there's not much time wasted:
+- cucumber parse error
+- ALKiln generator parse error (which is a cucumber error we can catch)
+- ALKiln invalid generator error 
+
+- Test using wrong variable (out of date)
+- Form code using wrong variable
+
+Confusing to try to show multiple errors at the same time:
+
+```gherkin
+Scenario: blah
+  Given ALKiln shows an error for the generator test:
+     """
+     ALK0000 Error message
+     """
+  And I start the interview at "foo.yml"
+  And I generate 10 random tests...
+  And I set "name" to "Ethan"
+```
+
+Potentially less confusing:
+
+```gherkin
+Scenario: blah
+  Given ALKiln shows an error for the generator test:
+     """
+     ALK0000 Error message
+
+     Your Scenario:
+
+     Scenario: blah
+       And I start the interview at "foo.yml"
+       And I generate 10 random tests...
+       And I set "name" to "Ethan"
+     """
+```
+
+
+### If ALKiln throws all its errors
+
+**Just cucumber throws file parsing errors at parse stage:**
+
+```
+GitHub starts stuff, lots of text
+Cucumber expected #Tag, #Feature, or #Example (more text)
+** Process exits **
+```
+
+**Just ALKiln throws cucumber parsing errors from  at parse stage:**
+
+```
+GitHub starts stuff, lots of text
+Cucumber expected #Tag, #Feature, or #Example (more text) (cucumber parsing error in ALKiln generator file)
+** Process exits **
+```
+
+**Just ALKiln throws generator parsing errors at parse stage:**
+
+```
+GitHub starts stuff, lots of text
+ALK0000 Error: generator 1 (file and Scenario with text of Scenario) test was bad
+ALK0000 Error: generator 2 (file and Scenario with text of Scenario) test was bad
+** Process exits **
+```
+
+**All 3:**
+
+```
+GitHub starts stuff, lots of text
+ALK0000 Error: generator 1 (file and Scenario with text of Scenario) test was bad
+ALK0000 Error: generator 2 (file and Scenario with text of Scenario) test was bad
+Cucumber expected #Tag, #Feature, or #Example (more text) (cucumber parsing error in ALKiln generator file)
+** Process exits **
+```
+
+Once we throw the errors, cucumber isn't going to get a chance to throw its own parsing errors
+
+## Tangents
+
+TODO: Make issue - log reports of failed Scenarios right under their progress line?
+TODO: Make issue - log Scenario cucumber error output right under each Scenario's ALKiln output?
+
+## Alternative ways to run generated tests
+
+- In their own action
+- At the very start before any of the other test stuff. Before we start any server or install any projects. Then we wouldn't have warnings, only errors, and we'd throw them all before doing anything else. These could fail really fast. We could fail these as soon as the action started so that problems with them could be seen right away. (we might want to parse the cucumber files and throw those errors right away regardless of what we do with the generators.)
+
+## Where errors/warnings?
+
+Can see them together
+Can see them as soon as possible
+Can see them in all places (duplication)
+  Put in report at parse time
+  Exclude from report at run time?
+  Show in console at run time in both places
+  Maybe put at the top of the file and console so they can be in the same place in both the report and the console output
+  Can we make sure generated tests are run first? Or maybe last? Two separate runs? Then splits up their location again. Want to find another way.
+
+### If ALKiln logs all its errors and warnings in at cucumber parse time
+
+**ALKiln logs generator and cucumber parsing errors at parse stage:**
+
+(console output)
+
+```
+GitHub starts stuff, lots of text
+
+ALK0000 ERROR: generator 1 file Scenario 1 (file and Scenario with text of Scenario) test was bad
+ALK0000 ERROR: generator 1 file Scenario 2 (file and Scenario with text of Scenario) test was bad
+Cucumber expected #Tag, #Feature, or #Example (more text) (cucumber parsing error in ALKiln generator file)
+
+ALK0000 Warning: generator 3 file Scenario 6 wasn't great
+ALK0000 Warning: generator 4 file Scenario 1 wasn't great
+
+{
+  cucumber config...
+}
+
+Tests started successfully
+
+## (report starts here)
+Scenario: 8 I do something ...***..>..**..F
+Scenario: 7 ..**v**>***...
+Scenario: generator 3 Scenario 6 ..
+Scenario: generator 4 Scenario 1 ..
+
+====
+Failed
+===
+
+Scenario: 8 I do something
+Step 1
+Step 2
+
+ALK0000 ERROR: Took too long to continue
+
+
+===
+Success
+===
+
+Scenario: 7
+Step 1
+Step 2
+Step 3
+
+Scenario: generator 3 Scenario 6
+Step 1
+Step 2
+Step 3
+Step 4
+
+Scenario: generator 4 Scenario 1
+Step 1
+Step 2
+
+
+Failures:
+
+1) Scenario: Sign in to server successfully (attempt 2) # docassemble/ALKilnTests/data/sources/reports.feature:591
+   ✔ Before # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:115
+   ✔ Given the Scenario report should include: # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1161
+       """
+       signed in
+       """
+   ✔ Given I sign in with the email "USER1_EMAIL" and the password "USER1_PASSWORD" # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:284
+   ✖ After # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1266
+       ReferenceError: errors is not defined
+           at World.<anonymous> (/opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1516:131)
+
+Warnings:
+
+1) Scenario: Sign in to server successfully (attempt 1, retried) # docassemble/ALKilnTests/data/sources/reports.feature:591
+   ✔ Before # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:115
+   ✔ Given the Scenario report should include: # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1161
+       """
+       signed in
+       """
+   ✔ Given I sign in with the email "USER1_EMAIL" and the password "USER1_PASSWORD" # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:284
+   ✖ After # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1266
+       ReferenceError: errors is not defined
+           at World.<anonymous> (/opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1516:131)
+
+103 scenarios (1 failed, 102 passed)
+582 steps (582 passed)
+8m18.453s (executing steps: 8m18.014s)
+```
+
+
+
+### ALK logs at test run time
+
+```
+GitHub starts stuff, lots of text
+
+{
+  cucumber config...
+}
+
+Tests started successfully
+
+## (report starts here)
+Scenario: 8 I do something ...***..>..**..F
+Scenario: 7 ..**v**>***...
+Scenario: generator 1 Scenario 1 ..F
+Scenario: generator 1 Scenario 2 ..F
+Scenario: generator 3 Scenario 6 ..
+Scenario: generator 4 Scenario 1 ..
+
+====
+Failed
+===
+
+Scenario: 8 I do something
+Step 1
+Step 2
+
+ALK0000 ERROR: Took too long to continue
+
+Scenario: generator 1 Scenario 1 ..F
+ALK0000 ERROR: Generator 1 Scenario 1 was bad
+
+Scenario: generator 1 Scenario 2 ..F
+ALK0000 ERROR: Generator 1 Scenario 2 was bad
+
+
+===
+Success
+===
+
+Scenario: 7
+Step 1
+Step 2
+Step 3
+
+Scenario: generator 3 Scenario 6
+Step 1
+Step 2
+Step 3
+Step 4
+ALK0000 WARNING: generator 3 Scenario 6 wasn't great
+
+Scenario: generator 4 Scenario 1
+Step 1
+Step 2
+ALK0000 WARNING: generator 4 Scenario 1 wasn't great
+
+
+Failures:
+
+1) Scenario: generator 1 Scenario 1 (attempt 2) # docassemble/ALKilnTests/data/sources/reports.feature:591
+   ✔ Before # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:115
+   ✔ Given the generator Scenario was invalid with this error: # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1161
+       """
+       ALK0000 Error: No rows in this generator Scenario
+
+       Scenario: generator 1 Scenario 1
+         Given I start the interview at "file.yml"
+         And I generate 10 tests:
+           | var | possible values |
+         And I take a screenshot
+       """
+   ✔ Given I sign in with the email "USER1_EMAIL" and the password "USER1_PASSWORD" # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:284
+   ✖ After # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1266
+       ReferenceError: errors is not defined
+           at World.<anonymous> (/opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1516:131)
+
+2) Scenario: generator 1 Scenario 2 (attempt 2) # docassemble/ALKilnTests/data/sources/reports.feature:591
+   ✔ Before # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:115
+   ✔ Given the generator Scenario was invalid with this error: # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1161
+       """
+       ALK0000 Error: No rows in this generator Scenario
+
+       Scenario: generator 1 Scenario 2
+         Given I start the interview at "file.yml"
+         And I generate 10 tests:
+           | var | possible values |
+         And I take a screenshot
+       """
+   ✔ Given I sign in with the email "USER1_EMAIL" and the password "USER1_PASSWORD" # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:284
+   ✖ After # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1266
+       ReferenceError: errors is not defined
+           at World.<anonymous> (/opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1516:131)
+
+3) Scenario: 8 I do something (attempt 2) # docassemble/ALKilnTests/data/sources/reports.feature:591
+   ✔ Before # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:115
+   ✔ Given the Scenario report should include: # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1161
+       """
+       signed in
+       """
+   ✔ Given I sign in with the email "USER1_EMAIL" and the password "USER1_PASSWORD" # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:284
+   ✖ After # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1266
+       ReferenceError: errors is not defined
+           at World.<anonymous> (/opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1516:131)
+
+Warnings:
+
+1) Scenario: Sign in to server successfully (attempt 1, retried) # docassemble/ALKilnTests/data/sources/reports.feature:591
+   ✔ Before # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:115
+   ✔ Given the Scenario report should include: # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1161
+       """
+       signed in
+       """
+   ✔ Given I sign in with the email "USER1_EMAIL" and the password "USER1_PASSWORD" # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:284
+   ✖ After # ../../../../../opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1266
+       ReferenceError: errors is not defined
+           at World.<anonymous> (/opt/hostedtoolcache/node/18.20.5/x64/lib/node_modules/@suffolklitlab/alkiln/lib/steps.js:1516:131)
+
+103 scenarios (1 failed, 102 passed)
+582 steps (582 passed)
+8m18.453s (executing steps: 8m18.014s)
+```
+
+
+
+## Author wrote multiple generator Steps in one Scenario
+
+Discuss: Warning or error
+
+We think the easiest and clearest reaction is to refuse to generate any test and give a warning or error.
+
+Other options:
+
+- Use the last Step and delete the others. This may be hard to do. We'd need to mutate the Scenario's file text lines, maybe as we were looping through them. It would take some research
+- Use the first Step and leave the others as they are. This would generate a broken test, but the test would at least work until it gets to the next generator Step line. We could add a Step in the core code for the generator Step and use it to give a meaningful error instead of just letting cucumber set a status of `undefined`. Maybe that would at least be a bit more clear, but I think not as clear as just refusing to generate and producing a warning or error.
+
+Complications:
+
+- What if there are other Steps between and after the generator Steps? Do we keep those and put them all at the bottom? Do we only keep the ones after the chosen generator Step?
+
+If we do decide to pick a generator Step, maybe we can make sure to pick one that is otherwise valid. That is, it has a dataTable that has rows, is otherwise correctly formatted, etc.
+
+## Grammar
+
+Post MVP, discuss: should generator Steps without a number, but with the word "tests" generate 2 Scenarios?

package/docs/decisions/006_which_log_level_to_pick_2024_09_5.md

@@ -0,0 +1,70 @@
+# Which log levels to pick
+
+## Context and Problem Statement
+
+Both authors and internal ALKiln developers see and use console logs and logs in debug files to track down bugs and understand how the tests and ALKiln are working.
+
+<!-- TODO: Should this be about `Log` or about log levels?
+
+`Log` uses the standard log levels `info`, `warn`, `error`, and `log`. It sometimes uses them differently than nodejs's `console` object would. For example, `Log` has a `.throw()` instead of a `.error()`, which stores the error information in the debug files and then throws the error.
+
+`Log` saves and prints these logs with metadata, like icons and date stamps. For example, `Log`'s `.success()` uses the console log level `info` and includes a rainbow icon.
+
+When we need to, we skip using the `Log` class and write straight to the console. -->
+
+We want to give the internal development team descriptive information. We also want to give authors the best signal we can about their tests. How do we balance these? For example, if we show an error icon whenever there's an "informational" error about an aborted promise - an expected event that we cause whenever we use `Promise.race()` - an author could waste a lot of time trying to track down a problem that doesn't exist. Using `Log`'s `.success()` too much could also be confusing - successfully deleting a Project from a server shouldn't look like a passing test.
+
+When picking the log level for a log, how do we know which level to choose?
+
+## Considered Options
+
+- Stay as true to technical behavior as possible and be very literal - throw all errors, log anything successfully completed action with `.success()`.
+- Stay as clear to authors as possible - use levels that will help authors pay attention to what affects them.
+
+Example options for `success`:
+
+- Use whenever anything completes successfully.
+- Only use for success of major operations.
+- Use in the summary report for every test success.
+- Use in the summary report for success of the whole test suite.
+
+Example options for `error`:
+
+- Always use `error` when any kind of error happens.
+- Use `error` when an unexpected error happens.
+- Only use `error` for author-relevant errors.
+
+(See the architecture design document about errors.)
+
+
+## Decision Outcome
+
+Distract the author as little as possible everywhere.
+
+"success" icon:
+
+- Only actually log to the console and debug files with `log.success()` once for success of all tests.
+- Include the success icon at the top of the Scenario "Passed" section of the report.
+- Include the success icon in individual Scenario reports for Scenarios that passed.
+
+Throwing an error[^1]:
+
+- Throw an error for a test that fails. cucumberjs will use this error to mark the test as failed.
+- Throw an error when ALKiln has to unexpectedly abort a test.
+- Throw an error when ALKiln's code has to unexpectedly abort one of its own runtime processes. This includes processes like setup and takedown.
+
+"error" log level:
+
+- Only use the "error" level with errors that the test author should investigate[^1].
+
+"info" log level:
+
+- Log expected errors (errors that are purely informational) with the "info" level. For example, when we wait for navigation, we use a `Promise.race()`. When one promise wins the race, the correct thing to do is to abort the other promises. That causes errors that we catch. We log that *information*[^1] as data about which promise won during that test.
+
+"warn" log level[^1]:
+
+- Use the "warn" log level in the console for messages the test author should investigate.
+- Use the "warn" log level in the debug log for messages that could give ALKiln devs clues about a potential problem.
+- Use the "warn" log level for messages for internal problems that do not impact the tests (non-critical errors). For example, errors that happen when ALKiln tries to save to log files or errors when ALKiln tries to delete finished interview sessions. These messages should be silent and should look nonthreatening. Do try to be clear about what happened, but also try to avoid using the word "error" if possible. E.g. use "Skipped creating metadata for the logs" instead of "Error while creating metadata for the logs". Include the error along with that log message, but avoid throwing it. ALKiln should only write the message and `Error` to the console if no other output is possible.
+
+[^1]: Read more about this in the documentation about errors.

package/docs/decisions/007_getting_and_storing_logs.md

@@ -0,0 +1,179 @@
+# Getting and storing logs
+
+## Context and Problem Statement
+
+There are multiple processes that log to the console - our own console logs, cucumberjs logs, and GitHub action logs[^1]. We want to save some of those to the reports and all of those to the debugging logs. We also want to save plain debug messages that get sent only to the instance of `Log` and not to the console[^2].
+
+Also, if we want to search one of the debugging logs for output that was sent to the console, it would be nice to just copy the text and search for the exact text.
+
+Every process that we trigger separately in the GitHub action - setup, server_install, run_cucumber, and takedown - each need their own instance of `Log`. There isn't a way to pass the instance between those processes[^1]. Also, until we wrap the entire cucumberjs process inside another manager of some kind (which we may do in the future to improve and manage internal tests for failure handling), it seems impossible to pass an instance of `Log` from our cucumberjs runner (currently `run_cucumber.js`) to the cucumberjs `World` object so we can use it while running the actual tests with the tests handler[^3].
+
+[^1]: See the architecture design doc about our 4 environments.
+
+[^2]: See the decision doc about our report and log architecture.
+
+## Goals
+
+MVP:
+- Have a unified way of getting and storing all messages including console logs, stdout, and internal silent debug logs.
+- Make the logs look like the console's logs to more easily visually match the output.
+
+Ideally:
+- Also capture information from cucumberjs.
+
+## Considered Options
+
+- Listen for write to `stdout`
+- Save with `fs` `console.Console` `writeStream`s
+- Cucumber formatters
+- `fs.appendFileSync` with node's `util.inspect`
+- A combination of methods
+
+## Decision Outcome
+
+Use fs.appendFileSync with node's `util.inspect`. In future, we might explore capturing cucumber logs in this process too.
+
+## Pros and Cons of the Options
+
+### Listen for write to `stdout`
+
+We'd listen for `stdout` and `stderr` events. We haven't yet experimented with detecting which logs are cucumber's and storing only those. See pseudo code in the supplementary material section.
+
+**Pros:**
+
+- We wouldn't have to make an intermediary for handling first saving to debug logs and then logging to the console. We could simply log straight to the console which would trigger a function that would save to the debug files.
+- Those saved strings would look the same as those console logs.
+- We might be able to catch some of cucumber's output more easily and accurately.
+
+**Cons:**
+
+- When we tried this, the data that came into the console listener was unable to handle emoji, even with various methods of encoding/decoding. It may have problems with other characters as well.
+- We'd still need to create metadata, which makes the process more complex than "simply logging to the console".
+- We might still need to make custom error throwing behavior, which might then make our code less consistent.
+- We haven't experimented with detecting what log level the caller used.
+
+### Save with `fs` Console write streams
+
+We would create one custom `console.Console` instance that saves to the debugging files and maybe another for the report logs.
+
+**Pros:**
+
+- The saved strings would look the same as console logs because we'd literally be using an actual console object's `.log()`.
+- We could avoid handling circular references and other annoying errors ourselves.
+
+**Cons:**
+
+- We have to manage the streams' lifecycles - opening, flushing, and closing the streams. That includes special ways to flush the streams to get their text into the debugging and report files with event listeners for `end` and `close` and such. We need to do that by wrapping node processes in a try/catch. There are problems with this.
+   1. It seems fragile.
+   2. In one case, it's potentially impossible - the tests handler[^3]. The cucumber runner has a clear beginning and end to its node process and we can open and close the write streams around that process. The tests handler is buried in that and we don't have the experience we need to understand how to wrap the node process in there. E.g. do we wrap the whole steps.js file in a try/catch? Do we wrap each hook? Do we wrap the code in each hook? We need to throw errors so that cucumber can catch them and detect failures. See example below in supplementary material.
+
+### Cucumber formatters
+
+`cucumberjs` recommends using its [formatters](https://github.com/cucumber/cucumber-js/blob/main/docs/formatters.md)
+
+**Pros:**
+
+- We would be working within the `cucumberjs` ecosystem. That could give us access to data we might not otherwise be able to get and may have other benefits that we haven't explored fully because of the cons.
+
+**Cons:**
+
+- We were unable to get anything to work.
+- We won't be able to use that with anything outside cucumber. For example, setup, takedown, possibly the cucumber runner, and (in future) with GitHub action logs.
+- We would need to keep up with `cucumberjs` updates for this too.
+
+### `fs.appendFileSync` with node's `util.inspect`
+
+**Pros:**
+
+- We're familiar with the functionality.
+- `util.inspect` is what node uses to write to the console, so it can give us very similar output, though there's some customization we still have to do.
+
+**Cons:**
+
+- We have to do some of our own formatting and it would probably take a lot of care to make it completely match the console logs. For example, we could easily inspect the list of logs, but then it would look like a list, not like individual logs.
+- We have to manage circular dependencies and possibly other details.
+
+### A combination of methods
+
+For example, the cucumber formatter as well as one of the other methods.
+
+**Pros:**
+
+- We'd get the functionality of the pros of each.
+
+**Cons:**
+
+- We'd get at least some of the cons of each.
+- It may be a more complex system and possibly harder to maintain.
+
+## Supplementary material
+
+### Use `fs` Console to write to custom streams
+
+Examples of using node `fs` custom [console.Console](https://nodejs.org/docs/latest-v18.x/api/console.html) and [writeStream](https://nodejs.org/docs/latest-v18.x/api/fs.html#filehandlecreatewritestreamoptions). Before exiting, either with 0 or with an error, we have to flush the stream with `.on("close")` and `.on("error")` to write to the file.
+
+Flushing streams:
+
+```js
+// Based on https://github.com/nodejs/node/issues/24688#issuecomment-442688666
+const fs = require(`fs`);
+const util = require(`util`);
+
+async function run() {
+  // Create the stream to save to the debug.log file
+  const stream = fs.createWriteStream(`debug.log`);
+  // We can use this later to wait for the stream to close
+  const streamPromise = new Promise((res, rej) => {
+    stream.once(`close`, res).once(`error`, rej);
+  });
+
+  // Write some things
+  stream.write(new Date().toISOString() + `\n`);
+  stream.write(new Date().toISOString() + `\n`);
+
+  try {
+    // Example of exiting early
+    throw new Error(`Without ending the stream and awaiting the promise, this causes a blank debug.log`);
+  } catch (error) {
+    // Add more information to the file
+    stream.write(util.inspect(error));
+    stream.write(`We managed to avoid a blank file`);
+    // Avoid leaving the file blank
+    stream.end();
+    await streamPromise;
+    throw error;
+  }
+}
+
+run();
+```
+
+To keep the logic in `Log`, `Log` could have a method that a process can use to wrap its functionality:
+
+```js
+const run = function() { /* Do stuff */ };
+log.with_flush( run );
+```
+
+`Log` would handle creating the promises. `log.with_flush()` would handle the try/catch logic to wait for the listeners and close the streams.
+
+### Listen for write to `stdout`
+
+For capturing console output, look into how to get the `stdin` for capturing various console output. https://stackoverflow.com/a/54202970:
+
+```js
+const stdin = process.openStdin()
+
+process.stdout.write('Enter name: ')
+
+stdin.addListener('data', text => {
+  const name = text.toString().trim()
+  console.log('Your name is: ' + name)
+
+  stdin.pause() // stop reading
+})
+```
+
+We couldn't figure out how to handle decoding emoji.
+
+[^3]: I'm not sure of good names to differentiate the code in run_cucumber.js, which triggers the rest of the test running, but is separate from it, and the code that handles the nitty gritty - like index.js, steps.js, scope.js, etc. "Running cucumber" is ambiguous. I'm calling run_cucumber.js code the "cucumber runner" and I'm calling the code that handles the actual tests the "tests handler".

package/docs/decisions/move_runtime_config_to_sources_folder.md

@@ -0,0 +1,3 @@
+Moving the runtime config to the sources folder would make it very easy for each Project to have its own runtime config which would make ALKilnInThePlayground work more smoothly. On the other hand, it could be confusing to authors (though it's fine if they check it in now, it may not be fine later). On the other other hand, we could delete it after every test suite run and and "temp" and "alkiln" to the name. If there are multiple sources folders with config files then...? We'll just have to pick one?
+
+Maybe this wouldn't be useful? Check if runtime_config file might be used after further tests are run and will only point to the most recent test (and thus be unable to get past tests). If so, that's what's going on now too.