@suffolklitlab/alkiln 5.13.2 → 5.13.3-feat-random

package/docs/decisions/006_which_log_level_to_pick_2024_09_5.md

@@ -0,0 +1,70 @@
+# Which log levels to pick
+
+## Context and Problem Statement
+
+Both authors and internal ALKiln developers see and use console logs and logs in debug files to track down bugs and understand how the tests and ALKiln are working.
+
+<!-- TODO: Should this be about `Log` or about log levels?
+
+`Log` uses the standard log levels `info`, `warn`, `error`, and `log`. It sometimes uses them differently than nodejs's `console` object would. For example, `Log` has a `.throw()` instead of a `.error()`, which stores the error information in the debug files and then throws the error.
+
+`Log` saves and prints these logs with metadata, like icons and date stamps. For example, `Log`'s `.success()` uses the console log level `info` and includes a rainbow icon.
+
+When we need to, we skip using the `Log` class and write straight to the console. -->
+
+We want to give the internal development team descriptive information. We also want to give authors the best signal we can about their tests. How do we balance these? For example, if we show an error icon whenever there's an "informational" error about an aborted promise - an expected event that we cause whenever we use `Promise.race()` - an author could waste a lot of time trying to track down a problem that doesn't exist. Using `Log`'s `.success()` too much could also be confusing - successfully deleting a Project from a server shouldn't look like a passing test.
+
+When picking the log level for a log, how do we know which level to choose?
+
+## Considered Options
+
+- Stay as true to technical behavior as possible and be very literal - throw all errors, log anything successfully completed action with `.success()`.
+- Stay as clear to authors as possible - use levels that will help authors pay attention to what affects them.
+
+Example options for `success`:
+
+- Use whenever anything completes successfully.
+- Only use for success of major operations.
+- Use in the summary report for every test success.
+- Use in the summary report for success of the whole test suite.
+
+Example options for `error`:
+
+- Always use `error` when any kind of error happens.
+- Use `error` when an unexpected error happens.
+- Only use `error` for author-relevant errors.
+
+(See the architecture design document about errors.)
+
+
+## Decision Outcome
+
+Distract the author as little as possible everywhere.
+
+"success" icon:
+
+- Only actually log to the console and debug files with `log.success()` once for success of all tests.
+- Include the success icon at the top of the Scenario "Passed" section of the report.
+- Include the success icon in individual Scenario reports for Scenarios that passed.
+
+Throwing an error[^1]:
+
+- Throw an error for a test that fails. cucumberjs will use this error to mark the test as failed.
+- Throw an error when ALKiln has to unexpectedly abort a test.
+- Throw an error when ALKiln's code has to unexpectedly abort one of its own runtime processes. This includes processes like setup and takedown.
+
+"error" log level:
+
+- Only use the "error" level with errors that the test author should investigate[^1].
+
+"info" log level:
+
+- Log expected errors (errors that are purely informational) with the "info" level. For example, when we wait for navigation, we use a `Promise.race()`. When one promise wins the race, the correct thing to do is to abort the other promises. That causes errors that we catch. We log that *information*[^1] as data about which promise won during that test.
+
+"warn" log level[^1]:
+
+- Use the "warn" log level in the console for messages the test author should investigate.
+- Use the "warn" log level in the debug log for messages that could give ALKiln devs clues about a potential problem.
+- Use the "warn" log level for messages for internal problems that do not impact the tests (non-critical errors). For example, errors that happen when ALKiln tries to save to log files or errors when ALKiln tries to delete finished interview sessions. These messages should be silent and should look nonthreatening. Do try to be clear about what happened, but also try to avoid using the word "error" if possible. E.g. use "Skipped creating metadata for the logs" instead of "Error while creating metadata for the logs". Include the error along with that log message, but avoid throwing it. ALKiln should only write the message and `Error` to the console if no other output is possible.
+
+[^1]: Read more about this in the documentation about errors.

package/docs/decisions/007_getting_and_storing_logs.md

@@ -0,0 +1,179 @@
+# Getting and storing logs
+
+## Context and Problem Statement
+
+There are multiple processes that log to the console - our own console logs, cucumberjs logs, and GitHub action logs[^1]. We want to save some of those to the reports and all of those to the debugging logs. We also want to save plain debug messages that get sent only to the instance of `Log` and not to the console[^2].
+
+Also, if we want to search one of the debugging logs for output that was sent to the console, it would be nice to just copy the text and search for the exact text.
+
+Every process that we trigger separately in the GitHub action - setup, server_install, run_cucumber, and takedown - each need their own instance of `Log`. There isn't a way to pass the instance between those processes[^1]. Also, until we wrap the entire cucumberjs process inside another manager of some kind (which we may do in the future to improve and manage internal tests for failure handling), it seems impossible to pass an instance of `Log` from our cucumberjs runner (currently `run_cucumber.js`) to the cucumberjs `World` object so we can use it while running the actual tests with the tests handler[^3].
+
+[^1]: See the architecture design doc about our 4 environments.
+
+[^2]: See the decision doc about our report and log architecture.
+
+## Goals
+
+MVP:
+- Have a unified way of getting and storing all messages including console logs, stdout, and internal silent debug logs.
+- Make the logs look like the console's logs to more easily visually match the output.
+
+Ideally:
+- Also capture information from cucumberjs.
+
+## Considered Options
+
+- Listen for write to `stdout`
+- Save with `fs` `console.Console` `writeStream`s
+- Cucumber formatters
+- `fs.appendFileSync` with node's `util.inspect`
+- A combination of methods
+
+## Decision Outcome
+
+Use fs.appendFileSync with node's `util.inspect`. In future, we might explore capturing cucumber logs in this process too.
+
+## Pros and Cons of the Options
+
+### Listen for write to `stdout`
+
+We'd listen for `stdout` and `stderr` events. We haven't yet experimented with detecting which logs are cucumber's and storing only those. See pseudo code in the supplementary material section.
+
+**Pros:**
+
+- We wouldn't have to make an intermediary for handling first saving to debug logs and then logging to the console. We could simply log straight to the console which would trigger a function that would save to the debug files.
+- Those saved strings would look the same as those console logs.
+- We might be able to catch some of cucumber's output more easily and accurately.
+
+**Cons:**
+
+- When we tried this, the data that came into the console listener was unable to handle emoji, even with various methods of encoding/decoding. It may have problems with other characters as well.
+- We'd still need to create metadata, which makes the process more complex than "simply logging to the console".
+- We might still need to make custom error throwing behavior, which might then make our code less consistent.
+- We haven't experimented with detecting what log level the caller used.
+
+### Save with `fs` Console write streams
+
+We would create one custom `console.Console` instance that saves to the debugging files and maybe another for the report logs.
+
+**Pros:**
+
+- The saved strings would look the same as console logs because we'd literally be using an actual console object's `.log()`.
+- We could avoid handling circular references and other annoying errors ourselves.
+
+**Cons:**
+
+- We have to manage the streams' lifecycles - opening, flushing, and closing the streams. That includes special ways to flush the streams to get their text into the debugging and report files with event listeners for `end` and `close` and such. We need to do that by wrapping node processes in a try/catch. There are problems with this.
+   1. It seems fragile.
+   2. In one case, it's potentially impossible - the tests handler[^3]. The cucumber runner has a clear beginning and end to its node process and we can open and close the write streams around that process. The tests handler is buried in that and we don't have the experience we need to understand how to wrap the node process in there. E.g. do we wrap the whole steps.js file in a try/catch? Do we wrap each hook? Do we wrap the code in each hook? We need to throw errors so that cucumber can catch them and detect failures. See example below in supplementary material.
+
+### Cucumber formatters
+
+`cucumberjs` recommends using its [formatters](https://github.com/cucumber/cucumber-js/blob/main/docs/formatters.md)
+
+**Pros:**
+
+- We would be working within the `cucumberjs` ecosystem. That could give us access to data we might not otherwise be able to get and may have other benefits that we haven't explored fully because of the cons.
+
+**Cons:**
+
+- We were unable to get anything to work.
+- We won't be able to use that with anything outside cucumber. For example, setup, takedown, possibly the cucumber runner, and (in future) with GitHub action logs.
+- We would need to keep up with `cucumberjs` updates for this too.
+
+### `fs.appendFileSync` with node's `util.inspect`
+
+**Pros:**
+
+- We're familiar with the functionality.
+- `util.inspect` is what node uses to write to the console, so it can give us very similar output, though there's some customization we still have to do.
+
+**Cons:**
+
+- We have to do some of our own formatting and it would probably take a lot of care to make it completely match the console logs. For example, we could easily inspect the list of logs, but then it would look like a list, not like individual logs.
+- We have to manage circular dependencies and possibly other details.
+
+### A combination of methods
+
+For example, the cucumber formatter as well as one of the other methods.
+
+**Pros:**
+
+- We'd get the functionality of the pros of each.
+
+**Cons:**
+
+- We'd get at least some of the cons of each.
+- It may be a more complex system and possibly harder to maintain.
+
+## Supplementary material
+
+### Use `fs` Console to write to custom streams
+
+Examples of using node `fs` custom [console.Console](https://nodejs.org/docs/latest-v18.x/api/console.html) and [writeStream](https://nodejs.org/docs/latest-v18.x/api/fs.html#filehandlecreatewritestreamoptions). Before exiting, either with 0 or with an error, we have to flush the stream with `.on("close")` and `.on("error")` to write to the file.
+
+Flushing streams:
+
+```js
+// Based on https://github.com/nodejs/node/issues/24688#issuecomment-442688666
+const fs = require(`fs`);
+const util = require(`util`);
+
+async function run() {
+  // Create the stream to save to the debug.log file
+  const stream = fs.createWriteStream(`debug.log`);
+  // We can use this later to wait for the stream to close
+  const streamPromise = new Promise((res, rej) => {
+    stream.once(`close`, res).once(`error`, rej);
+  });
+
+  // Write some things
+  stream.write(new Date().toISOString() + `\n`);
+  stream.write(new Date().toISOString() + `\n`);
+
+  try {
+    // Example of exiting early
+    throw new Error(`Without ending the stream and awaiting the promise, this causes a blank debug.log`);
+  } catch (error) {
+    // Add more information to the file
+    stream.write(util.inspect(error));
+    stream.write(`We managed to avoid a blank file`);
+    // Avoid leaving the file blank
+    stream.end();
+    await streamPromise;
+    throw error;
+  }
+}
+
+run();
+```
+
+To keep the logic in `Log`, `Log` could have a method that a process can use to wrap its functionality:
+
+```js
+const run = function() { /* Do stuff */ };
+log.with_flush( run );
+```
+
+`Log` would handle creating the promises. `log.with_flush()` would handle the try/catch logic to wait for the listeners and close the streams.
+
+### Listen for write to `stdout`
+
+For capturing console output, look into how to get the `stdin` for capturing various console output. https://stackoverflow.com/a/54202970:
+
+```js
+const stdin = process.openStdin()
+
+process.stdout.write('Enter name: ')
+
+stdin.addListener('data', text => {
+  const name = text.toString().trim()
+  console.log('Your name is: ' + name)
+
+  stdin.pause() // stop reading
+})
+```
+
+We couldn't figure out how to handle decoding emoji.
+
+[^3]: I'm not sure of good names to differentiate the code in run_cucumber.js, which triggers the rest of the test running, but is separate from it, and the code that handles the nitty gritty - like index.js, steps.js, scope.js, etc. "Running cucumber" is ambiguous. I'm calling run_cucumber.js code the "cucumber runner" and I'm calling the code that handles the actual tests the "tests handler".

package/docs/decisions/move_runtime_config_to_sources_folder.md

@@ -0,0 +1,5 @@
+For ALKilnInThePlayground tests
+
+Moving the runtime config to the sources folder would make it very easy for each Project to have its own runtime config which would make ALKilnInThePlayground work more smoothly. On the other hand, it could be confusing to authors (though it's fine if they check it in now, it may not be fine later). On the other other hand, we could delete it after every test suite run and and "temp" and "alkiln" to the name. If there are multiple sources folders with config files then...? We'll just have to pick one?
+
+Maybe this wouldn't be useful? Check if runtime_config file might be used after further tests are run and will only point to the most recent test (and thus be unable to get previous tests). If so, that's what's going on now too.

package/docs/decisions/reports_and_logs_2024_08_11.md

@@ -0,0 +1,161 @@
+# Shape and behavior of the report and log objects
+
+<!-- TODO: turn this into a architecture design doc? -->
+<!-- TODO: Rename to just "logs" -->
+
+## Context and scope
+
+We show logs to the user in the console and as test suite artifacts. We have logs we use for development. cucumberjs also has logs we want to include in the above until we are confident we are handling all cases ourselves.
+
+Right now, the system for logging is fractured, scattered, and duplicated. Logs that are not from the report don't get saved anywhere. Some logs only get shown in debug mode in the console, which can be frustrating and unhelpful. We lack verbose output information for tests that authors run in the wild.
+
+This document excludes discussion of report architecture, but it is useful to note that at the moment they are completely separate from logs and yet save to some of the same files, so they influence our current choices.
+
+## Goals
+
+- Make log information persistent and visible
+- Add additional meaningful information to logs (e.g. metadata)
+- Make sure logs are clear without needing to read the surrounding code
+- Unify and clarify log flow
+- Make log behavior consistent
+
+## Non/anti-goals
+
+- Disrupt the signals we're giving authors about their tests with the noise of internal ALKiln details
+- Create a complex and heavy system to try to abstract absolutely everything
+- Let the complexity of the relationship with report behavior too heavily influencing our log design goals
+
+## The actual design
+
+### `Log` internal errors
+
+Logs are informational and not related to the behavior of authors' tests. Because of that, these logs should avoid adding noise to what an author sees. That is, internal warnings or errors should look different than authors' test errors. To help with this:
+
+- All log methods` arguments have default values.
+- We use try/catch very aggressively in the `Log` class.
+
+Read more of the thoughts behind this choice in the decision doc on which log level to pick and the design doc on types of errors.
+
+### Purposes of methods
+
+Some of these notes have to do with reports because logs and reports are using some of the same files at the moment and yet are not integrated. We'll write a doc for reports when we integrate reports more fully with logs.
+
+1. `Log`: Messages or errors from wrapping functionality, like setup and takedown messages.
+2. `Log`: Test progress (a compact way to see what's happening/happened in each test):
+
+   Test1: .............F--.
+
+   Test 2: ....
+3. Report: A pretty report at the end with details about each test. It has sections, like "failed" and "passed", headings, etc.
+4. Report: cucumberjs's various console logs.
+5. `Log`: Lots of other informational logs for internal development and debugging purposes.
+
+Different logs belong in different places:
+
+- ALKiln always logs items 1, 2, 3, and 4 to the console.
+- Ideally save all items in a verbose debug file sequentially as the tests run.
+- During the tests, save items 2, 3, and 4 to a running report file sequentially as they come. Include metadata. Suggested filenames: `report_log.txt`/`running_report.txt`/`report_run.txt`. If we have to stop the tests early because of an infinite loop or something, this will at least show the test author some information.
+- At the end of the tests, format items 1, 2, 3, and 4 nicely in a file called `report.txt`. This will mostly exclude the metadata.
+- When there is a warning-level or error-level log, save 1, 3, 4, and 5 in an unexpected results file too.
+- In future, discuss the usefulness of having a debug mode where ALKiln logs all items to the console while the tests are running.
+
+We may also want copies of various files that have different formats for different environments (see docs on the different environments in which ALKiln runs).
+
+- Console logs with colors.
+- Plain text versions that exclude those color codes.
+- HTML versions to show with, for example ALKilnInThePlayground. Maybe in GitHub as well in the future.
+
+### Anatomy and behavior of logging methods
+
+Most of a `Log`'s logging methods add additional metadata to a log:
+
+1. The level of the log (success, info, warning, error, and custom values)
+2. An icon showing the level of a log
+3. A log code, like (e.g. ALK0105)
+4. Contextual info about where the log originated (e.g. "setup", "takedown", "invalid answer")
+9. A timestamp
+
+To be able to show that metadata, most methods will accept custom values for parts of that metadata. Most methods offer:
+
+- Log code
+- Contextual info
+- The error throwing information
+
+For the debug method, everything other than the timestamp can be customized.
+
+Callers can influence the content and formatting of most parts of the log message. For example, they can call `.warn` to show a warning icon and level metadata. The one exception would usually be the part before the metadata. Customizing that could be useful for things like creating a visual separator before ALKiln starts evaluating the fields of a web page. For that reason, the caller can add a `before` property to the metadata that a `Log` object will add before the metadata.
+
+A `Log` object should be able to throw errors so that it can log metadata and any additional styling and context along with the error. It can also make sure that the error will be an actual `Error` object (as opposed to a string or other object). Otherwise everywhere we needed to throw an error, we would need duplicate handling bad data (like objects with circular references), using `Log` to save to files (etc.), throwing the error, and other possible complications. For that, `Logs` also need to accept this additional information:
+
+- Whether to throw an error
+- The error to throw
+
+Those methods should also accept multiple individual log arguments the same way `console.log()` does. This lets the caller include strings separately from objects they want to log, including `Error` objects, so that a `Log` object can handle them safely and style them consistently. We could have required a list of logs instead, but the extra syntax was frustrating.
+
+Right now, the only way we differentiate methods that just store data vs. methods that also log to the console is with documentation. This seems less than ideal. Using the name `record` could be confusing because developers might think they need to log to the console separately from saving the information to debug files.
+
+## APIs
+
+Proposed external methods: `success`, `info`, `warn`, `throw`, `stdout`, `debug`. See the `Log` class and its tests for the most up-to-date API.
+
+<!-- Discuss: better name for `stdout`? -->
+
+## Data storage
+
+- `report.txt` - final pretty output for the user with headings, etc.
+- Each Scenario's `report.txt` - pretty output in each test's folder.
+- `report_log.txt` (name?) - ugly, but still sparse, giving the user some info about what happened during the tests without overwhelming them.
+- `debug_log.txt` (name? `debug`?) - very ugly and full of all the information from everywhere else and more. Useful for internal development and troubleshooting. Includes setup, takedown, etc.
+- `temp_unexpected_results_debug_log.txt` - A file containing messages about unexpected behavior, like test warnings and errors, as well as internal ALKiln warnings. We don't output this in our GitHub actions - the file itself is temporary, we just add its contents to the final unexpected results file. All its information exists in the debug file too.
+- `unexpected_results.txt` - Logs for failures and other unexpected behavior. It has information about test warnings and errors, as well as internal ALKiln warnings. At first we only save report information there. When the tests finish, we add the contents of the specific debug file that contains only unexpected results.
+
+## Degree of constraint
+
+- Right now we're a bit limited by report functionality staying separate. Other than that, we can design as we want. We should avoid the current behavior of reports substantially influencing our design decisions here. We do have to keep in mind how reports might integrate with this in the future, but we will also need to change the design of the report behavior significantly in the future, so we should keep an open mind.
+- We have not yet approached saving logs from GitHub actions, though this should be fairly simple. It may add some complication.
+- We have not yet solved getting all logs from the console (e.g. cucumberjs logs).
+
+## Alternatives considered
+
+- Avoid the complexity of storing setup and takedown output in the verbose/debug file. In GitHub, they're in the GitHub console. Locally, we see the results immediately. On the other hand, storing them ourselves lets us look for information in just one place.
+- Include GitHub action output in the debug files. Out of scope for the current iteration.
+- Catch cucumberjs's actual console output. Out of scope for the current iteration.
+
+### System-context diagram
+
+Current design before integration of reports, GitHub action logs, and other non-ALKiln console logs.
+
+```
+## `Log`
+
+console methods -> ._console() -> .debug() -> log to the console
+.stdout() -> .debug() -> sometimes write to stdout
+
+.debug() -> debug_log.txt with metadata
+         -> sometimes temp_unexpected_results_debug_log.txt with metadata
+
+## `Report`
+report and its methods -> report_log.txt (plain text)
+                       -> debug_log.txt (plain text)
+                       -> sometimes unexpected_results.txt (plain text)
+                       -> report summary at end of tests
+                          -> final report.txt (plain text summary)
+                          -> final console log (summary with console colors)
+                       -> individual Scenario reports at end of tests
+                          -> final individual report summary file in each Scenario's folder
+
+## takedown
+Add temp_unexpected_results_debug_log.txt to the end of  unexpected_results.txt
+```
+
+## Cross-cutting concerns
+
+_Security, privacy, observability, etc._
+
+Code that calls `Log` methods should avoid passing sensitive information. A `Log` object doesn't protect against exposing information.
+
+## Related documentation
+
+Decision doc about which log level to use.
+Architecture design doc about types of unexpected and expected "errors".
+Decision doc about getting and storing logs.

package/docs/kinds_of_errors.md

@@ -0,0 +1,26 @@
+# Kinds of Errors
+
+An error, or even an `Error` instance, can mean a lot of different things in the context of this testing framework. Throwing errors for failing tests is what we are most familiar with, but it is just one kind of error.
+
+## Interview `Error`s
+
+1. **Interview bugs**: For example, the interview says ALKiln's answer is invalid. The author expected that answer to be valid. ALKiln saves an error message to the report and throws an error to tell cucumber to mark the test as failed. Currently, these logs errors come with the 🤕 icon. When all the tests are done, the test suite will show as failing in the console, the report, and the various logs. In future, ALKiln will use a method like `report.throw()` to save to the various log files with the `error` level and then throw the error to let cucumber know the test failed.
+1. **Interview expected errors**: For example, the interview says ALKiln's answer is invalid. The author was testing their interview's field validation rules. The test passes.
+
+## Test file `Error`s
+
+1. **Test file syntax error**: For example, a row in a table in a test file has too many columns. cucumberjs throws an error and no tests are run. ALKiln should show the tests as failed. ALKiln is unable to get the output of that error into a log, but there should be an error in the console.
+1. **Test Step typo**: For example, `And I ta p to contcontinue`. cucumberjs will cause this test to fail, giving an "undefined" Step as the reason. The test suite will show as failing in the console, the report, and the various logs.
+1. **Test bugs**: For example, a test has old values that need to be updated. The test fails. This will look like an interview bug, as this document describes.
+
+## ALKiln `Error`s
+
+1. **ALKiln un-handled bugs**: For example, we have forgotten to define a variable and none of our tests has yet triggered that code. These are unexpected errors that force processes to abort or force individual tests to fail in uninformative ways. To an author that runs into that error, it looks like their test has failed, but the fault is ours and we need to troubleshoot it. We should make that clear.
+1. **ALKiln silent bugs**: For example, a debug log has an object that causes an error when ALKiln tries to turn it into a string. ALKiln should record the failure somewhere, but should avoid throwing it. We can troubleshoot those if we see them and add new internal tests. See decisions docs on when to use which kind of log level.
+1. **ALKiln expected errors**: For example, one promise won a `Promise.race()`, so ALKiln aborted the other promises. That causes those aborted promises to error. That is why ALKiln saves the "informational" Error to the debug files and it has a level of "info" or "debug".
+
+## ALKiln's internal tests
+
+1. **ALKiln internal success-handling test failures**: For example, the test fails to fill out radio buttons correctly.
+1. **ALKiln internal failure-handling test successes**: For example, we test that the summary report shows a specific log code when an interview fails to load and the log code shows up correctly. The "test" fails - the interview fails to load - but *our* test passes - the right log code was in the summary report.
+1. **ALKiln internal failure-handling test failures**: For example, we test that the report shows a specific log code when an interview fails to load, but the log code is missing. The "test" fails and *our* test fails.

package/docs/prioritize_constrained_random_tests.md

@@ -0,0 +1,60 @@
+# Prioritize constrained random answers tests over pure random answers tests
+
+## Context and Problem Statement
+
+We could develop constrained randomness first or pure randomness first. Is there a reason to prioritize one over another?
+
+## Considered Options
+
+- Purely random answers. They fill in fields without any guidance.
+- Constrained random answers tests. They make the test author list all the variable names and all the choices of answers they want to allow for those variables.
+- Hybrid approach. Allow an author to specify some values while letting other values stay completely random.
+
+## Decision Outcome
+
+Prioritize constrained random answers tests over pure random answers tests. Strangely, they seem to be the more flexible of the two options.
+
+## Pros and Cons of the Options
+
+The discussion about the general pros and cons of random answers tests will be in the `constrained_random_tests` doc for now.
+
+### Purely random answers
+
+Pros:
+
+- Very little work for test authors to write.
+- Because they're so free-form, they may do a better job finding unexpected behavior that authors didn't think to test for.
+
+Cons:
+
+- Unless we can find better ways of generating purely random test files for the authors, copying a random answers test to keep permanently might be very challenging.
+- Fields might get nonsensical answers. For example, a very large number for age or a very small number for income.
+- Impossible to use for fields that need very specific answers, like a real lawyer's bar number or a specific file name for uploading exhibits.
+
+### Constrained random answers
+
+Pros:
+
+- Useful for fields that need very specific answers, like a real lawyer's bar number or a specific file name for uploading exhibits.
+- An author can focus the tests' answers on important user stories. For example, they might want to focus some tests on users that have children while keeping other answers more random.
+- An author might be able to make more efficient random tests. For example, a multiple choice question may have multiple answers that, behind the scenes, set the same value. Authors might do this to help users who are confused about their options give more accurate answers. A purely random test would pick that repeated value too often and waste testing time.
+
+Cons:
+
+- They take more time to write.
+- They take more work to maintain.
+
+<!-- TODO: Add to docs - each `possible_values` cell should contain 1 or more possible answers. -->
+
+### Hybrid
+
+In future, we believe we can figure out how to let users have partially constrained tests where unspecified fields get purely random answers while constrained fields choose from a set of answers. It would need to be its own Step.
+
+Pros:
+
+- Offers the most flexibility and control.
+
+Cons:
+
+- Might be hard to avoid confusion about test behavior.
+- Means developing both methods of testing. That's out of scope for one PR.