@suffolklitlab/alkiln 5.13.4 → 5.14.0

package/docs/prioritize_constrained_random_tests.md

@@ -1,101 +1,60 @@
-# Constrained random input tests
+# Prioritize constrained random answers tests over pure random answers tests
 
-I do 10 constrained random tests
+## Context and Problem Statement
 
-## Why do people want purely random input tests?
+We could develop constrained randomness first or pure randomness first. Is there a reason to prioritize one over another?
 
-1. They don't have time to write individual specific tests
-2. They are worried they haven't covered all eventualities
+## Considered Options
 
-## Problems for purely random tests
+- Purely random answers. They fill in fields without any guidance.
+- Constrained random answers tests. They make the test author list all the variable names and all the choices of answers they want to allow for those variables.
+- Hybrid approach. Allow an author to specify some values while letting other values stay completely random.
 
-- Non-specific input. E.g. a name field might get a sentence. Can we do our best? That would be more work.
+## Decision Outcome
 
-## Why would people prefer constrained random input tests?
+Prioritize constrained random answers tests over pure random answers tests. Strangely, they seem to be the more flexible of the two options.
 
-- They want to make the tests more focused on certain scenarios and avoid wasting time on unimportant tests. For example, sometimes several options actually give the same answer. A purely random test would give more preference to the repeated answer.
-- **Some fields might require very specific answers, like a real lawyer's bar number or a specific file name for uploading exhibits.**
-- Narrow down the paths at some points, like always choosing that you have minor children
-- **Even if the user doesn't know it, this is the most flexible option**
+## Pros and Cons of the Options
 
-## Note
+The discussion about the general pros and cons of random answers tests will be in the `constrained_random_tests` doc for now.
 
-~~If in the future we want to add the option to not specify, then we can and it won't be a breaking change.~~
+### Purely random answers
 
-We have to decide what it means to allow a person to not specify. E.g if excluding a row means don't give an answer to that variable at all, then it can't in future mean "give a random answer to this field". Options:
+Pros:
 
-- (bad) Excluding a row means don't give an answer. Including a row with a blank answer means give a random answer.
-- (bad) Including a row with a blank answer means don't give an answer. Give unlisted variables random answers.
-- Or should we require a string (e.g. ALK_NO_ANSWER) for fields that should not get random inputs?
-- (most easy purely random input method to implement without changing current code?) Or specify randomness (e.g. ALK_RANDOM_ANSWER)?
-- **(backwards compatible because we can become more flexible later) Require both of the above?**
+- Very little work for test authors to write.
+- Because they're so free-form, they may do a better job finding unexpected behavior that authors didn't think to test for.
 
-Which of these is worse? Mind reading can be bad. On the other hand, specifying everything may be somewhat antithetical to the 2 goals of people who want purely random input tests. (added to the interview an unrequired value, but forgot to add it to the test, they won't test all eventualities).
+Cons:
 
-Possible solutions for people who want purely random tests or bits of tests:
-- Add a Step for answering randomly for a whole test.
-- Add a Step for answering randomly up to various page ids (list is allowed).
-- Add Step to answer randomly for one page.
+- Unless we can find better ways of generating purely random test files for the authors, copying a random answers test to keep permanently might be very challenging.
+- Fields might get nonsensical answers. For example, a very large number for age or a very small number for income.
+- Impossible to use for fields that need very specific answers, like a real lawyer's bar number or a specific file name for uploading exhibits.
 
-## Flexible constraints?
+### Constrained random answers
 
-Post MVP
+Pros:
 
-- Allow number range "ALKiln: [1, 5)"? Or they have to do "1, 2, 3, 4"
-- "ALKiln: year 1965 - 1999"
-- "ALKiln: date [today, today - 10)"
-- "ALKiln: max length 3" (of 5 checkbox answers)
-- "ALKiln: name"
-- "ALKiln: number" (and other types like "currency") (we can probably figure those out)
+- Useful for fields that need very specific answers, like a real lawyer's bar number or a specific file name for uploading exhibits.
+- An author can focus the tests' answers on important user stories. For example, they might want to focus some tests on users that have children while keeping other answers more random.
+- An author might be able to make more efficient random tests. For example, a multiple choice question may have multiple answers that, behind the scenes, set the same value. Authors might do this to help users who are confused about their options give more accurate answers. A purely random test would pick that repeated value too often and waste testing time.
 
-## Note
+Cons:
 
-MVP of random input tests
+- They take more time to write.
+- They take more work to maintain.
 
-## no seeds
+<!-- TODO: Add to docs - each `possible_values` cell should contain 1 or more possible answers. -->
 
-seeds can break more easily. instead we save the spec of the test(s)
+### Hybrid
 
-## MVP
+In future, we believe we can figure out how to let users have partially constrained tests where unspecified fields get purely random answers while constrained fields choose from a set of answers. It would need to be its own Step.
 
-save the spec of the test...somewhere. Probably using special name to say
-   "rerun" and then change the name at the end of the run.
-retry the test somehow (if needed)...
-present the test and results to the user
+Pros:
 
-## Better
+- Offers the most flexibility and control.
 
-Avoid repeating previous random tests
-Avoid repeating tests of users (which is much(?) harder)
+Cons:
 
-## Specifics
-
-Create a file with a certain # of Scenarios and run it
-   - Does cucumber decide the tests it'll run at runtime or before? I think the former
-   - Is it better to split tests up into separate files if there are a lot?
-   - Is it better or worse to help the developer feel the magnitude of the #
-     of tests they're running (using Example) [Answer: worse]
-   - How to handle separate constrained random tables?
-
-## right now
-[r1 opt 1, r1 opt 2, r1 opt 3, name input, date input, description input]
-
-## need for random
-
-detect r1 while looping through the fields
-[r1 opt 1, r1 opt 2, r1 opt 3, name input, date input, description input]
-Con: as we loop through, we don't know the number of options at each particular instance
-
-or make grouped fields into groups. Basically group r1:
-[[r1 opt 1, r1 opt 2, r1 opt 3], name input, date input, description input]
-
-## why not generate a test that uses all the constraints and thus is fully random?
-
-Because we'd have to parse docassembled *compiled* files, which is very hard
-
-Alternative:
-
-  When the developer runs a test, they can use a Step to have it collect all possible fields that ALKiln sees during that test run.
-  Q: How effectively can we decode the fields? Or can we make a tool that will help a developer decode their fields? (Include an input that just decodes one encoded string (`atob()`))
-  Q: If the fields are multiple choice and the developer's test picks at least one choice from that group of fields, can we use that to properly decode all the names of the fields? Hesitant to do this because some values may be encoded differently from other values (I think).
-  Q: Is this more work than just refactoring how we group fields?
+- Might be hard to avoid confusion about test behavior.
+- Means developing both methods of testing. That's out of scope for one PR.

package/lib/scope.js

@@ -888,7 +888,7 @@ module.exports = {
   },  // Ends scope.normalizeTable()
 
   getAllFields:  async function ( scope, { html }) {
-    /* Given HTML string of a page, returns a list of all the fields in it
+    /** Given HTML string of a page, returns a list of all the fields in it
     *  as some kind of sophisticated data? With their selectors, for example.
     *  We will choose selectors that will continue to identify the node, eve
     *  after a hidden field is revealed, etc.
@@ -1019,12 +1019,15 @@ module.exports = {
       // button, radio, checkbox yesno, and maybe others need the value of the `value` attribute
       let values = [];
       if ( node.name === `select` ) {
+        // Get possible values
         values = await scope.getOptionValues( scope, { $select: $node });
       }
       if ( values.length === 0 && node.attribs.value ) {
         // TODO: Add links to docs about da YAML values that devs never fill in themselves and
         // so don't know what they are. E.g. yesnomabye -> True/False/None.
         // TODO: Text inputs and textareas must ignore the value, etc. How?
+
+        // Get the current value for not `select` fields
         values.push( node.attribs.value );
       }
 
@@ -1387,6 +1390,14 @@ module.exports = {
     * @param <field[n].guesses[n].trigger> {str} - var name that triggered this page
     * 
     * @returns [ var_data[n] ]
+    * @returns [{
+    *   table_row: var_data[n],
+    *   field: {
+    *     selector: str,
+    *     guesses: [{ values: [str] }]
+    *   }
+    * }]
+    * What does it return when there's no match?
     */
 
     let rows_that_match_field_names = [];

package/lib/steps.js

@@ -707,39 +707,48 @@ Then(/the "([^"]+)" link opens in (a new window|the same window)/, async (linkTe
   await scope.afterStep(scope);
 });
 
-Then(/(?:the )?text(?: in)?(?: the)?(?: JSON)?(?: var)?(?:iable)?(?: )?"([^"]+)" should be/i, async (var_name, expected_value) => {
-  /* Check that the text string of the page's JSON var value matches
-  *    the given text string. Does not currently accept nested values
-  *    like "foo.bar". The word "text" is required.
-  *
-  * The format for this step is something like this:
-  * ```
-  * Then the text in the JSON variable "var_name" should be
-  * """
-  * A single or
-  * multiline string
-  * """
-  * ```
-  *
-  * Sentence combinations that will work:
-  * `the text in the JSON variable "variable_with_text_value" should be`
-  * `the text in the JSON var "variable_with_text_value" should be`
-  * `the text in "variable_with_text_value" should be`
-  * `text var "variable_with_text_value" should be`
-  */
+Then(/(?:the )?text(?: in)?(?: for)?(?: the)?(?: JSON)?(?: var)?(?:iable)?(?: )?"([^"]+)" should be/i, async (var_name, expected_value) => {
+  /** Check that the text string of the page's JSON var value matches
+   *    the given text string. Does not currently accept nested values
+   *    like "foo.bar". The word "text" is required.
+   *
+   * The format for this step is something like this:
+   * ```
+   * Then the text in the JSON variable "var_name" should be
+   * """
+   * A single or
+   * multiline string
+   * """
+   * ```
+   * 
+   * Note: ALKiln removes `\r` from both the expected and actual values.
+   *
+   * Sentence combinations that will work:
+   * the text in the JSON variable "variable_with_text_value" should be
+   * the text in the JSON var "variable_with_text_value" should be
+   * the text for "variable_with_text_value" should be
+   * text "variable_with_text_value" should be
+   * */
 
   // Get JSON variable value
   let { json_path, json } = await scope.savePageJSONData( scope );
+
+  // docassemble (at least sometimes) makes new lines with `\r\n`
   let actual_value = json.variables[ var_name ];
+  actual_value = actual_value.replace(/\r/, ``);
+  // Normalize `expected_value` as well for backwards compatibility in case any
+  //     authors are already compensating for docassemble
+  expected_value = expected_value.replace(/\r/, ``);
 
   // Add warning to report if value is not as expected
   if ( actual_value !== expected_value ) {
     reports.addToReport( scope, {
       type: `error`,
       code: `ALK0083`,
-      value: `The variable "${ var_name }" was not equal to the expected `
-        + `value on the "${ scope.page_id }" screen. Check `
-        + `${ json_path } to see the page's JSON variables.`
+      value: `The variable "${ var_name }" was different from the expected `
+        + `value on the "${ scope.page_id }" screen.`
+        + `The expected values was\n` + expected_value
+        + `\nwhile the actual value was\n` + actual_value
     });
   }
 
@@ -1044,6 +1053,22 @@ When(/I set the var(?:iable)? "([^"]+)" to "([^"]+)"/, { timeout: -1 }, async (
 
 });
 
+When(/I set the var(?:iable)? "([^"]+)" to\s*$/, { timeout: -1 }, async ( var_name, answer ) => {
+  /** Set a non-story table variable (with or without a choice associated with
+   *     it) to a value that can have new lines in it. Most useful for
+   *     `textarea`s.
+   *
+   * Example:
+   *   I set the var "benefits['SSI']" to
+   *     """
+   *     Some multline
+   *     value
+   *     """
+   * */
+  // `page` timeout will deal with custom timeout
+  await scope.steps.set_regular_var( scope, var_name, answer )
+});
+
 When(/I set the var(?:iable)? "([^"]+)" to ?(?:the)? ?(?:GitHub)? secret "([^"]+)"/i, { timeout: -1 }, async ( var_name, set_to_env_name ) => {
   /* Set a non-story table variable to a secrete value. Same as the above, but reads the value from `process.env` */
   // This could theoretically tap an element, so

package/lib/utils/Log.js

@@ -491,31 +491,28 @@ class Log {
   }  // Ends Log._stringify_logs()
 
   _stringify_inline({ prev_line_type, inline_log }) {
-    /** Formats the start of an inline log based on the previous log and returns
-     *  that combined string.
+    /** Formats the log to add a space or a new line separator. May end up with
+     *     a leading space for now.
      *
      * @param { "inline"|"block"|null } prev_line_type - Required. The type of
      *     the previous log.
      * @param { null|non-obj } inline_log - Required. The log.
      *
-     * @returns { string } The formatted log.
+     * @returns { string } The formatted line.
      *
      * @example
      * // Backticks are to make the surrounding white space more clear
      * log._stringify_inline({ prev_line_type: null, inline_log: "apple" });
-     * // `apple`
+     * // "apple "
      * log._stringify_inline({ prev_line_type: 'inline', inline_log: "apple" });
-     * // ` apple`
+     * // "apple "
      * log._stringify_inline({ prev_line_type: 'block', inline_log: "apple" });
-     * // `
-     * // apple`
+     * // "
+     * // apple"
      * */
-    // An inline log's string starts with nothing if it's the first string...
     let start = ``;
-    // ...starts after a space if its next to an inline string...
-    if ( prev_line_type === `inline` ) { start = ` `; }
-    // ...or starts on a new line if it's after a block of text, like an error
-    // or object.
+    if ( prev_line_type === `inline` && !inline_log.startsWith(`\n`) ) { start = ` `; }
+    // This is actually the end of the block-type log
     else if ( prev_line_type === `block` ) { start = `\n`; }
     return `${ start }${ inline_log }`;
   }
@@ -525,6 +522,8 @@ class Log {
      *  a "block"-type separator at the start. Can handle circular references.
      *  Catch ANY error.
      *
+     * @param { "inline"|"block"|null } prev_line_type - Required. The type of
+     *     the previous log.
      * @param { Object } obj_log - Required. The object to format.
      *
      * @returns { string } The obj as a string plus a separator
@@ -536,15 +535,15 @@ class Log {
      * // { fruit: 'bananas' }
      * */
     let start = ``;
-    if ( prev_line_type === `inline` || prev_line_type === `block` ) {
-      start = `\n`;
-    }
+    // Avoid adding a new line to the start of the first line
+    let is_first_line = prev_line_type === null;
+    if ( !is_first_line ) { start = `\n`; }
 
     try {
       // Try to stringify the object.
       return `${ start }${ this._inspect({ value: obj_log }) }`;
 
-    // Discuss: record these errors?
+    // Discuss: record the outer error?
     } catch ( outer_error ) {
       // If that fails, see if it has its own way of stringifying
       try {