@suffolklitlab/alkiln 5.13.4 → 5.15.0-deps-2

package/docs/prioritize_constrained_random_tests.md

@@ -1,101 +1,60 @@
-# Constrained random input tests
+# Prioritize constrained random answers tests over pure random answers tests
 
-I do 10 constrained random tests
+## Context and Problem Statement
 
-## Why do people want purely random input tests?
+We could develop constrained randomness first or pure randomness first. Is there a reason to prioritize one over another?
 
-1. They don't have time to write individual specific tests
-2. They are worried they haven't covered all eventualities
+## Considered Options
 
-## Problems for purely random tests
+- Purely random answers. They fill in fields without any guidance.
+- Constrained random answers tests. They make the test author list all the variable names and all the choices of answers they want to allow for those variables.
+- Hybrid approach. Allow an author to specify some values while letting other values stay completely random.
 
-- Non-specific input. E.g. a name field might get a sentence. Can we do our best? That would be more work.
+## Decision Outcome
 
-## Why would people prefer constrained random input tests?
+Prioritize constrained random answers tests over pure random answers tests. Strangely, they seem to be the more flexible of the two options.
 
-- They want to make the tests more focused on certain scenarios and avoid wasting time on unimportant tests. For example, sometimes several options actually give the same answer. A purely random test would give more preference to the repeated answer.
-- **Some fields might require very specific answers, like a real lawyer's bar number or a specific file name for uploading exhibits.**
-- Narrow down the paths at some points, like always choosing that you have minor children
-- **Even if the user doesn't know it, this is the most flexible option**
+## Pros and Cons of the Options
 
-## Note
+The discussion about the general pros and cons of random answers tests will be in the `constrained_random_tests` doc for now.
 
-~~If in the future we want to add the option to not specify, then we can and it won't be a breaking change.~~
+### Purely random answers
 
-We have to decide what it means to allow a person to not specify. E.g if excluding a row means don't give an answer to that variable at all, then it can't in future mean "give a random answer to this field". Options:
+Pros:
 
-- (bad) Excluding a row means don't give an answer. Including a row with a blank answer means give a random answer.
-- (bad) Including a row with a blank answer means don't give an answer. Give unlisted variables random answers.
-- Or should we require a string (e.g. ALK_NO_ANSWER) for fields that should not get random inputs?
-- (most easy purely random input method to implement without changing current code?) Or specify randomness (e.g. ALK_RANDOM_ANSWER)?
-- **(backwards compatible because we can become more flexible later) Require both of the above?**
+- Very little work for test authors to write.
+- Because they're so free-form, they may do a better job finding unexpected behavior that authors didn't think to test for.
 
-Which of these is worse? Mind reading can be bad. On the other hand, specifying everything may be somewhat antithetical to the 2 goals of people who want purely random input tests. (added to the interview an unrequired value, but forgot to add it to the test, they won't test all eventualities).
+Cons:
 
-Possible solutions for people who want purely random tests or bits of tests:
-- Add a Step for answering randomly for a whole test.
-- Add a Step for answering randomly up to various page ids (list is allowed).
-- Add Step to answer randomly for one page.
+- Unless we can find better ways of generating purely random test files for the authors, copying a random answers test to keep permanently might be very challenging.
+- Fields might get nonsensical answers. For example, a very large number for age or a very small number for income.
+- Impossible to use for fields that need very specific answers, like a real lawyer's bar number or a specific file name for uploading exhibits.
 
-## Flexible constraints?
+### Constrained random answers
 
-Post MVP
+Pros:
 
-- Allow number range "ALKiln: [1, 5)"? Or they have to do "1, 2, 3, 4"
-- "ALKiln: year 1965 - 1999"
-- "ALKiln: date [today, today - 10)"
-- "ALKiln: max length 3" (of 5 checkbox answers)
-- "ALKiln: name"
-- "ALKiln: number" (and other types like "currency") (we can probably figure those out)
+- Useful for fields that need very specific answers, like a real lawyer's bar number or a specific file name for uploading exhibits.
+- An author can focus the tests' answers on important user stories. For example, they might want to focus some tests on users that have children while keeping other answers more random.
+- An author might be able to make more efficient random tests. For example, a multiple choice question may have multiple answers that, behind the scenes, set the same value. Authors might do this to help users who are confused about their options give more accurate answers. A purely random test would pick that repeated value too often and waste testing time.
 
-## Note
+Cons:
 
-MVP of random input tests
+- They take more time to write.
+- They take more work to maintain.
 
-## no seeds
+<!-- TODO: Add to docs - each `possible_values` cell should contain 1 or more possible answers. -->
 
-seeds can break more easily. instead we save the spec of the test(s)
+### Hybrid
 
-## MVP
+In future, we believe we can figure out how to let users have partially constrained tests where unspecified fields get purely random answers while constrained fields choose from a set of answers. It would need to be its own Step.
 
-save the spec of the test...somewhere. Probably using special name to say
-   "rerun" and then change the name at the end of the run.
-retry the test somehow (if needed)...
-present the test and results to the user
+Pros:
 
-## Better
+- Offers the most flexibility and control.
 
-Avoid repeating previous random tests
-Avoid repeating tests of users (which is much(?) harder)
+Cons:
 
-## Specifics
-
-Create a file with a certain # of Scenarios and run it
-   - Does cucumber decide the tests it'll run at runtime or before? I think the former
-   - Is it better to split tests up into separate files if there are a lot?
-   - Is it better or worse to help the developer feel the magnitude of the #
-     of tests they're running (using Example) [Answer: worse]
-   - How to handle separate constrained random tables?
-
-## right now
-[r1 opt 1, r1 opt 2, r1 opt 3, name input, date input, description input]
-
-## need for random
-
-detect r1 while looping through the fields
-[r1 opt 1, r1 opt 2, r1 opt 3, name input, date input, description input]
-Con: as we loop through, we don't know the number of options at each particular instance
-
-or make grouped fields into groups. Basically group r1:
-[[r1 opt 1, r1 opt 2, r1 opt 3], name input, date input, description input]
-
-## why not generate a test that uses all the constraints and thus is fully random?
-
-Because we'd have to parse docassembled *compiled* files, which is very hard
-
-Alternative:
-
-  When the developer runs a test, they can use a Step to have it collect all possible fields that ALKiln sees during that test run.
-  Q: How effectively can we decode the fields? Or can we make a tool that will help a developer decode their fields? (Include an input that just decodes one encoded string (`atob()`))
-  Q: If the fields are multiple choice and the developer's test picks at least one choice from that group of fields, can we use that to properly decode all the names of the fields? Hesitant to do this because some values may be encoded differently from other values (I think).
-  Q: Is this more work than just refactoring how we group fields?
+- Might be hard to avoid confusion about test behavior.
+- Means developing both methods of testing. That's out of scope for one PR.

package/lib/docassemble/server_install.js

@@ -56,8 +56,6 @@ const server_install = async () => {
 session_vars.validateEnvironment();
 
 // 2. Prepare logs
-// Local devs: This will be separate from your test paths
-// unless you pass a `path` argument into the cucumber run command
 const argv = require(`minimist`)(process.argv.slice(2));
 const log = new Log({ path: argv.path, context, });
 

package/lib/docassemble/setup.js

@@ -54,7 +54,7 @@ const setup = async () => {
       + `dainstall this package on your server. See above for more details.`;
     log.throw({ code: `ALK0047`, context,
       before: `☝️ ☝️ ☝️  Error data ☝️ ☝️ ☝️\n`,
-      error: `Error while installing project` },
+      error: new Error(`Error while installing project on your docassemble server during GitHub+You™ tests`) },
       install_err_msg
     );
   }
@@ -68,8 +68,6 @@ const setup = async () => {
 session_vars.validateEnvironment();
 
 // 2. Prepare logs
-// Local devs: This will be separate from your test paths
-// unless you pass a `path` argument into the cucumber run command
 const argv = require(`minimist`)(process.argv.slice(2));
 const log = new Log({ path: argv.path, context, });
 

package/lib/docassemble/takedown.js

@@ -40,8 +40,8 @@ const takedown = async () => {
 session_vars.validateEnvironment();
 
 // 2. Prepare logs
-// Local devs: This will be separate from your test artifacts paths
-// unless you pass a `path` argument into the `cucumber` run command
+// Local devs: This will be separate from your test artifacts paths unless you
+//    pass a `path` argument into the `cucumber` run command
 const argv = require(`minimist`)( process.argv.slice(2) );
 const log = new Log({ path: argv.path, context, });
 

package/lib/globals.js

@@ -0,0 +1,10 @@
+let globals = {
+  // File name prefix ALKiln can use to identify random answer generator files
+  generators_prefix: `alkiln_random`,
+  generated_files_folder_name: `_alkiln_generated`,
+}
+
+module.exports = globals;
+
+// Discuss: Would this be clearer as a config file instead? If not, change name
+// to `constants`?

package/lib/run_cucumber.js

@@ -1,21 +1,37 @@
 #!/usr/bin/env node
 
 const fs = require('fs');
+const fast_glob = require(`fast-glob`);
 const path = require(`path`);
 const cucumber_api = require('@cucumber/cucumber/api');
 
+const globals = require(`./globals`);
 const Log = require('./utils/Log');
 const session_vars = require('./utils/session_vars');
+const set_sources_paths = require(`./utils/set_sources_paths`);
 
 const SUMMARY_FILENAME = `cucumber-report.txt`;
 
 
-function build_config( command_args ) {
+function build_config() {
+  /** Use the command line arguments to build the cucumberjs config object.
+   * 
+   * @param {string?} - Arguments passed to the command line.
+   * 
+   * @returns {object} - cucumberjs config object
+   * */
   // Turn all the paths to the sources folders into paths to the feature files
   let sources_paths = session_vars.get_sources_paths();
   let feature_paths = [];
-  for ( let source of sources_paths ) {
-    feature_paths.push( `${ source }/*.feature` );
+
+  // Get all the paths for the right feature files (for cucumber to use)
+  for ( let sources_path of sources_paths ) {
+    let glob = fast_glob.sync(
+      `${ sources_path }/**/*.feature`,
+      // Ignore "generator" feature files that we used to generate actual tests
+      { ignore: [`${ sources_path }/${ globals.generators_prefix }*.feature`] }
+    );
+    feature_paths.push( ...glob );
   }
 
   const config = {
@@ -26,9 +42,10 @@ function build_config( command_args ) {
     retry: 1,
   }
 
+  const cli_args = require(`minimist`)( process.argv.slice(2) );
   // Normalize tag expressions, if there are any
   // Joining and splitting handles both tags surrounded with `""` or those without `""`
-  let tags = command_args[`_`].join(` `).split(` `);
+  let tags = cli_args[`_`].join(` `).split(` `);
   // If there was no tag expression, `tags` is ['']. Remove that item.
   tags = tags.filter(function(entry) { return entry !== ""; })
   if ( tags.length > 0 ) {
@@ -56,46 +73,12 @@ function build_config( command_args ) {
 
 async function main() {
   /** Create the cucumber config, run the tests, create and
-  *   manipluate the output. */
+  *   manipulate the output. */
 
   const environment = { cwd: process.cwd() };
   log.info({ code: `ALK0049`, context: `pre_run environment` },
     environment
   );
-
-  // We'll use the user info to construct the possible paths to /sources
-  let id = session_vars.get_user_id();
-  // `project` is the name in the server docassemble "Project" page
-  let project = session_vars.get_user_project_name();
-
-  // process.argv is a list of strings of commands and args.
-  // If using `npm run cucumber` (as opposed to the bin commands),
-  // use `--` before `--sources=./foo`
-  const argv = require(`minimist`)( process.argv.slice(2), {
-    default: {
-      sources: [
-        `./docassemble/*/data/sources`,  // In GitHub folder
-        `/usr/share/docassemble/files/playgroundsources/${id}/${project}/*.feature`, // From playground without S3
-        `/tmp/playgroundsources/${id}/${project}/*.feature` // From playground with S3
-      ],
-    }
-  });
-  
-  // Make sure `sources` is always a list
-  let sources_list = argv.sources;
-  if ( typeof argv.sources === `string` ) {
-    sources_list = [ argv.sources ];
-  }
-
-  let one_dir_exists = session_vars.set_sources_paths( sources_list );
-  if ( !one_dir_exists ) {
-    // If we don't throw, it'll be more confusing when no tests run
-    log.throw({ code: `ALK0050`, context: `pre_run`,
-      error: new ReferenceError(
-        `ALKiln can't find your "sources" folder(s). See above for details.`
-      ),
-    });
-  }
   
   const { runConfiguration } = await cucumber_api.loadConfiguration({
     provided: build_config( argv ),
@@ -122,9 +105,9 @@ async function main() {
 
   /** Write to files.
    *
-   * Note: Don't add to the purely report-focused files. Those are files where
-   *    authors can focus on the info we give them, which they often miss in
-   *    the GitHub action.
+   * Note: Avoid adding to the purely report-focused files. Those are files
+   *    where authors can focus on the info we give them, which they often miss
+   *    in the GitHub action.
    */
 
   if ( success ) {
@@ -134,7 +117,7 @@ async function main() {
   } else {
     // Don't want to throw error here, so don't pass error object
     log.warn({ code: `ALK0055`, context: `post_run`, },
-      `The tests ran into something unexpected. Look above for more details.`
+      `The tests ran into something unexpected. See warnings and errors above.`
     );
   }
   // Custom exit code. May overlap with other systems.
@@ -142,14 +125,12 @@ async function main() {
   const exitCode = success ? 0 : 50;
   process.exitCode = exitCode;
 
-  // This is hardcoded information from `provide_config` above.
-  // This code isn't used for anything other than detecting and removing color codes.
   let summary_file_name = SUMMARY_FILENAME;
   fs.readFile(summary_file_name, 'utf8', (read_error, text) => {
     if ( read_error ) {
       // Something else will probably show an error that's more relevant
       log.warn({ code: `ALK0056`, context: `post_run error`, },
-        `Failed to read cucumber report`, read_error
+        `Failed to read the report of the 3rd party library, cucumberjs. There may be more information above or below.`, read_error
       );
       return;
     }
@@ -184,7 +165,13 @@ async function main() {
         try {
           fs.readFileSync(`${ log.path }/${ log.unexpected_filename }`);  // Does it have contents?
           fs.appendFileSync(`${ log.path }/${ log.unexpected_filename }`, `\n\n\n\n\n` );
-        } catch ( ignore_err ) { /* Missing unexpected file is ok */ }
+        } catch ( ignore_err ) {
+          /* Missing "unexpected results" file is ok */
+        }
+
+        // TODO: Is this meant to write to the debug file or is this debug info
+        // really supposed to be in the "unexpected results" file? Or are these
+        // comments old?
 
         // Add the heading for debug content
         fs.appendFileSync( `${ log.path }/${ log.unexpected_filename }`,
@@ -198,8 +185,8 @@ async function main() {
 
     } catch ( writing_cucumber_error ) {
       let code = `ALK0182`;
-      let context = `post_run error`
-      let msg = `Something probably went wrong earlier. Look above for more details. Skipped writing to ${ log.path }/${ log.debug_log_filename }.`;
+      let context = `post_run`
+      let msg = `Something probably went wrong earlier. See warnings and errors above. Skipped writing to ${ log.path }/${ log.debug_log_filename }.`;
       try {
         log.warn({ code, context, }, msg, writing_cucumber_error );
       } catch ( all_writing_error ) {
@@ -223,10 +210,10 @@ session_vars.validateEnvironment();
 // If using `npm run cucumber` (as opposed to the bin commands),
 // use `--` before `--sources=./foo`
 const argv = require(`minimist`)( process.argv.slice(2) );
-// The artifacts folder might be getting created for the first
-// time right here. The tests themselves should avoid creating
-// their own file.
 const log = new Log({ path: argv.path, context: `run` });
 
-// 3. run
+// 3. Prep the sources paths to find .feature files
+let sources_paths = set_sources_paths({ log });
+
+// 4. run
 main();

package/lib/scope.js

@@ -888,7 +888,7 @@ module.exports = {
   },  // Ends scope.normalizeTable()
 
   getAllFields:  async function ( scope, { html }) {
-    /* Given HTML string of a page, returns a list of all the fields in it
+    /** Given HTML string of a page, returns a list of all the fields in it
     *  as some kind of sophisticated data? With their selectors, for example.
     *  We will choose selectors that will continue to identify the node, eve
     *  after a hidden field is revealed, etc.
@@ -1019,12 +1019,15 @@ module.exports = {
       // button, radio, checkbox yesno, and maybe others need the value of the `value` attribute
       let values = [];
       if ( node.name === `select` ) {
+        // Get possible values
         values = await scope.getOptionValues( scope, { $select: $node });
       }
       if ( values.length === 0 && node.attribs.value ) {
         // TODO: Add links to docs about da YAML values that devs never fill in themselves and
         // so don't know what they are. E.g. yesnomabye -> True/False/None.
         // TODO: Text inputs and textareas must ignore the value, etc. How?
+
+        // Get the current value for not `select` fields
         values.push( node.attribs.value );
       }
 
@@ -1387,6 +1390,14 @@ module.exports = {
     * @param <field[n].guesses[n].trigger> {str} - var name that triggered this page
     * 
     * @returns [ var_data[n] ]
+    * @returns [{
+    *   table_row: var_data[n],
+    *   field: {
+    *     selector: str,
+    *     guesses: [{ values: [str] }]
+    *   }
+    * }]
+    * What does it return when there's no match?
     */
 
     let rows_that_match_field_names = [];
@@ -2118,6 +2129,38 @@ module.exports = {
     // await scope.page.mouse.move(1, 1);  // Too big? Wait a while before drawing?
     await scope.page.mouse.up();
   },
+  
+  get_page_id: async function ( scope ) {
+    /** Return the current page's docassemble `question` block ID or `null`
+     *
+     * Note: The ID is already sanitized by docassemble.
+     *
+     * @param {object} scope - Tests state object
+     * @param {object} scope.page - puppeteerjs page object
+     *
+     * @returns {null | string}
+     * */
+    let question_id_match = await scope.page.$eval(`body`, function (elem) {
+      return elem.className.match(/question-([^ ]+)/);
+    });
+    if ( question_id_match !== null ) {
+      return question_id_match[ 1 ];
+    }
+    return null;
+  },
+
+  to_docassemble_sanitized_id: function ( risky_id ) {
+    /** Return a version of `risky_id` that's sanitized the same way docassemble
+     *    sanitizes its IDs to turn them into HTML class names in python:
+     *      re.sub(r'[^A-Za-z0-9]+', '-', interview_status.question.id.lower())
+     *    (reference needed)
+     *
+     * @param {string} risky_id - The un-sanitized id
+     *
+     * @returns {string} - Sanitized id
+     * */
+    return risky_id.toLowerCase().replace(/[^A-Za-z0-9]+/g, '-');
+  },
 
   examinePageID: async function ( scope, question_id = '' ) {
     /* Looks for a sanitized version of the question id as it's written