@nzz/q-cli 1.6.0 → 1.8.0

package/README.md

@@ -164,7 +164,29 @@ Q new-custom-code my-project-name
 Q new-custom-code my-project-name -d my-project-directory
 ```
 
-### Updating existing Q items
+### Creating new ed-tech utility package project
+
+Once `Q` cli is installed one can create the skeleton of a new ed-tech utility package project by executing
+
+```bash
+Q new-et-utils-package package-name package-author package-description
+```
+
+- The directory name where the new ed-tech utility package project is being created defaults to the project name and can be overwritten by using option `-d` or `--dir`
+
+```bash
+Q new-et-utils-package package-name -d my-project-directory
+```
+
+#### Notes
+
+New utility package projects should only be created inside the [ed-tech-utilities](https://github.com/nzzdev/ed-tech-utilities) repository.
+
+### Q item actions
+
+The `Q` cli can copy and/or update existing Q items.
+
+#### Updating existing Q items
 
 Once `Q` cli installed one can update one or many Q items by executing:
 
@@ -193,63 +215,116 @@ Q update-item -r
 - Credentials can be provided as environment variables to avoid user prompts. The variable names are `Q_ENV_SERVER`, `Q_ENV_USERNAME`, `Q_ENV_PASSWORD`, `Q_ENV_ACCESSTOKEN`, where `ENV` is the uppercase version of the environment name.
 
 ```bash
-Q_TEST_SERVER=https://q-server.st-test.nzz.ch/ Q_TEST_USERNAME=[username] Q_TEST_PASSWORD=[password] Q update-item
+Q_TEST_SERVER=[server_route] Q_TEST_USERNAME=[username] Q_TEST_PASSWORD=[password] Q update-item
 ```
 
 or
 
 ```bash
-Q_TEST_SERVER=https://q-server.st-test.nzz.ch/ Q_TEST_ACCESSTOKEN=[accessToken] Q update-item
+Q_TEST_SERVER=[server_route] Q_TEST_ACCESSTOKEN=[accessToken] Q update-item
 ```
 
-Alternatively
-
-The config file has to follow [this json-schema](./bin/commands/updateItem/schema.json). Here an example:
+The config file has to follow [this json-schema](./bin/commands/qItem/updateItem/updateSchema.json). This schema will be extended by the respective tool schema of your Q item.
+Here's an example:
 
 ```json
 {
   "items": [
     {
       "environments": [
+        // "environments" references the desired q items to be updated, at least 1 environment is required
         {
           "name": "production",
-          "id": "6dcf203a5c5f74b61aeea0cb0eef7e0b"
+          "id": "6dcf203a5c5f74b61aeea0cb0eef7e0b" // Id of your q item in the production environment
         },
         {
           "name": "staging",
-          "id": "6dcf203a5c5f74b61aeea0cb0ef2ca9f"
+          "id": "6dcf203a5c5f74b61aeea0cb0ef2ca9f" // Id of your q item in the staging environment
         }
       ],
       "item": {
+        // The actual content you want to update for your referenced q items listed in "environments"
         "title": "Der Konsum in der Schweiz springt wieder an",
         "subtitle": "Wöchentliche Ausgaben mittels Bankkarten in Mio. Fr. im Jahr 2020, zum Vergleich 2019",
         "data": [
+          // "data" represents the data table of your q item inside the q-editor
           ["Datum", "2020", "2019"],
           ["2020-01-06", "690004302", "641528028"],
           ["2020-01-13", "662122373", "617653790"],
           ["2020-01-20", "688208667", "654303249"]
         ]
       }
-    },
+    }
+  ]
+}
+```
+
+#### Copy existing Q items
+
+Once `Q` cli installed one can copy one or many Q items by executing:
+
+```bash
+Q copy-item
+```
+
+- The path to the config file can be set by using option `-c` or `--config`. By default the `copy-item` command will look for a config file called `q.config.json` in the current directory
+
+```bash
+Q copy-item -c [path]
+```
+
+- Items of a specified environment can be updated by using the option `-e` or `--environment`. By default the `copy-item` command updates all item specified in the config file
+
+```bash
+Q copy-item -e [env]
+```
+
+- Stored configuration properties like Q-Server url or access tokens can be reset by using option `-r` or `--reset`
+
+```bash
+Q copy-item -r
+```
+
+- Credentials can be provided as environment variables to avoid user prompts. The variable names are `Q_ENV_SERVER`, `Q_ENV_USERNAME`, `Q_ENV_PASSWORD`, `Q_ENV_ACCESSTOKEN`, where `ENV` is the uppercase version of the environment name.
+
+```bash
+Q_TEST_SERVER=[server_route] Q_TEST_USERNAME=[username] Q_TEST_PASSWORD=[password] Q update-item
+```
+
+or
+
+```bash
+Q_TEST_SERVER=[server_route] Q_TEST_ACCESSTOKEN=[accessToken] Q update-item
+```
+
+The config file has to follow [this json-schema](./bin/commands/qItem/updateItem/updateSchema.json). This schema will be extended by the respective tool schema of your Q item.
+Here's an example:
+
+```json
+{
+  "items": [
     {
       "environments": [
         {
           "name": "production",
-          "id": "6dcf203a5c5f74b61aeea0cb0ef2edea"
+          "id": "6dcf203a5c5f74b61aeea0cb0eef7e0b" // Id of your q item in the production environment
         },
         {
           "name": "staging",
-          "id": "6dcf203a5c5f74b61aeea0cb0ef68480"
+          "id": "6dcf203a5c5f74b61aeea0cb0ef2ca9f" // Id of your q item in the staging environment
         }
       ],
       "item": {
-        "title": "Der Lastwagenverkehr in Deutschland nimmt wieder zu",
-        "subtitle": "Täglicher Lkw-Maut-Fahrleistungsindex (2015 = 100, saison- und kalenderbereinigt) im Jahr 2020, zum Vergleich 2019\t\t",
-        "data": [
-          ["Datum", "2020", "2019"],
-          ["2020-01-07", "105.9", "112.1"],
-          ["2020-01-08", "108.9", "111.4"],
-          ["2020-01-09", "112.2", "113.5"]
+        "title": "Russische Angriffe auf die Ukraine",
+        "subtitle": "Verzeichnete Angriffe in der ganzen Ukraine",
+        "files": [
+          // Adds or overwrites the listed files in your q item
+          {
+            "loadSyncBeforeInit": false, // Has to be set for the file upload to work
+            "file": {
+              "path": "./angriffsFlaechen.json" // Your local path to your file. The path is relative to where you execute the command.
+            }
+          }
         ]
       }
     }
@@ -257,8 +332,6 @@ The config file has to follow [this json-schema](./bin/commands/updateItem/schem
 }
 ```
 
-The configuration object has a property `items` which contains an object for each Q item. A Q item has a property `environments` and `item`. The `environments` array contains an objects with properties `name` and `id` for each environment the item is deployed on. The `item` contains the data of the Q item. The structure of the item can vary between each graphic type (chart, map, table ect.).
-
 [to the top](#table-of-contents)
 
 ## License

package/bin/commands/bootstrap.js

@@ -2,11 +2,19 @@ const fs = require("fs-extra");
 const path = require("path");
 const replaceInFile = require("replace-in-file");
 const chalk = require("chalk");
+const { replace } = require("nunjucks/src/filters");
 const errorColor = chalk.red;
 const successColor = chalk.green;
 const warningColor = chalk.yellow;
 
-module.exports = async function (type, name, basedir) {
+/**
+ *
+ * @param {string} type - Skeleton type
+ * @param {string} name - Name of the project
+ * @param {string} basedir - Base directory name to be created
+ * @param {Array.<{regex: RegExp, replaceWith: string}>} textReplacements
+ */
+module.exports = async function (type, basedir, textReplacements) {
   if (fs.existsSync(basedir)) {
     console.error(
       errorColor(`directory ${basedir} already exists or is not writable`)
@@ -16,34 +24,46 @@ module.exports = async function (type, name, basedir) {
     fs.mkdirSync(basedir);
   }
 
-  const replaceOptions = {
-    files: `${basedir}/**`,
-    from: new RegExp(`${type}-skeleton`, "g"),
-    to: name,
-    glob: {
-      dot: true, // Include file names starting with a dot
-    },
-  };
-
   try {
     await fs.copySync(
       path.join(__dirname, `../../skeletons/${type}-skeleton`),
       basedir
     );
-    await replaceInFile(replaceOptions);
+
+    if (textReplacements) {
+      for (const txtRe of textReplacements) {
+        await replaceText(txtRe.regex, txtRe.replaceWith, basedir);
+      }
+    }
+
     console.log(successColor(`Q ${type} is now bootstrapped in ${basedir}`));
 
-    if (type === "tool")
+    if (type === "tool" || type === "et-utils-package")
       console.log(
-        warningColor("Search for 'TODO:' inside the new tool to get started!")
+        warningColor(
+          "Search for 'TODO' inside the new tool/package to get started!"
+        )
       );
   } catch (error) {
     console.error(
       errorColor(
-        `An unexpected error occured. Please check the entered information and try again. ${JSON.stringify(
+        `An unexpected error occurred. Please check the entered information and try again. ${JSON.stringify(
           error
         )}`
       )
     );
   }
 };
+
+async function replaceText(regex, replaceWith, basedir) {
+  const replaceOptions = {
+    files: `${basedir}/**`, // Replace in all files
+    from: regex,
+    to: replaceWith,
+    glob: {
+      dot: true, // Include file names starting with a dot
+    },
+  };
+
+  return await replaceInFile(replaceOptions);
+}

package/bin/commands/qItem/configStore.js

@@ -0,0 +1,142 @@
+const helpers = require("./helpers.js");
+const promptly = require("promptly");
+const Configstore = require("configstore");
+const package = require("./../../../package.json");
+const configStore = new Configstore(package.name, {});
+
+async function setAuthenticationConfig(environment, qServer) {
+  const result = await authenticate(environment, qServer);
+  configStore.set(`${environment}.accessToken`, result.accessToken);
+  configStore.set(`${environment}.cookie`, result.cookie);
+}
+
+async function setupStore(qConfig, environmentFilter, reset) {
+  if (reset) {
+    configStore.clear();
+  }
+  for (const environment of helpers.getEnvironments(
+    qConfig,
+    environmentFilter
+  )) {
+    await setupConfigFromEnvVars(environment);
+
+    if (!configStore.get(`${environment}.qServer`)) {
+      const qServer = await promptly.prompt(
+        `Enter the Q-Server url for ${environment} environment: `,
+        {
+          validator: (qServer) => {
+            return new URL(qServer).toString();
+          },
+          retry: true,
+        }
+      );
+      configStore.set(`${environment}.qServer`, qServer);
+    }
+
+    const qServer = configStore.get(`${environment}.qServer`);
+    if (!configStore.get(`${environment}.accessToken`)) {
+      await setAuthenticationConfig(environment, qServer);
+    }
+
+    const accessToken = configStore.get(`${environment}.accessToken`);
+    const cookie = configStore.get(`${environment}.cookie`);
+    const isAccessTokenValid = await helpers.checkValidityOfAccessToken(
+      environment,
+      qServer,
+      accessToken,
+      cookie
+    );
+
+    // Get a new access token in case its not valid anymore
+    if (!isAccessTokenValid) {
+      await setAuthenticationConfig(environment, qServer);
+    }
+  }
+
+  return configStore;
+}
+
+async function setupConfigFromEnvVars(environment) {
+  const environmentPrefix = environment.toUpperCase();
+
+  const qServer = process.env[`Q_${environmentPrefix}_SERVER`];
+  if (qServer) {
+    configStore.set(`${environment}.qServer`, qServer);
+  }
+  const accessToken = process.env[`Q_${environmentPrefix}_ACCESSTOKEN`];
+  const username = process.env[`Q_${environmentPrefix}_USERNAME`];
+  const password = process.env[`Q_${environmentPrefix}_PASSWORD`];
+  if (qServer && accessToken) {
+    configStore.set(`${environment}.accessToken`, accessToken);
+  } else if (qServer && username && password) {
+    const cookie = configStore.get(`${environment}.cookie`);
+    const result = await helpers.getAccessToken(
+      environment,
+      qServer,
+      username,
+      password,
+      cookie
+    );
+
+    if (!result) {
+      console.error(
+        errorColor(
+          `A problem occured while authenticating to the ${environment} environment using environment variables. Please check your credentials and try again.`
+        )
+      );
+      process.exit(1);
+    }
+
+    configStore.set(`${environment}.accessToken`, result.accessToken);
+    configStore.set(`${environment}.cookie`, result.cookie);
+  }
+}
+
+async function authenticate(environment, qServer) {
+  let username = configStore.get(`${environment}.username`);
+  if (!username) {
+    username = await promptly.prompt(
+      `Enter your username on ${environment} environment: `,
+      { validator: (username) => username.trim() }
+    );
+    configStore.set(`${environment}.username`, username);
+  }
+
+  const password = await promptly.password(
+    `Enter your password on ${environment} environment: `,
+    {
+      validator: async (password) => password.trim(),
+      replace: "*",
+    }
+  );
+
+  const cookie = configStore.get(`${environment}.cookie`);
+  let result = await helpers.getAccessToken(
+    environment,
+    qServer,
+    username,
+    password,
+    cookie
+  );
+
+  while (!result) {
+    console.error(
+      errorColor(
+        "A problem occured while authenticating. Please check your credentials and try again."
+      )
+    );
+
+    result = await authenticate(environment, qServer);
+
+    if (result.accessToken) {
+      break;
+    }
+  }
+
+  return result;
+}
+
+module.exports = {
+  store: configStore,
+  setupStore: setupStore,
+};

package/bin/commands/qItem/copyItem/copyItem.js

@@ -0,0 +1,103 @@
+const schemaService = require("./../schemaService.js");
+const configStore = require("./../configStore.js");
+const itemService = require("./../itemService.js");
+const fs = require("fs");
+const path = require("path");
+const chalk = require("chalk");
+const errorColor = chalk.red;
+const successColor = chalk.green;
+
+module.exports = async function (command) {
+  try {
+    const qConfigPath = path.resolve(command.config);
+
+    if (fs.existsSync(qConfigPath)) {
+      const qConfig = JSON.parse(fs.readFileSync(qConfigPath));
+      const validationResult = schemaService.validateConfig(
+        qConfig,
+        "copyItem"
+      );
+
+      if (validationResult.isValid) {
+        const config = await configStore.setupStore(
+          qConfig,
+          command.environment,
+          command.reset
+        );
+
+        for (const item of itemService.getItems(qConfig, command.environment)) {
+          for (const environment of item.environments) {
+            const qServer = config.get(`${environment.name}.qServer`);
+            const accessToken = config.get(`${environment.name}.accessToken`);
+            const cookie = config.get(`${environment.name}.cookie`);
+
+            const existingItem = await itemService.getItem(
+              qServer,
+              environment,
+              accessToken,
+              cookie
+            );
+
+            delete existingItem.updatedBy;
+            delete existingItem.createdBy;
+            delete existingItem.createdDate;
+            delete existingItem._id;
+            delete existingItem._rev;
+
+            let newItem = await itemService.createItem(
+              existingItem,
+              environment,
+              config
+            );
+            // Save for success message
+            const newItemId = newItem._id;
+            const existingItemId = environment.id;
+
+            const hasOverwrites =
+              item.item &&
+              Object.keys(item.item).length > 0 &&
+              Object.getPrototypeOf(item.item) === Object.prototype;
+
+            if (hasOverwrites) {
+              environment.id = newItemId;
+
+              newItem = await itemService.updateItem(
+                item.item,
+                environment,
+                config,
+                qConfigPath
+              );
+            }
+
+            if (newItem) {
+              console.log(
+                successColor(
+                  `Successfully copied item with id ${existingItemId} on ${environment.name} environment. Copied item id ${newItemId}`
+                )
+              );
+            }
+          }
+        }
+      } else {
+        console.error(
+          errorColor(
+            `A problem occured while validating the config file: ${validationResult.errorsText}`
+          )
+        );
+        process.exit(1);
+      }
+    } else {
+      console.error(
+        errorColor(
+          "Couldn't find config file named q.config.json in the current directory. Create a config file in the current directory or pass the path to the config file with the option -c <path>"
+        )
+      );
+    }
+  } catch (error) {
+    console.error(
+      errorColor(
+        `A problem occured while parsing the config file at ${command.config}. Please make sure it is valid JSON.`
+      )
+    );
+  }
+};

package/bin/commands/qItem/copyItem/copySchema.json

@@ -0,0 +1,37 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "title": "Q Config",
+  "description": "Config used by the Q CLI to copy items",
+  "type": "object",
+  "properties": {
+    "items": {
+      "description": "Array of Q items",
+      "type": "array",
+      "minItems": 1,
+      "items": {
+        "type": "object",
+        "properties": {
+          "environments": {
+            "type": "array",
+            "minItems": 1,
+            "items": {
+              "type": "object",
+              "properties": {
+                "id": {
+                  "type": "string",
+                  "description": "Id of Q item"
+                }
+              },
+              "required": ["id"]
+            }
+          },
+          "item": {
+            "type": "object"
+          }
+        },
+        "required": ["environments", "item"]
+      }
+    }
+  },
+  "required": ["items"]
+}

package/bin/commands/qItem/helpers.js

@@ -0,0 +1,102 @@
+const fetch = require("node-fetch");
+const chalk = require("chalk");
+const errorColor = chalk.red;
+
+function getEnvironments(qConfig, environmentFilter) {
+  try {
+    const environments = new Set();
+    for (const item of qConfig.items) {
+      for (const environment of item.environments) {
+        if (environmentFilter) {
+          if (environmentFilter === environment.name) {
+            environments.add(environment.name);
+          }
+        } else {
+          environments.add(environment.name);
+        }
+      }
+    }
+
+    if (environments.size > 0) {
+      return Array.from(environments);
+    } else {
+      throw new Error(
+        `No items with environment ${environmentFilter} found. Please check your configuration and try again.`
+      );
+    }
+  } catch (error) {
+    console.error(errorColor(error.message));
+    process.exit(1);
+  }
+}
+
+async function getAccessToken(
+  environment,
+  qServer,
+  username,
+  password,
+  cookie
+) {
+  try {
+    const response = await fetch(`${qServer}authenticate`, {
+      method: "POST",
+      headers: {
+        "user-agent": "Q Command-line Tool",
+        origin: qServer,
+        Cookie: cookie ? cookie : "",
+      },
+      body: JSON.stringify({
+        username: username,
+        password: password,
+      }),
+    });
+
+    if (response.ok) {
+      const body = await response.json();
+      return {
+        accessToken: body.access_token,
+        cookie: response.headers.get("set-cookie"),
+      };
+    }
+
+    return false;
+  } catch (error) {
+    console.error(
+      errorColor(
+        `A problem occured while authenticating on ${environment} environment. Please check your connection and try again.`
+      )
+    );
+    process.exit(1);
+  }
+}
+
+async function checkValidityOfAccessToken(
+  environment,
+  qServer,
+  accessToken,
+  cookie
+) {
+  try {
+    const response = await fetch(`${qServer}user`, {
+      headers: {
+        "user-agent": "Q Command-line Tool",
+        Authorization: `Bearer ${accessToken}`,
+        Cookie: cookie ? cookie : "",
+      },
+    });
+    return response.ok;
+  } catch (error) {
+    console.error(
+      errorColor(
+        `A problem occured while checking the validity of your access token on ${environment} environment. Please check your connection and try again.`
+      )
+    );
+    process.exit(1);
+  }
+}
+
+module.exports = {
+  getEnvironments: getEnvironments,
+  getAccessToken,
+  checkValidityOfAccessToken,
+};