@google-cloud/nodejs-common 0.9.4-beta3 → 0.9.9-alpha

package/README.md

@@ -1,69 +1,78 @@
 # NodeJS Common Library
 
-<!--* freshness: { owner: 'lushu' reviewed: '2021-06-01' } *-->
-
-A NodeJs common library for other projects, e.g. [GMP and Google Ads 
-Connector] and [Data Tasks Coordinator]. This library includes:
-
-1.  Authentication wrapper based on google auth library to support OAuth, JWT
-    and ADC authentication;
-
-1.  Wrapper for some Google APIs for integration, mainly for [GMP and Google Ads 
- Connector]:
-
-    *   Google Analytics data import
-    *   Google Analytics measurement protocol
-    *   Campaign Manager offline conversion upload    
-    *   Search Ads 360 conversions upload
-    *   Google Ads click conversions upload
-    *   Google Ads customer match upload
-    *   Google Ads conversions scheduled uploads based on Google Sheets
-    *   Measurement Protocol Google Analytics 4
-
-1.  Wrapper for some Google APIs for reporting, mainly for [Data Tasks Coordinator]:
-
-    *   Google Ads reporting
-    *   Campaign Manager reporting
-    *   Search Ads 360 reporting
-    *   Display and Video 360 reporting
-    *   Ads Data Hub querying
-
-1.  Utilities wrapper class for Google Cloud Products:
-
-    *   **Firestore Access Object**: Firestore has two modes which are excluded
-        to each other and can't be changed once selected in a Cloud
-        Project[[2]]. This class, with its two successors offer a unified
-        interface to operate data objects on either Firestore or Datastore.
-
-    *  **AutoMl Tables API**: Offers a unified entry to use this API based on 
-        Google Cloud client library combined with REST requests to service 
-        directly due to some functionalities missed in the client library.
-
-    *   **Pub/Sub Utilities**: Offers utilities functions to create topics and
-        subscriptions for Pub/Sub, as well as the convenient way to publish a
-        message.
-
-    *   **Storage Utilities**: Offers functions to manipulate the files on Cloud
-        Storage. The main functions are:
-
-        *   Reading a given length (or slightly less) content without breaking a
-            line;
-        *   Splitting a file into multiple files with the given length (or
-            slightly less) without breaking a line;
-        *   Merging files into one file.
-    
-    *  **Cloud Scheduler Adapter**: A wrapper to pause and resume Cloud 
-       Scheduler jobs.
-    
-    *  **Cloud Functions Adapter**: Cloud Functions have different parameters
-       in different environments, e.g. Node6 vs Node8[[1]]. This utility file
-       offers an adapter to wrap a Node8 Cloud Functions into Node6 and Node8
-       compatible functions.
-       
+<!--* freshness: { owner: 'lushu' reviewed: '2021-12-02' } *-->
+
+A NodeJs common library for other projects, e.g. [GMP and Google Ads Connector]
+and [Data Tasks Coordinator]. This library includes:
+
+1. Authentication wrapper based on google auth library to support OAuth, JWT and
+   ADC authentication;
+
+1. Wrapper for some Google APIs for integration, mainly
+   for [GMP and Google Ads Connector]:
+
+    * Google Analytics data import
+    * Google Analytics measurement protocol
+    * Campaign Manager offline conversion upload
+    * Search Ads 360 conversions upload
+    * Google Ads click conversions upload
+    * Google Ads customer match upload
+    * Google Ads conversions scheduled uploads based on Google Sheets
+    * Measurement Protocol Google Analytics 4
+
+1. Wrapper for some Google APIs for reporting, mainly
+   for [Data Tasks Coordinator]:
+
+    * Google Ads reporting
+    * Campaign Manager reporting
+    * Search Ads 360 reporting
+    * Display and Video 360 reporting
+    * YouTube Data API
+    * Ads Data Hub querying
+
+1. Utilities wrapper class for Google Cloud Products:
+
+    * **Firestore Access Object**: Firestore has two modes which are excluded to
+      each other and can't be changed once selected in a Cloud Project[[2]].
+      This class, with its two successors offer a unified interface to operate
+      data objects on either Firestore or Datastore.
+
+    * **AutoMl Tables API**: Offers a unified entry to use this API based on
+      Google Cloud client library combined with REST requests to service
+      directly due to some functionalities missed in the client library.
+
+    * **Vertex AI API**: Offers a unified entry to use this API based on Google
+      Cloud client library.
+
+    * **Pub/Sub Utilities**: Offers utilities functions to create topics and
+      subscriptions for Pub/Sub, as well as the convenient way to publish a
+      message.
+
+    * **Storage Utilities**: Offers functions to manipulate the files on Cloud
+      Storage. The main functions are:
+
+        * Reading a given length (or slightly less) content without breaking a
+          line;
+        * Splitting a file into multiple files with the given length (or
+          slightly less) without breaking a line;
+        * Merging files into one file.
+
+    * **Cloud Scheduler Adapter**: A wrapper to pause and resume Cloud Scheduler
+      jobs.
+
+    * **Cloud Functions Adapter**: Cloud Functions have different parameters in
+      different environments, e.g. Node6 vs Node8[[1]]. This utility file offers
+      an adapter to wrap a Node8 Cloud Functions into Node6 and Node8 compatible
+      functions.
+
 1. A share library for [Bash] to facilitate installation tasks.
 
 [GMP and Google Ads Connector]:https://github.com/GoogleCloudPlatform/cloud-for-marketing/tree/master/marketing-analytics/activation/gmp-googleads-connector
+
 [Data Tasks Coordinator]:https://github.com/GoogleCloudPlatform/cloud-for-marketing/tree/master/marketing-analytics/activation/data-tasks-coordinator
+
 [1]:https://cloud.google.com/functions/docs/writing/background#functions-writing-background-hello-pubsub-node8-10
+
 [2]:https://cloud.google.com/datastore/docs/concepts/overview#comparison_with_traditional_databases
+
 [Bash]:https://www.gnu.org/software/bash/

package/bin/install_functions.sh

@@ -108,6 +108,7 @@ declare EXTERNAL_APIS=(
   "sheets.googleapis.com"
   "googleads.googleapis.com"
   "adsdatahub.googleapis.com"
+  "youtube.googleapis.com"
 )
 
 # Some APIs only support OAuth which needs an process to generate refresh token.
@@ -115,6 +116,7 @@ declare EXTERNAL_APIS=(
 declare EXTERNAL_APIS_OAUTH_ONLY=(
   "googleads.googleapis.com"
   "doubleclickbidmanager.googleapis.com"
+  "youtube.googleapis.com"
 )
 
 # API scopes for OAuth authentication.
@@ -133,6 +135,8 @@ EXTERNAL_API_SCOPES=(
   ["sheets.googleapis.com"]="https://www.googleapis.com/auth/spreadsheets"
   ["googleads.googleapis.com"]="https://www.googleapis.com/auth/adwords"
   ["adsdatahub.googleapis.com"]="https://www.googleapis.com/auth/adsdatahub"
+  ["youtube.googleapis.com"]=\
+"https://www.googleapis.com/auth/youtube.force-ssl"
 )
 
 # Enabled APIs' OAuth scopes.
@@ -739,8 +743,8 @@ select_dataset_location() {
     "ASIA_PACIFIC"
   )
   local MULTI_REGIONAL=(
-    "Data centers within member states of the European Union (EU)"
-    "Data centers in the United States (US)"
+    "Data centers within member states of the European Union (eu)"
+    "Data centers in the United States (us)"
   )
   local AMERICAS=(
     "${NORTH_AMERICA[@]}"
@@ -1195,21 +1199,17 @@ create_or_update_sink() {
   local existingFilter
   existingFilter=$(gcloud logging sinks list --filter="name:${sinkName}" \
 --format="value(filter)")
-  if [[ "${existingFilter}" != "${logFilter}" ]]; then
-    local action
-    if [[ -z "${existingFilter}" ]]; then
-      action="create"
-      printf '%s\n' "  Logging Export [${sinkName}] doesn't exist. Creating..."
-    else
-      action="update"
-      printf '%s\n' "  Logging Export [${sinkName}] exists with a different \
-filter. Updating..."
-    fi
-    gcloud -q logging sinks ${action} "${sinkName}" "${sinkDestAndFlags[@]}" \
---log-filter="${logFilter}"
+  local action
+  if [[ -z "${existingFilter}" ]]; then
+    action="create"
+    printf '%s\n' "  Logging Export [${sinkName}] doesn't exist. Creating..."
   else
-    printf '%s\n' "  Logging Export [${sinkName}] exists. Continue..."
+    action="update"
+    printf '%s\n' "  Logging Export [${sinkName}] exists with a different \
+filter. Updating..."
   fi
+  gcloud -q logging sinks ${action} "${sinkName}" "${sinkDestAndFlags[@]}" \
+--log-filter="${logFilter}"
   if [[ $? -gt 0 ]];then
     printf '%s\n' "Failed to create or update Logs router sink."
     return 1
@@ -1829,9 +1829,8 @@ get_cloud_functions_service_account() {
 #   None.
 #######################################
 check_firestore_existence() {
-  local firestore_status
-  firestore_status=$(gcloud firestore operations list 2>&1)
-  while [[ ${firestore_status} =~ .*NOT_FOUND.* ]]; do
+  gcloud firestore indexes fields list >/dev/null 2>&1
+  while [[ $? -gt 0 ]]; do
     cat <<EOF
 Cannot find Firestore or Datastore in current project. Please visit \
 https://console.cloud.google.com/firestore?project=${GCP_PROJECT} to create a \
@@ -1842,7 +1841,7 @@ EOF
     local any
     read -n1 -s any
     printf '\n'
-    firestore_status=$(gcloud firestore operations list 2>&1)
+    gcloud firestore indexes fields list >/dev/null 2>&1
   done
 }
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@google-cloud/nodejs-common",
-  "version": "0.9.4-beta3",
+  "version": "0.9.9-alpha",
   "description": "A NodeJs common library for solutions based on Cloud Functions",
   "author": "Google Inc.",
   "license": "Apache-2.0",
@@ -16,21 +16,22 @@
   },
   "homepage": "https://github.com/GoogleCloudPlatform/cloud-for-marketing/blob/master/marketing-analytics/activation/common-libs/nodejs-common/README.md",
   "dependencies": {
-    "@google-cloud/aiplatform": "^1.10.1",
-    "@google-cloud/automl": "^2.4.2",
-    "@google-cloud/bigquery": "^5.6.0",
-    "@google-cloud/datastore": "^6.4.0",
-    "@google-cloud/firestore": "^4.12.2",
-    "@google-cloud/logging-winston": "^4.0.5",
-    "@google-cloud/pubsub": "^2.12.0",
-    "@google-cloud/storage": "^5.8.5",
-    "gaxios": "^4.3.0",
-    "google-ads-api": "^8.1.0",
-    "google-ads-node": "^6.1.3",
-    "google-auth-library": "^7.1.0",
-    "googleapis": "^74.2.0",
-    "soap": "^0.38.0",
-    "winston": "^3.3.3"
+    "@google-cloud/aiplatform": "^1.13.0",
+    "@google-cloud/automl": "^2.5.1",
+    "@google-cloud/bigquery": "^5.9.2",
+    "@google-cloud/datastore": "^6.6.2",
+    "@google-cloud/firestore": "^4.15.1",
+    "@google-cloud/logging-winston": "^4.1.1",
+    "@google-cloud/pubsub": "^2.18.2",
+    "@google-cloud/storage": "^5.16.0",
+    "gaxios": "^4.3.2",
+    "google-ads-api": "^9.0.0",
+    "google-ads-node": "^7.0.0",
+    "google-auth-library": "^7.10.2",
+    "googleapis": "^91.0.0",
+    "soap": "^0.43.0",
+    "winston": "^3.3.3",
+    "lodash": "^4.17.21"
   },
   "devDependencies": {
     "jasmine": "^3.5.0"

package/src/apis/auth_client.js

@@ -110,6 +110,7 @@ class AuthClient {
    * @return {!OAuth2Client}
    */
   getOAuth2Client(keyFile = this.oauthTokenFile) {
+    this.ensureFileExisting_(keyFile, 'OAUTH token');
     const key = JSON.parse(fs.readFileSync(keyFile).toString());
     console.log(`Get OAuth token with client Id: ${key.client_id}`);
     const oAuth2Client = new OAuth2Client(key.client_id, key.client_secret);
@@ -123,6 +124,7 @@ class AuthClient {
    * @return {!JWT}
    */
   getServiceAccount(keyFile = this.serviceAccountKeyFile) {
+    this.ensureFileExisting_(keyFile, 'Service Account key');
     console.log(`Get Service Account's key file: ${keyFile}`);
     return new JWT({keyFile, scopes: this.scopes,});
   }
@@ -141,6 +143,7 @@ class AuthClient {
    * }}
    */
   getOAuth2Token(keyFile = this.oauthTokenFile) {
+    this.ensureFileExisting_(keyFile, 'OAuth token');
     const key = JSON.parse(fs.readFileSync(keyFile).toString());
     return {
       clientId: key.client_id,
@@ -148,6 +151,18 @@ class AuthClient {
       refreshToken: key.token.refresh_token,
     };
   }
+
+  /**
+   * Some APIs only support one authorization type, so we have to designate a
+   * specific auth method. If the related key/token file is not available, the
+   * auth will fail. The function throws errors with more meaningful message.
+   * @param {string} file Target file path
+   * @param {string} type Key type name, 'OAuth token' or 'Service Account key'
+   * @private
+   */
+  ensureFileExisting_(file, type) {
+    if (!file) throw new Error(`Required ${type} file doesn't exist.`);
+  }
 }
 
 /**

package/src/apis/dfa_reporting.js

@@ -26,6 +26,7 @@ const {
   getLogger,
   getFilterFunction,
   SendSingleBatch,
+  BatchResult,
 } = require('../components/utils.js');
 
 const API_SCOPES = Object.freeze([
@@ -64,7 +65,7 @@ let InsertConversionsConfig;
 /**
  * List of properties that will be take from the data file as elements of a
  * conversion.
- * See https://developers.google.com/doubleclick-advertisers/v3.3/conversions
+ * See https://developers.google.com/doubleclick-advertisers/rest/v3.5/Conversion
  * @type {Array<string>}
  */
 const PICKED_PROPERTIES = [
@@ -79,8 +80,14 @@ const PICKED_PROPERTIES = [
  * see https://developers.google.com/doubleclick-advertisers/service_accounts
  */
 class DfaReporting {
-  constructor() {
-    const authClient = new AuthClient(API_SCOPES);
+
+  /**
+   * @constructor
+   * @param {!Object<string,string>=} env The environment object to hold env
+   *     variables.
+   */
+  constructor(env = process.env) {
+    const authClient = new AuthClient(API_SCOPES, env);
     this.auth = authClient.getDefaultAuth();
     /** @const {!google.dfareporting} */
     this.instance = google.dfareporting({
@@ -126,7 +133,7 @@ class DfaReporting {
      * @param {!Array<string>} lines Data for single request. It should be
      *     guaranteed that it doesn't exceed quota limitation.
      * @param {string} batchId The tag for log.
-     * @return {!Promise<boolean>}
+     * @return {!Promise<BatchResult>}
      */
     return async (lines, batchId) => {
       /** @type {function} Gets the conversion elements from the data object. */
@@ -153,6 +160,11 @@ class DfaReporting {
       if (config.idType === 'encryptedUserId') {
         requestBody.encryptionInfo = config.encryptionInfo;
       }
+      /** @const {BatchResult} */
+      const batchResult = {
+        result: true,
+        numberOfLines: lines.length,
+      };
       try {
         const response = await this.instance.conversions.batchinsert({
           profileId: config.profileId,
@@ -160,25 +172,61 @@ class DfaReporting {
         });
         const failed = response.data.hasFailures;
         if (failed) {
-          console.error(`Dfareporting[${batchId}] has failures.`);
-          const errorMessages = response.data.status
-              .filter((record) => typeof record.errors !== 'undefined')
-              .map((record) => {
-                this.logger.debug(record);
-                return record.errors.map((error) => error.message).join(',');
-              });
-          console.error(errorMessages.join('\n'));
+          this.logger.warn(`CM [${batchId}] has failures.`);
+          this.extraFailedLines_(batchResult, response.data.status, lines);
         }
         this.logger.debug('Configuration: ', config);
         this.logger.debug('Response: ', response);
-        return !failed;
+        return batchResult;
       } catch (error) {
-        console.error(`Dfareporting[${batchId}] failed.`, error);
-        return false;
+        this.logger.error(`CM[${batchId}] failed.`, error);
+        batchResult.result = false;
+        batchResult.errors = [error.message || error.toString()];
+        return batchResult;
       }
     };
   };
 
+  /**
+   * Campaign Manager API returns an array of ConversionStatus for the status of
+   * uploaded conversions. If there are errors related to the conversion, then
+   * an array of 'ConversionError' named 'errors' will be available in the
+   * ConversionStatus object. This function extras failed lines and error
+   * messages based on the 'errors'.
+   * For 'ConversionStatus', see:
+   *   https://developers.google.com/doubleclick-advertisers/rest/v3.5/ConversionStatus
+   * For 'ConversionError', see:
+   *   https://developers.google.com/doubleclick-advertisers/rest/v3.5/ConversionStatus#ConversionError
+   * @param {!BatchResult} batchResult
+   * @param {!Array<!Schema$ConversionStatus>} statuses
+   * @param {!Array<string>} lines The original input data.
+   * @private
+   */
+  extraFailedLines_(batchResult, statuses, lines) {
+    batchResult.result = false;
+    batchResult.failedLines = [];
+    batchResult.groupedFailed = {};
+    const errors = new Set();
+    statuses.forEach((conversionStatus, index) => {
+      if (conversionStatus.errors) {
+        const failedLine = lines[index];
+        batchResult.failedLines.push(failedLine);
+        conversionStatus.errors.forEach(({message}) => {
+          // error messages have detailed IDs. Need to generalize them.
+          const generalMessage = message.replace(/.*error: /, '');
+          errors.add(generalMessage);
+          const groupedFailed = batchResult.groupedFailed[generalMessage]
+              || [];
+          groupedFailed.push(failedLine);
+          if (groupedFailed.length === 1) {
+            batchResult.groupedFailed[generalMessage] = groupedFailed;
+          }
+        });
+      }
+      batchResult.errors = Array.from(errors);
+    });
+  }
+
   /**
    * Lists all UserProfiles.
    * @return {!Promise<!Array<string>>}
@@ -203,8 +251,8 @@ class DfaReporting {
    * @return {!Promise<string>} Profile Id.
    * @private
    */
-  getProfileForOperation_(config) {
-    if (config.profileId) return Promise.resolve(config.profileId);
+  async getProfileForOperation_(config) {
+    if (config.profileId) return config.profileId;
     if (config.accountId) return this.getProfileId(config.accountId);
     throw new Error('There is no profileId or accountId in the configuration.');
   }
@@ -213,7 +261,7 @@ class DfaReporting {
    * Runs a report and return the file Id. As an asynchronized process, the
    * returned file Id will be a placeholder until the status changes to
    * 'REPORT_AVAILABLE' in the response of `getFile`.
-   * @see https://developers.google.com/doubleclick-advertisers/v3.3/reports/run
+   * @see https://developers.google.com/doubleclick-advertisers/rest/v3.5/reports/run
    *
    * @param {{
    *   accountId:(string|undefined),
@@ -236,7 +284,7 @@ class DfaReporting {
    * Returns file url from a report. If the report status is 'REPORT_AVAILABLE',
    * then return the apiUrl from the response; if the status is 'PROCESSING',
    * returns undefined; otherwise throws an error.
-   * @see https://developers.google.com/doubleclick-advertisers/v3.3/reports/files/get
+   * @see https://developers.google.com/doubleclick-advertisers/rest/v3.5/reports/get
    *
    * @param {{
    *   accountId:(string|undefined),