ac-useractionlog-connector 4.0.25

package/README.md

@@ -0,0 +1,190 @@
+# AC UserActionLog Connector
+Connect an app to our userActionLog service with this module.
+
+The module receives some log payload from backend applications, sends them to AWS firehose for processing/converting and stores the data (as original as well as converted) to S3.
+
+[![Node.js CI](https://github.com/AdmiralCloud/ac-useractionlog-connector/actions/workflows/node.js.yml/badge.svg)](https://github.com/AdmiralCloud/c-useractionlog-connector/actions/workflows/node.js.yml)
+
+# Breaking change in version 4 / Upgrading from version 3
++ improved compatibility with AWS SDK
++ use AWS_PROFILE instead of custom profile
++ let SDK handle session management (refresh tokens)
++ function is now a class - so initializing slightly changes
+
+To upgrade (your local or non EC2 machines), please read the section "Local development/non EC2 instances"
+
+# Breaking change in version 3
++ works with Node16 or higher
++ async/await - no callback!
++ uses AWS IAM roles or AWS IAM profiles instead of IAM credentials
+
+# Upgrading from version 1 to 2
+Please note that version 1 is no longer supported. Version 2 uses AWS Kinesis Firehose, AWS Glue and AWS Athena.
+
+The data is stored in S3 and you can export/download it from there any time.
+
+# Usage 
+
+```
+// Init during bootstrap
+
+const ualConnector = require('ac-useractionlog-connector')
+
+ualConnector = new ual({
+  // optional config parameters (see below)
+})
+
+// now send logs like this
+await ualConnector.log({ ip, processingTime, statusCode, logMeta: { body: req.body } }, req)
+```
+
+### Configuration options
+|Parameter|Default value|Description|
+|-----|----|----|
+|region|eu-central-1|Region for Firehose|
+|deliveryStreamName|UserActionLogs|Name of the stream in Firehose|
+|applicationName|AC-UAL-AppNameNotSet|Name of the logging application|
+|applicationVersion|AppVersionNotSet|Version of the logging application|
+
+# AWS Credentials
+Credentials are fetched using the approach described here:
+https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-credentials-node.html
+
+## Local development/non EC2 instances
+AWS SDK automatically handles expired session tokens if you follow these instructions:
+
+1 Download this script -> https://github.com/AdmiralCloud/bashHelper/blob/master/awsConnect/getRoleSession.sh
+2 Prepare the file ~/.aws/config and add a configuration for the role (e.g. ac-api-instance-role) you want to use:
+```
+// dev is the profile name of the IAM user (your IAM user) that is allowed to assume the role. 
+// The profile must be defined in ~/.aws/credentials
+
+[profile ac-api-instance-role]
+credential_process = /ABSOLUTE_PATH/BashHelpers/awsConnect/getRoleSession.sh  "ac-api-instance-role" "dev"
+```
+
+3 export AWS_PROFILE=role in the project where this connector is used. You can also add it to pm2 config json.
+4 Run the script in your project that uses this connector. The SDK will handle everything else for you!
+
+
+# Preparation
+## Glue
+Create a manual table in Glue:
+
+Name it "UAL-Default", select the database "useractionlog" (or create it if not existing), choose Type "Kinesis" with Location "UserActionLog" and the Kinesis URL "https://glue.eu-central-1.amazonaws.com". Select data format JSON. And then use the following fields:
+
+
+| Pos | Field | Type |
+| --- | --- | --- |
+1 | ip | string |
+2 | method | string |
+3 | controller | string |
+4 | action | string |
+5 | statuscode | int |
+6 | processingtime | int |
+7 | userId | int |
+8 | customerId | int |
+9 | mediaContainerId | int |
+10 | created | bigint |
+11 | createdAt | timestamp |
+12 | platform | string |
+13 | platformVersion | string |
+14 | clientVersion | string |
+15 | token | string |
+16 | clientid | string |
+17 | deviceidentifier | string |
+18 | useragent | string |
+19 | payload | string |
+20 | truncated | boolean |
+21 | response | string |
+
+
+## AWS Firehose
+First you have to create a delivery stream (Firehose) in AWS Kinesis. Name it "UserActionLogs", select "Direct PUT or other sources". 
+
+On the next page, choose "Convert record format" and select "Apache Parquet" format. Use the table definition from Glue (above).
+
+The destination should be S3 logbucket with prefix "ual/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/". The error prefixx can be "ual_errors/".
+
+Also select S3 backup and use the logbucket and prefix "userActionLogs.
+
+Finally, create the delivery stream.
+
+# Analysis
+To analyze your data you have to create a Athena database and query it.
+
+
+### Create the table
+Before running the query below, consider the following:
++ If you don't need the whole month, but only a day, remove "day" from partition and update the location with the day (day=xx)
++ The fewer data is indexed, the faster the process.
+
+```
+// Create the table in database useractionlog
+CREATE EXTERNAL TABLE IF NOT EXISTS ual_202011 (
+  ip string,
+  method string,
+  controller string,
+  action string,
+  statusCode int,
+  processingtime int,
+  userId int,
+  customerId int,
+  mediaContainerId int,
+  createdAt timestamp,
+  platform string,
+  clientVersion string,
+  payload string,
+  token string,
+  clientId string,
+  deviceIdentifier string,
+  response boolean,
+  response string      
+  userAgent string,
+  created bigint,
+) 
+PARTITIONED BY (
+  day string,
+  hour string
+)
+STORED AS PARQUET 
+LOCATION 's3://logs.admiralcloud.live.fra/ual/year=2020/month=11/'
+```
+
+After creation and before querying, you need to load the partition using the following command:
+
+```
+MSCK REPAIR TABLE ual_202011;
+```
+
+When querying data, please consider reducing the data to parse using the partions created above
+
+```
+// Example Query using partitions
+
+SELECT * FROM "useractionlog"."ual_202011" 
+where hour >= '14' AND hour < '15'
+limit 10;
+
+```
+
+# Error handling
+If creating a Glue table is not working via AWS console, you can use CLI but have to edit/add field definitions etc later. So use this only as fallback!
+```
+// FALLBACK
+  aws glue create-table \
+  --database-name useractionlog \
+  --table-input  '{
+    "Name":"ual-default", "StorageDescriptor":{}}' \
+  --endpoint https://glue.eu-central-1.amazonaws.com
+```
+
+
+
+## Links
+- [Website](https://www.admiralcloud.com/)
+- [Twitter (@admiralcloud)](https://twitter.com/admiralcloud)
+- [Facebook](https://www.facebook.com/MediaAssetManagement/)
+
+## License
+Copyright mmpro

package/debug.js

@@ -0,0 +1,43 @@
+const ual = require('./index')
+
+const connector = new ual({}) 
+
+const params = {
+  ip: '8.8.8.8',
+  statusCode: 200,
+  processingTime: 51,
+  userId: 123,
+  customerId: 147,
+  platform: 'APIv5',
+  requestPayload: {
+    type: 'video'
+  },
+  responsePayload: {
+    id: '123',
+    type: 'video'
+  }
+}
+
+const req = {
+  authInfo: {
+    token: 'token-123-abc',
+    clientId: 'clientId-456-111',
+    deviceIdentifier: 'myDevice',
+    clientVersion: '11.04.34455'
+  },
+  method: 'post',
+  options: {
+    controller: 'mediacontainer',
+    action: 'create'
+  },
+  headers: {
+    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36'
+  },
+  allParams: () => {
+    return params.requestPayload
+  }
+}
+
+setInterval(async() => {
+  await connector.log(params, req)
+}, 15000)

package/eslint.config.js

@@ -0,0 +1,34 @@
+const globals = require('globals')
+
+module.exports = [
+  {
+    files: ['index.js', 'test/test.js'],
+    languageOptions: {
+      ecmaVersion: 2022,
+      sourceType: 'module',
+      globals: {
+        ...globals.commonjs,
+        ...globals.es2015,
+        ...globals.node,
+        expect: 'readonly',
+        describe: 'readonly',
+        it: 'readonly'
+      }
+    },
+    rules: {
+      'no-const-assign': 'error',
+      'space-before-function-paren': 'off',
+      'no-extra-semi': 'off',
+      'object-curly-spacing': ['error', 'always'],
+      'brace-style': ['error', 'stroustrup', { allowSingleLine: true }],
+      'block-spacing': 'error',
+      'no-useless-escape': 'off',
+      'no-console': ['warn', { allow: ['warn', 'error'] }],
+      'no-unused-vars': 'error',
+      'eqeqeq': 'error',
+      'no-var': 'error',
+      'curly': 'error',
+      'prefer-const': ['error', { ignoreReadBeforeAssign: true }]
+    }
+  }
+]

package/index.js

@@ -0,0 +1,167 @@
+const _ = require('lodash')
+
+const { FirehoseClient, PutRecordCommand } = require("@aws-sdk/client-firehose")
+const https = require('https')
+const agent = new https.Agent({
+  keepAlive: true
+})
+
+// ATTN: export AWS_PROFILE=ac-api-instance-role (or whatever role is used)
+
+class FirehoseManager {
+  constructor({ region = 'eu-central-1', deliveryStreamName = 'UserActionLogs', applicationName= 'AC-UAL-AppNameNotSet', applicationVersion = 'AppVersionNotSet', debug }) {
+    this.debug = debug
+    const awsParams = {
+      region,
+      httpOptions: { agent }
+    }
+
+    this.firehose = {}
+    this.firehose.deliveryStreamName = deliveryStreamName
+    this.firehose.applicationName = applicationName
+    this.firehose.applicationVersion = applicationVersion
+    this.firehose.client = new FirehoseClient(awsParams)
+  }
+
+  async microTimeString () {
+    const time = new Date().getTime()
+    const hrtime = process.hrtime()
+    const id = time + '.' + (hrtime[0] * 1e9 + hrtime[1])
+    return id.toString().substring(0, 20)
+  }
+
+  // Trim response object without deep recursion:
+  // - keeps all top-level keys
+  // - truncates arrays at top level and one level deep to max 3 items
+  trimData(data) {
+    try {
+      const obj = JSON.parse(data)
+      for (const key of Object.keys(obj)) {
+        if (Array.isArray(obj[key])) {
+          obj[key] = obj[key].slice(0, 3).map(item => {
+            if (item && typeof item === 'object') {
+              for (const k of Object.keys(item)) {
+                if (typeof item[k] === 'string' && item[k].length > 500) {
+                  item[k] = item[k].slice(0, 500) + '...'
+                }
+              }
+            }
+            return item
+          })
+        }
+        else if (obj[key] && typeof obj[key] === 'object') {
+          for (const subKey of Object.keys(obj[key])) {
+            if (Array.isArray(obj[key][subKey])) {
+              obj[key][subKey] = obj[key][subKey].slice(0, 3)
+            }
+          }
+        }
+        else if (typeof obj[key] === 'string' && obj[key].length > 500) {
+          obj[key] = obj[key].slice(0, 500) + '...'
+        }
+      }
+
+      return JSON.stringify(obj)
+    }
+    catch {
+      console.error('AC-UserActionLog-Connector | Failed to parse JSON for trimming')
+      return null
+    }
+  }
+
+  async log(params, req) {
+    const created = parseInt(this.microTimeString())
+    const platform = _.toLower(_.get(params, 'platform', this.firehose.applicationName))
+    const platformVersion = _.toLower(_.get(params, 'platformVersion', this.firehose.applicationVersion))
+
+    const requestParams = _.get(params, 'requestPayload', req && req.allParams())
+
+    const obscureFields = [
+      { field: 'password', search: /.+/g, replace: 'XXXX-XXXX' }
+    ]
+
+    _.forEach(obscureFields, field => {
+      if (_.get(requestParams, field.field)) { _.set(requestParams, field.field, _.get(requestParams, field.field).replace(field.search, field.replace)) }
+    })
+
+    let mediaContainerId = _.get(params, 'mediaContainerId') || _.get(requestParams, 'mediaContainerId') || _.get(params, 'responsePayload.mediaContainerId')
+    if (!mediaContainerId && _.get(req, 'options.controller') === 'mediacontainer' && _.get(req, 'options.action') === 'create') { mediaContainerId = _.get(params, 'responsePayload.id') }
+
+    const clientId = _.get(req, 'authInfo.clientId', _.get(requestParams, 'client_id'))
+    const deviceIdentifier = _.get(req, 'authInfo.deviceIdentifier', _.get(requestParams, 'device'))
+    const clientVersion = _.get(req, 'authInfo.clientVersion', _.get(requestParams, 'clientVersion'))
+    const loginuid = _.get(params, 'loginuid')
+
+    const now = new Date()
+    const ualPayload = {
+      ip: _.get(params, 'ip'),
+      iso2: _.get(params, 'iso2'),
+      method: _.get(req, 'method'),
+      controller: _.get(req, 'options.controller'),
+      action: _.get(req, 'options.action'),
+      statusCode: _.get(params, 'statusCode'),
+      processingTime: _.get(params, 'processingTime'),
+      userId: _.get(req, 'user.id', _.get(params, 'userId')),
+      customerId: _.get(req, 'user.customerId', _.get(params, 'customerId')),
+      accessKey: _.get(params, 'accessKey'),
+      mediaContainerId,
+      created,
+      createdAt: now.toISOString().slice(0,10) + ' ' + now.toISOString().slice(11,23),
+      platform,
+      platformVersion,
+      clientVersion,
+      loginuid,
+      clientId,
+      deviceIdentifier,
+      userAgent: _.get(req, 'headers.user-agent'),
+      responseLength: 0,
+      payload: JSON.stringify(requestParams),
+      response: JSON.stringify(_.get(params, 'responsePayload'))
+    }
+
+    const recordLength = Buffer.byteLength(JSON.stringify(ualPayload), 'utf8')
+
+    if (recordLength > 1024000) {
+      const t0 = performance.now()
+      ualPayload.truncatedResponse = true
+      ualPayload.response = this.trimData(ualPayload.response)
+
+      // reduce request payload if still too large after trimming response
+      if (Buffer.byteLength(JSON.stringify(ualPayload), 'utf8') > 1024000) {
+        ualPayload.payload = this.trimData(ualPayload.payload)
+      }
+
+      // final fallback if still too large - drop response entirely but keep request payload
+      if (Buffer.byteLength(JSON.stringify(ualPayload), 'utf8') > 1024000) {
+        ualPayload.response = null
+      }
+
+      console.warn('AC-UserActionLog-Connector | Took %sms | Record truncated from %s to %s bytes', (performance.now() - t0).toFixed(2), recordLength, Buffer.byteLength(JSON.stringify(ualPayload), 'utf8'))
+    }
+
+    if (process.env.NODE_ENV === 'test') {
+      return ualPayload
+    }
+
+    const awsParams = {
+      DeliveryStreamName: this.firehose.deliveryStreamName,
+      Record: {
+        Data: Buffer.from(JSON.stringify(ualPayload))
+      }
+    }
+    const command = new PutRecordCommand(awsParams)
+    try {
+      const r = await this.firehose.client.send(command)
+      if (this.debug) {
+        const c = await this.firehose.client.config.credentials()
+        console.warn('UAL | FIREHOSE', c?.accessKeyId, c?.expiration, r?.$metadata?.httpStatusCode)
+      }
+    }
+    catch(e) {
+      console.error('AC-UserActionLog-Connector | Failed %j', e?.message)
+    }
+  }
+}
+
+
+module.exports = FirehoseManager

package/package.json

@@ -0,0 +1,35 @@
+{
+  "name": "ac-useractionlog-connector",
+  "author": "Mark Poepping (https://www.admiralcloud.com)",
+  "license": "MIT",
+  "repository": "admiralcloud/ac-useractionlog-connector",
+  "version": "4.0.25",
+  "dependencies": {
+    "@aws-sdk/client-firehose": "^3.1014.0",
+    "@eslint/js": "^10.0.1",
+    "globals": "^17.4.0",
+    "install": "^0.13.0",
+    "lodash": "^4.17.23"
+  },
+  "devDependencies": {
+    "ac-jenkins": "git+ssh://git@ac-jenkins.github.com/admiralcloud/ac-jenkins#v1.2.10",
+    "ac-semantic-release": "^0.4.10",
+    "chai": "^4.5.0",
+    "eslint": "^10.1.0",
+    "mocha": "^11.7.5"
+  },
+  "scripts": {
+    "test": "NODE_ENV=test mocha --reporter spec"
+  },
+  "engines": {
+    "node": ">=20.0.0"
+  },
+  "resolutions": {
+    "mocha/chokidar/braces": "^3.0.3",
+    "mocha/diff": "^8.0.3",
+    "minimatch": "^10.2.1",
+    "fast-xml-parser": "^5.3.8",
+    "mocha/serialize-javascript": "^7.0.3"
+  },
+  "packageManager": "yarn@1.22.22+sha512.a6b2f7906b721bba3d67d4aff083df04dad64c399707841b7acf00f6b133b7ac24255f2652fa22ae3534329dc6180534e98d17432037ff6fd140556e2bb3137e"
+}