@mojaloop/central-ledger 9.2.2 → 15.1.3-snapshot.2

package/README.md

@@ -1,8 +1,9 @@
 # central-ledger
+
 [![Git Commit](https://img.shields.io/github/last-commit/mojaloop/central-ledger.svg?style=flat)](https://github.com/mojaloop/central-ledger/commits/master)
 [![Git Releases](https://img.shields.io/github/release/mojaloop/central-ledger.svg?style=flat)](https://github.com/mojaloop/central-ledger/releases)
 [![Docker pulls](https://img.shields.io/docker/pulls/mojaloop/central-ledger.svg?style=flat)](https://hub.docker.com/r/mojaloop/central-ledger)
-[![CircleCI](https://circleci.com/gh/mojaloop/central-ledger.svg?style=svg)](https://circleci.com/gh/mojaloop/central-ledger)
+[![CircleCI](https://circleci.com/gh/mojaloop/central-ledger.svg?style=svg)](https://app.circleci.com/pipelines/github/mojaloop/central-ledger)
 
 The central ledger is a series of services that facilitate clearing and settlement of transfers between DFSPs, including the following functions:
 
@@ -12,21 +13,29 @@ The central ledger is a series of services that facilitate clearing and settleme
 
 The following documentation represents the services, APIs and endpoints responsible for various ledger functions.
 
-Contents:
+## Contents
 
-- [Deployment](#deployment)
-- [Configuration](#configuration)
-- [API](#api)
-- [Logging](#logging)
-- [Tests](#tests)
+- [central-ledger](#central-ledger)
+  - [Contents](#contents)
+  - [Running Locally](#running-locally)
+  - [Configuration](#configuration)
+    - [Environment variables](#environment-variables)
+  - [API](#api)
+  - [Logging](#logging)
+  - [Tests](#tests)
+    - [Running Integration Tests interactively](#running-integration-tests-interactively)
+  - [Container Scans](#container-scans)
+  - [Automated Releases](#automated-releases)
+    - [Potential problems](#potential-problems)
 
-## Deployment
+## Running Locally
 
-See the [Onboarding guide](Onboarding.md) for running the service locally.
+Please follow the instruction in [Onboarding Document](Onboarding.md) to setup and run the service locally.
 
 ## Configuration
 
 ### Environment variables
+
 The Central Ledger has many options that can be configured through environment variables.
 
 | Environment variable | Description | Example values |
@@ -46,7 +55,6 @@ The Central Ledger has many options that can be configured through environment v
 | CLEDG\_AMOUNT__PRECISION | Numeric value used to determine precision recorded for transfer amounts on this ledger. | 10 |
 | CLEDG\_AMOUNT__SCALE | Numeric value used to determine scale recorded for transfer amounts on this ledger. | 2 |
 
-
 ## API
 
 For endpoint documentation, see the [API documentation](API.md).
@@ -59,7 +67,7 @@ Logs are sent to standard output by default.
 
 ## Tests
 
-Tests include unit, functional, and integration. 
+Tests include unit, functional, and integration.
 
 Running the tests:
 
@@ -69,31 +77,100 @@ Running the tests:
 
 Tests include code coverage via istanbul. See the test/ folder for testing scripts.
 
-### Running Integration Tests interactively-ish
+### Running Integration Tests interactively
 
-If you want to run integration tests in a repetitive manner, you can startup the test containers using `docker-compose`, login to running `central-ledger` container like so:
+If you want to run integration tests in a repetitive manner, you can startup the test containers using `docker-compose` via one of the following methods:
 
-```bash
-docker-compose -f docker-compose.yml -f docker-compose.integration.yml up kafka mysql central-ledger
+- Running locally
 
-#in a new shell
-docker exec -it cl_central-ledger sh
-npm run migrate #first time only
-npm run test:int
-```
+    Start containers required for Integration Tests
+
+    ```bash
+    docker-compose -f docker-compose.yml up -d kafka mysql
+    ```
+
+    Run wait script which will report once all required containers are up and running
+
+    ```bash
+    npm run wait-4-docker
+    ```
+
+    Run the Integration Tests
+
+    ```bash
+    npm run test:int
+    ```
+
+- Running inside docker
+
+    Start containers required for Integration Tests, including a `central-ledger` container which will be used as a proxy shell.
+
+    ```bash
+    docker-compose -f docker-compose.yml -f docker-compose.integration.yml up -d kafka mysql central-ledger
+    ```
+
+    Run the Integration Tests from the `central-ledger` container
+
+    ```bash
+    docker exec -it cl_central-ledger sh
+    export CL_DATABASE_HOST=mysql
+    npm run test:int
+  ```
 
 ## Auditing Dependencies
 
-We use `npm-audit-resolver` along with `npm audit` to check dependencies for vulnerabilities, and keep track of resolved dependencies with an `audit-resolv.json` file.
+We use `npm-audit-resolver` along with `npm audit` to check dependencies for node vulnerabilities, and keep track of resolved dependencies with an `audit-resolve.json` file.
 
 To start a new resolution process, run:
+
 ```bash
 npm run audit:resolve
 ```
 
 You can then check to see if the CI will pass based on the current dependencies with:
+
 ```bash
 npm run audit:check
 ```
 
-And commit the changed `audit-resolv.json` to ensure that CircleCI will build correctly.
+And commit the changed `audit-resolve.json` to ensure that CircleCI will build correctly.
+
+## Container Scans
+
+As part of our CI/CD process, we use anchore-cli to scan our built docker container for vulnerabilities upon release.
+
+If you find your release builds are failing, refer to the [container scanning](https://github.com/mojaloop/ci-config#container-scanning) in our shared Mojaloop CI config repo. There is a good chance you simply need to update the `mojaloop-policy-generator.js` file and re-run the circleci workflow.
+
+For more information on anchore and anchore-cli, refer to:
+- [Anchore CLI](https://github.com/anchore/anchore-cli)
+- [Circle Orb Registry](https://circleci.com/orbs/registry/orb/anchore/anchore-engine)
+
+## Automated Releases
+
+As part of our CI/CD process, we use a combination of CircleCI, standard-version
+npm package and github-release CircleCI orb to automatically trigger our releases
+and image builds. This process essentially mimics a manual tag and release.
+
+On a merge to master, CircleCI is configured to use the mojaloopci github account
+to push the latest generated CHANGELOG and package version number.
+
+Once those changes are pushed, CircleCI will pull the updated master, tag and
+push a release triggering another subsequent build that also publishes a docker image.
+
+### Potential problems
+
+- There is a case where the merge to master workflow will resolve successfully, triggering
+  a release. Then that tagged release workflow subsequently failing due to the image scan,
+  audit check, vulnerability check or other "live" checks.
+
+  This will leave master without an associated published build. Fixes that require
+  a new merge will essentially cause a skip in version number or require a clean up
+  of the master branch to the commit before the CHANGELOG and bump.
+
+  This may be resolved by relying solely on the previous checks of the
+  merge to master workflow to assume that our tagged release is of sound quality.
+  We are still mulling over this solution since catching bugs/vulnerabilities/etc earlier
+  is a boon.
+
+- It is unknown if a race condition might occur with multiple merges with master in
+  quick succession, but this is a suspected edge case.

package/audit-resolve.json

@@ -1 +1,136 @@
-
+{
+  "decisions": {
+    "1075703|@mojaloop/central-services-shared>@mojaloop/event-sdk>grpc>protobufjs": {
+      "decision": "ignore",
+      "madeAt": 1657029291345,
+      "expiresAt": 1659621287320
+    },
+    "1075703|@mojaloop/event-sdk>grpc>protobufjs": {
+      "decision": "ignore",
+      "madeAt": 1657029291345,
+      "expiresAt": 1659621287320
+    },
+    "1075704|@mojaloop/central-services-shared>@mojaloop/event-sdk>grpc>protobufjs": {
+      "decision": "ignore",
+      "madeAt": 1657029292596,
+      "expiresAt": 1659621287320
+    },
+    "1075704|@mojaloop/event-sdk>grpc>protobufjs": {
+      "decision": "ignore",
+      "madeAt": 1657029292596,
+      "expiresAt": 1659621287320
+    },
+    "1070030|@mojaloop/central-services-shared>@mojaloop/event-sdk>grpc>protobufjs>shins>markdown-it": {
+      "decision": "ignore",
+      "madeAt": 1657029293742,
+      "expiresAt": 1659621287320
+    },
+    "1070030|widdershins>markdown-it": {
+      "decision": "ignore",
+      "madeAt": 1657547585781,
+      "expiresAt": 1660139581210
+    },
+    "1068155|@mojaloop/central-services-shared>@mojaloop/event-sdk>grpc>protobufjs>shins>markdown-it>sanitize-html": {
+      "decision": "ignore",
+      "madeAt": 1657029294926,
+      "expiresAt": 1659621287320
+    },
+    "1070260|@mojaloop/central-services-shared>@mojaloop/event-sdk>grpc>protobufjs>shins>markdown-it>sanitize-html": {
+      "decision": "ignore",
+      "madeAt": 1657029296035,
+      "expiresAt": 1659621287320
+    },
+    "1070412|ejs": {
+      "decision": "ignore",
+      "madeAt": 1657029299037,
+      "expiresAt": 1659621287320
+    },
+    "1068386|hapi-auth-basic>hapi": {
+      "decision": "ignore",
+      "madeAt": 1657029300211,
+      "expiresAt": 1659621287320
+    },
+    "1068399|hapi-auth-basic>hapi>ammo": {
+      "decision": "ignore",
+      "madeAt": 1657029301421,
+      "expiresAt": 1659621287320
+    },
+    "1068389|hapi-auth-basic>hapi>ammo>subtext": {
+      "decision": "ignore",
+      "madeAt": 1657029302848,
+      "expiresAt": 1659621287320
+    },
+    "1068390|hapi-auth-basic>hapi>ammo>subtext": {
+      "decision": "ignore",
+      "madeAt": 1657029304128,
+      "expiresAt": 1659621287320
+    },
+    "1067553|swagger2openapi>better-ajv-errors>jsonpointer": {
+      "decision": "ignore",
+      "madeAt": 1657029305419,
+      "expiresAt": 1659621287320
+    },
+    "1067946|swagger2openapi>better-ajv-errors>jsonpointer>oas-validator>ajv": {
+      "decision": "ignore",
+      "madeAt": 1657029307042,
+      "expiresAt": 1659621287320
+    },
+    "1068310|widdershins>markdown-it>yargs>yargs-parser": {
+      "decision": "ignore",
+      "madeAt": 1657029308195,
+      "expiresAt": 1659621287320
+    },
+    "1080969|@mojaloop/central-services-shared>@mojaloop/event-sdk>grpc>protobufjs>moment": {
+      "decision": "ignore",
+      "madeAt": 1657547584804,
+      "expiresAt": 1660139581210
+    },
+    "1080969|@mojaloop/event-sdk>grpc>protobufjs>moment": {
+      "decision": "ignore",
+      "madeAt": 1657547584804,
+      "expiresAt": 1660139581210
+    },
+    "1070030|@mojaloop/central-services-shared>@mojaloop/event-sdk>grpc>protobufjs>moment>shins>markdown-it": {
+      "decision": "ignore",
+      "madeAt": 1657547585781,
+      "expiresAt": 1660139581210
+    },
+    "1068155|@mojaloop/central-services-shared>@mojaloop/event-sdk>grpc>protobufjs>moment>shins>markdown-it>sanitize-html": {
+      "decision": "ignore",
+      "madeAt": 1657547586612,
+      "expiresAt": 1660139581210
+    },
+    "1070260|@mojaloop/central-services-shared>@mojaloop/event-sdk>grpc>protobufjs>moment>shins>markdown-it>sanitize-html": {
+      "decision": "ignore",
+      "madeAt": 1657547587381,
+      "expiresAt": 1660139581210
+    },
+    "1070030|@mojaloop/central-services-shared>@mojaloop/event-sdk>grpc>protobufjs>moment>widdershins>markdown-it": {
+      "decision": "ignore",
+      "madeAt": 1657548277077,
+      "expiresAt": 1660140273753
+    },
+    "1070030|shins>markdown-it": {
+      "decision": "ignore",
+      "madeAt": 1657548277077,
+      "expiresAt": 1660140273753
+    },
+    "1068310|@mojaloop/central-services-shared>@mojaloop/event-sdk>grpc>protobufjs>moment>widdershins>markdown-it>yargs>yargs-parser": {
+      "decision": "ignore",
+      "madeAt": 1657548277911,
+      "expiresAt": 1660140273753
+    },
+    "1068155|shins>markdown-it>sanitize-html": {
+      "decision": "ignore",
+      "madeAt": 1657548278663,
+      "expiresAt": 1660140273753
+    },
+    "1070260|shins>markdown-it>sanitize-html": {
+      "decision": "ignore",
+      "madeAt": 1657548279466,
+      "expiresAt": 1660140273753
+    }
+  },
+  "rules": {},
+  "version": 1
+}

package/config/default.json

@@ -26,15 +26,9 @@
     "PRECISION": 18,
     "SCALE": 4
   },
-  "SIDECAR": {
-    "DISABLED": true,
-    "HOST": "localhost",
-    "PORT": 5678,
-    "CONNECT_TIMEOUT": 45000,
-    "RECONNECT_INTERVAL": 5000
-  },
   "MONGODB": {
-    "DISABLED": false,
+    "DISABLED": true,
+    "DEBUG": false,
     "URI": "mongodb://localhost:27017/mlos"
   },
   "ERROR_HANDLING": {
@@ -46,11 +40,6 @@
     "API": {
       "DISABLED": false
     },
-    "CRON": {
-      "DISABLED": false,
-      "TIMEXP": "*/10 * * * * *",
-      "TIMEZONE": "UTC"
-    },
     "TIMEOUT": {
       "DISABLED": false,
       "TIMEXP": "*/15 * * * * *",
@@ -83,11 +72,13 @@
     ]
   },
   "INTERNAL_TRANSFER_VALIDITY_SECONDS": "432000",
+  "ENABLE_ON_US_TRANSFERS": false,
   "CACHE": {
     "CACHE_ENABLED": false,
     "MAX_BYTE_SIZE": 10000000,
     "EXPIRES_IN_MS": 1000
   },
+  "API_DOC_ENDPOINTS_ENABLED": true,
   "KAFKA": {
     "TOPIC_TEMPLATES": {
       "PARTICIPANT_TOPIC_TEMPLATE": {
@@ -117,7 +108,8 @@
               "client.id": "cl-con-bulk-prepare",
               "group.id": "cl-group-bulk-prepare",
               "metadata.broker.list": "localhost:9092",
-              "socket.keepalive.enable": true
+              "socket.keepalive.enable": true,
+              "allow.auto.create.topics": true
             },
             "topicConf": {
               "auto.offset.reset": "earliest"
@@ -140,7 +132,8 @@
               "client.id": "cl-con-bulk-processing",
               "group.id": "cl-group-bulk-processing",
               "metadata.broker.list": "localhost:9092",
-              "socket.keepalive.enable": true
+              "socket.keepalive.enable": true,
+              "allow.auto.create.topics": true
             },
             "topicConf": {
               "auto.offset.reset": "earliest"
@@ -163,7 +156,32 @@
               "client.id": "cl-con-bulk-fulfil",
               "group.id": "cl-group-bulk-fulfil",
               "metadata.broker.list": "localhost:9092",
-              "socket.keepalive.enable": true
+              "socket.keepalive.enable": true,
+              "allow.auto.create.topics": true
+            },
+            "topicConf": {
+              "auto.offset.reset": "earliest"
+            }
+          }
+        },
+        "GET": {
+          "config": {
+            "options": {
+              "mode": 2,
+              "batchSize": 1,
+              "pollFrequency": 10,
+              "recursiveTimeout": 100,
+              "messageCharset": "utf8",
+              "messageAsJSON": true,
+              "sync": true,
+              "consumeTimeout": 1000
+            },
+            "rdkafkaConf": {
+              "client.id": "cl-con-bulk-get",
+              "group.id": "cl-group-bulk-get",
+              "metadata.broker.list": "localhost:9092",
+              "socket.keepalive.enable": true,
+              "allow.auto.create.topics": true
             },
             "topicConf": {
               "auto.offset.reset": "earliest"
@@ -188,7 +206,8 @@
               "client.id": "cl-con-transfer-prepare",
               "group.id": "cl-group-transfer-prepare",
               "metadata.broker.list": "localhost:9092",
-              "socket.keepalive.enable": true
+              "socket.keepalive.enable": true,
+              "allow.auto.create.topics": true
             },
             "topicConf": {
               "auto.offset.reset": "earliest"
@@ -211,7 +230,8 @@
               "client.id": "cl-con-transfer-get",
               "group.id": "cl-group-transfer-get",
               "metadata.broker.list": "localhost:9092",
-              "socket.keepalive.enable": true
+              "socket.keepalive.enable": true,
+              "allow.auto.create.topics": true
             },
             "topicConf": {
               "auto.offset.reset": "earliest"
@@ -234,7 +254,8 @@
               "client.id": "cl-con-transfer-fulfil",
               "group.id": "cl-group-transfer-fulfil",
               "metadata.broker.list": "localhost:9092",
-              "socket.keepalive.enable": true
+              "socket.keepalive.enable": true,
+              "allow.auto.create.topics": true
             },
             "topicConf": {
               "auto.offset.reset": "earliest"
@@ -257,7 +278,8 @@
               "client.id": "cl-con-transfer-position",
               "group.id": "cl-group-transfer-position",
               "metadata.broker.list": "localhost:9092",
-              "socket.keepalive.enable": true
+              "socket.keepalive.enable": true,
+              "allow.auto.create.topics": true
             },
             "topicConf": {
               "auto.offset.reset": "earliest"
@@ -282,7 +304,34 @@
               "client.id": "cl-con-transfer-admin",
               "group.id": "cl-group-transfer-admin",
               "metadata.broker.list": "localhost:9092",
-              "socket.keepalive.enable": true
+              "socket.keepalive.enable": true,
+              "allow.auto.create.topics": true
+            },
+            "topicConf": {
+              "auto.offset.reset": "earliest"
+            }
+          }
+        }
+      },
+      "NOTIFICATION": {
+        "EVENT": {
+          "config": {
+            "options": {
+              "mode": 2,
+              "batchSize": 1,
+              "pollFrequency": 10,
+              "recursiveTimeout": 100,
+              "messageCharset": "utf8",
+              "messageAsJSON": true,
+              "sync": true,
+              "consumeTimeout": 1000
+            },
+            "rdkafkaConf": {
+              "client.id": "TESTONLY",
+              "group.id": "TESTONLY",
+              "metadata.broker.list": "localhost:9092",
+              "socket.keepalive.enable": true,
+              "allow.auto.create.topics": true
             },
             "topicConf": {
               "auto.offset.reset": "earliest"
@@ -307,7 +356,8 @@
               "queue.buffering.max.messages": 10000000
             },
             "topicConf": {
-              "request.required.acks": "all"
+              "request.required.acks": "all",
+              "partitioner": "murmur2_random"
             }
           }
         }
@@ -327,7 +377,8 @@
               "queue.buffering.max.messages": 10000000
             },
             "topicConf": {
-              "request.required.acks": "all"
+              "request.required.acks": "all",
+              "partitioner": "murmur2_random"
             }
           }
         },
@@ -345,7 +396,8 @@
               "queue.buffering.max.messages": 10000000
             },
             "topicConf": {
-              "request.required.acks": "all"
+              "request.required.acks": "all",
+              "partitioner": "murmur2_random"
             }
           }
         },
@@ -363,7 +415,8 @@
               "queue.buffering.max.messages": 10000000
             },
             "topicConf": {
-              "request.required.acks": "all"
+              "request.required.acks": "all",
+              "partitioner": "murmur2_random"
             }
           }
         }
@@ -383,7 +436,8 @@
               "queue.buffering.max.messages": 10000000
             },
             "topicConf": {
-              "request.required.acks": "all"
+              "request.required.acks": "all",
+              "partitioner": "murmur2_random"
             }
           }
         }
@@ -403,7 +457,8 @@
               "queue.buffering.max.messages": 10000000
             },
             "topicConf": {
-              "request.required.acks": "all"
+              "request.required.acks": "all",
+              "partitioner": "murmur2_random"
             }
           }
         }