@into-cps-association/libms 0.3.1 → 0.4.3

package/DEVELOPER.md

@@ -10,32 +10,49 @@ file. Please see [README](./README.md) for this information.
 ```bash
 yarn install    # Install dependencies for the microservice
 yarn syntax     # Analyze code for errors and style issues
+yarn format     #format .ts[x] and .js[x] files with prettier
 yarn graph      # Generate dependency graphs in the code
 yarn build      # Compile ES6 to ES5 and copy JS files to build/ directory
-yarn test:unit  # Run all tests
-yarn test:e2e   # Run end-to-end tests
+yarn test:unit  # Run unit tests
 yarn test:int   # Run integration tests
-yarn test:all   # Run unit tests
+yarn test:e2e   # Run end-to-end tests
+yarn test:http  # Run tests for HTTP file service provided by cloudcmd
+yarn test:all   # Run all tests except http tests
 yarn start -h   # List of all the CLI commands
 yarn start      # Start the application
+yarn start:pm2  # Start the application with process manager 2
+yarn stop:pm2   # Stop the application managed by process manager 2
 yarn clean      # Deletes "build", "coverage", "dist" and other temp files
 ```
 
+**NOTE:** The integration, end-to-end and http tests require a valid
+`.env` file. Here is a sample file.
+
+```ini
+PORT='4001'
+MODE='local'
+LOCAL_PATH ='/Users/<Username>/DTaaS/files'
+LOG_LEVEL='debug'
+APOLLO_PATH='/lib'
+GRAPHQL_PLAYGROUND='true'
+```
+
 ## :package: :ship: NPM package
 
 ### Github Package Registry
 
 The Github actions workflow of
-[lib microservice](../../.github/workflows/lib-ms.yml) publishes the __libms__
-into [public packages](https://github.com/orgs/INTO-CPS-Association/packages).
+[lib microservice](../../.github/workflows/lib-ms.yml) publishes the **libms**
+into
+[packages](https://github.com/orgs/INTO-CPS-Association/packages?repo_name=DTaaS).
 
 ### Verdaccio - Local Package Registry
 
 Use the instructions in
 [publish npm package](../../docs/developer/npm-packages.md) for help
-with publishing __libms npm package__ in local computer.
+with publishing **libms npm package** in local computer.
 
-Application of the advice given on that page for __libms__ will require
+Application of the advice given on that page for **libms** will require
 running the following commands.
 
 ### Publish

package/README.md

@@ -1,28 +1,10 @@
 # Overview
 
 The **lib microservice** is a simplified file manager providing graphQL API.
-It has three features:
+It has two features:
 
 * provide a listing of directory contents.
 * transfer a file to user.
-* Source files can either come from local file system or from a gitlab instance.
-
-## Gitlab setup
-
-For this microserivce to be functional,
-a certain directory or gitlab project structure is expected.
-The microservice expects that the gitlab consisting of one group, dtaas (by convention),
-and within that group, all of the projects be located,
-**user1**, **user2**, ... , as well as a **commons** projects.
-Each project corresponds to files of one user.
-A sample file structure can be seen in [gitlab dtaas group](https://gitlab.com/dtaas).
-You can visit the gitlab documentation on
-[groups](https://docs.gitlab.com/ee/user/group/)
-for help on the management of gitlab groups.
-
-You can clone the git repositories from
-the [gitlab dtaas](https://gitlab.com/dtaas) group
-to get a sample file system structure for the lib microservice.
 
 ## :arrow_down: Install
 
@@ -51,9 +33,6 @@ The template configuration file is:
 PORT='4001'
 MODE='local' or 'gitlab'
 LOCAL_PATH ='/Users/<Username>/DTaaS/files'
-GITLAB_GROUP ='dtaas'
-GITLAB_URL='https://gitlab.com/api/graphql'
-TOKEN='123-sample-token'
 LOG_LEVEL='debug'
 APOLLO_PATH='/lib' or ''
 GRAPHQL_PLAYGROUND='false' or 'true'
@@ -61,34 +40,34 @@ GRAPHQL_PLAYGROUND='false' or 'true'
 
 The `LOCAL_PATH` variable is the absolute filepath to the
 location of the local directory which will be served to users
-by the Library microservice.
-
-The `GITLAB_URL`, `GITLAB_GROUP` and `TOKEN` are only relevant for `gitlab` mode.
-The `TOKEN` should be set to your GitLab Group access API token.
-For more information on how to create and use your access token,
-[gitlab page](https://docs.gitlab.com/ee/user/group/settings/group_access_tokens.html).
-
-Once you've generated a token, copy it and replace
-the value of `TOKEN` with your token for the gitlab group,
+by the Library microservice. This
+[sample configuration file](./config/.env.default) can be used.
 
 Replace the default values the appropriate values for your setup.
 
-**NOTE**:
-
-1. When \__MODE=local_, only _LOCAL_PATH_ is used.
-   Other environment variables are unused.
-1. When _MODE=gitlab_, _GITLAB_URL, TOKEN_,
-   and _GITLAB_GROUP_ are used; _LOCAL_PATH_ is unused.
-
 ## :rocket: Use
 
 Display help.
 
 ```bash
-libms -h
+$libms -h
+Usage: libms [options]
+
+The lib microservice is a file server. It supports file transfer
+over GraphQL and HTTP protocols.
+
+Options:
+  -c, --config <file>  provide the config file (default .env)
+  -H, --http <file>    enable the HTTP server with the specified config
+  -h, --help           display help for libms
 ```
 
-The config is saved `.env` file by convention. The **libms** looks for
+Both the options are not mandatory.
+
+### Configuration file
+
+The config is saved `.env` file by convention. If `-c` is not specified
+The **libms** looks for
 `.env` file in the working directory from which it is run.
 If you want to run **libms** without explicitly specifying the configuration
 file, run
@@ -105,15 +84,35 @@ libms --config FILE-PATH
 ```
 
 If the environment file is named something other than `.env`,
-for example as `.env.development`, you can run
+for example as `config/.env.default`, you can run
 
 ```sh
-libms -c ".env.development"
+libms -c "config/.env.default"
 ```
 
 You can press `Ctl+C` to halt the application.
 
-The microservice is available at: 'localhost:PORT/lib'
+### Protocol Support
+
+The **libms** supports GraphQL protocol by default.
+It is possible to enable the HTTP protocol by setting
+the `-H` option.
+
+To run **libms** with a custom config for HTTP protocol, use
+
+```bash
+libms -H FILE-PATH
+libms --http FILE-PATH
+```
+
+A sample configuration is [available](./config/http.json).
+
+### Accessible URLs
+
+The microservice is available at:
+
+**GraphQL protocol**: 'localhost:PORT/lib'
+**HTTP protocol**: 'localhost:PORT/lib/files'
 
 <!-- markdownlint-disable-next-line MD013 -->
 The [API](https://into-cps-association.github.io/DTaaS/development/user/servers/lib/LIB-MS.html) page shows sample queries and responses.

package/config/.env.default

@@ -0,0 +1,6 @@
+PORT='4001'
+MODE='local'
+LOCAL_PATH ='/Users/<Username>/DTaaS/files'
+LOG_LEVEL='debug'
+APOLLO_PATH='/lib'
+GRAPHQL_PLAYGROUND='true'

package/config/http.json

@@ -0,0 +1,30 @@
+{
+  "name": "DTaaS Fileserver",
+  "auth": false,
+  "editor": "edward",
+  "packer": "zip",
+  "diff": true,
+  "zip": true,
+  "buffer": true,
+  "dirStorage": true,
+  "online": false,
+  "open": false,
+  "oneFilePanel": true,
+  "keysPanel": false,
+  "prefix": "/lib/files",
+  "confirmCopy": true,
+  "confirmMove": true,
+  "showConfig": false,
+  "showFileName": true,
+  "contact": false,
+  "configDialog": false,
+  "console": false,
+  "terminal": false,
+  "vim": false,
+  "columns": "name-size-date-owner-mode",
+  "export": false,
+  "import": false,
+  "dropbox": false,
+  "dropboxToken": "",
+  "log": true
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@into-cps-association/libms",
-  "version": "0.3.1",
+  "version": "0.4.3",
   "description": "microservices that handles request by fetching and returning the file-names and folders of given directory",
   "author": "phillip.boe.jensen@gmail.com",
   "contributors": [
@@ -16,53 +16,63 @@
     "format": "prettier --ignore-path ../.gitignore --write \"**/*.{ts,tsx,css,scss}\"",
     "graph": "npx madge --image src.svg src && npx madge --image test.svg test",
     "start": "node dist/src/main.js",
+    "start:pm2": "pm2 start pm2.config.js",
+    "stop:pm2": "pm2 delete libms",
     "syntax": "npx eslint . --fix",
     "pretest": "npx shx cp test/data/user2/tools/README.md ../../files/user2/tools/README.md",
-    "test:all": "jest --coverage",
-    "test:e2e": "npx shx cp test/data/user2/tools/README.md ../../files/user2/tools/ && npx cross-env TEST_PATH=test/data jest --config ./test/jest-e2e.json --coverage && npx rimraf ../../files/user2/tools/README.md",
-    "test:int": "npx shx cp test/data/user2/tools/README.md ../../files/user2/tools/ && npx cross-env TEST_PATH=test/data jest ../test/integration --coverage && npx rimraf ../../files/user2/tools/README.md",
-    "test:unit": "npx shx cp test/data/user2/tools/README.md ../../files/user2/tools/ && npx cross-env TEST_PATH=test/data jest ../test/unit --coverage && npx rimraf ../../files/user2/tools/README.md"
+    "posttest": "npx rimraf ../../files/user2/tools/README.md",
+    "test:all": "npx cross-env LOCAL_PATH=test/data jest --testPathIgnorePatterns=cloudcmd --coverage",
+    "test:e2e": "npx cross-env LOCAL_PATH=test/data jest --config ./test/jest-e2e.json --coverage",
+    "test:http": "yarn build && pm2 start -f --name libms-test dist/src/main.js -- -c .env -H ./config/http.json && jest test/cloudcmd --coverage --coverageThreshold=\"{}\" && pm2 delete libms-test",
+    "test:http-nocov": "yarn build && pm2 start -f --name libms-test dist/src/main.js -- -c .env -H ./config/http.json && jest test/cloudcmd  --coverage=false && pm2 delete libms-test",
+    "test:http-github": "yarn build && yarn start:pm2 && jest test/cloudcmd --coverage --coverageThreshold=\"{}\" && pm2 delete libms",
+    "test:int": "npx cross-env LOCAL_PATH=test/data jest ../test/integration --coverage",
+    "test:nocov": "yarn test:http-nocov && npx cross-env LOCAL_PATH=test/data jest --testPathIgnorePatterns=cloudcmd --coverage=false",
+    "test:unit": "npx cross-env LOCAL_PATH=test/data jest ../test/unit --coverage"
   },
   "bin": "./dist/src/main.js",
   "dependencies": {
-    "@apollo/client": "^3.8.4",
-    "@apollo/server": "^4.9.3",
-    "@nestjs/apollo": "^12.0.9",
-    "@nestjs/common": "^10.2.6",
-    "@nestjs/config": "^3.1.1",
-    "@nestjs/core": "^10.2.6",
-    "@nestjs/graphql": "^12.0.9",
-    "@nestjs/platform-express": "^10.2.6",
+    "@apollo/client": "^3.8.9",
+    "@apollo/server": "^4.10.0",
+    "@nestjs/apollo": "^12.0.11",
+    "@nestjs/common": "^10.3.7",
+    "@nestjs/config": "^3.2.0",
+    "@nestjs/core": "^10.3.3",
+    "@nestjs/graphql": "^12.0.11",
+    "@nestjs/platform-express": "^10.3.7",
     "axios": "^1.5.1",
+    "cloudcmd": "^16.17.7",
     "commander": "^11.1.0",
     "dotenv": "^16.3.1",
-    "express": "^4.18.2",
     "graphql": "^16.8.1",
     "mock-fs": "^5.2.0",
-    "reflect-metadata": "^0.1.13",
+    "reflect-metadata": "^0.2.1",
     "rxjs": "^7.8.1",
+    "socket.io": "^4.7.2",
     "type-graphql": "^2.0.0-beta.3"
   },
   "devDependencies": {
-    "@nestjs/cli": "^10.1.18",
-    "@nestjs/schematics": "^10.0.2",
-    "@nestjs/testing": "^10.2.6",
-    "@types/express": "^4.17.18",
-    "@types/jest": "^29.5.5",
-    "@types/node": "20.7.2",
-    "@types/supertest": "^2.0.13",
-    "@typescript-eslint/eslint-plugin": "^6.7.3",
-    "@typescript-eslint/parser": "^6.7.3",
-    "eslint": "^8.50.0",
+    "@nestjs/cli": "^10.3.2",
+    "@nestjs/schematics": "^10.1.1",
+    "@nestjs/testing": "^10.3.3",
+    "@types/express": "^4.17.21",
+    "@types/jest": "^29.5.12",
+    "@types/node": "20.12.5",
+    "@types/supertest": "^6.0.2",
+    "@typescript-eslint/eslint-plugin": "^6.19.0",
+    "@typescript-eslint/parser": "^6.19.0",
+    "cross-fetch": "^4.0.0",
+    "eslint": "^8.56.0",
     "eslint-config-airbnb-base": "^15.0.0",
-    "eslint-config-prettier": "^9.0.0",
-    "eslint-plugin-import": "^2.28.1",
-    "eslint-plugin-jest": "^27.4.0",
-    "eslint-plugin-prettier": "^5.0.0",
+    "eslint-config-prettier": "^9.1.0",
+    "eslint-plugin-import": "^2.29.1",
+    "eslint-plugin-jest": "^28.2.0",
+    "eslint-plugin-prettier": "^5.1.3",
     "graphql-scalars": "^1.22.2",
     "jest": "29.7.0",
-    "prettier": "^3.0.3",
-    "supertest": "^6.3.3",
+    "prettier": "^3.2.2",
+    "react": "^18.2.0",
+    "supertest": "^6.3.4",
     "ts-jest": "^29.1.1",
     "ts-node": "^10.9.1",
     "tsconfig-paths": "4.2.0",

package/pm2.config.js

@@ -0,0 +1,10 @@
+module.exports = {
+  apps : [
+      {
+        name: "libms",
+        script: "./dist/src/main.js",
+        watch: false,
+        args: "-H ./config/http.json"
+      }
+  ]
+}

package/src/bootstrap.ts

@@ -2,9 +2,11 @@ import { NestFactory } from '@nestjs/core';
 import { ConfigService } from '@nestjs/config';
 import * as dotenv from 'dotenv';
 import AppModule from './app.module';
+import cloudCMD from './cloudcmd/cloudcmd';
 
 type BootstrapOptions = {
   config?: string;
+  httpServer?: string;
   runHelp?: CallableFunction;
 };
 
@@ -22,8 +24,14 @@ export default async function bootstrap(options?: BootstrapOptions) {
       process.exit(1);
     }
   }
+
   const app = await NestFactory.create(AppModule);
   const configService = app.get(ConfigService);
   const port = configService.get<number>('PORT');
+
+  if (options.httpServer) {
+    cloudCMD(app, options.httpServer, configService.get<string>('LOCAL_PATH'));
+  }
+
   await app.listen(port);
 }

package/src/cloudcmd/cloudcmd.ts

@@ -0,0 +1,32 @@
+import { INestApplication } from '@nestjs/common';
+import { Server } from 'socket.io';
+import * as cloudcmd from 'cloudcmd';
+import { join } from 'path';
+
+const runCloudCMD = (
+  app: INestApplication,
+  optionsPath: string,
+  filesPath: string,
+) => {
+  const { createConfigManager } = cloudcmd;
+  const configManager = createConfigManager({
+    configPath: join(process.cwd(), optionsPath),
+  });
+
+  configManager('root', filesPath);
+
+  const server = app.getHttpServer();
+
+  const socket = new Server(server, {
+    path: `${configManager('prefix')}/socket.io`,
+  });
+  app.use(
+    configManager('prefix'),
+    cloudcmd({
+      configManager,
+      socket,
+    }),
+  );
+};
+
+export default runCloudCMD;

package/src/files/files.module.ts

@@ -1,15 +1,9 @@
 import { Module } from '@nestjs/common';
 import FilesResolver from './resolvers/files.resolver';
-import GitlabFilesService from './services/gitlab-files.service';
 import FilesServiceFactory from './services/files-service.factory';
 import LocalFilesService from './services/local-files.service';
 
 @Module({
-  providers: [
-    FilesResolver,
-    LocalFilesService,
-    GitlabFilesService,
-    FilesServiceFactory,
-  ],
+  providers: [FilesResolver, LocalFilesService, FilesServiceFactory],
 })
 export default class FilesModule {}

package/src/files/services/files-service.factory.ts

@@ -1,7 +1,6 @@
 import { Injectable, Inject } from '@nestjs/common';
 import { ConfigService } from '@nestjs/config';
 import { IFilesService } from '../interfaces/files.service.interface';
-import GitlabFilesService from './gitlab-files.service';
 import LocalFilesService from './local-files.service';
 
 @Injectable()
@@ -9,7 +8,6 @@ export default class FilesServiceFactory {
   /* eslint-disable no-useless-constructor, no-empty-function */
   constructor(
     private configService: ConfigService,
-    @Inject(GitlabFilesService) private gitlabFilesService: GitlabFilesService,
     @Inject(LocalFilesService) private localFilesService: LocalFilesService,
   ) {}
   /* eslint-enable no-useless-constructor, no-empty-function */
@@ -19,9 +17,6 @@ export default class FilesServiceFactory {
     if (mode === 'local') {
       return this.localFilesService;
     }
-    if (mode === 'gitlab') {
-      return this.gitlabFilesService;
-    }
     throw new Error(`Invalid MODE: ${mode}`);
   }
 }

package/src/files/services/local-files.service.ts

@@ -34,11 +34,11 @@ export default class LocalFilesService implements IFilesService {
 
   async readFile(path: string): Promise<Project> {
     const dataPath = this.configService.get('LOCAL_PATH');
-    const fullpath = join(dataPath, path);
+    const fullPath = join(dataPath, path);
 
     try {
       const content = await (
-        await fs.promises.readFile(fullpath, 'utf8')
+        await fs.promises.readFile(fullPath, 'utf8')
       ).trim();
 
       const name = path.split('/').pop(); // extract file name from the path

package/src/main.ts

@@ -4,15 +4,20 @@ import bootstrap from './bootstrap';
 
 type ProgramOptions = {
   config?: string;
+  http?: string;
 };
 
 const program = new Command();
 
 program
   .description(
-    'The lib microservice is responsible for handling and serving the contents of library assets of the DTaaS platform. It provides API endpoints for clients to query, and fetch these assets.',
+    'The lib microservice is a file server. It supports file transfer over GraphQL and HTTP protocols.',
+  )
+  .option('-c, --config <file>', 'provide the config file (default .env)')
+  .option(
+    '-H, --http <file>',
+    'enable the HTTP server with the specified config',
   )
-  .option('-c, --config <path>', 'set the config path (default .env)')
   .helpOption('-h, --help', 'display help for libms')
   .showHelpAfterError();
 
@@ -20,8 +25,8 @@ program.parse(process.argv);
 
 const options: ProgramOptions = program.opts();
 
-if (options.config) {
-  bootstrap({ config: options.config, runHelp: () => program.help() });
-} else {
-  bootstrap({ runHelp: () => program.help() });
-}
+bootstrap({
+  config: options.config,
+  httpServer: options.http,
+  runHelp: () => program.help(),
+});

package/test/cloudcmd/cloudcmd.spec.ts

@@ -0,0 +1,52 @@
+import axios from 'axios';
+import { unlink, writeFile } from 'fs/promises';
+import { config } from 'dotenv';
+
+describe('cloudcmd test for the application', () => {
+  beforeAll(async () => {
+    config();
+
+    await writeFile(`${process.env.LOCAL_PATH}/test.txt`, 'content12345');
+    // eslint-disable-next-line no-promise-executor-return
+    await new Promise((resolve) => setTimeout(resolve, 8000));
+  }, 10000);
+
+  afterAll(async () => {
+    await unlink(`${process.env.LOCAL_PATH}/test.txt`);
+    await unlink(`${process.env.LOCAL_PATH}/uploadTest.txt`);
+  }, 10000);
+
+  it('should return the correct directory that is set as root', async () => {
+    const response = await axios.get(
+      `http://localhost:${process.env.PORT}${process.env.APOLLO_PATH}/files/api/v1/fs`,
+      {
+        responseType: 'json',
+      },
+    );
+    /* eslint-disable no-console */
+    console.log(response.data);
+    /* eslint-enable no-console */
+    expect(response.data.path).toEqual('/');
+    expect(response.data.files[0].name).toEqual('common');
+    expect(response.data.files[1].name).toEqual('user1');
+    expect(response.data.files[2].name).toEqual('user2');
+  }, 10000);
+
+  it('should return the content of a file that is uplaoded to cloudcmd ', async () => {
+    const response = await axios.get(
+      `http://localhost:${process.env.PORT}${process.env.APOLLO_PATH}/files/api/v1/fs/test.txt`,
+    );
+
+    expect(response.data).toEqual('content12345');
+  }, 10000);
+
+  it('should upload a file to cloudcmd', async () => {
+    const response = await axios.put(
+      `http://localhost:${process.env.PORT}${process.env.APOLLO_PATH}/files/api/v1/fs/uploadTest.txt`,
+      'some content',
+      { responseType: 'text' },
+    );
+
+    expect(response.data).toEqual('save: ok("uploadTest.txt")');
+  }, 10000);
+});