@gokiteam/goki-dev 0.2.0

package/guidelines/HttpApiImplementationGuideline.md

@@ -0,0 +1,137 @@
+# HTTP API Implementation Guideline
+
+We don't follow REST. We only use `POST` method in the APIs. The document ID and other fields should be sent in the request's **body**.
+
+## Endpoints
+
+```plain
+create 		    POST 	/<version>/<collection>/create
+update 		    POST 	/<version>/<collection>/update
+details 	    POST 	/<version>/<collection>/details
+list 		    POST 	/<version>/<collection>/list
+delete 		    POST 	/<version>/<collection>/delete
+batch create    POST 	/<version>/<collection>/batchCreate
+batch update    POST 	/<version>/<collection>/batchUpdate
+batch details   POST 	/<version>/<collection>/batchDetails
+batch delete    POST 	/<version>/<collection>/batchDelete
+batch action    POST	/<version>/<collection>/batch<action>
+action 		    POST 	/<version>/<collection>/<action>
+```
+
+## Request
+### API query param convention
+The following table shows the common parameters to be used in API request.
+
+| **Field** | **Type**                                            | **Description**                                                                                                                                                 |
+|-----------|-----------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| id        | Any                                                 | the ID of the resource                                                                                                                                          |
+| ids       | Any[]                                               | the list of IDs of the resource                                                                                                                                 |
+| query     | String                                              | will be globally searched in the resource                                                                                                                       |
+| searchOn  | String\[\]                                          | the fields on which the "query" can be searched on (if the "query" is present)                                                                                  |
+| filter    | Object                                              | an object with predefined keys (filters)                                                                                                                        |
+| page      | Object                                              | an object with two (limit, offset) keys that are used for pagination                                                                                            |
+| sort      | Object\[{ field: String, order: 'asc' \| 'desc' }\] | an ordered list of keys that indicates how the list should be sorted                                                                                            |
+| extend    | String\[\]                                          | a list of related objects which should be added to each item in the list. Singular or Plural form depends on the response which is an array or a single object. |
+| join      | String\[\]                                          | a list of un-related objects which should be added to each item in the list                                                                                     |
+| mask      | Object                                              | specifies what fields should be included in the response documents. It's an **object** with the singular name of the documents.                                 |
+| notFilter | Object                                              | an object with predefined keys (not filters)                                                                                                                    |
+
+## Response
+It should always be **an object** with the name of the resources it returns. If the resource is an array, it's plural and if it's a single object, it's singular.
+In the listing APIs, a field named `total` can be presented which indicates the count of the main resource in the DB.
+### Structure
+If the API returns a single resource
+
+```json
+{
+  "success": true,
+  "status": 200,
+  "message": "The request succeeded.",
+  "data": { "[RESOURCE]": RESOURCE }
+}
+```
+
+If the API returns an array of resources
+
+```json
+{
+  "success": true,
+  "status": 200,
+  "message": "The request succeeded.",
+  "data": {
+    "[RESOURCE]s": RESOURCE[],
+    "total": NUMBER
+  }
+}
+```
+
+If the API returns a resource and extends other resources
+
+```json
+{
+  "success": true,
+  "status": 200,
+  "message": "The request succeeded.",
+  "data": {
+    "[RESOURCE]": RESOURCE,
+    "[EXTENDED_RESOURCE]": EXTENDED_RESOURCE,
+    "[EXTENDED_RESOURCE]s": EXTENDED_RESOURCE[]
+  }
+}
+```
+
+If the API returns an array of resources and extends other resources
+
+```json
+{
+  "success": true,
+  "status": 200,
+  "message": "The request succeeded.",
+  "data": {
+    "[RESOURCE]s": RESOURCE[],
+    "total": NUMBER,
+    "[EXTENDED_RESOURCE]": EXTENDED_RESOURCE,
+    "[EXTENDED_RESOURCE]s": EXTENDED_RESOURCE[]
+  }
+}
+```
+
+If the API throws an error (data may include some other fields related to the error)
+
+```json
+{
+  "success": false,
+  "status": 5XX | 4XX,
+  "errorReason": "REASON",
+  "message": "MESSAGE",
+  "data": {
+    "traceId": "ID",
+    "[METADATA]": METADATA,
+    ...
+  }
+}
+```
+
+**NOTE:** If the API is not defined for retrieving data (not details and list), its response may have a specific structure.
+
+### **Status codes**
+
+```plain
+----- Success -----
+200 - The request is successfully done, and data is returned in response
+202 - THe request is accepted (it doesn't mean done because maybe it's async or background process. The requester can be notified in 2 ways (polling, socket))
+204 - The request is successfully done, and there is no data to return in response
+
+----- Client Errors -----
+400 - validation failed
+401 - unauthorized (the JWT, api-key, etc. is not valid)
+403 - have no permission to call the endpoint (auth token is valid but permission is not granted)
+404 - The ID is not found
+406 - the request logically is not allowed
+409 - the resource already exists
+423 - the resource is locked
+429 - too many requests (this is rate-limiting status)
+
+----- Server Errors -----
+500 - an unknown internal error occurred
+```

package/guidelines/NamingGuideline.md

@@ -0,0 +1,182 @@
+# Naming Guideline
+
+## Variables & Classes
+
+1. All imported modules (third-party NPM libraries or local modules) names should be added with pascal-case.
+
+```javascript
+import Moment from 'moment'
+import Fs from 'fs'
+import Lodash from 'lodash'
+import { find as Find } from 'lodash' // not recommended but conventionally is correct
+import lodash from 'lodash' // wrong
+import { find } from 'lodash' // wrong
+```
+
+2. All variables should be in camel-case even if they contain abbreviations. For example **apiKey**.
+3. The variables which contain a list of objects with the same type should be named in plural form. For example, an array of **hostel** objects should be named **hostels**.
+4. The class/enum/interface name should be a pascal-case.
+5. For boolean variables, you need to start with **is**, **has**, or **does** prefixes. For example **isMale** and **hasChild**.
+
+## Methods
+
+Names are a combination of words, we have 6 basic actions:
+1. get - when you want to get a single object
+2. list - when you list a few objects
+3. create - when you create an object
+4. update - when you update an existing object (The updated properties should exist)
+5. set - when you update an existing object (You don’t know if the properties exist or not)
+6. remove - when you remove an existing object
+   As an example, if you are in a class called User and you want to get a user object your method name is:
+
+```javascript
+class User {
+    get() {}
+}
+```
+
+And if you are in the same class but want to get a users profile and get a list of permissions
+
+```javascript
+class User {
+    get() {},
+    getProfile() {},
+    listPermissions() {}
+}
+```
+
+As you can see we do not postfix the method with its own namespace but anything else will be prefixed by their name.
+If you are doing multiple **create**, **update**, and **remove** in a method you should prefix it with **batch**.
+
+```javascript
+batchCreate () {} 
+batchRemove () {}
+batchUpdate () {}
+```
+
+If it is a background job or a delayed job to process actions it is postfixed with "job".
+
+```javascript
+// a job to create an object
+createJob () {}
+
+
+// a job to create many objects in batch
+batchCreateJob () {}
+```
+
+If you have methods that are doing two actions, such as (create or update) or (create and delete), we order the names by the sequence of the actions.
+If you are using the OR operation between actions, they should be ordered by the (**get, list, create, update, set, remove**).
+
+```javascript
+class User {
+  //create a user or update a user based on a condition
+  createOrUpdate () {}
+
+
+  //first remove actions in batch then update user
+  batchRemoveActionsAndUpdate () {} 
+
+
+  //Remove actions in batch in background then update user
+  batchRemoveActionsJobAndUpdate () {}
+}
+```
+
+If there are methods that are answering a question and returning a boolean, they should be prefixed by "is", "has" or "does".
+
+```plain
+isMale() {}
+hasChild() {}
+```
+
+If there are methods that add or remove items in a field of type array, they should be prefixed with "add" or "remove".
+
+```plain
+addImage() {}
+removeCard() {}
+```
+
+For any other method that doesn’t match the previous rules, the name of the method should start with action.
+
+```plain
+calculateDebt() {}               //debtCalculation() is wrong
+notifyTravelers() {}             //travelersNotification() is wrong
+batchPublish() {}                //batchPublishing() is wrong
+```
+
+## Directories & Files
+
+Directories should be named in camel-case style, and files should be named in pascal-case. Each file should export a class, function, or object with a name exactly matching the file name.
+
+## Pub/Sub Topics & Subscriptions
+
+### **Topic name**
+The topic name should represent an event inside the publisher service. For example, if a reservation is updated inside the microservice, the topic name for this event would be: "**reservationUpdated"**. The topic name should be in the past tense and camel-case.
+### **Subscription name**
+The subscription name is a composition of multiple parts. All parts in the name should be camel-case:
+**`<topic name>-<microservice name>-sub`**
+For example**,** if the **pms** microservice wants to subscribe to the "**reservationUpdated"** topic, its subscription name would be "**reservationUpdated-pms-sub**".
+
+Add `dev-` and `test-` prefixes for development and test environments. For example:
+- Topic Name: `dev-reservationUpdated`
+- Subscription Name: `dev-reservationUpdated-pms-sub`
+
+## Redis Keys & Redlock resource names
+
+A Redis key or Redlock resource name should be:
+- Unique
+- Predictable
+- Not overlapping (for example, with other microservices)
+- Can be a combination of multiple identifiers and literals, separated with a ":".
+- Follow this convention:
+  - Redis key: `<key>:<microservice name>`
+  - Redlock resource name: `<name>:<microservice name>:lock`
+
+Examples:
+```javascript
+// microservice name is "bunny"
+const redisKey = `passportNumber:${userId}:bunny`
+const redlockResourceName = `order:${orderId}:bunny:lock`
+```
+
+## Collections & Models
+
+1. The database collection, table, or index should be in the plural form.
+```
+# Bad 😖
+user
+staff
+hostel
+
+# Good 😌
+users
+staffs
+hostels
+```
+
+2. The firestore collections should be prefixed with the environment short name.
+```
+# Bad 😖
+users
+staffs
+hostels
+
+# Good 😌
+dev_users
+stg_staffs
+prd_hostels
+```
+
+3. The models should be in the singular form
+```
+# Bad 😖
+Users
+Staffs
+Hostels
+
+# Good 😌
+User
+Staff
+Hostel
+```

package/package.json

@@ -0,0 +1,138 @@
+{
+  "name": "@gokiteam/goki-dev",
+  "version": "0.2.0",
+  "description": "Unified local development platform for Goki services",
+  "type": "module",
+  "main": "./client/dist/index.js",
+  "types": "./client/dist/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./client/dist/index.d.ts",
+      "import": "./client/dist/index.js",
+      "require": "./client/dist/index.js"
+    }
+  },
+  "typesVersions": {
+    "*": {
+      ".": [
+        "./client/dist/index.d.ts"
+      ]
+    }
+  },
+  "bin": {
+    "goki-dev": "./bin/goki-dev.js",
+    "goki-dev-mcp": "./bin/mcp-server.js",
+    "goki-secrets": "./bin/secrets-cli.js"
+  },
+  "files": [
+    "bin/",
+    "cli/",
+    "src/",
+    "client/dist/",
+    "config.development",
+    "config.test",
+    "guidelines/",
+    "patterns/"
+  ],
+  "scripts": {
+    "start": "dotenv -e config.development node src/Server.js",
+    "dev": "dotenv -e config.development node src/Server.js",
+    "dev:watch": "dotenv -e config.development node --watch src/Server.js",
+    "dev:ui": "concurrently \"npm run dev\" \"cd ui && npm start\" --names \"backend,frontend\" --prefix-colors \"blue,green\"",
+    "ui:start": "cd ui && npm start",
+    "ui:build": "cd ui && npm run build",
+    "build:client": "cd client && npx tsc && echo '{\"type\":\"commonjs\"}' > dist/package.json",
+    "prepublishOnly": "npm run build:client",
+    "test": "dotenv -e config.test mocha 'tests/**/*.test.js'",
+    "test:api": "dotenv -e config.test mocha 'tests/api/**/*.test.js'",
+    "test:emulation": "dotenv -e config.test mocha 'tests/emulation/**/*.test.js'",
+    "test:integration": "dotenv -e config.test mocha 'tests/integration/**/*.test.js' --timeout 10000",
+    "test:all": "npm run test:integration",
+    "test:e2e": "playwright test",
+    "test:e2e:ui": "playwright test tests/e2e/ui",
+    "test:e2e:workflows": "playwright test tests/e2e/workflows",
+    "test:e2e:headed": "playwright test --headed",
+    "test:e2e:debug": "playwright test --debug",
+    "lint": "standard --fix",
+    "docker:up": "docker-compose up -d",
+    "docker:up:build": "docker-compose up -d --build",
+    "docker:down": "docker-compose down",
+    "docker:logs": "docker-compose logs -f",
+    "docker:services": "docker-compose -f docker-compose.services.yml up -d",
+    "mcp": "node bin/mcp-server.js",
+    "cleanup:jobs": "node scripts/cleanup-bull-jobs.js"
+  },
+  "keywords": [
+    "goki",
+    "dev-tools",
+    "pubsub",
+    "mqtt",
+    "emulator",
+    "local-development"
+  ],
+  "author": "Goki Team",
+  "license": "UNLICENSED",
+  "engines": {
+    "node": ">=18.0.0"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "dependencies": {
+    "@gokiteam/koa": "^3.0.6",
+    "@gokiteam/oops": "*",
+    "@google-cloud/functions-framework": "^5.0.2",
+    "@koa/cors": "^5.0.0",
+    "@modelcontextprotocol/sdk": "^1.27.0",
+    "@portalteam/protocols": "latest",
+    "@zowe/secrets-for-zowe-sdk": "^8.29.4",
+    "aedes": "^0.51.0",
+    "axios": "^1.6.0",
+    "better-sqlite3": "^12.6.2",
+    "bull": "^3.29.3",
+    "chalk": "^5.3.0",
+    "commander": "^11.0.0",
+    "concurrently": "^9.2.1",
+    "dockerode": "^3.3.5",
+    "dotenv": "^16.4.0",
+    "execa": "^8.0.1",
+    "firebase-admin": "^13.6.1",
+    "http-proxy": "^1.18.1",
+    "inquirer": "^9.2.0",
+    "ioredis": "^5.9.2",
+    "joi": "^17.13.0",
+    "jsonpath-plus": "^10.4.0",
+    "koa": "^2.15.0",
+    "koa-bodyparser": "^4.4.1",
+    "koa-router": "^12.0.1",
+    "koa-static": "^5.0.0",
+    "macos-touchid": "^1.0.3",
+    "moment": "^2.30.0",
+    "mqtt-packet": "^9.0.0",
+    "node-cron": "^4.2.1",
+    "ora": "^7.0.1",
+    "pg": "^8.18.0",
+    "socket.io": "^4.7.0",
+    "uuid": "^9.0.1"
+  },
+  "devDependencies": {
+    "@google-cloud/logging": "^11.2.1",
+    "@playwright/test": "^1.58.1",
+    "@types/better-sqlite3": "^7.6.13",
+    "chai": "^5.1.0",
+    "dotenv-cli": "^7.4.0",
+    "eventsource": "^4.1.0",
+    "mocha": "^10.3.0",
+    "mqtt": "^5.15.0",
+    "playwright": "^1.58.1",
+    "standard": "^17.1.0",
+    "supertest": "^6.3.0",
+    "typescript": "^5.9.3"
+  },
+  "standard": {
+    "env": [
+      "mocha"
+    ]
+  },
+  "packageManager": "yarn@3.8.7+sha512.bbe7e310ff7fd20dc63b111110f96fe18192234bb0d4f10441fa6b85d2b644c8923db8fbe6d7886257ace948440ab1f83325ad02af457a1806cdc97f03d2508e"
+}

package/patterns/api/[collectionName]/Controllers.md

@@ -0,0 +1,62 @@
+## API Controller definition
+For each API endpoints:
+- There should be a controller function which is a koa middleware
+- The controller exports data from request, then calls the endpoint logic, then attaches response/status to koa context
+
+The common way of defining controller is like:
+```javascript
+import { Logic } from './Logic.js'
+
+export const Controllers = {
+  // other api endpoints controllers...
+  async create (ctx) {
+    const { traceId } = ctx.custom
+    const { body: data } = ctx.request
+    ctx.body = await Logic.create({ data, traceId })
+  }
+}
+```
+
+If the endpoint doesn't return data on API response:
+```javascript
+import { Logic } from './Logic.js'
+
+export const Controllers = {
+  // other api endpoints controllers...
+  async create (ctx) {
+    const { traceId } = ctx.custom
+    const { body: data } = ctx.request
+    ctx.status = 204
+  }
+}
+```
+
+If the endpoint returns data on response with a status other than 200:
+```javascript
+import { Logic } from './Logic.js'
+
+export const Controllers = {
+  // other api endpoints controllers...
+  async create (ctx) {
+    const { traceId } = ctx.custom
+    const { body: data } = ctx.request
+    ctx.body = await Logic.create({ data, traceId })
+    ctx.status = 202
+  }
+}
+```
+
+If the response should bypass the default response formatting rules:
+```javascript
+import { Logic } from './Logic.js'
+
+export const Controllers = {
+  // other api endpoints controllers...
+  async create (ctx) {
+    const { traceId } = ctx.custom
+    const { body: data } = ctx.request
+    ctx.body = await Logic.create({ data, traceId })
+    ctx.custom.isRawBody = true
+  }
+}
+```

package/patterns/api/[collectionName]/Logic.md

@@ -0,0 +1,154 @@
+## API Logic definition
+For each API endpoint:
+- There is logic function in "src/api/[collection]/Logic.js"
+- The logic implementation includes (in order)
+  - Checking for all logical edge cases and throw Oops errors with proper reasons
+    - Check if the supplied entity IDs in the request exist
+    - Check value boundaries, logical relationships, limits enforced by configs, etc.
+    - The edge cases should be added to "src/api/[collection]/Schemas.js" as error responses
+    - All reasons should be defined in "src/enums/ErrorReason.js" enum
+  - DB operations, including ACID transaction
+  - Call shared logic from `src/logic` directory if required
+  - Publish pub/sub messages if required
+    - If there is an operator in the request, add
+  - Return data for controller to be set in API response
+
+Coding pattern:
+```javascript
+import { ErrorReason } from '../../enums/ErrorReason.js'
+import { TryPublishUserCreatedMessage } from '../../logic/TryPublishUserCreatedMessage.js'
+import { User } from '../../models/User.js'
+
+export const Logic = {
+  // other endpoints logic implementation...
+  async create (params) {
+    const { data, traceId } = params || {}
+    const { firstName, lastName, email } = data
+    const logicName = Logic.create.name
+    const exists = await User.doesExistCompositeId({ email })
+    if (exists) {
+      throw Oops.notAcceptable('The email already exists')
+        .reason(ErrorReason.userAlreadyExists)
+        .resource(logicName)
+        .meta({ traceId, email })
+    }
+    const user = new User({ firstName, lastName, email })
+    await user.create()
+    return { user: user.mask() }
+  }
+}
+```
+
+If an endpoint needs to update multiple models atomically, use transactions:
+
+```javascript
+import { PostgresTransactionHandler } from '@gokiteam/odm'
+import { PostgresPool } from '../../singletons/PostgresPool.js'
+import { Device } from '../../models/Device.js'
+import { DeviceStatus } from '../../models/DeviceStatus.js'
+
+export const Logic = {
+  // other endpoints logic implementation...
+  async create (params) {
+    const { data, traceId } = params || {}
+    const { propertyId, macAddress, model } = data
+    const logicName = Logic.create.name
+    const exists = await Device.doesExist(macAddress)
+    if (exists) {
+      throw Oops.notAcceptable('Device already exists')
+        .reason(ErrorReason.deviceAlreadyExists)
+        .resource(logicName)
+        .meta({ traceId, macAddress })
+    }
+    const device = new Device({ propertyId, macAddress, model })
+    const deviceStatus = new DeviceStatus({ deviceId: macAddress })
+    const handler = new PostgresTransactionHandler(PostgresPool)
+    await handler
+      .create(device)
+      .create(deviceStatus)
+      .run()
+    return { device: device.mask() }
+  }
+}
+```
+
+For more transaction patterns, refer to `patterns/transactions/Patterns.md`.
+
+If a request includes an "operator" data, a pub/sub message should be published like this:
+
+```javascript
+import Moment from 'moment'
+import { v4 as IdGenerator } from 'uuid'
+
+import { Application } from '../../configs/Application.js'
+import { ErrorReason } from '../../enums/ErrorReason.js'
+import { TryPublishBatchEventLogRequired } from '../../logic/TryPublishBatchEventLogRequired.js'
+import { User } from '../../models/User.js'
+
+export const Logic = {
+  // other endpoints logic implementation...
+  async create (params) {
+    const { data, traceId } = params || {}
+    const { propertyId, firstName, lastName, email, operator } = data
+    const logicName = Logic.create.name
+    const exists = await User.doesExistCompositeId({ email })
+    if (exists) {
+      throw Oops.notAcceptable('The email already exists')
+        .reason(ErrorReason.userAlreadyExists)
+        .resource(logicName)
+        .meta({ traceId, email })
+    }
+    const user = new User({ firstName, lastName, email })
+    await user.create()
+    TryPublishBatchEventLogRequired({
+      propertyId,
+      records: [
+        {
+          operator,
+          targetId: user.id,
+          targetType: EventTargetType.user,
+          data: { new: user.mask(), old: {} },
+          event: Event.userCreated
+        }
+      ],
+      traceId
+    })
+    return { user: user.mask() }
+  }
+}
+```
+
+For complex Logic files with shared logic between multiple endpoints, use internal methods object:
+
+```javascript
+const methods = {
+  async createDevice (params) {
+    const { propertyId, macAddress, model, supportedProtocols } = params || {}
+    const logicName = methods.createDevice.name
+    // ... shared logic implementation
+    const device = new Device({ propertyId, macAddress, model })
+    const deviceStatus = new DeviceStatus({ deviceId: macAddress })
+    const handler = new PostgresTransactionHandler(PostgresPool)
+    await handler.create(device).create(deviceStatus).run()
+    return { device: device.mask(), deviceStatus: deviceStatus.mask() }
+  }
+}
+
+export const Logic = {
+  // other endpoints logic implementation...
+  async create (params) {
+    const { data, traceId } = params || {}
+    return methods.createDevice({ ...data, traceId })
+  },
+  async createWithModel (params) {
+    const { data, traceId } = params || {}
+    const { model } = data
+    return methods.createDevice({ ...data, model, traceId })
+  }
+}
+```
+
+This pattern is useful when:
+- Multiple endpoints share similar logic
+- You need to call the same logic from different entry points
+- You want to keep the exported Logic object clean and focused on endpoint routing