gcf-common-lib 0.35.0 → 0.36.0

package/docs/tasks.md

@@ -0,0 +1,88 @@
+# Improvement Tasks Checklist
+
+Note: Each item is actionable and ordered to minimize risk: start with repo hygiene and tests, then architecture and code-level refactors, followed by observability and CI.
+
+1. [x] Documentation and repository metadata
+   - [x] Expand README with project overview, installation, quick start, and usage examples for Pub/Sub and AMQP flows.
+   - [x] Add API documentation for GcfCommon, AmqpHelper, MongoHelper, and utils (function signatures and examples).
+   - [x] Add CONTRIBUTING.md (dev setup, coding standards, testing, release process) and CODE_OF_CONDUCT.md if desired.
+   - [x] Set a valid SPDX license in package.json and add a LICENSE file.
+   - [x] Fill in meaningful package.json description and keywords.
+
+2. [ ] Testing strategy and coverage
+   - [x] Make tests runnable without transpilation by default: convert or add JS tests under test/*.test.js for pure utilities and core flows.
+   - [x] Gate any network-dependent tests behind environment flags and skip by default in CI.
+   - [x] Add unit tests for src/utils.ts (ms, sec, A1 conversions, safeJsonParse edge cases).
+   - [x] Add unit tests for GcfCommon.process(...) that avoid network I/O (no topic/exchange/queue), including error path and buildResponse.
+   - [x] Add tests for getMetadataOrAttribute(...) across GCS, Pub/Sub, and HTTP request payloads.
+   - [x] If keeping TS tests, add loader support (e.g., tsx) and an npm script: "test:ts": "node --test --import tsx .\\test\\**/*.ts".
+   - [x] Add coverage reporting with c8 and an npm script: "test:coverage".
+
+3. [ ] Build and packaging
+   - [ ] Decide on distribution model: TS-only consumers vs emitting JS. Document decision in README.
+   - [ ] If emitting JS: add tsconfig.build.json (noEmit: false, outDir: dist) and update scripts (build, clean, prepare) and package exports.
+   - [ ] If TS-only: add "types" entry and proper "exports" map pointing to source types; document that consumers need a TS-aware build.
+   - [ ] Validate that main/exports/types interop works for both ESM/CJS consumers.
+
+4. [ ] Architectural separation of concerns
+   - [ ] Extract transport publishing from GcfCommon.response into dedicated transport modules (PubSubTransport, AmqpTransport) behind a common interface.
+   - [ ] Extract metadata/attributes parsing into a separate utility with explicit type guards.
+   - [ ] Keep GcfCommon.process focused on orchestration (handler execution, timeout, and delegating response publishing).
+
+5. [ ] Reliability: retries, timeouts, and backoff
+   - [ ] Implement configurable retries with exponential backoff for Pub/Sub publishes and AMQP publishes (with max attempts and jitter).
+   - [ ] Add a configurable timeout to GcfCommon.process using AbortController or a Promise.race helper (remove/comment RxJS-based approach entirely or move behind a feature flag).
+   - [ ] Ensure idempotency considerations (e.g., correlationId/request_id usage) are documented and supported.
+
+6. [ ] Logging and observability
+   - [ ] Replace console.log with a minimal logger interface supporting levels (debug/info/warn/error) and structured context (request_id, app_id, env).
+   - [ ] Add debug logs around metadata extraction and transport selection; avoid logging sensitive data (redaction rules).
+   - [ ] Provide hooks to integrate with external loggers or APM (e.g., OpenTelemetry) without hard dependency.
+
+7. [ ] Input validation and typing
+   - [ ] Validate and normalize metadata/attributes (coerce to strings where required; verify topic/exchange/queue formats).
+   - [ ] Make GcfCommon.getOptions generic: getOptions<T = Dict<any>>(...): Promise<T> with runtime validation of JSON shape (optional schema param).
+   - [ ] Tighten TResponse typing (readonly where possible); consider branded types for identifiers (RequestId, ConsumerId).
+
+8. [ ] AMQP helper modernization
+   - [ ] Replace Bluebird disposers with async/await and try/finally; keep confirm channel variant where needed.
+   - [ ] Add connection URL validation and explicit error messages when amqpOptions.url is missing.
+   - [ ] Support configurable prefetch and confirm mode via options.
+   - [ ] Implement publish backpressure handling and retries with reconnection on channel/connection errors.
+
+9. [ ] Mongo client lifecycle and locking
+   - [ ] Review MongoHelper.withMongoClient to avoid global client leaks; either manage a shared client with health checks or expose a close method.
+   - [ ] Add ping/healthcheck and resilience to transient network errors.
+   - [ ] Deprecate MongoLock class formally (already annotated) and document a recommended alternative (e.g., using findOneAndUpdate with TTL index).
+
+10. [ ] Dead code and comments cleanup
+   - [ ] Remove or finalize the commented-out RxJS timeout code in GcfCommon.
+   - [ ] Remove the commented safeGetAttributes path or implement a proper version that fetches Storage metadata when needed.
+   - [ ] Add JSDoc for public APIs and rationale where behavior is non-obvious.
+
+11. [ ] Pub/Sub specifics and payload handling
+   - [ ] Validate attribute sizes and JSON payload limits; document constraints.
+   - [ ] Consider enabling message ordering and document how to set ordering keys when needed.
+   - [ ] Ensure attributes are consistently strings; add a helper for attribute coercion.
+
+12. [ ] Security and configuration
+   - [ ] Centralize configuration (AMQP URL, Pub/Sub project/topic defaults) with environment variable support and a typed config loader.
+   - [ ] Redact sensitive fields in logs and propagate only safe metadata to responses.
+   - [ ] Run "npm audit" regularly; schedule dependency update checks (e.g., weekly).
+
+13. [ ] Linting, formatting, and scripts
+   - [ ] Add npm scripts: "lint", "lint:fix", "format", and ensure Prettier integration.
+   - [ ] Consider enabling additional ESLint rules suitable for this repo (promise rules, no-floating-promises) while keeping pragmatism.
+   - [ ] Configure EditorConfig to enforce CRLF and common conventions.
+
+14. [ ] Continuous Integration
+   - [ ] Add GitHub Actions workflow: Node 20.x matrix to run lint, type-check, and unit tests (JS tests only by default).
+   - [ ] Cache npm and enable problem matchers for faster feedback.
+
+15. [ ] Examples and developer experience
+   - [ ] Provide an examples/ directory with a minimal handler using GcfCommon.process demonstrating no-network and networked responses.
+   - [ ] Document how to stub response publishing in tests and how to configure amqpOptions at runtime.
+
+16. [ ] Versioning and release process
+   - [ ] Adopt semantic versioning explicitly; add CHANGELOG and automate release notes generation (e.g., via GitHub Releases).
+   - [ ] Document any planned breaking changes from architectural refactors and provide migration guidance.

package/eslint.config.mjs

@@ -1,31 +1,31 @@
-import { defineConfig } from 'eslint/config';
-import globals from 'globals';
-import js from '@eslint/js';
-import tseslint from 'typescript-eslint';
-import eslintPluginUnicorn from 'eslint-plugin-unicorn';
-import pluginPromise from 'eslint-plugin-promise';
-
-export default defineConfig([
-  { files: ['**/*.{js,mjs,cjs,ts}'] },
-  { files: ['**/*.js'], languageOptions: { sourceType: 'commonjs' } },
-  { files: ['**/*.{js,mjs,cjs,ts}'], languageOptions: { globals: globals.node } },
-  { files: ['**/*.{js,mjs,cjs,ts}'], plugins: { js }, extends: ['js/recommended'] },
-  tseslint.configs.recommended,
-  eslintPluginUnicorn.configs.recommended,
-  pluginPromise.configs['flat/recommended'],
-  {
-    rules: {
-      'block-scoped-var': 'error',
-      'no-loop-func': 'error',
-      //
-      '@typescript-eslint/no-explicit-any': 'off',
-      '@typescript-eslint/no-unused-vars': 'off',
-      '@typescript-eslint/no-empty-function': 'off',
-      '@typescript-eslint/no-empty-interface': 'off',
-      //
-      'unicorn/prevent-abbreviations': 'off',
-      // 'lodash/prefer-lodash-method': 'off',
-      // 'rxjs/no-implicit-any-catch': 'off',
-    },
-  },
-]);
+import { defineConfig } from 'eslint/config';
+import globals from 'globals';
+import js from '@eslint/js';
+import tseslint from 'typescript-eslint';
+import eslintPluginUnicorn from 'eslint-plugin-unicorn';
+import pluginPromise from 'eslint-plugin-promise';
+
+export default defineConfig([
+  { files: ['**/*.{js,mjs,cjs,ts}'] },
+  { files: ['**/*.js'], languageOptions: { sourceType: 'commonjs' } },
+  { files: ['**/*.{js,mjs,cjs,ts}'], languageOptions: { globals: globals.node } },
+  { files: ['**/*.{js,mjs,cjs,ts}'], plugins: { js }, extends: ['js/recommended'] },
+  tseslint.configs.recommended,
+  eslintPluginUnicorn.configs.recommended,
+  pluginPromise.configs['flat/recommended'],
+  {
+    rules: {
+      'block-scoped-var': 'error',
+      'no-loop-func': 'error',
+      //
+      '@typescript-eslint/no-explicit-any': 'off',
+      '@typescript-eslint/no-unused-vars': 'off',
+      '@typescript-eslint/no-empty-function': 'off',
+      '@typescript-eslint/no-empty-interface': 'off',
+      //
+      'unicorn/prevent-abbreviations': 'off',
+      // 'lodash/prefer-lodash-method': 'off',
+      // 'rxjs/no-implicit-any-catch': 'off',
+    },
+  },
+]);

package/package.json

@@ -1,52 +1,63 @@
-{
-  "name": "gcf-common-lib",
-  "description": "",
-  "version": "0.35.0",
-  "publishConfig": {
-    "access": "public",
-    "branches": [
-      "master"
-    ]
-  },
-  "engines": {
-    "node": ">=20"
-  },
-  "main": "src/index",
-  "scripts": {
-    "build": "tsc",
-    "test": "node --test ./test/"
-  },
-  "repository": {
-    "type": "git",
-    "url": "https://github.com/TopTechnologies/gcf-common.git"
-  },
-  "dependencies": {
-    "@google-cloud/pubsub": "^4.10.0",
-    "@google-cloud/storage": "^7.15.2",
-    "@types/amqplib": "^0.10.7",
-    "@types/bluebird": "^3.5.42",
-    "@types/express": "^4.17.21",
-    "@types/lodash": "^4.17.16",
-    "amqplib": "^0.10.5",
-    "bluebird": "^3.7.2",
-    "lodash": "^4.17.21",
-    "moment": "^2.30.1",
-    "mongodb": "^6.14.1",
-    "rxjs": "^7.8.2"
-  },
-  "devDependencies": {
-    "@tsconfig/node20": "^20.1.4",
-    "@types/node": "^20.17.23",
-    "@eslint/js": "^9.21.0",
-    "eslint": "^9.21.0",
-    "eslint-plugin-lodash": "^8.0.0",
-    "eslint-plugin-promise": "^7.2.1",
-    "eslint-plugin-unicorn": "^57.0.0",
-    "globals": "^15.15.0",
-    "prettier": "^3.5.2",
-    "typescript-eslint": "^8.25.0",
-    "typescript": "^5.3.3"
-  },
-  "author": "alert83@gmail.com",
-  "license": ""
-}
+{
+  "name": "gcf-common-lib",
+  "description": "Common helpers for Google Cloud Functions: Pub/Sub publishing, Mongo helpers, and utilities for Node 20+ (TypeScript).",
+  "version": "0.36.0",
+  "publishConfig": {
+    "access": "public",
+    "branches": [
+      "master"
+    ]
+  },
+  "engines": {
+    "node": ">=20"
+  },
+  "main": "src/index",
+  "scripts": {
+    "build": "tsc",
+    "test": "node --test .\\test\\sample.test.js",
+    "test:ts": "node --test --import tsx .\\test",
+    "test:coverage": "c8 --reporter=text --reporter=html node --test .\\test\\sample.test.js"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/TopTechnologies/gcf-common.git"
+  },
+  "keywords": [
+    "node",
+    "typescript",
+    "google-cloud-functions",
+    "gcf",
+    "pubsub",
+    "mongodb",
+    "utilities"
+  ],
+  "dependencies": {
+    "@google-cloud/pubsub": "^5.2.0",
+    "@google-cloud/storage": "^7.18.0",
+    "@types/bluebird": "^3.5.42",
+    "@types/express": "^4.17.25",
+    "@types/lodash": "^4.17.21",
+    "bluebird": "^3.7.2",
+    "lodash": "^4.17.21",
+    "moment": "^2.30.1",
+    "mongodb": "^6.21.0",
+    "rxjs": "^7.8.2"
+  },
+  "devDependencies": {
+    "@tsconfig/node20": "^20.1.8",
+    "@types/node": "^20.19.27",
+    "@eslint/js": "^9.21.0",
+    "eslint": "^9.21.0",
+    "eslint-plugin-lodash": "^8.0.0",
+    "eslint-plugin-promise": "^7.2.1",
+    "eslint-plugin-unicorn": "^57.0.0",
+    "globals": "^15.15.0",
+    "prettier": "^3.7.4",
+    "typescript-eslint": "^8.25.0",
+    "typescript": "^5.8.3",
+    "tsx": "^4.16.2",
+    "c8": "^9.1.0"
+  },
+  "author": "alert83@gmail.com",
+  "license": "MIT"
+}

package/src/{amqp-helper.ts → amqp-helper.ts.old}

@@ -1,52 +1,52 @@
-import { Channel, ChannelModel, ConfirmChannel, connect, Options } from 'amqplib';
-import Bluebird from 'bluebird';
-import Dict = NodeJS.Dict;
-
-export const AmqpHelper = {
-  async withAmqpConn(fn: (channelModel: ChannelModel) => Promise<any>, url: string) {
-    function withDisposer() {
-      return Bluebird.method(async () => {
-        const amqpConn = await connect(url);
-        amqpConn.on('close', () => console.info('Amqp connection closed!'));
-        return amqpConn;
-      })().disposer((channelModel, promise) => channelModel.close());
-    }
-
-    return Bluebird.using(withDisposer(), channelModel => fn(channelModel));
-  },
-
-  async withAmqpCh(fn: (ch: Channel) => Promise<any>, url: string, useConfirmChannel = false, prefetch = 1) {
-    return AmqpHelper.withAmqpConn(async channelModel => {
-      function withDisposer() {
-        return Bluebird.method(async () => {
-          const ch = useConfirmChannel ? await channelModel.createConfirmChannel() : await channelModel.createChannel();
-          await ch.prefetch(prefetch);
-          return ch;
-        })().disposer((ch, promise) => ch.close());
-      }
-
-      return Bluebird.using(withDisposer(), ch => fn(ch));
-    }, url);
-  },
-
-  async publishAmqp(
-    ch: Channel,
-    exchange: string | undefined,
-    routingKey: string,
-    json: Dict<any>,
-    options?: Options.Publish,
-  ) {
-    const payload = Buffer.from(JSON.stringify(json));
-    const keepSending = exchange
-      ? ch.publish(exchange, routingKey, payload, options)
-      : ch.sendToQueue(routingKey, payload, options);
-    if (!keepSending) await new Promise<void>(resolve => ch.once('drain', () => resolve()));
-  },
-
-  async sendToQueueConfAmqp(ch: ConfirmChannel, queue: string, json: Dict<any>, options?: Options.Publish) {
-    const payload = Buffer.from(JSON.stringify(json));
-    await new Promise((resolve, reject) =>
-      ch.sendToQueue(queue, payload, options, (err, ok) => (err ? reject(err) : resolve(ok))),
-    );
-  },
-};
+import { Channel, ChannelModel, ConfirmChannel, connect, Options } from 'amqplib';
+import Bluebird from 'bluebird';
+import Dict = NodeJS.Dict;
+
+export const AmqpHelper = {
+  async withAmqpConn(fn: (channelModel: ChannelModel) => Promise<any>, url: string) {
+    function withDisposer() {
+      return Bluebird.method(async () => {
+        const amqpConn = await connect(url);
+        amqpConn.on('close', () => console.info('Amqp connection closed!'));
+        return amqpConn;
+      })().disposer((channelModel, promise) => channelModel.close());
+    }
+
+    return Bluebird.using(withDisposer(), channelModel => fn(channelModel));
+  },
+
+  async withAmqpCh(fn: (ch: Channel) => Promise<any>, url: string, useConfirmChannel = false, prefetch = 1) {
+    return AmqpHelper.withAmqpConn(async channelModel => {
+      function withDisposer() {
+        return Bluebird.method(async () => {
+          const ch = useConfirmChannel ? await channelModel.createConfirmChannel() : await channelModel.createChannel();
+          await ch.prefetch(prefetch);
+          return ch;
+        })().disposer((ch, promise) => ch.close());
+      }
+
+      return Bluebird.using(withDisposer(), ch => fn(ch));
+    }, url);
+  },
+
+  async publishAmqp(
+    ch: Channel,
+    exchange: string | undefined,
+    routingKey: string,
+    json: Dict<any>,
+    options?: Options.Publish,
+  ) {
+    const payload = Buffer.from(JSON.stringify(json));
+    const keepSending = exchange
+      ? ch.publish(exchange, routingKey, payload, options)
+      : ch.sendToQueue(routingKey, payload, options);
+    if (!keepSending) await new Promise<void>(resolve => ch.once('drain', () => resolve()));
+  },
+
+  async sendToQueueConfAmqp(ch: ConfirmChannel, queue: string, json: Dict<any>, options?: Options.Publish) {
+    const payload = Buffer.from(JSON.stringify(json));
+    await new Promise((resolve, reject) =>
+      ch.sendToQueue(queue, payload, options, (err, ok) => (err ? reject(err) : resolve(ok))),
+    );
+  },
+};