@devopsplaybook.io/common-utils 1.0.0-beta.5.9c5ecde → 1.0.0

package/AGENTS.md

@@ -0,0 +1,84 @@
+# AGENTS.md
+
+This file provides context for AI coding agents working on this repository.
+
+## Repository Overview
+
+`@devopsplaybook.io/common-utils` is a shared npm library that centralizes utility modules used across devopsplaybook.io server projects. It serves two purposes:
+
+1. **Node.js library** (`src/`): Database access (SQLite via `better-sqlite3`, PostgreSQL via `pg.Pool`), configuration loading, OpenTelemetry context management, and small helpers.
+2. **Reusable GitHub Actions workflows** (`.github/workflows/reusable-*.yml`): Standardized CI/CD pipelines adopted by all projects in the organization.
+
+## Architecture
+
+```
+index.ts                    # Barrel re-exports for all modules
+src/
+  OTelContext.ts            # createOTelContext() factory (tracer/meter/logger singletons)
+  ConfigBase.ts             # Abstract base class for 3-layer config (env > config.json > defaults)
+  SqlDbUtils.ts             # SQLite operations (better-sqlite3, synchronous)
+  PostgresDbUtils.ts        # PostgreSQL operations (pg.Pool, async/Promise)
+  DbUtils.ts                # Unified facade dispatching to Sql or Postgres
+  DbUtilsNoTelemetry.ts     # Same DB ops without OTel span overhead
+  SystemCommand.ts          # Promise wrapper around child_process.exec
+  Timeout.ts                # Promise wrapper around setTimeout
+  *.spec.ts                 # Co-located test files
+.github/workflows/
+  main-build.yml            # Caller: push to main -> reusable-npm-merge
+  pr-check.yml              # Caller: PR to main -> reusable-npm-pr
+  npm-upgrade.yml           # Caller: weekly schedule -> reusable-npm-upgrade
+  reusable-npm-merge.yml    # Lint + test + build + publish release to npm
+  reusable-npm-pr.yml       # Lint + test + build + publish beta tag + comment PR
+  reusable-npm-upgrade.yml  # npm-check-updates + auto PR
+  reusable-pr-verify.yml    # Matrix Node.js + multi-platform Docker build for PRs
+  reusable-merge-build.yml  # Matrix Node.js + Docker build with version tags on merge
+```
+
+## Key Conventions
+
+- **TypeScript**: Target ES2019, CommonJS output, strict mode enabled, declarations generated.
+- **ESLint**: Uses `typescript-eslint` with `strict` and `stylistic` rule sets. Minimize `eslint-disable` comments; use them only when the type system cannot express the constraint (e.g., `@typescript-eslint/no-explicit-any` for `this as any` dynamic field access in `ConfigBase`).
+- **Tests**: Jest with `ts-jest`. Spec files live next to source (`*.spec.ts`). Run with `npm run test`. The `tsconfig.spec.json` includes jest types.
+- **No default exports**: All modules use named exports only.
+- **OTel dependency injection**: Every DB module exposes a `*SetOTel(tracer, logger)` function that must be called before `*Init()`. OTel instances are stored as module-level singletons.
+- **ModuleLogger pattern**: `StandardLogger` only exposes `createModuleLogger(name)`. DB modules call `logger.createModuleLogger("ModuleName")` internally. Never call `.info()` or `.error()` directly on a `StandardLogger`.
+- **SQLite-first SQL**: Write SQL with `?` placeholders. The `DbUtils` facade and `DbUtilsNoTelemetry` module auto-convert to `$1, $2, ...` for Postgres via `convertToPostgresPlaceholders()`.
+- **Migration convention**: SQL files named `init-NNNN.sql`. `init-0000.sql` must create the `metadata` table. Subsequent files are applied in lexicographic order; applied versions are tracked in `metadata` for idempotency.
+
+## Build and Verification
+
+```bash
+npm install
+npm run build    # tsc -> dist/
+npm run lint     # eslint src (must pass with 0 errors)
+npm run test     # jest --coverage (all tests must pass)
+```
+
+All three commands must pass before committing. The CI pipeline (`reusable-npm-merge.yml`) runs the same checks.
+
+## Dependencies
+
+| Package                         | Role                                                                                       |
+| ------------------------------- | ------------------------------------------------------------------------------------------ |
+| `@devopsplaybook.io/otel-utils` | `StandardTracer`, `StandardLogger`, `StandardMeter`, `ModuleLogger`, `ConfigOTelInterface` |
+| `better-sqlite3`                | Synchronous SQLite driver (NOT the callback-based `sqlite3`)                               |
+| `pg`                            | PostgreSQL client with connection pooling                                                  |
+| `uuid`                          | v14+ (ESM -- requires `jest.mock("uuid")` in tests)                                        |
+| `fs-extra`                      | Async/sync file operations, `readJson`/`ensureDir`                                         |
+
+## Known Gotchas
+
+- **uuid ESM**: `uuid` v14+ ships ESM. In any test that transitively imports `uuid`, add `jest.mock("uuid", () => ({ v4: () => "mock-uuid-1234" }))` to avoid `SyntaxError: Unexpected token 'export'`.
+- **better-sqlite3 is synchronous**: `SqlDbUtils` functions return values directly (not Promises). `PostgresDbUtils` functions return Promises. The `DbUtils` facade returns `number | Promise<number>` depending on the active backend.
+- **eslint-disable placement**: `eslint-disable-next-line` applies to the **immediately following line only**. When disabling a rule inside a function call argument, place the comment directly before the offending expression, not before the function call.
+- **pg callback typing**: Always explicitly type pg callback parameters: `(error: Error | null, result: { rowCount: number | null })`. TypeScript cannot infer these from the overloaded `pool.query` signature.
+
+## Adopting in Other Projects
+
+See the [README](./README.md) for full adoption instructions. The typical pattern:
+
+1. Add `@devopsplaybook.io/common-utils` as a dependency.
+2. Replace the project's `OTelContext.ts` with `createOTelContext()`.
+3. Replace the project's `Config` class to `extend ConfigBase`.
+4. Replace DB utility files with re-exports from `common-utils`.
+5. Update `App.ts` to call `*SetOTel()` and `*Init()` from `common-utils`.

package/README.md

@@ -1 +1,413 @@
-# common-utils
+# @devopsplaybook.io/common-utils
+
+Shared utility modules for [devopsplaybook.io](https://github.com/devopsplaybook-io) projects. Provides OpenTelemetry-aware database access (SQLite and PostgreSQL), configuration loading, telemetry context management, and reusable GitHub Actions CI/CD workflows.
+
+## Contents
+
+- [Node.js Library](#nodejs-library)
+  - [Installation](#installation)
+  - [Modules](#modules)
+  - [Quick Start](#quick-start)
+- [Shared GitHub Actions Workflows](#shared-github-actions-workflows)
+  - [Reusable Workflows](#reusable-workflows)
+  - [Adopting in Your Project](#adopting-in-your-project)
+
+---
+
+## Node.js Library
+
+### Installation
+
+```bash
+npm install @devopsplaybook.io/common-utils
+```
+
+**Peer dependencies** (installed automatically):
+
+| Package                         | Purpose                                             |
+| ------------------------------- | --------------------------------------------------- |
+| `@devopsplaybook.io/otel-utils` | `StandardTracer`, `StandardLogger`, `StandardMeter` |
+| `@opentelemetry/api`            | OTel API (`SpanStatusCode`)                         |
+| `@opentelemetry/sdk-trace-base` | `Span` type                                         |
+| `better-sqlite3`                | Synchronous SQLite driver                           |
+| `pg`                            | PostgreSQL client (`Pool`)                          |
+| `fs-extra`                      | File system helpers                                 |
+| `uuid`                          | UUID generation for JWT keys                        |
+
+### Modules
+
+#### `OTelContext` -- Telemetry Singleton Factory
+
+Creates an isolated set of OTel singletons (tracer, meter, logger) for a server process.
+
+```ts
+import { createOTelContext } from "@devopsplaybook.io/common-utils";
+import { StandardTracer, StandardMeter } from "@devopsplaybook.io/otel-utils";
+
+const otel = createOTelContext();
+otel.OTelSetTracer(new StandardTracer(config));
+otel.OTelSetMeter(new StandardMeter(config));
+otel.OTelLogger().initOTel(config);
+
+// Later:
+const tracer = otel.OTelTracer();
+const span = tracer.startSpan("my-operation");
+```
+
+| Export                | Description                                                                                   |
+| --------------------- | --------------------------------------------------------------------------------------------- |
+| `createOTelContext()` | Returns `{ OTelTracer, OTelSetTracer, OTelMeter, OTelSetMeter, OTelLogger, OTelRequestSpan }` |
+
+---
+
+#### `ConfigBase` -- Configuration Base Class
+
+Abstract class implementing the three-layer override strategy:
+
+1. **Environment variable** (highest priority)
+2. **config.json** file value
+3. **Default** declared on the class property
+
+```ts
+import { ConfigBase } from "@devopsplaybook.io/common-utils";
+
+class MyConfig extends ConfigBase {
+  public MY_SETTING = "default";
+
+  constructor() {
+    super("my-service");
+    this.addConfigField({ field: "MY_SETTING" });
+  }
+
+  async reload(): Promise<void> {
+    await super.reload((msg) => console.log(msg));
+  }
+}
+
+const config = new MyConfig();
+await config.reload();
+```
+
+**Built-in fields** (pre-registered, no `addConfigField` needed):
+
+| Field                                  | Default              | Sensitive                           |
+| -------------------------------------- | -------------------- | ----------------------------------- |
+| `API_PORT`                             | `8080`               | No                                  |
+| `JWT_VALIDITY_DURATION`                | `8035200` (3 months) | No                                  |
+| `CORS_POLICY_ORIGIN`                   | `""`                 | No                                  |
+| `DATA_DIR`                             | `/data`              | No                                  |
+| `JWT_KEY`                              | `uuidv4()`           | Yes                                 |
+| `LOG_LEVEL`                            | `"info"`             | No                                  |
+| `DATABASE_TYPE`                        | `"sqlite"`           | No                                  |
+| `DATABASE_POSTGRES_HOST`               | `""`                 | No                                  |
+| `DATABASE_POSTGRES_PORT`               | `5432`               | No                                  |
+| `DATABASE_POSTGRES_USER`               | `""`                 | No                                  |
+| `DATABASE_POSTGRES_PASSWORD`           | `""`                 | Yes                                 |
+| `DATABASE_POSTGRES_DATABASE`           | `""`                 | No                                  |
+| All `OPENTELEMETRY_COLLECTOR_*` fields | Various              | No (except `_AUTHORIZATION_HEADER`) |
+
+---
+
+#### `SqlDbUtils` -- SQLite Database Access
+
+Synchronous database operations using `better-sqlite3`, with OTel tracing on every call.
+
+```ts
+import {
+  SqlDbUtilsSetOTel,
+  SqlDbUtilsInit,
+  SqlDbUtilsExecSQL,
+  SqlDbUtilsQuerySQL,
+} from "@devopsplaybook.io/common-utils";
+
+// At startup
+SqlDbUtilsSetOTel(tracer, logger);
+await SqlDbUtilsInit(span, config, path.resolve(__dirname, "../sql"));
+
+// Read
+const rows = SqlDbUtilsQuerySQL(span, "SELECT * FROM users WHERE id = ?", [
+  userId,
+]);
+
+// Write
+const changes = SqlDbUtilsExecSQL(
+  span,
+  "UPDATE users SET name = ? WHERE id = ?",
+  [name, userId],
+);
+```
+
+| Export                  | Signature                      | Description                                      |
+| ----------------------- | ------------------------------ | ------------------------------------------------ |
+| `SqlDbUtilsSetOTel`     | `(tracer, logger)`             | Inject OTel instances                            |
+| `SqlDbUtilsInit`        | `(span, config, sqlDir)`       | Open DB and run migrations                       |
+| `SqlDbUtilsExecSQL`     | `(span, sql, params?)`         | Execute write, returns `number` (changes)        |
+| `SqlDbUtilsQuerySQL`    | `(span, sql, params?, debug?)` | Execute read, returns `any[]`                    |
+| `SqlDbUtilsExecSQLFile` | `(span, filename)`             | Execute an entire SQL file                       |
+| `SqlDbUtilsGetDatabase` | `()`                           | Returns the `better-sqlite3` `Database` instance |
+
+**Migration convention**: Files named `init-NNNN.sql` in `sqlDir`, applied in order. A `metadata` table tracks applied versions for idempotent re-runs. `init-0000.sql` must exist (creates the `metadata` table).
+
+---
+
+#### `PostgresDbUtils` -- PostgreSQL Database Access
+
+Async (Promise-based) database operations using `pg.Pool`, with OTel tracing.
+
+| Export                             | Signature                      | Description                              |
+| ---------------------------------- | ------------------------------ | ---------------------------------------- |
+| `PostgresDbUtilsSetOTel`           | `(tracer, logger)`             | Inject OTel instances                    |
+| `PostgresDbUtilsInit`              | `(span, config, sqlDir)`       | Create pool and run migrations           |
+| `PostgresDbUtilsExecSQL`           | `(span, sql, params?)`         | Execute write, returns `Promise<number>` |
+| `PostgresDbUtilsQuerySQL`          | `(span, sql, params?, debug?)` | Execute read, returns `Promise<any[]>`   |
+| `PostgresDbUtilsExecSQLFile`       | `(span, filename)`             | Execute an entire SQL file               |
+| `PostgresDbUtilsGetPool`           | `()`                           | Returns the `pg.Pool` instance           |
+| `PostgresDbUtilsTransactionStart`  | `(span)`                       | Begin a transaction (`BEGIN`)            |
+| `PostgresDbUtilsTransactionCommit` | `(span)`                       | Commit a transaction (`COMMIT`)          |
+
+Pool defaults: `max: 20`, `idleTimeoutMillis: 30000`, `connectionTimeoutMillis: 10000`.
+
+---
+
+#### `DbUtils` -- Unified Database Facade
+
+Dispatches to SQLite or Postgres based on `config.DATABASE_TYPE`. Write SQL using SQLite-style `?` placeholders; they are automatically converted to `$1, $2, ...` for Postgres.
+
+```ts
+import {
+  DbUtilsSetOTel,
+  DbUtilsInit,
+  DbUtilsExecSQL,
+  DbUtilsQuerySQL,
+} from "@devopsplaybook.io/common-utils";
+
+DbUtilsSetOTel(tracer, logger);
+await DbUtilsInit(span, config, sqlDir);
+
+// Works with both SQLite and Postgres -- placeholders auto-converted
+const rows = DbUtilsQuerySQL(span, "SELECT * FROM users WHERE id = ?", [
+  userId,
+]);
+```
+
+| Export                                        | Description                                  |
+| --------------------------------------------- | -------------------------------------------- |
+| `DbUtilsSetOTel(tracer, logger)`              | Set OTel on both backends                    |
+| `DbUtilsInit(span, config, sqlDir)`           | Init the active backend                      |
+| `DbUtilsExecSQL(span, sql, params?)`          | Write with auto-conversion                   |
+| `DbUtilsQuerySQL(span, sql, params?, debug?)` | Read with auto-conversion                    |
+| `DbUtilsGetDatabase()`                        | Returns native handle (`Database` or `Pool`) |
+| `DbUtilsGetType()`                            | Returns `"sqlite"` or `"postgres"`           |
+| `convertToPostgresPlaceholders(sql)`          | Converts `?` to `$1, $2, ...`                |
+
+---
+
+#### `DbUtilsNoTelemetry` -- High-Throughput Path
+
+Same SQL operations but **without** creating OTel spans. Use on hot paths where span overhead matters.
+
+```ts
+import {
+  DbUtilsNoTelemetrySetLogger,
+  DbUtilsNoTelemetryExecSQL,
+  DbUtilsNoTelemetryBatchInsert,
+} from "@devopsplaybook.io/common-utils";
+
+DbUtilsNoTelemetrySetLogger(logger);
+
+DbUtilsNoTelemetryExecSQL("INSERT INTO log (msg) VALUES (?)", [message]);
+
+// Batch insert: builds multi-row INSERT
+DbUtilsNoTelemetryBatchInsert(
+  "INTO prices (token, price, ts)", // table + columns
+  3, // number of columns
+  [
+    ["BTC", 65000, "2024-01-01"],
+    ["ETH", 3200, "2024-01-01"],
+  ], // rows
+);
+```
+
+| Export                                                    | Description                       |
+| --------------------------------------------------------- | --------------------------------- |
+| `DbUtilsNoTelemetrySetLogger(logger)`                     | Inject logger for error reporting |
+| `DbUtilsNoTelemetryExecSQL(sql, params?)`                 | Write without spans               |
+| `DbUtilsNoTelemetryQuerySQL(sql, params?, debug?)`        | Read without spans                |
+| `DbUtilsNoTelemetryBatchInsert(tableCols, numCols, rows)` | Optimized multi-row INSERT        |
+
+---
+
+#### `SystemCommand` -- Shell Command Execution
+
+```ts
+import { SystemCommandExecute } from "@devopsplaybook.io/common-utils";
+
+const output = await SystemCommandExecute("ls -la /tmp", { cwd: "/home" });
+```
+
+#### `Timeout` -- Promise-based Delay
+
+```ts
+import { TimeoutWait } from "@devopsplaybook.io/common-utils";
+
+await TimeoutWait(5000); // wait 5 seconds
+```
+
+### Quick Start
+
+```ts
+import {
+  createOTelContext,
+  ConfigBase,
+  DbUtilsSetOTel,
+  DbUtilsInit,
+} from "@devopsplaybook.io/common-utils";
+import { StandardTracer, StandardMeter } from "@devopsplaybook.io/otel-utils";
+
+// 1. Config
+class AppConfig extends ConfigBase {
+  constructor() {
+    super("my-app");
+  }
+  async reload() {
+    await super.reload((m) => console.log(m));
+  }
+}
+const config = new AppConfig();
+await config.reload();
+
+// 2. OTel
+const otel = createOTelContext();
+otel.OTelSetTracer(new StandardTracer(config));
+otel.OTelSetMeter(new StandardMeter(config));
+otel.OTelLogger().initOTel(config);
+
+// 3. Database
+DbUtilsSetOTel(otel.OTelTracer(), otel.OTelLogger());
+const span = otel.OTelTracer().startSpan("init");
+await DbUtilsInit(span, config, path.resolve(__dirname, "../sql"));
+span.end();
+```
+
+---
+
+## Shared GitHub Actions Workflows
+
+The `.github/workflows/` directory contains **reusable workflows** that other repositories can call to standardize their CI/CD pipelines.
+
+### Reusable Workflows
+
+| Workflow        | File                       | Trigger         | Purpose                                                                                                                                                       |
+| --------------- | -------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **NPM Merge**   | `reusable-npm-merge.yml`   | `workflow_call` | Lint, test, build, and publish a release to npm on merge to main. Uploads coverage to Quality Dashboard. Only publishes if the version doesn't already exist. |
+| **NPM PR**      | `reusable-npm-pr.yml`      | `workflow_call` | Lint, test, and build on PR. Publishes a **beta** version tagged `beta` and comments the PR with install instructions.                                        |
+| **NPM Upgrade** | `reusable-npm-upgrade.yml` | `workflow_call` | Runs `npm-check-updates -u`, bumps the patch version, and opens a PR. Supports monorepo sub-folders via `npm_services` input.                                 |
+| **PR Verify**   | `reusable-pr-verify.yml`   | `workflow_call` | Matrix build/lint/test for multiple Node.js apps plus multi-platform Docker build. For monorepos with Docker images.                                          |
+| **Merge Build** | `reusable-merge-build.yml` | `workflow_call` | Same as PR Verify but on merge. Tags Docker images with `latest`, version, major, and minor tags.                                                             |
+
+### Inputs and Secrets
+
+#### NPM Workflows (`reusable-npm-merge`, `reusable-npm-pr`)
+
+| Input              | Required | Default | Description                                               |
+| ------------------ | -------- | ------- | --------------------------------------------------------- |
+| `node_version`     | No       | `"18"`  | Node.js version                                           |
+| `npm_package_name` | Yes      | --      | npm package name (e.g. `@devopsplaybook.io/common-utils`) |
+
+| Secret                    | Required | Description                               |
+| ------------------------- | -------- | ----------------------------------------- |
+| `NPM_TOKEN`               | Yes      | npm publish token                         |
+| `QUALITY_DASHBOARD_URL`   | No       | Quality Dashboard URL for coverage upload |
+| `QUALITY_DASHBOARD_TOKEN` | No       | Quality Dashboard upload token            |
+
+#### NPM Upgrade (`reusable-npm-upgrade`)
+
+| Input          | Required | Default                                 | Description                                                   |
+| -------------- | -------- | --------------------------------------- | ------------------------------------------------------------- |
+| `npm_services` | Yes      | --                                      | JSON array of sub-folder paths (e.g. `'["server","client"]'`) |
+| `pr_branch`    | No       | `feature/YYYY.MM.DD-dependency-updates` | Branch name for the PR                                        |
+
+#### Docker/Node Workflows (`reusable-pr-verify`, `reusable-merge-build`)
+
+| Input                  | Required | Default                      | Description                           |
+| ---------------------- | -------- | ---------------------------- | ------------------------------------- |
+| `docker_platforms`     | No       | `linux/arm64/v8,linux/amd64` | Docker build platforms                |
+| `node_app_directories` | No       | `""`                         | JSON array of Node.js app directories |
+| `node_version`         | No       | `"22"`                       | Node.js version                       |
+
+| Secret                    | Required | Description                    |
+| ------------------------- | -------- | ------------------------------ |
+| `DOCKER_HUB_USERNAME`     | Yes      | Docker Hub username            |
+| `DOCKER_HUB_ACCESS_TOKEN` | Yes      | Docker Hub access token        |
+| `QUALITY_DASHBOARD_URL`   | No       | Quality Dashboard URL          |
+| `QUALITY_DASHBOARD_TOKEN` | No       | Quality Dashboard upload token |
+
+### Adopting in Your Project
+
+Create a caller workflow in `.github/workflows/main-build.yml`:
+
+```yaml
+name: Main Build
+on:
+  push:
+    branches: ["main"]
+jobs:
+  npm-merge:
+    uses: devopsplaybook-io/common-utils/.github/workflows/reusable-npm-merge.yml@main
+    with:
+      npm_package_name: "@your-scope/your-package"
+    secrets:
+      NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
+```
+
+And `.github/workflows/pr-check.yml`:
+
+```yaml
+name: PR Check
+on:
+  pull_request:
+    branches: ["main"]
+permissions:
+  contents: read
+  pull-requests: write
+  issues: write
+jobs:
+  npm-pr:
+    uses: devopsplaybook-io/common-utils/.github/workflows/reusable-npm-pr.yml@main
+    with:
+      npm_package_name: "@your-scope/your-package"
+    secrets:
+      NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
+```
+
+And `.github/workflows/npm-upgrade.yml` for weekly dependency updates:
+
+```yaml
+name: NPM Upgrade
+on:
+  schedule:
+    - cron: "0 6 * * 1" # Monday 6am UTC
+jobs:
+  npm-upgrade:
+    uses: devopsplaybook-io/common-utils/.github/workflows/reusable-npm-upgrade.yml@main
+    permissions:
+      contents: write
+      pull-requests: write
+    with:
+      npm_services: "[]"
+```
+
+---
+
+## Development
+
+```bash
+npm install
+npm run build    # TypeScript compilation -> dist/
+npm run lint     # ESLint (strict + stylistic)
+npm run test     # Jest with coverage
+```
+
+## License
+
+ISC

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@devopsplaybook.io/common-utils",
-  "version": "1.0.0-beta.5.9c5ecde",
+  "version": "1.0.0",
   "description": "Shared utility modules for devopsplaybook.io projects (DB, Config, OTel context, system helpers)",
   "keywords": [
     "Open Telemetry",