@ciq-dev/neoiq-foundation-node 1.1.2-beta.0 → 1.1.2-beta.10

package/README.md

@@ -1,295 +1,136 @@
 # @ciq-dev/neoiq-foundation-node
 
-Node.js observability foundation for CommerceIQ services. Integrates with **Groundcover** via **OpenTelemetry**.
+Node.js observability foundation for CommerceIQ services — OpenTelemetry + Groundcover.
 
 ```
-┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
-│   Our Service   │────▶│  OTEL Collector │────▶│   Groundcover   │
-│  (auth-service) │     │   (in-cluster)  │     │   (dashboard)   │
-└─────────────────┘     └─────────────────┘     └─────────────────┘
-        │                        │
-        ├── Traces (OTLP) ───────┤
-        └── Metrics (OTLP) ──────┘
-
-        │
-        └── Logs (stdout) ──────▶ Groundcover Agent scrapes container logs
+Service → OTEL Collector → Groundcover
+  ├── Traces (OTLP gRPC)
+  ├── Metrics (OTLP gRPC)
+  └── Logs (stdout → scraped by Groundcover Agent)
 ```
 
 ## Features
 
-- **Traces**: Automatic span creation for HTTP requests (incoming & outgoing)
-- **Metrics**: HTTP request counts, durations, errors + custom business metrics
-- **Logs**: Structured JSON logs with automatic trace context (traceId, spanId, correlationId)
-- **Selective Enablement**: Enable only the features you need
-- **Type-Safe Configuration**: Zod validation with helpful error messages
-- **Fastify Plugin**: One-line integration for request lifecycle
-- **HTTP Client**: Axios wrapper with retry, circuit breaker, and trace propagation
-- **Object Store Adapter**: Provider-agnostic wrapper for cloud object storage (e.g. S3) to avoid direct SDK coupling
+- Distributed tracing, metrics, structured logs via OpenTelemetry + Pino
+- Fastify plugin for automatic request lifecycle instrumentation
+- Axios HTTP client with retry, circuit breaker, and trace propagation
+- AsyncLocalStorage context manager (correlationId across async calls)
+- HashiCorp Vault secrets client
+- Provider-agnostic object store (S3 + in-memory)
+- Zod-validated config with helpful error messages
 
 ---
 
-## Quick Start
-
-### 1. Install
+## Install
 
 ```bash
 npm install @ciq-dev/neoiq-foundation-node
 
-# Peer dependencies
-npm install zod @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/sdk-metrics \
+# Peer deps
+npm install zod pino pino-pretty axios axios-retry opossum fastify fastify-plugin \
+  @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/sdk-metrics \
   @opentelemetry/exporter-trace-otlp-grpc @opentelemetry/exporter-metrics-otlp-grpc \
-  @opentelemetry/auto-instrumentations-node @opentelemetry/resources @opentelemetry/semantic-conventions \
-  pino pino-pretty axios axios-retry opossum fastify-plugin
+  @opentelemetry/auto-instrumentations-node @opentelemetry/resources \
+  @opentelemetry/semantic-conventions
 
-# Optional (only if you want AWS S3 implementation of ObjectStore)
+# Optional — AWS S3 object store
 npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner
 ```
 
-### 2. Initialize (First Line of the App)
+---
+
+## Quick Start
 
 ```typescript
+// src/foundation.ts — import this FIRST in the app
 import { createFoundation } from '@ciq-dev/neoiq-foundation-node';
 
-// Create foundation instance (must be called BEFORE other imports)
-const foundation = createFoundation({
-  serviceName: 'auth-service',
-  serviceVersion: '1.0.0',
+export const foundation = createFoundation({
+  serviceName: 'my-service',
+  serviceVersion: '1.0.0',        // default: $SERVICE_VERSION or '1.0.0'
+  environment: 'production',       // 'development' | 'staging' | 'qa' | 'production'
 });
-
-// Export for use in other files
-export { foundation };
 ```
 
-### 3. Add Fastify Plugin
-
 ```typescript
+// src/index.ts
 import Fastify from 'fastify';
 import { foundation } from './foundation';
 
 const app = Fastify();
-app.register(foundation.fastifyPlugin);
-```
-
-### 4. Set Environment Variable in Kubernetes
-
-```yaml
-env:
-  - name: OTEL_EXPORTER_OTLP_ENDPOINT
-    value: 'http://otel-stack-deployment-collector.observability.svc.cluster.local:4317'
-```
-
-That's it! Traces and metrics will flow to Groundcover.
-
----
-
-## Selective Feature Enablement
+app.register(foundation.fastifyPlugin); // automatic request tracing/metrics/logging
 
-Enable only the features you need for lightweight deployments:
-
-```typescript
-import { createFoundation } from '@ciq-dev/neoiq-foundation-node';
-
-// Full observability (default)
-const foundation = createFoundation({
-  serviceName: 'auth-service',
-});
-
-// Logging only (lightweight mode)
-const foundation = createFoundation({
-  serviceName: 'simple-worker',
-  features: {
-    tracing: false,
-    metrics: false,
-    logging: true,
-  },
+process.on('SIGTERM', async () => {
+  await app.close();
+  await foundation.lifecycle.shutdown();
 });
+```
 
-// Custom auto-instrumentation
-const foundation = createFoundation({
-  serviceName: 'api-gateway',
-  features: {
-    tracing: true,
-    metrics: true,
-    autoInstrumentation: {
-      http: true,
-      fastify: true,
-      mongodb: false,  // Not using MongoDB
-      redis: false,    // Not using Redis
-      pg: true,        // Using PostgreSQL
-    },
-  },
-});
+```yaml
+# Kubernetes env
+- name: OTEL_EXPORTER_OTLP_ENDPOINT
+  value: 'http://otel-stack-deployment-collector.observability.svc.cluster.local:4317'
 ```
 
 ---
 
-## API Reference
+## API
 
-### `createFoundation(options)`
+The foundation instance exposes top-level shortcuts for the most common operations and sub-modules for everything else.
 
-Create a foundation instance with full observability capabilities.
+### Top-level shortcuts
 
 ```typescript
-import { createFoundation } from '@ciq-dev/neoiq-foundation-node';
-
-const foundation = createFoundation({
-  // Required
-  serviceName: 'my-service',
-  
-  // Optional
-  serviceVersion: '1.0.0',        // Default: '1.0.0' or $SERVICE_VERSION
-  environment: 'production',       // Default: 'development' or $NODE_ENV
-  
-  // Feature toggles
-  features: {
-    tracing: true,                 // Enable distributed tracing
-    metrics: true,                 // Enable metrics collection
-    logging: true,                 // Enable structured logging
-    autoInstrumentation: {         // Fine-grained control
-      http: true,
-      fastify: true,
-      express: true,
-      mongodb: true,
-      pg: true,
-      mysql: true,
-      redis: true,
-      ioredis: true,
-      fs: false,                   // Disabled by default (noisy)
-      dns: false,                  // Disabled by default
-    },
-  },
-  
-  // OTEL configuration
-  otel: {
-    endpoint: 'http://...:4317',   // Default: cluster OTEL Collector
-    metricsIntervalMs: 5000,       // Default: 5000 (5 seconds)
-    traceSampleRate: 1.0,          // Default: 1.0 (100%)
-  },
-  
-  // Logging configuration
-  logging: {
-    level: 'info',                 // 'debug' | 'info' | 'warn' | 'error'
-    prettyPrint: false,            // Auto-detect from environment
-  },
-});
+foundation.logger        // Pino logger with trace context auto-injected
+foundation.context       // AsyncLocalStorage context manager
+foundation.fastifyPlugin // Fastify plugin (register directly)
+foundation.features      // { tracing: boolean, metrics: boolean, logging: boolean }
+foundation.config        // Full resolved config
 ```
 
-### Foundation Instance
-
-The foundation instance provides access to all observability features:
+### `foundation.observability`
 
 ```typescript
-// Logging
+// Structured logging — logs include traceId, spanId, correlationId automatically
 foundation.logger.info({ userId: '123' }, 'User logged in');
-foundation.logger.error({ error: err.message }, 'Failed to process');
+const routeLogger = foundation.logger.child({ component: 'auth' });
 
-// Metrics
-const meter = foundation.getMeter('my-service');
-const counter = meter.createCounter('logins.total');
-counter.add(1, { provider: 'workos' });
+// Custom metrics
+const meter = foundation.observability.getMeter('my-service');
+const loginCounter = meter.createCounter('auth.logins.total');
+loginCounter.add(1, { provider: 'workos' });
 
-// Tracing
-const tracer = foundation.getTracer();
+const latency = meter.createHistogram('auth.token.duration');
+latency.record(42, { status: 'success' });
+
+// Custom spans
+const tracer = foundation.observability.getTracer();
 tracer.startActiveSpan('validateToken', (span) => {
-  // ... do work
+  // ... work
   span.end();
 });
 
-// HTTP Client
-const client = foundation.createHttpClient({
-  baseURL: 'http://user-service:3000',
-  serviceName: 'user-service',
+// Convenience: span with auto error handling
+await foundation.observability.trace('validateToken', async () => {
+  // ... work
 });
 
-// Fastify Plugin
-app.register(foundation.fastifyPlugin);
-
-// Shutdown
-await foundation.shutdown();
-
-// Check features
-console.log(foundation.features);
-// { tracing: true, metrics: true, logging: true }
+// Get current trace context
+const { traceId, spanId } = foundation.observability.getTraceContext();
 ```
 
-### `foundation.logger`
-
-Structured logger with automatic trace context injection.
+### `foundation.http`
 
 ```typescript
-foundation.logger.info({ userId: '123' }, 'User logged in');
-foundation.logger.error({ error: err.message }, 'Failed to process');
-foundation.logger.warn({ attempt: 3 }, 'Retry attempt');
-foundation.logger.debug({ payload: data }, 'Processing request');
-
-// Child logger for component-specific logging
-const authLogger = foundation.logger.child({ component: 'auth' });
-authLogger.info({}, 'Auth module initialized');
-```
-
-**Log Output:**
-```json
-{
-  "level": "info",
-  "time": 1703318400000,
-  "service": "auth-service",
-  "traceId": "abc123...",
-  "spanId": "def456...",
-  "correlationId": "req-789",
-  "userId": "123",
-  "msg": "User logged in"
-}
-```
-
-### `foundation.getMeter(name, version?)`
-
-Get a meter for custom metrics.
-
-```typescript
-const meter = foundation.getMeter('auth-service');
-
-// Counter
-const counter = meter.createCounter('logins.total');
-counter.add(1, { provider: 'workos' });
-
-// Histogram
-const histogram = meter.createHistogram('token.validation.duration');
-histogram.record(150, { status: 'success' });
-
-// Observable Gauge
-meter.createObservableGauge('active_sessions', {}, (result) => {
-  result.observe(getActiveSessions());
-});
-```
-
-### `foundation.fastifyPlugin`
-
-Fastify plugin for automatic request handling.
-
-```typescript
-app.register(foundation.fastifyPlugin);
-```
-
-**Automatically:**
-- Extracts/generates correlation ID (x-request-id header)
-- Creates OpenTelemetry spans for each request
-- Logs request received/completed with full context
-- Records HTTP metrics (http.server.requests.total, http.server.request.duration)
-- Skips health check endpoints (/health, /healthz, /ready, /live)
-
-### `foundation.createHttpClient(options)`
-
-Create an Axios client with full observability.
-
-```typescript
-const userClient = foundation.createHttpClient({
+const userClient = foundation.http.createClient({
   baseURL: 'http://user-service:3000',
   serviceName: 'user-service',
   timeout: 10000,
   retry: {
     retries: 3,
     retryDelay: 1000,
-    retryStatusCodes: [408, 429, 500, 502, 503, 504],
+    retryStatusCodes: [429, 500, 502, 503, 504],
+    retryNonIdempotent: false,  // default: don't retry POST/PATCH
   },
   circuitBreaker: {
     enabled: true,
@@ -298,192 +139,200 @@ const userClient = foundation.createHttpClient({
   },
 });
 
-const response = await userClient.get('/api/users/123');
+const { data } = await userClient.get('/api/users/123');
+// Trace context (traceparent) and correlationId propagated automatically
 ```
 
-**Automatically:**
-- Propagates trace context (traceparent header)
-- Propagates correlation ID (x-request-id header)
-- Logs outbound requests/responses
-- Records HTTP client metrics
-- Retries on 5xx errors with exponential backoff
-- Circuit breaker protection
-
-### `foundation.shutdown()`
-
-Gracefully shutdown OTEL providers. Call on application shutdown.
+### `foundation.lifecycle`
 
 ```typescript
-process.on('SIGTERM', async () => {
-  await app.close();
-  await foundation.shutdown(); // Flush telemetry
-});
-```
+// Graceful shutdown — flushes OTEL providers
+await foundation.lifecycle.shutdown();
 
----
+// Readiness check — false if providers are still initializing
+foundation.lifecycle.isReady();
 
-## Object Store Adapter (S3 / Cloud Abstraction)
+// Structured health status
+const health = foundation.lifecycle.health();
+// { status: 'healthy'|'degraded'|'unhealthy', components: { tracing, metrics, logging } }
 
-Use this when you want to **decouple application code** from `aws-sdk` / cloud vendor SDKs.
+// Run with error capture (useful for background tasks)
+const result = await foundation.lifecycle.safeRun(
+  () => riskyOperation(),
+  defaultValue,  // returned on error
+);
+```
 
-### AWS S3 (optional peer deps)
+### `foundation.secrets` (Vault)
 
 ```typescript
-import { AwsS3ObjectStore } from '@ciq-dev/neoiq-foundation-node';
-
-const store = new AwsS3ObjectStore({
-  clientOptions: { region: process.env.AWS_REGION || 'us-east-1' },
+// Enable in config:
+const foundation = createFoundation({
+  serviceName: 'my-service',
+  vault: {
+    enabled: true,
+    address: 'https://vault.beta.commerceiq.ai',
+    authMethod: 'kubernetes',  // or 'token'
+    role: 'my-service',
+  },
 });
 
-const ref = { bucket: 'my-bucket', key: 'path/to/file.json' };
-
-// Upload
-await store.putObject(ref, JSON.stringify({ ok: true }), { contentType: 'application/json' });
-
-// Presigned PUT URL (browser upload)
-const uploadUrl = await store.presignPutObject(ref, { expiresInSeconds: 60, contentType: 'application/json' });
+// Use:
+const secrets = foundation.secrets!;
+const creds = await secrets.getSecret('path/to/secret');
+const apiKey = await secrets.getSecretValue('path/to/secret', 'API_KEY');
 ```
 
-### In-memory (local/tests)
+---
+
+## Full Config Reference
 
 ```typescript
-import { InMemoryObjectStore } from '@ciq-dev/neoiq-foundation-node';
+createFoundation({
+  serviceName: 'my-service',         // required
+  serviceVersion: '1.0.0',           // default: $SERVICE_VERSION
+  environment: 'development',         // 'development'|'staging'|'qa'|'production'
 
-const store = new InMemoryObjectStore();
-await store.putObject({ bucket: 'b', key: 'k' }, 'hello');
-const obj = await store.getObject({ bucket: 'b', key: 'k' });
-```
+  features: {
+    tracing: true,
+    metrics: true,
+    logging: true,
+    autoInstrumentation: {
+      http: true, fastify: true, express: true,
+      mongodb: true, pg: true, mysql: true,
+      redis: true, ioredis: true, grpc: true,
+      fs: false,   // disabled by default (noisy)
+      dns: false,
+    },
+  },
 
----
+  otel: {
+    endpoint: 'http://...:4317',   // default: $OTEL_EXPORTER_OTLP_ENDPOINT or cluster URL
+    metricsIntervalMs: 5000,
+  },
 
-## Complete Example
+  logging: {
+    level: 'info',          // 'debug'|'info'|'warn'|'error'
+    prettyPrint: false,     // default: true in development
+  },
 
-```typescript
-// src/foundation.ts
-import { createFoundation } from '@ciq-dev/neoiq-foundation-node';
+  requestLogging: {
+    logHeaders: true,
+    logBody: false,
+    logResponseBody: false,
+    maxBodySize: 10240,
+    redactHeaders: ['authorization', 'cookie'],
+  },
 
-export const foundation = createFoundation({
-  serviceName: 'auth-service',
-  environment: process.env.NODE_ENV as 'development' | 'production',
+  redaction: {
+    additionalPaths: ['req.body.password', 'res.body.token'],
+  },
+
+  shutdown: {
+    flushOnCrash: false,    // flush telemetry on uncaughtException/unhandledRejection
+    flushTimeoutMs: 5000,
+  },
+
+  vault: {
+    enabled: false,
+    address: 'https://vault.beta.commerceiq.ai',
+    authMethod: 'kubernetes',
+    role: 'my-service',
+    mountPoint: 'secret',
+    tokenPath: '/var/run/secrets/kubernetes.io/serviceaccount/token',
+    k8sAuthMount: 'kubernetes',
+  },
 });
 ```
 
-```typescript
-// src/index.ts
-import Fastify from 'fastify';
-import { foundation } from './foundation';
+---
 
-const app = Fastify();
+## Object Store
 
-// Register observability plugin
-app.register(foundation.fastifyPlugin);
+Provider-agnostic abstraction over cloud object storage.
 
-// Create HTTP clients
-const userClient = foundation.createHttpClient({
-  baseURL: process.env.USER_SERVICE_URL!,
-  serviceName: 'user-service',
-});
+```typescript
+import { AwsS3ObjectStore, InMemoryObjectStore } from '@ciq-dev/neoiq-foundation-node';
 
-// Custom metrics
-const meter = foundation.getMeter('auth-service');
-const loginCounter = meter.createCounter('auth.logins.total');
+// Production
+const store = new AwsS3ObjectStore({ clientOptions: { region: 'us-east-1' } });
 
-// Routes
-app.post('/api/v1/auth/login', async (req, reply) => {
-  const { provider } = req.body as { provider: string };
-  
-  loginCounter.add(1, { provider });
-  foundation.logger.info({ provider }, 'Login attempt');
-  
-  // Call user service (trace context automatically propagated)
-  const { data: user } = await userClient.get('/api/users/me');
-  
-  return { success: true, user };
-});
+// Tests
+const store = new InMemoryObjectStore();
 
-// Start with graceful shutdown
-app.listen({ port: 3000 });
+const ref = { bucket: 'my-bucket', key: 'path/to/file.json' };
+await store.putObject(ref, JSON.stringify(data), { contentType: 'application/json' });
+const obj = await store.getObject(ref);
 
-process.on('SIGTERM', async () => {
-  await app.close();
-  await foundation.shutdown();
-});
+// Presigned upload URL
+const url = await store.presignPutObject(ref, { expiresInSeconds: 60, contentType: 'application/json' });
 ```
 
 ---
 
-## Environment Variables
+## Fastify Plugin — What it does
 
-| Variable | Description | Default |
-|----------|-------------|---------|
-| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTEL Collector URL | `http://otel-stack-deployment-collector.observability.svc.cluster.local:4317` |
-| `SERVICE_VERSION` | Service version | `1.0.0` |
-| `NODE_ENV` | Environment | `development` |
-| `LOG_LEVEL` | Log level | `info` |
+`foundation.fastifyPlugin` automatically:
+- Extracts/generates `correlationId` from `x-request-id`
+- Creates an OTEL span per request
+- Logs `request received` / `request completed` with full context
+- Records `http.server.requests.total` and `http.server.request.duration` metrics
+- Skips `/health`, `/healthz`, `/ready`, `/live`
 
 ---
 
-## What Gets Sent to Groundcover
-
-### Traces
-- Every HTTP request (incoming and outgoing)
-- Correlation ID linking requests across services
-- Span attributes: method, url, status_code, duration
+## Metrics Reference
 
-### Metrics
-- `http.server.requests.total` - Incoming request count
-- `http.server.request.duration` - Incoming request latency
-- `http.server.requests.errors` - Incoming request errors
-- `http.client.requests.total` - Outgoing request count
-- `http.client.request.duration` - Outgoing request latency
-- Custom business metrics you define
-
-### Logs (via stdout)
-- Structured JSON logs written to stdout (Pino)
-- Include traceId, spanId, correlationId for correlation
-- Groundcover Agent scrapes container logs
-- Automatically correlated with traces in Groundcover dashboard
+| Metric | Description |
+|--------|-------------|
+| `http.server.requests.total` | Incoming request count |
+| `http.server.request.duration` | Incoming request latency (histogram) |
+| `http.server.requests.errors` | Incoming request errors |
+| `http.client.requests.total` | Outgoing request count |
+| `http.client.request.duration` | Outgoing request latency (histogram) |
 
 ---
 
-## Project Structure
+## Environment Variables
 
-```
-src/
-├── index.ts                    # Main exports
-├── config.ts                   # Zod configuration schemas
-├── foundation.ts               # createFoundation() entry point
-├── features/
-│   ├── context.ts              # AsyncLocalStorage context manager
-│   ├── logging.ts              # Pino logger setup
-│   ├── tracing.ts              # OTEL trace provider
-│   └── metrics.ts              # OTEL meter provider
-└── integrations/
-    ├── fastify-plugin.ts       # Fastify observability plugin
-    ├── http-client.ts          # Axios wrapper with observability
-    └── object-store/           # Cloud object store abstraction (S3, etc.)
-```
+| Variable | Default |
+|----------|---------|
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://otel-stack-deployment-collector.observability.svc.cluster.local:4317` |
+| `SERVICE_VERSION` | `1.0.0` |
+| `NODE_ENV` | `development` |
+| `LOG_LEVEL` | `info` |
 
 ---
 
 ## Development
 
 ```bash
-# Install dependencies
-npm install
-
-# Build
-npm run build
-
-# Type check
-npm run typecheck
+npm run build          # tsdown: CJS + ESM dual output → dist/
+npm run typecheck      # tsc --noEmit
+npm run lint           # ESLint
+npm run prettier:write # Prettier format
+npm test               # Vitest
+npm run test:watch     # Vitest watch mode
+```
 
-# Lint
-npm run lint
+---
 
-# Format
-npm run prettier:write
-```
+## Deprecated API
+
+The following flat methods on `foundation` still work but emit a deprecation warning. Migrate to sub-modules:
+
+| Deprecated | Use instead |
+|------------|-------------|
+| `foundation.getMeter()` | `foundation.observability.getMeter()` |
+| `foundation.getTracer()` | `foundation.observability.getTracer()` |
+| `foundation.getTraceContext()` | `foundation.observability.getTraceContext()` |
+| `foundation.getActiveSpan()` | `foundation.observability.getActiveSpan()` |
+| `foundation.trace()` | `foundation.observability.trace()` |
+| `foundation.createHttpClient()` | `foundation.http.createClient()` |
+| `foundation.shutdown()` | `foundation.lifecycle.shutdown()` |
+| `foundation.isReady()` | `foundation.lifecycle.isReady()` |
+| `foundation.health()` | `foundation.lifecycle.health()` |
+| `foundation.safeRun()` | `foundation.lifecycle.safeRun()` |
 
 ---