@qlibs/apm 0.1.2 → 0.3.0

package/README.md

@@ -70,17 +70,124 @@ dependency of this package.
 
 | Variable | Default | Purpose |
 |---|---|---|
-| `APM_ENABLED` | `true` | Master switch. `false` fully disables tracing + remote log transports (console logging still works). |
+| `APM_ENABLED` | `true` | `false` disables tracing. Log transports are controlled only by `APM_LOG_TRANSPORT`. |
 | `APM_SERVICE_NAME` | `ats-be` | Service name shown in traces/logs — set per app, e.g. `my-app-api`. |
-| `APM_ENVIRONMENT` | `NODE_ENV` | `deployment.environment` resource attribute. |
+| `APM_ENVIRONMENT` | `NODE_ENV`, else `production` | `deployment.environment` resource attribute. Also drives the log level/format defaults below. Unset `NODE_ENV` is treated as `production` (never as local). |
 | `OTEL_EXPORTER_OTLP_ENDPOINT` | _(empty)_ | Base OTLP endpoint. Empty = tracing disabled. |
 | `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` | `http/protobuf` (port 4318 typically) or `grpc` (port 4317). Must match what your collector listens on. |
 | `OTEL_EXPORTER_OTLP_HEADERS` | _(empty)_ | Comma-separated `key=value` pairs, e.g. ingestion keys / auth tokens. |
-| `LOG_LEVEL` | `info` (`debug` locally) | Winston log level. |
-| `APM_LOG_FORMAT` | `pretty` locally, `json` otherwise | `pretty` = colorized console; `json` = structured stdout. |
+| `LOG_LEVEL` | `info` (`debug` when env is `local`) | Winston log level (`error`, `warn`, `info`, `http`, `verbose`, `debug`). At `info`, Nest `verbose`/`debug` logs are dropped — set `verbose` or `debug` to see them. HTTP access logs and Nest `fatal` (logged as `error` with `fatal: true`) are always visible at `info`. On the console, `error`/`warn` go to stderr and everything else to stdout. |
+| `APM_LOG_FORMAT` | `pretty` for `local`/`development`, `json` otherwise | `pretty` = colorized console; `json` = structured stdout. |
 | `APM_LOG_TRANSPORT` | `stdout` | Comma list of `stdout`, `elasticsearch`, `otlp`. Multiple can be active at once. |
-| `ELASTICSEARCH_NODE` / `_USERNAME` / `_PASSWORD` / `_INDEX` | _(empty)_ | Only used when `APM_LOG_TRANSPORT` includes `elasticsearch`. |
-| `APM_HTTP_IGNORE_PATHS` | `/health,/favicon.ico` | Comma list of path prefixes excluded from the HTTP access log. |
+| `ELASTICSEARCH_NODE` / `_USERNAME` / `_PASSWORD` / `_INDEX` | _(empty)_ | Only used when `APM_LOG_TRANSPORT` includes `elasticsearch`. While ES is unreachable at most 1000 log lines are buffered (the rest are dropped) and each is retried 5 times. Shipping via stdout + Filebeat/Fluent Bit is still preferred in production. |
+| `APM_HTTP_IGNORE_PATHS` | `/health,/favicon.ico` | Comma list of path prefixes excluded from the HTTP access log and from tracing. |
+| `OTEL_NODE_ENABLED_INSTRUMENTATIONS` | `http,undici,express,nestjs-core,pg,ioredis,redis,amqplib` | Which auto-instrumentations to load (names without the `@opentelemetry/instrumentation-` prefix). Add e.g. `mongodb`, `kafkajs` if your app needs them. |
+| `OTEL_TRACES_SAMPLER` / `OTEL_TRACES_SAMPLER_ARG` | `parentbased_always_on` | Standard OTel sampling. For high-traffic services use `parentbased_traceidratio` with e.g. `0.1` (10%). |
+
+## Example configurations
+
+`LOG_LEVEL` is a threshold: it keeps that level and everything more severe
+(`error` < `warn` < `info` < `http` < `verbose` < `debug` < `silly`). Nest's
+`logger.log()` is `info`, `logger.fatal()` is `error`. Use lowercase values
+exactly as listed — an unknown value (e.g. `WARN`, `log`, `trace`) currently
+drops **all** logs, errors included.
+
+What each setting affects:
+
+| | stdout (pm2 / docker logs) | Logs in SigNoz | Traces in SigNoz |
+|---|---|---|---|
+| `LOG_LEVEL` | ✅ | ✅ | ❌ |
+| `APM_LOG_TRANSPORT` | ✅ | ✅ (needs `otlp`) | ❌ |
+| `OTEL_TRACES_SAMPLER` | ❌ | ❌ | ✅ |
+
+Errors are still marked on the trace span by `recordException()` regardless of
+`LOG_LEVEL`. After changing env vars, restart the app (pm2:
+`pm2 restart <app> --update-env`).
+
+**1. Production, minimal — only warn + error in pm2/docker, traces to SigNoz**
+```env
+NODE_ENV=production
+APM_SERVICE_NAME=my-app-api
+LOG_LEVEL=warn
+APM_LOG_TRANSPORT=stdout
+OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz-otel-collector:4318
+```
+Note: at `warn` the HTTP access log (`GET /api/... 200`, level `info`) is
+dropped too.
+
+**2. Production, standard — info + access log in pm2/docker, logs and traces to SigNoz**
+```env
+NODE_ENV=production
+APM_SERVICE_NAME=my-app-api
+LOG_LEVEL=info
+APM_LOG_TRANSPORT=stdout,otlp
+OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz-otel-collector:4318
+```
+
+**3. Production, high traffic — trace only 10% of requests**
+```env
+NODE_ENV=production
+APM_SERVICE_NAME=my-app-api
+LOG_LEVEL=warn
+APM_LOG_TRANSPORT=stdout,otlp
+OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz-otel-collector:4318
+OTEL_TRACES_SAMPLER=parentbased_traceidratio
+OTEL_TRACES_SAMPLER_ARG=0.1
+```
+
+**4. Staging, investigating a bug — everything**
+```env
+NODE_ENV=staging
+APM_SERVICE_NAME=my-app-api
+LOG_LEVEL=debug
+APM_LOG_TRANSPORT=stdout,otlp
+OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz-otel-collector:4318
+```
+
+**5. Local development, no SigNoz**
+```env
+NODE_ENV=local
+APM_SERVICE_NAME=my-app-api
+# LOG_LEVEL defaults to debug, format to colorized pretty output
+# OTEL_EXPORTER_OTLP_ENDPOINT left empty -> tracing off
+```
+
+**6. Logs to pm2/docker only, no tracing at all**
+```env
+NODE_ENV=production
+APM_SERVICE_NAME=my-app-api
+LOG_LEVEL=warn
+APM_LOG_TRANSPORT=stdout
+# OTEL_EXPORTER_OTLP_ENDPOINT not set -> nothing is sent to SigNoz
+```
+
+**7. Collector listening on gRPC (port 4317)**
+```env
+OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz-otel-collector:4317
+OTEL_EXPORTER_OTLP_PROTOCOL=grpc
+```
+
+**8. Logs to Elasticsearch / Kibana**
+```env
+NODE_ENV=production
+APM_SERVICE_NAME=my-app-api
+LOG_LEVEL=warn
+APM_LOG_TRANSPORT=stdout,elasticsearch
+ELASTICSEARCH_NODE=https://es.internal:9200
+ELASTICSEARCH_USERNAME=elastic
+ELASTICSEARCH_PASSWORD=changeme
+```
+
+To stop shipping logs remotely, remove `otlp` / `elasticsearch` from
+`APM_LOG_TRANSPORT` — `APM_ENABLED=false` currently only turns tracing off.
+
+## Shutdown
+
+On `SIGTERM`/`SIGINT` the tracing SDK flushes pending spans (max 5s) but does
+**not** call `process.exit()` — your app's own graceful shutdown (e.g.
+`app.enableShutdownHooks()` in Nest) stays in control. If no other signal
+handler is registered, the signal is re-raised after the flush so the process
+still terminates normally.
 
 ## Pointing at a specific backend
 

package/dist/apm.config.js

@@ -29,7 +29,8 @@ function getApmConfig() {
     if (cachedConfig)
         return cachedConfig;
     const env = process.env;
-    const nodeEnv = env.NODE_ENV || 'local';
+    const nodeEnv = env.APM_ENVIRONMENT || env.NODE_ENV || 'production';
+    const isLocal = nodeEnv === 'local' || nodeEnv === 'development';
     const otlpEndpoint = env.OTEL_EXPORTER_OTLP_ENDPOINT || undefined;
     const transports = parseList(env.APM_LOG_TRANSPORT);
     cachedConfig = {
@@ -45,8 +46,7 @@ function getApmConfig() {
         },
         logging: {
             level: env.LOG_LEVEL || (nodeEnv === 'local' ? 'debug' : 'info'),
-            format: env.APM_LOG_FORMAT ||
-                (nodeEnv === 'local' || nodeEnv === 'development' ? 'pretty' : 'json'),
+            format: env.APM_LOG_FORMAT || (isLocal ? 'pretty' : 'json'),
             transports: transports.length ? transports : ['stdout'],
             elasticsearch: {
                 node: env.ELASTICSEARCH_NODE || undefined,

package/dist/filters/apm-exception.filter.js

@@ -14,7 +14,7 @@ let ApmExceptionFilter = class ApmExceptionFilter {
         (0, record_exception_1.recordException)(err, 'ApmExceptionFilter');
         const ctx = host.switchToHttp();
         const response = ctx.getResponse();
-        if (!(response === null || response === void 0 ? void 0 : response.status))
+        if (!(response === null || response === void 0 ? void 0 : response.status) || response.headersSent)
             return;
         const status = err instanceof common_1.HttpException ? err.getStatus() : 500;
         const body = err instanceof common_1.HttpException

package/dist/http/apm-logging.interceptor.js

@@ -30,7 +30,7 @@ let ApmLoggingInterceptor = class ApmLoggingInterceptor {
         return next.handle().pipe((0, rxjs_1.tap)({
             next: () => {
                 const response = context.switchToHttp().getResponse();
-                winston_instance_1.winstonLogger.http(`${method} ${path} ${response.statusCode}`, {
+                winston_instance_1.winstonLogger.info(`${method} ${path} ${response.statusCode}`, {
                     context: 'HTTP',
                     method,
                     path,
@@ -41,7 +41,7 @@ let ApmLoggingInterceptor = class ApmLoggingInterceptor {
                 });
             },
             error: (err) => {
-                winston_instance_1.winstonLogger.http(`${method} ${path} failed`, {
+                winston_instance_1.winstonLogger.info(`${method} ${path} failed`, {
                     context: 'HTTP',
                     method,
                     path,

package/dist/logger/apm-logger.service.d.ts

@@ -4,6 +4,7 @@ export declare class ApmLoggerService implements LoggerService {
     error(message: any, trace?: string, context?: string): void;
     warn(message: any, context?: string): void;
     debug(message: any, context?: string): void;
+    fatal(message: any, context?: string): void;
     verbose(message: any, context?: string): void;
     private stringify;
 }

package/dist/logger/apm-logger.service.js

@@ -22,6 +22,9 @@ let ApmLoggerService = class ApmLoggerService {
     debug(message, context) {
         winston_instance_1.winstonLogger.debug(this.stringify(message), { context });
     }
+    fatal(message, context) {
+        winston_instance_1.winstonLogger.error(this.stringify(message), { context, fatal: true });
+    }
     verbose(message, context) {
         winston_instance_1.winstonLogger.verbose(this.stringify(message), { context });
     }

package/dist/logger/otlp-log.transport.js

@@ -28,25 +28,33 @@ const SEVERITY_MAP = {
     silly: { number: 1, text: 'TRACE' },
 };
 let otelLogger;
+let emitFailureReported = false;
 function getOtelLogger() {
     if (otelLogger)
         return otelLogger;
     const config = (0, apm_config_1.getApmConfig)();
     const { LoggerProvider, BatchLogRecordProcessor, } = require('@opentelemetry/sdk-logs');
-    const { OTLPLogExporter } = require('@opentelemetry/exporter-logs-otlp-http');
     const { resourceFromAttributes } = require('@opentelemetry/resources');
     const { ATTR_SERVICE_NAME } = require('@opentelemetry/semantic-conventions');
-    const base = (config.tracing.otlpEndpoint || '').replace(/\/$/, '');
-    const exporter = new OTLPLogExporter({
-        url: `${base}/v1/logs`,
-        headers: config.tracing.headers,
-    });
+    let exporter;
+    if (config.tracing.protocol === 'grpc') {
+        const { OTLPLogExporter, } = require('@opentelemetry/exporter-logs-otlp-grpc');
+        exporter = new OTLPLogExporter({ url: config.tracing.otlpEndpoint });
+    }
+    else {
+        const { OTLPLogExporter, } = require('@opentelemetry/exporter-logs-otlp-http');
+        const base = (config.tracing.otlpEndpoint || '').replace(/\/$/, '');
+        exporter = new OTLPLogExporter({
+            url: `${base}/v1/logs`,
+            headers: config.tracing.headers,
+        });
+    }
     const provider = new LoggerProvider({
         resource: resourceFromAttributes({
             [ATTR_SERVICE_NAME]: config.serviceName,
         }),
+        processors: [new BatchLogRecordProcessor({ exporter })],
     });
-    provider.addLogRecordProcessor(new BatchLogRecordProcessor(exporter));
     otelLogger = provider.getLogger(config.serviceName);
     return otelLogger;
 }
@@ -64,7 +72,11 @@ class OtlpLogTransport extends winston_transport_1.default {
                 context: api_1.context.active(),
             });
         }
-        catch (_a) {
+        catch (err) {
+            if (!emitFailureReported) {
+                emitFailureReported = true;
+                console.error('[apm] otlp log transport failed, logs are not being exported:', err instanceof Error ? err.message : err);
+            }
         }
         callback();
     }

package/dist/logger/winston.instance.js

@@ -61,7 +61,10 @@ function buildConsoleTransport() {
             return `${timestamp} ${level}${ctx} ${message}${trace}${extra}`;
         }))
         : winston.format.combine((0, trace_context_format_1.traceContextFormat)(), winston.format.timestamp(), winston.format.json());
-    return new winston.transports.Console({ format });
+    return new winston.transports.Console({
+        format,
+        stderrLevels: ['error', 'warn'],
+    });
 }
 function buildElasticsearchTransport() {
     const config = (0, apm_config_1.getApmConfig)();
@@ -73,6 +76,8 @@ function buildElasticsearchTransport() {
     return new ElasticsearchTransport({
         level: config.logging.level,
         indexPrefix: config.logging.elasticsearch.index,
+        bufferLimit: 1000,
+        retryLimit: 5,
         clientOpts: {
             node: config.logging.elasticsearch.node,
             auth: config.logging.elasticsearch.username &&
@@ -122,3 +127,6 @@ exports.winstonLogger = winston.createLogger({
     defaultMeta: { service: config.serviceName, environment: config.environment },
     transports: buildTransports(),
 });
+exports.winstonLogger.on('error', (err) => {
+    console.error('[apm] log transport error:', err instanceof Error ? err.message : err);
+});

package/dist/tracing.js

@@ -3,17 +3,16 @@ Object.defineProperty(exports, "__esModule", { value: true });
 const apm_config_1 = require("./apm.config");
 const config = (0, apm_config_1.getApmConfig)();
 if (config.enabled && config.tracing.enabled && config.tracing.otlpEndpoint) {
+    if (!process.env.OTEL_NODE_ENABLED_INSTRUMENTATIONS) {
+        process.env.OTEL_NODE_ENABLED_INSTRUMENTATIONS =
+            'http,undici,express,nestjs-core,pg,ioredis,redis,amqplib';
+    }
     const { NodeSDK } = require('@opentelemetry/sdk-node');
     const { getNodeAutoInstrumentations, } = require('@opentelemetry/auto-instrumentations-node');
-    const { OTLPTraceExporter, } = require('@opentelemetry/exporter-trace-otlp-http');
     const { resourceFromAttributes } = require('@opentelemetry/resources');
     const { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION, } = require('@opentelemetry/semantic-conventions');
-    const traceExporter = new OTLPTraceExporter({
-        url: config.tracing.protocol === 'grpc'
-            ? config.tracing.otlpEndpoint
-            : `${config.tracing.otlpEndpoint.replace(/\/$/, '')}/v1/traces`,
-        headers: config.tracing.headers,
-    });
+    const traceExporter = createTraceExporter();
+    const ignorePaths = config.http.ignorePaths;
     const sdk = new NodeSDK({
         resource: resourceFromAttributes({
             [ATTR_SERVICE_NAME]: config.serviceName,
@@ -24,20 +23,42 @@ if (config.enabled && config.tracing.enabled && config.tracing.otlpEndpoint) {
         instrumentations: [
             getNodeAutoInstrumentations({
                 '@opentelemetry/instrumentation-fs': { enabled: false },
+                '@opentelemetry/instrumentation-http': {
+                    ignoreIncomingRequestHook: (req) => ignorePaths.some((p) => { var _a; return (_a = req.url) === null || _a === void 0 ? void 0 : _a.startsWith(p); }),
+                },
             }),
         ],
     });
     sdk.start();
-    const shutdown = () => {
-        sdk
-            .shutdown()
-            .catch((err) => console.error('Error shutting down OTel SDK', err))
-            .finally(() => process.exit(0));
+    const SHUTDOWN_TIMEOUT_MS = 5000;
+    const onSignal = (signal) => {
+        Promise.race([
+            sdk.shutdown(),
+            new Promise((resolve) => setTimeout(resolve, SHUTDOWN_TIMEOUT_MS).unref()),
+        ])
+            .catch((err) => console.error('[apm] error shutting down OTel SDK', err))
+            .finally(() => {
+            if (process.listenerCount(signal) === 0) {
+                process.kill(process.pid, signal);
+            }
+        });
     };
-    process.on('SIGTERM', shutdown);
-    process.on('SIGINT', shutdown);
-    console.info(`[apm] tracing enabled -> ${config.tracing.otlpEndpoint} (service: ${config.serviceName})`);
+    process.once('SIGTERM', onSignal);
+    process.once('SIGINT', onSignal);
+    console.info(`[apm] tracing enabled -> ${config.tracing.otlpEndpoint} (${config.tracing.protocol}, service: ${config.serviceName})`);
 }
 else {
     console.info('[apm] tracing disabled (set OTEL_EXPORTER_OTLP_ENDPOINT to enable)');
 }
+function createTraceExporter() {
+    const { otlpEndpoint, protocol, headers } = config.tracing;
+    if (protocol === 'grpc') {
+        const { OTLPTraceExporter, } = require('@opentelemetry/exporter-trace-otlp-grpc');
+        return new OTLPTraceExporter({ url: otlpEndpoint });
+    }
+    const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
+    return new OTLPTraceExporter({
+        url: `${otlpEndpoint.replace(/\/$/, '')}/v1/traces`,
+        headers,
+    });
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@qlibs/apm",
-  "version": "0.1.2",
+  "version": "0.3.0",
   "description": "Vendor-neutral APM for NestJS: OpenTelemetry tracing (OTLP) + structured logging + error monitoring. Works with SigNoz, Elastic/Kibana, Jaeger, or any OTel collector.",
   "author": "QBIT Developer",
   "license": "MIT",
@@ -28,7 +28,9 @@
     "@opentelemetry/api": "^1.9.1",
     "@opentelemetry/api-logs": "^0.221.0",
     "@opentelemetry/auto-instrumentations-node": "^0.79.0",
+    "@opentelemetry/exporter-logs-otlp-grpc": "^0.221.0",
     "@opentelemetry/exporter-logs-otlp-http": "^0.221.0",
+    "@opentelemetry/exporter-trace-otlp-grpc": "^0.221.0",
     "@opentelemetry/exporter-trace-otlp-http": "^0.221.0",
     "@opentelemetry/resources": "^2.10.0",
     "@opentelemetry/sdk-logs": "^0.221.0",