ntlogger 2.7.0 → 2.8.2

package/README.md

@@ -11,6 +11,13 @@ NightTimeLogger is a custom logging wrapper built on top of the Winston logging
 - Custom session ID generation for tracking log sessions.
 - Support for both file and console log formatters.
 - Ability to configure log levels and formats to suit specific requirements.
+- Native call site path reporting - automatically capture file path, line number, and call chain where each log statement is executed.
+- **Child loggers** - Create contextual loggers with persistent metadata (perfect for multi-threaded applications).
+- **Log sampling and rate limiting** - Reduce console spam with configurable sampling rates and rate limits per log level.
+- **Log deduplication** - Automatically group and squish duplicate log messages (e.g., "Ticket already exists (x11)").
+- **Performance metrics** - Development-only performance tracking with `time()` and `timeEnd()` methods.
+- **Non-blocking operations** - All logging operations are asynchronous and won't block your application.
+- **TypeScript support** - Full TypeScript definitions included for excellent IDE integration (IntelliJ, VS Code, etc.).
 
 ## Installation
 
@@ -59,6 +66,143 @@ Check out [Full Configuration](https://github.com/NightSquawk/NightTimeLogger/bl
 - `maxFiles`: The maximum number of log files to retain (rotating file strategy).
 - `timestamp`: Whether to include timestamps in log messages. Defaults to `true`.
 - `debug`: Whether to enable debug mode, which logs internal messages. Defaults to `false`.
+- `reportPath`: Whether to enable call site path reporting. When enabled, automatically captures the file path, line number, column number, and call chain where each log statement is executed. The path is added as metadata (JSON field `filePath`), not in the formatted message string. Defaults to `false`.
+- `sampling`: Object with level-based sampling rates (e.g., `{ debug: 0.01, trace: 0.001 }`). Values between 0.0 and 1.0, where 1.0 = log all, 0.1 = log 10%. Defaults to `{}` (no sampling).
+- `rateLimit`: Object with level-based rate limits (e.g., `{ error: { max: 10, window: 60000 } }`). Prevents console spam by limiting logs per level within a time window. Defaults to `{}` (no rate limiting).
+- `deduplication`: Object with `{ enabled: boolean, threshold: number, window: number }`. Groups similar log messages and squishes duplicates (e.g., "Ticket already exists (x11)"). Defaults to `{ enabled: false, threshold: 3, window: 60000 }`.
+- `performanceMetrics`: Enable performance tracking (default: `NODE_ENV === 'development'`). Adds `time()` and `timeEnd()` methods.
+- `statsInterval`: Interval in milliseconds to automatically log statistics (default: `0` = disabled). Useful for tuning sampling/rate limiting parameters.
+
+### Call Site Path Reporting
+
+When `reportPath` is enabled, each log entry includes a `filePath` field in its metadata showing where the log was called:
+
+```javascript
+const logger = require('ntlogger');
+
+const log = logger('MyApp', {
+    reportPath: true
+});
+
+log.info('User logged in'); // filePath will show: "./src/routes/auth.js:45:12 [handleLogin ← router.post]"
+```
+
+The `filePath` field appears in:
+- JSON metadata (for plugins like OpenObserve)
+- Console output (appended to location)
+- File logs (appended to location)
+
+**Smart Internal Function Filtering**: The logger automatically filters out internal Node.js functions (like `_onTimeout`, `listOnTimeout`, `setImmediate`, etc.) from the call chain, ensuring you only see your actual application code. This works even when logs are called from within `setTimeout`, `setImmediate`, or Promise callbacks.
+
+Note: The `location` field represents the logger instance name, while `filePath` shows the actual call site where the log statement was written.
+
+### Child Loggers
+
+Create child loggers with persistent context that's automatically merged into all log entries. Perfect for request-scoped logging or multi-threaded applications:
+
+```javascript
+const logger = require('ntlogger');
+
+const log = logger('MyApp');
+
+// Create a child logger with context
+const requestLogger = log.child({ 
+    requestId: 'abc123', 
+    userId: 456,
+    endpoint: '/api/users' 
+});
+
+// All logs from requestLogger automatically include the context
+requestLogger.info('Processing request'); 
+// Logs: { requestId: 'abc123', userId: 456, endpoint: '/api/users', message: 'Processing request' }
+
+// Support nested children
+const operationLogger = requestLogger.child({ operation: 'validate' });
+operationLogger.debug('Validating input'); // Includes all parent context
+```
+
+### Log Sampling and Rate Limiting
+
+Reduce console spam with configurable sampling and rate limiting:
+
+```javascript
+const log = logger('MyApp', {
+    // Sample 1% of debug logs, 0.1% of trace logs
+    sampling: {
+        debug: 0.01,
+        trace: 0.001
+    },
+    // Limit errors to 10 per minute
+    rateLimit: {
+        error: { max: 10, window: 60000 }
+    },
+    // Log statistics every 5 minutes to help tune parameters
+    statsInterval: 300000
+});
+
+// Get statistics
+const stats = log.getStats();
+console.log(stats);
+// {
+//   sampling: { total: { error: 150 }, sampled: { debug: 99 }, rateLimited: { error: 5 } },
+//   deduplication: { totalDeduplicated: 50, uniqueMessages: 20 },
+//   performance: { enabled: true, avgLogProcessingTime: 0.5 }
+// }
+```
+
+### Log Deduplication
+
+Automatically group and squish duplicate log messages:
+
+```javascript
+const log = logger('MyApp', {
+    deduplication: {
+        enabled: true,
+        threshold: 3,  // After 3 duplicates, start squishing
+        window: 60000  // 60 second window
+    }
+});
+
+// If this message appears 11 times:
+log.debug('Ticket already exists for threshold a481199f-ed1c-47c6-834d-9cf54cdc394e and device 2642');
+
+// It will be logged once as:
+// "Ticket already exists for threshold * and device * (x11)"
+```
+
+### Performance Metrics (Development Only)
+
+Track performance in development mode:
+
+```javascript
+const log = logger('MyApp', {
+    performanceMetrics: true  // Auto-enabled in development
+});
+
+log.time('database-query');
+// ... do work ...
+const duration = log.timeEnd('database-query');
+// Logs: "Timer 'database-query' completed in 45.23ms"
+```
+
+### Non-Blocking Operations
+
+All logging operations are non-blocking and return immediately:
+
+```javascript
+const log = logger('MyApp');
+
+// All these return immediately, processing happens asynchronously
+log.info('Message 1');
+log.info('Message 2');
+log.info('Message 3');
+
+// Flush all pending logs (returns Promise)
+await log.flush();
+
+// Close logger gracefully (returns Promise)
+await log.close();
+```
 
 ## Custom Levels and Colors
 NightTimeLogger provides custom log levels and colors for enhanced logging experience:

package/index.d.ts

@@ -0,0 +1,225 @@
+/**
+ * TypeScript definitions for NightTimeLogger
+ * @file index.d.ts
+ */
+
+/**
+ * Log levels supported by NightTimeLogger
+ */
+export type LogLevel = 'trace' | 'debug' | 'info' | 'warn' | 'error' | 'fatal' | 'internal';
+
+/**
+ * Sampling configuration for log levels
+ */
+export interface SamplingConfig {
+    [level: string]: number; // 0.0 to 1.0, where 1.0 = log all, 0.1 = log 10%
+}
+
+/**
+ * Rate limit configuration for a log level
+ */
+export interface RateLimitConfig {
+    max: number;    // Maximum number of logs
+    window: number; // Time window in milliseconds
+}
+
+/**
+ * Rate limiting configuration for log levels
+ */
+export interface RateLimitConfigMap {
+    [level: string]: RateLimitConfig;
+}
+
+/**
+ * Deduplication configuration
+ */
+export interface DeduplicationConfig {
+    enabled: boolean;
+    threshold: number;  // Number of duplicates before squishing (default: 3)
+    window: number;     // Time window in milliseconds (default: 60000)
+}
+
+/**
+ * Plugin configuration
+ */
+export interface PluginConfig {
+    name: string;
+    enabled?: boolean;
+    config?: Record<string, any>;
+}
+
+/**
+ * Logger configuration options
+ */
+export interface LoggerConfig {
+    /** Minimum log level to be logged */
+    level?: LogLevel;
+    /** Enable console logging (default: true) */
+    console?: boolean;
+    /** Enable file logging (default: true) */
+    file?: boolean;
+    /** Filename for log file (default: `${location}.log`) */
+    filename?: string;
+    /** Directory path where log files will be saved (default: './logs') */
+    path?: string;
+    /** Maximum size of log file in bytes (default: 1048576 = 1MB) */
+    maxSize?: number;
+    /** Maximum number of log files to retain (default: 5) */
+    maxFiles?: number;
+    /** Include timestamp in log messages (default: true) */
+    timestamp?: boolean;
+    /** Skip cache and create new logger instance (default: false) */
+    skipCache?: boolean;
+    /** Enable call site path reporting (default: false) */
+    reportPath?: boolean;
+    /** Enable debug mode for logger itself (default: false) */
+    debug?: boolean;
+    /** Log sampling rates per level */
+    sampling?: SamplingConfig;
+    /** Rate limiting configuration per level */
+    rateLimit?: RateLimitConfigMap;
+    /** Log deduplication configuration */
+    deduplication?: DeduplicationConfig;
+    /** Enable performance metrics (default: NODE_ENV === 'development') */
+    performanceMetrics?: boolean;
+    /** Interval in milliseconds to log statistics (default: 0 = disabled) */
+    statsInterval?: number;
+    /** Plugin configurations */
+    plugins?: PluginConfig[];
+}
+
+/**
+ * Statistics object returned by getStats()
+ */
+export interface LoggerStats {
+    sampling: {
+        total: Record<string, number>;
+        sampled: Record<string, number>;
+        rateLimited: Record<string, number>;
+    } | null;
+    deduplication: {
+        totalDeduplicated: number;
+        uniqueMessages: number;
+        activeEntries: number;
+    } | null;
+    performance: {
+        enabled: boolean;
+        avgLogProcessingTime?: number;
+        avgTransportTime?: number;
+        logProcessingSamples?: number;
+        transportSamples?: number;
+    } | null;
+}
+
+/**
+ * Logger instance interface
+ */
+export interface Logger {
+    /**
+     * Log an informational message
+     * @param message - The message to log
+     * @param meta - Optional metadata object
+     */
+    info(message: string, meta?: Record<string, any>): void;
+    info(message: Record<string, any>): void;
+
+    /**
+     * Log a warning message
+     * @param message - The message to log
+     * @param meta - Optional metadata object
+     */
+    warn(message: string, meta?: Record<string, any>): void;
+    warn(message: Record<string, any>): void;
+
+    /**
+     * Log an error message
+     * @param message - The message to log
+     * @param meta - Optional metadata object
+     */
+    error(message: string, meta?: Record<string, any>): void;
+    error(message: Record<string, any>): void;
+
+    /**
+     * Log a debug message
+     * @param message - The message to log
+     * @param meta - Optional metadata object
+     */
+    debug(message: string, meta?: Record<string, any>): void;
+    debug(message: Record<string, any>): void;
+
+    /**
+     * Log a trace message
+     * @param message - The message to log
+     * @param meta - Optional metadata object
+     */
+    trace(message: string, meta?: Record<string, any>): void;
+    trace(message: Record<string, any>): void;
+
+    /**
+     * Log a fatal message
+     * @param message - The message to log
+     * @param meta - Optional metadata object
+     */
+    fatal(message: string, meta?: Record<string, any>): void;
+    fatal(message: Record<string, any>): void;
+
+    /**
+     * Log an internal message (logger system messages)
+     * @param message - The message to log
+     * @param meta - Optional metadata object
+     */
+    internal(message: string, meta?: Record<string, any>): void;
+    internal(message: Record<string, any>): void;
+
+    /**
+     * Create a child logger with persistent context
+     * @param context - Context object to merge into all logs
+     * @returns Child logger instance
+     */
+    child(context: Record<string, any>): Logger;
+
+    /**
+     * Start a performance timer (development mode only)
+     * @param label - Timer label
+     */
+    time?(label: string): void;
+
+    /**
+     * End a performance timer and log duration (development mode only)
+     * @param label - Timer label
+     * @returns Duration in milliseconds or null if timer not found
+     */
+    timeEnd?(label: string): number | null;
+
+    /**
+     * Get logging statistics
+     * @returns Statistics object
+     */
+    getStats(): LoggerStats;
+
+    /**
+     * Flush all pending logs (returns Promise)
+     * @returns Promise that resolves when all logs are flushed
+     */
+    flush(): Promise<void>;
+
+    /**
+     * Close the logger and flush all pending logs (returns Promise)
+     * @returns Promise that resolves when logger is closed
+     */
+    close(): Promise<void>;
+
+    /** Current log level */
+    level: string;
+}
+
+/**
+ * Main logger function
+ * @param location - The location/name of the logger instance
+ * @param config - Configuration options for the logger
+ * @returns Logger instance
+ */
+declare function logger(location?: string, config?: LoggerConfig): Logger;
+
+export = logger;
+

package/lib/logDeduplicator.js

@@ -0,0 +1,146 @@
+/**
+ * @file /lib/logDeduplicator.js
+ * @description Handles log deduplication/squishing by grouping similar messages.
+ */
+
+const crypto = require('crypto');
+const { normalizeMessage } = require('./messageNormalizer');
+
+/**
+ * Creates a fingerprint for a log message
+ * @param {string} level - Log level
+ * @param {string} message - Log message
+ * @param {string} location - Logger location
+ * @param {string} filePath - File path (if available)
+ * @returns {string} - Message fingerprint
+ */
+function createFingerprint(level, message, location, filePath) {
+    const normalized = normalizeMessage(message);
+    const key = `${level}:${normalized}:${location || ''}:${filePath || ''}`;
+    return crypto.createHash('md5').update(key).digest('hex');
+}
+
+/**
+ * Log deduplicator class
+ */
+class LogDeduplicator {
+    constructor(config = {}) {
+        this.enabled = config.enabled || false;
+        this.threshold = config.threshold || 3;
+        this.window = config.window || 60000; // 60 seconds default
+        this.entries = new Map(); // fingerprint -> { count, firstSeen, lastSeen, sample }
+        this.stats = {
+            totalDeduplicated: 0,
+            uniqueMessages: 0,
+        };
+
+        // Cleanup old entries periodically
+        if (this.enabled) {
+            this.cleanupInterval = setInterval(() => this.cleanup(), this.window);
+            // Use unref() to allow process to exit if this is the only timer
+            if (this.cleanupInterval.unref) {
+                this.cleanupInterval.unref();
+            }
+        }
+    }
+
+    /**
+     * Check if a log should be deduplicated
+     * @param {string} level - Log level
+     * @param {string} message - Log message
+     * @param {object} meta - Log metadata
+     * @returns {object|null} - Returns { shouldLog: boolean, count: number, message: string } or null if not deduplicated
+     */
+    check(level, message, meta = {}) {
+        if (!this.enabled) {
+            return null;
+        }
+
+        const fingerprint = createFingerprint(level, message, meta.location, meta.filePath);
+        const now = Date.now();
+        const entry = this.entries.get(fingerprint);
+
+        if (!entry) {
+            // First occurrence
+            this.entries.set(fingerprint, {
+                count: 1,
+                firstSeen: now,
+                lastSeen: now,
+                sample: { level, message, meta },
+            });
+            this.stats.uniqueMessages++;
+            return { shouldLog: true, count: 1, message };
+        }
+
+        // Update entry
+        entry.count++;
+        entry.lastSeen = now;
+
+        if (entry.count < this.threshold) {
+            // Still below threshold, log normally
+            return { shouldLog: true, count: entry.count, message };
+        } else if (entry.count === this.threshold) {
+            // Just reached threshold, log with count
+            this.stats.totalDeduplicated += entry.count - 1; // Count suppressed logs
+            return {
+                shouldLog: true,
+                count: entry.count,
+                message: `${message} (x${entry.count})`,
+            };
+        } else {
+            // Above threshold, suppress but update count
+            this.stats.totalDeduplicated++;
+            return { shouldLog: false, count: entry.count, message: `${message} (x${entry.count})` };
+        }
+    }
+
+    /**
+     * Clean up old entries
+     */
+    cleanup() {
+        const now = Date.now();
+        for (const [fingerprint, entry] of this.entries.entries()) {
+            if (now - entry.lastSeen > this.window) {
+                this.entries.delete(fingerprint);
+            }
+        }
+    }
+
+    /**
+     * Get statistics
+     * @returns {object} - Statistics object
+     */
+    getStats() {
+        return {
+            ...this.stats,
+            activeEntries: this.entries.size,
+        };
+    }
+
+    /**
+     * Reset statistics
+     */
+    resetStats() {
+        this.stats = {
+            totalDeduplicated: 0,
+            uniqueMessages: 0,
+        };
+    }
+
+    /**
+     * Destroy the deduplicator
+     */
+    destroy() {
+        if (this.cleanupInterval) {
+            clearInterval(this.cleanupInterval);
+            this.cleanupInterval = null;
+        }
+        this.entries.clear();
+    }
+}
+
+module.exports = {
+    LogDeduplicator,
+    createFingerprint,
+};
+

package/lib/logSampler.js

@@ -0,0 +1,139 @@
+/**
+ * @file /lib/logSampler.js
+ * @description Handles log sampling and rate limiting with statistics.
+ */
+
+/**
+ * Log sampler and rate limiter class
+ */
+class LogSampler {
+    constructor(config = {}) {
+        this.sampling = config.sampling || {};
+        this.rateLimit = config.rateLimit || {};
+        this.stats = {
+            total: {},
+            sampled: {},
+            rateLimited: {},
+        };
+
+        // Rate limit tracking: level -> { count: number, windowStart: number }
+        this.rateLimitCounters = new Map();
+
+        // Cleanup old rate limit entries periodically
+        if (Object.keys(this.rateLimit).length > 0) {
+            this.cleanupInterval = setInterval(() => this.cleanupRateLimits(), 60000);
+            // Use unref() to allow process to exit if this is the only timer
+            if (this.cleanupInterval.unref) {
+                this.cleanupInterval.unref();
+            }
+        }
+    }
+
+    /**
+     * Check if a log should be processed based on sampling and rate limiting
+     * @param {string} level - Log level
+     * @returns {boolean} - True if log should be processed, false if skipped
+     */
+    shouldProcess(level) {
+        // Initialize stats for level if needed
+        if (!this.stats.total[level]) {
+            this.stats.total[level] = 0;
+            this.stats.sampled[level] = 0;
+            this.stats.rateLimited[level] = 0;
+        }
+
+        this.stats.total[level]++;
+
+        // Check rate limiting first (more restrictive)
+        if (this.rateLimit[level]) {
+            const limit = this.rateLimit[level];
+            const now = Date.now();
+            const key = level;
+            const counter = this.rateLimitCounters.get(key);
+
+            if (counter) {
+                // Check if we're still in the same window
+                if (now - counter.windowStart < limit.window) {
+                    if (counter.count >= limit.max) {
+                        // Rate limit exceeded
+                        this.stats.rateLimited[level]++;
+                        return false;
+                    }
+                    counter.count++;
+                } else {
+                    // New window
+                    this.rateLimitCounters.set(key, { count: 1, windowStart: now });
+                }
+            } else {
+                // First log in this level
+                this.rateLimitCounters.set(key, { count: 1, windowStart: now });
+            }
+        }
+
+        // Check sampling
+        if (this.sampling[level] !== undefined) {
+            const rate = this.sampling[level];
+            if (rate < 1.0 && Math.random() > rate) {
+                // Sample rejected
+                this.stats.sampled[level]++;
+                return false;
+            }
+        }
+
+        return true;
+    }
+
+    /**
+     * Clean up old rate limit entries
+     */
+    cleanupRateLimits() {
+        const now = Date.now();
+        for (const [level, limit] of Object.entries(this.rateLimit)) {
+            const key = level;
+            const counter = this.rateLimitCounters.get(key);
+            if (counter && now - counter.windowStart >= limit.window) {
+                // Entry expired, will be reset on next use
+                this.rateLimitCounters.delete(key);
+            }
+        }
+    }
+
+    /**
+     * Get statistics
+     * @returns {object} - Statistics object
+     */
+    getStats() {
+        return {
+            total: { ...this.stats.total },
+            sampled: { ...this.stats.sampled },
+            rateLimited: { ...this.stats.rateLimited },
+        };
+    }
+
+    /**
+     * Reset statistics
+     */
+    resetStats() {
+        this.stats = {
+            total: {},
+            sampled: {},
+            rateLimited: {},
+        };
+    }
+
+    /**
+     * Destroy the sampler
+     */
+    destroy() {
+        if (this.cleanupInterval) {
+            clearInterval(this.cleanupInterval);
+            this.cleanupInterval = null;
+        }
+        this.rateLimitCounters.clear();
+    }
+}
+
+module.exports = {
+    LogSampler,
+};
+