http-log-replay 1.0.0

package/README.md

@@ -0,0 +1,197 @@
+# 🚦 Traffic Mirror (http-log-replay)
+
+> **Regressions No More.** Record production traffic and replay it against your changes to verify correctness with zero effort.
+
+[![npm version](https://img.shields.io/npm/v/http-log-replay.svg)](https://www.npmjs.com/package/http-log-replay)
+[![License: ISC](https://img.shields.io/badge/License-ISC-yellow.svg)](https://opensource.org/licenses/ISC)
+
+**Traffic Mirror** is a powerful regression testing tool designed for modern engineering teams. It allows you to **record** HTTP traffic from a live environment (like Production) and **replay** it against two different target environments (e.g., Stable vs. Canary) to instantly detect regressions, bugs, or side-effects.
+
+---
+
+## 📋 Table of Contents
+
+- [🌟 Features](#-features)
+- [🚀 Quick Start (Zero Setup)](#-quick-start-zero-setup)
+- [💻 Usage Guide: Web UI](#-usage-guide-web-ui)
+- [🖥️ Usage Guide: CLI](#️-usage-guide-cli)
+- [🐳 Usage Guide: Docker](#-usage-guide-docker)
+- [🛠️ Development & Contribution](#️-development--contribution)
+- [📦 Maintenance](#-maintenance)
+
+---
+
+## 🌟 Features
+
+- **🛡️ Zero-Config Recording**: Acts as a transparent HTTP proxy to capture real requests.
+- **⚡ High-Performance Replay**: Replay thousands of requests in parallel with customizable concurrency.
+- **🔍 Intelligent Diffing**: compares JSON responses and ignores dynamic fields (like timestamps, UUIDs) that cause false positives.
+- **📊 Rich Reporting**: Generates detailed HTML reports showing exactly what broke.
+- **🔌 Swagger Integration**: Auto-generate test traffic from your OpenAPI definitions if you don't have live traffic.
+- **🏗️ Two Modes**: Full interactive **Web UI** for debugging and a lightweight **CLI** for CI/CD pipelines.
+
+---
+
+## 🚀 Quick Start (Zero Setup)
+
+You don't need to install anything if you have Node.js (v18+) installed. Just run:
+
+```bash
+npx http-log-replay ui --port 4200
+```
+
+This will launch the **Traffic Mirror Dashboard** at [http://localhost:4200](http://localhost:4200).
+
+---
+
+## 💻 Usage Guide: Web UI
+
+The Web UI is the best way to get started. It guides you through the entire workflow in three simple tabs.
+
+### 1. 🔴 Record / Generate
+- **Manual Mode**: Start the proxy, point your application/client to it, and use your app normally. Requests are saved to a file.
+- **Auto-Generate**: Upload an OpenAPI/Swagger JSON file to automatically generate realistic traffic patterns.
+
+### 2. ▶️ Replay
+- Configure your **Primary** (Stable) and **Secondary** (Test) environments.
+- Set **Concurrency** to speed up large suites.
+- Add **Ignore Fields** (e.g., `createdAt`, `traceId`) to filter out noise in the comparison.
+- Click **Replay & Compare**.
+
+### 3. 📄 Report
+- Instantly view the results.
+- **Green**: Exact match.
+- **Red**: Mismatch (click to expand the JSON diff).
+
+---
+
+## 🖥️ Usage Guide: CLI
+
+Perfect for **CI/CD pipelines** or headless environments.
+
+### 1. Record Traffic
+Start a recording proxy on port `3000` forwarding to your real API at `localhost:8080`.
+
+```bash
+npx http-log-replay record --target http://localhost:8080 --port 3000 --out traffic.jsonl
+```
+
+### 2. Auto-Generate from Swagger
+Generate traffic without manual clicking.
+
+```bash
+npx http-log-replay generate --file ./openapi.json --target http://localhost:3000
+```
+
+### 3. Replay & Verify
+Replay recorded traffic against two environments and generate an HTML report.
+
+```bash
+npx http-log-replay replay \
+  --log traffic.jsonl \
+  --primary http://prod-api.com \
+  --secondary http://staging-api.com \
+  --report report.html \
+  --ignore "timestamp,id"
+```
+
+---
+
+## 🐳 Usage Guide: Docker
+
+We provide optimized Docker images for both the UI and CLI.
+
+### Prerequisites
+- Docker & Docker Compose installed.
+
+### Option A: Complete Environment (Recommended)
+Use the included `docker-compose.yml` to run everything.
+
+**Start the GUI:**
+```bash
+docker-compose up gui
+```
+> Access at [http://localhost:4200](http://localhost:4200). Data is persisted to your host folder.
+
+**Run CLI Commands:**
+```bash
+docker-compose run cli record --target http://host.docker.internal:8080 ...
+```
+
+### Option B: Manual Docker Run
+
+**UI Image:**
+```bash
+docker run -p 4200:4200 -v $(pwd):/app traffic-mirror-gui
+```
+
+**CLI Image:**
+```bash
+docker run -v $(pwd):/app traffic-mirror-cli --help
+```
+
+---
+
+## 🛠️ Development & Contribution
+
+We welcome contributions! Here is how to run the project locally for development.
+
+### 1. Setup
+```bash
+git clone https://github.com/your-username/http-log-replay.git
+cd http-log-replay
+npm install
+```
+
+### 2. Build the UI
+The UI is built with Angular. You must build it before running the app.
+```bash
+cd ui
+npm install
+npm run build
+cd ..
+```
+
+### 3. Run Locally (Dev Mode)
+```bash
+# Start the full app (Frontend + Backend)
+node index.js ui --port 4200
+```
+
+### 4. Testing & Code Quality
+We use **Jest** for testing and **ESLint** for code quality.
+
+```bash
+# Run Unit Tests
+npm test
+
+# Lint Code
+npm run lint
+
+# Format Code
+npm run format
+```
+
+---
+
+## 📦 Maintenance
+
+### Publishing to NPM
+1.  **Bump Version**: Update `version` in `package.json`.
+2.  **Build UI**: The `prepublishOnly` script will automatically build the Angular UI.
+3.  **Publish**:
+    ```bash
+    npm publish
+    ```
+
+### File Structure
+- `index.js`: CLI entry point.
+- `recorder.js`: Proxy logic.
+- `replayer.js`: Replay & Diff logic.
+- `server.js`: Express server for the UI.
+- `ui/`: Angular frontend source code.
+- `tests/`: Jest unit tests.
+
+---
+
+_Built with ❤️ by the Traffic Mirror Team._

package/ecosystem.config.js

@@ -0,0 +1,19 @@
+module.exports = {
+    apps: [
+        {
+            name: 'traffic-mirror-ui',
+            script: './index.js',
+            args: 'ui --port 4200',
+            instances: 4, // Keep 1 instance to avoid port conflicts with the recorder
+            autorestart: true,
+            watch: false,
+            max_memory_restart: '1G',
+            env: {
+                NODE_ENV: 'development',
+            },
+            env_production: {
+                NODE_ENV: 'production',
+            },
+        },
+    ],
+};

package/index.js

@@ -0,0 +1,95 @@
+#!/usr/bin/env node
+/* eslint-disable no-console */
+const { Command } = require('commander');
+const recorder = require('./recorder');
+const replayAndDiff = require('./replayer');
+const startServer = require('./server');
+const generator = require('./traffic-generator');
+
+const program = new Command();
+
+program
+  .name('traffic-mirror')
+  .description('Record and Replay HTTP traffic to detect regressions')
+  .version('1.0.0');
+
+// Command: Record
+program
+  .command('record')
+  .description('Start a proxy to record traffic to a JSONL file')
+  .requiredOption('-t, --target <url>', 'The target URL to proxy to (e.g., http://localhost:8080)')
+  .option('-p, --port <number>', 'Port to listen on', '3000')
+  .option('-o, --out <file>', 'Output file for logs', 'traffic.jsonl')
+  .action((options) => {
+    recorder.start(options.target, options.port, options.out);
+  });
+
+// Command: Replay
+program
+  .command('replay')
+  .description('Replay logs against two environments and diff the results')
+  .requiredOption('-l, --log <file>', 'The log file to replay')
+  .requiredOption('-a, --primary <url>', 'Primary environment URL (e.g., Stable)')
+  .requiredOption('-b, --secondary <url>', 'Secondary environment URL (e.g., Staging)')
+  .option('-r, --report <file>', 'Path to save HTML report', 'report.html')
+  .option('-i, --ignore <items>', 'Comma separated list of JSON fields to ignore', (val) =>
+    val.split(',')
+  )
+  .option('-x, --exclude-endpoints <items>', 'List of URL paths to skip', (val) => val.split(',')) // <--- NEW FLAG
+  .option('-c, --concurrency <number>', 'Number of concurrent requests', (val) => parseInt(val, 10), 5) // Default 5
+  .option('--auth <token>', 'Inject Authorization header (e.g. "Bearer eyJhb...")')
+  .action((options) => {
+    const injectedHeaders = options.auth ? { Authorization: options.auth } : {};
+    // Pass injectedHeaders to the replayer
+    replayAndDiff(
+      options.log,
+      options.primary,
+      options.secondary,
+      options.report,
+      options.ignore || [],
+      injectedHeaders,
+      options.excludeEndpoints || [],
+      () => { }, // Empty callback for CLI
+      options.concurrency || 1
+    );
+  });
+
+program
+  .command('ui')
+  .description('Start the Web Interface')
+  .option('-p, --port <number>', 'Port for the UI', '4200')
+  .action((options) => {
+    startServer(options.port);
+  });
+
+program
+  .command('generate')
+  .description('Auto-generate traffic from Swagger file')
+  .option('-t, --target <url>', 'Proxy URL', 'http://localhost:3000')
+  .option('-f, --file <path>', 'Swagger file path', './full_documentation.json')
+  .option('-x, --exclude <items>', 'Comma separated list of endpoints to exclude', (val) =>
+    val.split(',')
+  ) // <--- NEW OPTION
+  .option('-s, --source <url>', 'Source Server URL', 'http://localhost:1338')
+  .action(async (options) => {
+    try {
+      console.log('🚀 Starting Traffic Generation...');
+
+      // Pass options.exclude (or empty array) as 3rd arg
+      await generator.run(
+        options.target,
+        options.file,
+        options.exclude || [],
+        (log) => {
+          console.log(log.message);
+        },
+        options.source
+      );
+
+      console.log('✅ Done.');
+    } catch (e) {
+      console.error('❌ Error:', e.message);
+    }
+  });
+
+program.parse();

package/package.json

@@ -0,0 +1,55 @@
+{
+  "name": "http-log-replay",
+  "version": "1.0.0",
+  "description": "Traffic Mirror is a regression testing tool that records HTTP traffic from a live environment and replays it against two different environments (e.g., Stable vs. Canary) to detect differences in responses.",
+  "author": "Xhani Manolis Trungu",
+  "main": "index.js",
+  "bin": {
+    "traffic-mirror": "./index.js"
+  },
+  "files": [
+    "*.js",
+    "ui/dist"
+  ],
+  "scripts": {
+    "test": "jest tests/",
+    "lint": "eslint .",
+    "format": "prettier --write .",
+    "start:pm2": "pm2 start ecosystem.config.js",
+    "prepublishOnly": "cd ui && npm install && npm run build"
+  },
+  "keywords": [
+    "http-log",
+    "http-log-replay",
+    "traffic-replay",
+    "regression-testing",
+    "api-regression",
+    "api-testing"
+  ],
+  "license": "ISC",
+  "type": "commonjs",
+  "dependencies": {
+    "axios": "^1.13.2",
+    "colors": "^1.4.0",
+    "commander": "^14.0.2",
+    "cors": "^2.8.5",
+    "deep-diff": "^1.0.2",
+    "diff": "^8.0.2",
+    "express": "^5.2.1",
+    "http-proxy": "^1.18.1",
+    "open": "^11.0.0",
+    "readline": "^1.3.0",
+    "socket.io": "^4.8.1"
+  },
+  "devDependencies": {
+    "eslint": "^9.39.2",
+    "eslint-config-prettier": "^10.1.8",
+    "jest": "^30.2.0",
+    "pm2": "^6.0.14",
+    "prettier": "^3.7.4"
+  },
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/xhani-manolis-trungu/traffic-mirror"
+  }
+}

package/recorder.js

@@ -0,0 +1,114 @@
+/* eslint-disable no-console */
+const http = require('http');
+const httpProxy = require('http-proxy');
+const fs = require('fs');
+
+class Recorder {
+  constructor() {
+    this.server = null;
+    this.proxy = httpProxy.createProxyServer({});
+
+    // 1. Handle Proxy Errors (Critical for preventing crashes)
+    this.proxy.on('error', (err, req, res) => {
+      console.error(`❌ Proxy Error [${req.url}]:`, err.message);
+      if (!res.headersSent) {
+        res.writeHead(502, { 'Content-Type': 'application/json' });
+      }
+      res.end(JSON.stringify({ error: 'Proxy Request Failed', details: err.message }));
+    });
+  }
+
+  start(target, port, outputFile, onLog = () => { }) {
+    if (this.server) throw new Error('Recorder already running');
+
+    console.log(`📝 Recording to file: ${outputFile}`);
+    const stream = fs.createWriteStream(outputFile, { flags: 'a' });
+
+    // Handle file write errors
+    stream.on('error', (err) => {
+      console.error('❌ File Write Error:', err.message);
+    });
+
+    this.server = http.createServer((req, res) => {
+      // 2. Capture Request Body safely
+      // We use a separate array but we DO NOT attach 'data' listeners directly
+      // unless we plan to buffer it for the proxy.
+      // For simplicity, let's rely on the response capture primarily.
+
+      const reqChunks = [];
+      // We tap into the stream without consuming it destructively if possible,
+      // but the safest way with http-proxy is often just to listen to the proxy events.
+      // HOWEVER, for your specific "Request Stealing" fix:
+      req.on('data', (chunk) => reqChunks.push(chunk));
+
+      // --- Response Capture Logic ---
+      const originalWrite = res.write;
+      const originalEnd = res.end;
+      const resChunks = [];
+
+      res.write = function (chunk, ...args) {
+        if (chunk) resChunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
+        return originalWrite.apply(res, [chunk, ...args]);
+      };
+
+      res.end = function (chunk, ...args) {
+        if (chunk) resChunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
+
+        // WRAP IN TRY/CATCH to prevent silent crashes
+        try {
+          const reqBody = Buffer.concat(reqChunks).toString();
+          const resBody = Buffer.concat(resChunks).toString('utf8');
+
+          const logData = {
+            timestamp: new Date().toISOString(),
+            method: req.method,
+            url: req.url,
+            status: res.statusCode,
+            requestBody: reqBody || '',
+            responseBody: resBody || '',
+            // requestHeaders: req.headers, // Uncomment if needed (verbose)
+          };
+
+          const jsonLine = JSON.stringify(logData) + '\n';
+
+          // 3. Write and Log
+          stream.write(jsonLine);
+          onLog(logData);
+
+          // Debug Log to prove it worked
+          process.stdout.write('.'); // Prints a dot for every logged request
+        } catch (e) {
+          console.error('\n❌ Error generating log:', e.message);
+        }
+
+        return originalEnd.apply(res, [chunk, ...args]);
+      };
+
+      // 4. Force Plain Text (Disable Gzip)
+      delete req.headers['accept-encoding'];
+
+      // 5. Forward to Target
+      // We must set 'buffer' to the request stream if we consumed it,
+      // but since we are just tapping 'data' without pausing, http-proxy might miss it.
+      // A simple workaround for node streams in this context:
+      this.proxy.web(req, res, { target });
+    });
+
+    this.server.listen(port, () => {
+      console.log(`\n⏺️  Recorder listening on port ${port}`);
+      console.log(`➡️  Forwarding to ${target}`);
+    });
+
+    return `Recorder started`;
+  }
+
+  stop() {
+    if (this.server) {
+      this.server.close();
+      this.server = null;
+      return 'Recorder stopped';
+    }
+  }
+}
+
+module.exports = new Recorder();