@hackylabs/deep-redact 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Benjamin Green (https://bengreen.dev)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,107 @@
+# Deep Redact
+
+Faster than fast-redact <sup>1</sup> as well as being safer and more configurable than many other redaction libraries,
+Deep Redact is a zero-dependency tool that redacts sensitive information from strings and objects. It is designed to be
+used in a production environment where sensitive information needs to be redacted from logs, error messages, files,
+and other outputs.
+
+Circular references and other unsupported are handled gracefully, and the library is designed to be as fast as possible
+while still being configurable.
+
+Supporting both CommonJS and ESM, with named and default exports, Deep Redact is designed to be versatile and easy to
+use in any modern JavaScript or TypeScript project in Node or the browser.
+
+## Installation
+
+```bash
+npm install @hackylabs/deep-redact
+```
+
+## Usage
+
+<h4 style="color: red">In order to maintain a consistent usage throughout your project, it is not advised to call this
+library outside of your global logging/error-reporting libraries.</h4>
+
+```typescript
+// ./src/example.ts
+import {DeepRedact} from 'deep-redact'; // If you're using CommonJS, import with require('deep-redact') instead. Both CommonJS and ESM support named and default imports.
+
+const redaction = new DeepRedact({
+  replacement: '*',
+  replaceStringByLength: true,
+  blacklistedKeys: ['password'],
+  stringTests: [
+    /^[\d]{13,16}$/, // payment card number
+    /^[\d]{3,4}$/ // CVV
+  ],
+});
+
+const obj = {
+  password: '<h1><strong>Password</strong></h1>',
+  cardNumber: '1234567812345678',
+  cvv: '123',
+};
+
+redaction.redact(obj) // { password: '**********************************', cardNumber: '****************', cvv: '***' }
+```
+
+## Configuration
+
+### Main Options
+
+| key | description | type | options | default | required |
+| --- | --- | --- | --- | --- | --- |
+| blacklistedKeys | Deeply compare names of these keys against the keys in your object. | array | Array<string￨BlacklistKeyConfig> | [] | N |
+| stringTests | Array of regular expressions to perform against string values, whether that value is a flat string or nested within an object. | array | RegExp[] | [] | N |
+| fuzzyKeyMatch | Loosely compare key names by checking if the key name of your unredacted object is included anywhere within the name of your blacklisted key. For example, is "pass" (your key) included in "password" (from config). | boolean |  | false | N |
+| caseSensitiveKeyMatch | Loosely compare key names by normalising the strings. This involves removing non-word characters and transforms the string to lowercase. This means you never have to worry having to list duplicate keys in different formats such as snake_case, camelCase, PascalCase or any other case. | boolean |  | true | N |
+| remove | Determines whether or not to remove the key from the object when it is redacted. | boolean |  | false | N |
+| retainStructure | Determines whether or not keep all nested values of a key that is going to be redacted. Circular references are always removed. | boolean |  | false | N |
+| replacement | When a value is going to be redacted, what would you like to replace it with? | string |  | [REDACTED] | N |
+| replaceStringByLength | When a string value is going to be replaced, optionally replace it by repeating the `replacement` to match the length of the value. For example, if `replaceStringByLength` were set to `true` and `replacement` was set to "x", then redacting "secret" would return "xxxxxx". This is sometimes useful for debugging purposes, although it may be less secure as it could give hints to the original value. | boolean |  | false | N |
+| types | JS types (values of `typeof` keyword). Only values with a typeof equal to `string`, `number`, `bigint`, `boolean` or `object` may be redacted. The other types are only listed as options to keep TypeScript happy, so you never need to list them. | array | Array<'string'￨'number'￨'bigint'￨'boolean'￨'symbol'￨'undefined'￨'object'￨'function'> | ['string'] | N |
+| serialise | Determines whether or not to serialise the object after redacting. Typical use cases for this are when you want to send it over the network or save to a file, both of which are common use cases for redacting sensitive information. | boolean |  | true | N |
+| unsupportedTransformer | When an unsafe value is encountered or a value that cannot be serialised. By default, this function will transform an unsupported value `Unsupported` object. BigInt values are converted a string. Dates are returned using their own `toISOString` method. Regular expressions are returned as objects with their `source` and `flags` values. Errors are converted objects. This is useful when you have a custom class that you would like to redact. For safety reasons, you should always transform a BigInt to avoid JSON.stringify throwing an error. | (value: unknown) => unknown |  | DeepRedact.transformUnsupported | N |
+
+### BlacklistKeyConfig
+
+| key | type | default | required |
+| --- | --- | --- | --- |
+| key | string￨RegExp |  | Y |
+| fuzzyKeyMatch | boolean | Main options `fuzzyKeyMatch` | N |
+| caseSensitiveKeyMatch | boolean | Main options `caseSensitiveKeyMatch` | N |
+| remove | boolean | Main options `remove` | N |
+| retainStructure | boolean | Main options `retainStructure` | N |
+
+### Benchmark
+Comparisons are made against JSON.stringify and fast-redact as well as different configurations of deep-redact, using
+[this test object](./test/setup/dummyUser.ts). The benchmark is run on a 2021 iMac with an M1 chip with 16GB memory
+running Sonoma 14.5.
+
+JSON.stringify is included as a benchmark because it is the fastest way to deeply iterate over an object although it
+doesn't redact any sensitive information. Fast-redact is included as a benchmark because it's the next fastest redaction
+library available. Neither JSON.stringify nor fast-redact offer the same level of configurability as deep-redact.
+
+![Benchmark](./benchmark.png)
+
+| scenario | ops / sec | op duration (ms) | margin of error | sample count |
+| --- | --- | --- | --- | --- |
+| JSON.stringify, tiny object | 3878848.93 | 0.0002578084 | 0 | 1939425 |
+| DeepRedact, default config, tiny object | 1530332.28 | 0.0006534529 | 0.00001 | 765167 |
+| JSON.stringify, large object | 295526.11 | 0.0033837958 | 0.00001 | 147764 |
+| fast redact, tiny object | 228053.93 | 0.0043849277 | 0.00002 | 114027 |
+| DeepRedact, default config, large object | 92714.28 | 0.0107858256 | 0.00006 | 46358 |
+| DeepRedact, remove item, single object | 92349.45 | 0.0108284349 | 0.00005 | 46175 |
+| DeepRedact, fuzzy matching, single object | 89414.82 | 0.0111838282 | 0.00007 | 44708 |
+| DeepRedact, fuzzy and case insensitive matching, single object | 87852.36 | 0.0113827334 | 0.00006 | 43927 |
+| DeepRedact, case insensitive matching, single object | 86797.23 | 0.0115211045 | 0.00006 | 43399 |
+| DeepRedact, replace string by length, single object | 84150.95 | 0.011883407 | 0.00006 | 42076 |
+| DeepRedact, config per key, single object | 71236.85 | 0.0140376786 | 0.00009 | 35619 |
+| DeepRedact, retain structure, single object | 69738.86 | 0.0143392076 | 0.00007 | 34870 |
+| fast redact, large object | 19480.95 | 0.0513321865 | 0.00038 | 9741 |
+| JSON.stringify, 1000 tiny objects | 16003.16 | 0.0624876526 | 0.00017 | 8002 |
+| DeepRedact, default config, 1000 tiny objects | 15827.22 | 0.0631822932 | 0.00043 | 7914 |
+| DeepRedact, default config, 1000 large objects | 13705.52 | 0.072963285 | 0.00054 | 6853 |
+| fast redact, 1000 tiny objects | 7430.88 | 0.1345735164 | 0.00067 | 3716 |
+| JSON.stringify, 1000 large objects | 423.52 | 2.3611880519 | 0.00982 | 212 |
+| fast redact, 1000 large objects | 77.32 | 12.9327971026 | 0.25939 | 39 |

package/dist/cjs/index.d.ts

@@ -0,0 +1,34 @@
+export type Types = 'string' | 'number' | 'bigint' | 'boolean' | 'object' | 'function';
+export interface BlacklistKeyConfig {
+    fuzzyKeyMatch?: boolean;
+    caseSensitiveKeyMatch?: boolean;
+    retainStructure?: boolean;
+    remove?: boolean;
+    key: string | RegExp;
+}
+export interface DeepRedactConfig {
+    blacklistedKeys?: Array<string | BlacklistKeyConfig>;
+    stringTests?: RegExp[];
+    fuzzyKeyMatch?: boolean;
+    caseSensitiveKeyMatch?: boolean;
+    retainStructure?: boolean;
+    replaceStringByLength?: boolean;
+    replacement?: string;
+    remove?: boolean;
+    types?: Types[];
+    serialise?: boolean;
+    unsupportedTransformer?: (value: unknown) => unknown;
+}
+declare class DeepRedact {
+    private circularReference;
+    private readonly config;
+    constructor(config: DeepRedactConfig);
+    private static unsupportedTransformer;
+    private removeCircular;
+    private redactString;
+    private static complexShouldRedact;
+    private shouldRedactObjectValue;
+    private deepRedact;
+    redact: (value: unknown) => unknown;
+}
+export { DeepRedact as default, DeepRedact };

package/dist/cjs/index.js

@@ -0,0 +1,129 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.DeepRedact = exports.default = void 0;
+const normaliseString = (key) => key.toLowerCase().replace(/\W/g, '');
+class DeepRedact {
+    constructor(config) {
+        var _a, _b;
+        this.circularReference = new WeakSet();
+        this.config = {
+            blacklistedKeys: [],
+            stringTests: [],
+            fuzzyKeyMatch: false,
+            caseSensitiveKeyMatch: true,
+            retainStructure: false,
+            remove: false,
+            replaceStringByLength: false,
+            replacement: '[REDACTED]',
+            types: ['string'],
+            serialise: true,
+            unsupportedTransformer: DeepRedact.unsupportedTransformer,
+        };
+        this.removeCircular = (value) => {
+            var _a, _b;
+            if (!(value instanceof Object))
+                return value;
+            if (!((_a = this.circularReference) === null || _a === void 0 ? void 0 : _a.has(value))) {
+                (_b = this.circularReference) === null || _b === void 0 ? void 0 : _b.add(value);
+                return value;
+            }
+            return '__circular__';
+        };
+        this.redactString = (value, parentShouldRedact = false) => {
+            if (!this.config.stringTests.some((test) => test.test(value)) && !parentShouldRedact)
+                return value;
+            if (this.config.replaceStringByLength)
+                return this.config.replacement.repeat(value.length);
+            return this.config.remove ? undefined : this.config.replacement;
+        };
+        this.shouldRedactObjectValue = (key) => {
+            return this.config.blacklistedKeys.some((redactableKey) => (typeof redactableKey === 'string'
+                ? key === redactableKey
+                : DeepRedact.complexShouldRedact(key, redactableKey)));
+        };
+        this.deepRedact = (value, parentShouldRedact = false) => {
+            if (value === undefined || value === null)
+                return value;
+            let safeValue = this.removeCircular(value);
+            safeValue = this.config.unsupportedTransformer(safeValue);
+            if (!(safeValue instanceof Object)) {
+                // @ts-expect-error - we already know that safeValue is not a function, symbol, undefined, null, or an object
+                if (!this.config.types.includes(typeof safeValue))
+                    return safeValue;
+                if (typeof safeValue === 'string')
+                    return this.redactString(safeValue, parentShouldRedact);
+                if (!parentShouldRedact)
+                    return safeValue;
+                return this.config.remove
+                    ? undefined
+                    : this.config.replacement;
+            }
+            if (parentShouldRedact && (!this.config.retainStructure || this.config.remove)) {
+                return this.config.remove ? undefined : this.config.replacement;
+            }
+            if (Array.isArray(safeValue))
+                return safeValue.map((val) => this.deepRedact(val, parentShouldRedact));
+            return Object.fromEntries(Object.entries(safeValue).map(([key, val]) => {
+                const shouldRedact = parentShouldRedact || this.shouldRedactObjectValue(key);
+                return [key, this.deepRedact(val, shouldRedact)];
+            }));
+        };
+        this.redact = (value) => {
+            this.circularReference = new WeakSet();
+            const redacted = this.deepRedact(value);
+            this.circularReference = null;
+            return this.config.serialise ? JSON.stringify(redacted) : redacted;
+        };
+        this.config = Object.assign(Object.assign(Object.assign({}, this.config), config), { blacklistedKeys: (_b = (_a = config.blacklistedKeys) === null || _a === void 0 ? void 0 : _a.map((key) => {
+                if (typeof key === 'string')
+                    return key;
+                return Object.assign({ fuzzyKeyMatch: this.config.fuzzyKeyMatch, caseSensitiveKeyMatch: this.config.caseSensitiveKeyMatch, retainStructure: this.config.retainStructure, remove: this.config.remove }, key);
+            })) !== null && _b !== void 0 ? _b : [] });
+    }
+}
+exports.default = DeepRedact;
+exports.DeepRedact = DeepRedact;
+DeepRedact.unsupportedTransformer = (value) => {
+    if (typeof value === 'bigint') {
+        return {
+            __unsupported: {
+                type: 'bigint',
+                value: value.toString(),
+                radix: 10,
+            },
+        };
+    }
+    if (value instanceof Error) {
+        return {
+            __unsupported: {
+                type: 'error',
+                name: value.name,
+                message: value.message,
+                stack: value.stack,
+            },
+        };
+    }
+    if (value instanceof RegExp) {
+        return {
+            __unsupported: {
+                type: 'regexp',
+                source: value.source,
+                flags: value.flags,
+            },
+        };
+    }
+    if (value instanceof Date)
+        return value.toISOString();
+    return value;
+};
+DeepRedact.complexShouldRedact = (key, config) => {
+    if (config.key instanceof RegExp)
+        return config.key.test(key);
+    if (config.fuzzyKeyMatch && config.caseSensitiveKeyMatch)
+        return key.includes(config.key);
+    if (config.fuzzyKeyMatch && !config.caseSensitiveKeyMatch)
+        return normaliseString(key).includes(normaliseString(config.key));
+    if (!config.fuzzyKeyMatch && config.caseSensitiveKeyMatch)
+        return key === config.key;
+    return normaliseString(config.key) === normaliseString(key);
+};

package/dist/esm/index.d.ts

@@ -0,0 +1,34 @@
+export type Types = 'string' | 'number' | 'bigint' | 'boolean' | 'object' | 'function';
+export interface BlacklistKeyConfig {
+    fuzzyKeyMatch?: boolean;
+    caseSensitiveKeyMatch?: boolean;
+    retainStructure?: boolean;
+    remove?: boolean;
+    key: string | RegExp;
+}
+export interface DeepRedactConfig {
+    blacklistedKeys?: Array<string | BlacklistKeyConfig>;
+    stringTests?: RegExp[];
+    fuzzyKeyMatch?: boolean;
+    caseSensitiveKeyMatch?: boolean;
+    retainStructure?: boolean;
+    replaceStringByLength?: boolean;
+    replacement?: string;
+    remove?: boolean;
+    types?: Types[];
+    serialise?: boolean;
+    unsupportedTransformer?: (value: unknown) => unknown;
+}
+declare class DeepRedact {
+    private circularReference;
+    private readonly config;
+    constructor(config: DeepRedactConfig);
+    private static unsupportedTransformer;
+    private removeCircular;
+    private redactString;
+    private static complexShouldRedact;
+    private shouldRedactObjectValue;
+    private deepRedact;
+    redact: (value: unknown) => unknown;
+}
+export { DeepRedact as default, DeepRedact };

package/dist/esm/index.js

@@ -0,0 +1,125 @@
+const normaliseString = (key) => key.toLowerCase().replace(/\W/g, '');
+class DeepRedact {
+    constructor(config) {
+        var _a, _b;
+        this.circularReference = new WeakSet();
+        this.config = {
+            blacklistedKeys: [],
+            stringTests: [],
+            fuzzyKeyMatch: false,
+            caseSensitiveKeyMatch: true,
+            retainStructure: false,
+            remove: false,
+            replaceStringByLength: false,
+            replacement: '[REDACTED]',
+            types: ['string'],
+            serialise: true,
+            unsupportedTransformer: DeepRedact.unsupportedTransformer,
+        };
+        this.removeCircular = (value) => {
+            var _a, _b;
+            if (!(value instanceof Object))
+                return value;
+            if (!((_a = this.circularReference) === null || _a === void 0 ? void 0 : _a.has(value))) {
+                (_b = this.circularReference) === null || _b === void 0 ? void 0 : _b.add(value);
+                return value;
+            }
+            return '__circular__';
+        };
+        this.redactString = (value, parentShouldRedact = false) => {
+            if (!this.config.stringTests.some((test) => test.test(value)) && !parentShouldRedact)
+                return value;
+            if (this.config.replaceStringByLength)
+                return this.config.replacement.repeat(value.length);
+            return this.config.remove ? undefined : this.config.replacement;
+        };
+        this.shouldRedactObjectValue = (key) => {
+            return this.config.blacklistedKeys.some((redactableKey) => (typeof redactableKey === 'string'
+                ? key === redactableKey
+                : DeepRedact.complexShouldRedact(key, redactableKey)));
+        };
+        this.deepRedact = (value, parentShouldRedact = false) => {
+            if (value === undefined || value === null)
+                return value;
+            let safeValue = this.removeCircular(value);
+            safeValue = this.config.unsupportedTransformer(safeValue);
+            if (!(safeValue instanceof Object)) {
+                // @ts-expect-error - we already know that safeValue is not a function, symbol, undefined, null, or an object
+                if (!this.config.types.includes(typeof safeValue))
+                    return safeValue;
+                if (typeof safeValue === 'string')
+                    return this.redactString(safeValue, parentShouldRedact);
+                if (!parentShouldRedact)
+                    return safeValue;
+                return this.config.remove
+                    ? undefined
+                    : this.config.replacement;
+            }
+            if (parentShouldRedact && (!this.config.retainStructure || this.config.remove)) {
+                return this.config.remove ? undefined : this.config.replacement;
+            }
+            if (Array.isArray(safeValue))
+                return safeValue.map((val) => this.deepRedact(val, parentShouldRedact));
+            return Object.fromEntries(Object.entries(safeValue).map(([key, val]) => {
+                const shouldRedact = parentShouldRedact || this.shouldRedactObjectValue(key);
+                return [key, this.deepRedact(val, shouldRedact)];
+            }));
+        };
+        this.redact = (value) => {
+            this.circularReference = new WeakSet();
+            const redacted = this.deepRedact(value);
+            this.circularReference = null;
+            return this.config.serialise ? JSON.stringify(redacted) : redacted;
+        };
+        this.config = Object.assign(Object.assign(Object.assign({}, this.config), config), { blacklistedKeys: (_b = (_a = config.blacklistedKeys) === null || _a === void 0 ? void 0 : _a.map((key) => {
+                if (typeof key === 'string')
+                    return key;
+                return Object.assign({ fuzzyKeyMatch: this.config.fuzzyKeyMatch, caseSensitiveKeyMatch: this.config.caseSensitiveKeyMatch, retainStructure: this.config.retainStructure, remove: this.config.remove }, key);
+            })) !== null && _b !== void 0 ? _b : [] });
+    }
+}
+DeepRedact.unsupportedTransformer = (value) => {
+    if (typeof value === 'bigint') {
+        return {
+            __unsupported: {
+                type: 'bigint',
+                value: value.toString(),
+                radix: 10,
+            },
+        };
+    }
+    if (value instanceof Error) {
+        return {
+            __unsupported: {
+                type: 'error',
+                name: value.name,
+                message: value.message,
+                stack: value.stack,
+            },
+        };
+    }
+    if (value instanceof RegExp) {
+        return {
+            __unsupported: {
+                type: 'regexp',
+                source: value.source,
+                flags: value.flags,
+            },
+        };
+    }
+    if (value instanceof Date)
+        return value.toISOString();
+    return value;
+};
+DeepRedact.complexShouldRedact = (key, config) => {
+    if (config.key instanceof RegExp)
+        return config.key.test(key);
+    if (config.fuzzyKeyMatch && config.caseSensitiveKeyMatch)
+        return key.includes(config.key);
+    if (config.fuzzyKeyMatch && !config.caseSensitiveKeyMatch)
+        return normaliseString(key).includes(normaliseString(config.key));
+    if (!config.fuzzyKeyMatch && config.caseSensitiveKeyMatch)
+        return key === config.key;
+    return normaliseString(config.key) === normaliseString(key);
+};
+export { DeepRedact as default, DeepRedact };

package/package.json

@@ -0,0 +1,73 @@
+{
+  "name": "@hackylabs/deep-redact",
+  "version": "1.0.0",
+  "description": "A fast, safe and configurable zero-dependency library for redacting strings or deeply redacting arrays and objects.",
+  "private": false,
+  "license": "MIT",
+  "author": "Benjamin Green (https://bengreen.dev)",
+  "keywords": [
+    "redact",
+    "redaction",
+    "redactor",
+    "redacting",
+    "redacted",
+    "secrets",
+    "sensitive data",
+    "sensitive information",
+    "personal data",
+    "personally identifiable information",
+    "general data protection regulation",
+    "data protection",
+    "data security",
+    "GDPR",
+    "PII"
+  ],
+  "exports": {
+    ".": {
+      "import": "./dist/esm/index.js",
+      "require": "./dist/cjs/index.js",
+      "types": "./dist/esm/index.d.ts"
+    }
+  },
+  "files": [
+    "dist"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/hackylabs/deep-redact"
+  },
+  "scripts": {
+    "lint": "eslint",
+    "build": "npm run lint && npm run test && npm run bench && npm run build:esm && npm run build:cjs && npm run update-readme && npm run update-license",
+    "build:esm": "tsc --project tsconfig.esm.json",
+    "build:cjs": "tsc --project tsconfig.cjs.json",
+    "bench": "npx vitest bench --watch=false",
+    "bench:dev": "npx vitest bench",
+    "test:dev": "npx vitest",
+    "test": "npx vitest run",
+    "update-readme": "npx ts-node ./scripts/update-readme.ts",
+    "update-license": "npx ts-node ./scripts/update-license.ts"
+  },
+  "//": [
+    "fast-redact installed only as a benchmark comparison and is not used in the library",
+    "all dependencies are for development purposes only"
+  ],
+  "devDependencies": {
+    "@memlab/core": "1.1.34",
+    "@types/fast-redact": "3.0.4",
+    "@types/node": "20.14.12",
+    "@typescript-eslint/eslint-plugin": "7.14.1",
+    "@typescript-eslint/parser": "7.14.1",
+    "@vitest/coverage-v8": "2.0.4",
+    "eslint": "8.57.0",
+    "eslint-config-airbnb-base": "15.0.0",
+    "eslint-config-airbnb-typescript": "18.0.0",
+    "eslint-plugin-import": "2.29.1",
+    "eslint-plugin-n": "17.9.0",
+    "eslint-plugin-promise": "7.0.0",
+    "fast-redact": "3.5.0",
+    "image-charts": "6.1.19",
+    "typescript": "5.5.4",
+    "vitest": "2.0.4"
+  }
+}