@sepiariver/unique-set 2.0.1 → 2.0.3

package/PERF.md

@@ -0,0 +1,153 @@
+# Performance
+
+Comparison against the native Set is irrelevant. Not only does it lack value deduplication, it's so much faster it makes comparison meaningless.
+
+BloomSet was initialized with the default sized bit array `6553577` and either `1` or `7` hash iterations.
+
+All timing is expressed in milliseconds.
+
+## Shallow Data
+
+Shallow plain objects and Arrays, with 5 - 10% duplicates
+
+### BloomSet hashCount: 1
+
+Trial 1
+
+| Count | Unique   | Bloom   |
+| ----- | -------- | ------- |
+| 400   | 10.17    | 8.79    |
+| 1000  | 49.37    | 10.46   |
+| 20000 | 19812.43 | 2530.15 |
+
+Trial 2
+
+| Count | Unique   | Bloom   |
+| ----- | -------- | ------- |
+| 400   | 10.06    | 8.75    |
+| 1000  | 48.99    | 10.36   |
+| 20000 | 19663.32 | 2536.92 |
+
+It's clear BloomSet has something to offer starting at just a few hundred elements.
+
+### BloomSet hashCount: 7
+
+Trial 1
+
+| Count | Unique   | Bloom   |
+| ----- | -------- | ------- |
+| 400   | 10.53    | 9.65    |
+| 1000  | 48.60    | 10.39   |
+| 20000 | 19242.54 | 2490.88 |
+
+Trial 2
+
+| Count | Unique   | Bloom   |
+| ----- | -------- | ------- |
+| 400   | 9.79     | 9.02    |
+| 1000  | 48.85    | 10.49   |
+| 20000 | 19255.17 | 2489.50 |
+
+Performance is fairly stable and predictable with small datasets of shallow objects, regardless of hashCount.
+
+## Nested Data
+
+Plain objects and Arrays nested 1 or 2 levels deep, with 10-20% duplicates.
+
+### BloomSet hashCount: 1
+
+Trial 1
+
+| Count | Unique   | Bloom   |
+| ----- | -------- | ------- |
+| 400   | 26.32    | 12.78   |
+| 1000  | 91.30    | 16.86   |
+| 20000 | 37671.41 | 5116.11 |
+
+Trial 2
+
+| Count | Unique   | Bloom   |
+| ----- | -------- | ------- |
+| 400   | 21.15    | 12.65   |
+| 1000  | 115.2    | 16.33   |
+| 20000 | 37169.66 | 5031.50 |
+
+UniqueSet starts to suffer on > 1000 elements. It gets exponentially worse, especially with nested objects. Whereas BloomSet's optimizations keep it in the realm of usable at 20k elements. Subjectively I feel I'm willing to spend 5 seconds processing 20k elements if I need guaranteed uniqueness-by-value (but not 30 seconds).
+
+### BloomSet hashCount: 7
+
+Trial 1
+
+| Count | Unique   | Bloom   |
+| ----- | -------- | ------- |
+| 400   | 20.58    | 13.57   |
+| 1000  | 91.23    | 16.81   |
+| 20000 | 37639.03 | 5151.90 |
+
+Running 7 hashes doesn't add a lot of clock time for BloomSet, even with nested objects. Rather than recalculating the hash over the entire serialized value, BloomSet does some bit-mixing to distribute the value's representation across the bit array.
+
+Trial 2
+
+| Count | Unique   | Bloom   |
+| ----- | -------- | ------- |
+| 400   | 22.86    | 13.48   |
+| 1000  | 94.64    | 17.80   |
+| 20000 | 37673.08 | 5276.09 |
+
+## Large (relatively)
+
+Still using the nested dataset. Very roughly 15% duplicates, distributed in a contrived manner using modulo.
+
+### hashCount: 7, size: 6553577
+
+Trial 1
+
+| Count  | Unique     | Bloom      |
+| ------ | ---------- | ---------- |
+| 100000 | 982,727.79 | 142,716.46 |
+
+```txt
+UniqueSet size: 100000 Expected size: 100000
+BloomSet size: 100000 Expected size: 100000
+Native Set size: 114458 Expected size: 114458
+```
+
+With a (relatively) large dataset, UniqueSet is slow enough to make me not want to test it again. It might be possible to squeeze extra performance from BloomSet by tweaking the config options.
+
+Trial 2
+
+| Count  | Unique     | Bloom      |
+| ------ | ---------- | ---------- |
+| 100000 | n/a        | 149600.27  |
+
+### hashCount: 1, size: 6553577
+
+Trial 1
+
+| Count  | Unique     | Bloom      |
+| ------ | ---------- | ---------- |
+| 100000 | n/a        | 135919.53  |
+
+Trial 2
+
+| Count  | Unique     | Bloom      |
+| ------ | ---------- | ---------- |
+| 100000 | n/a        | 135913.43  |
+
+Reducing the hashCount predictably improves performance by 5-10%. Collisions fallback to `fast-deep-equal`, so we can tolerate false positives unless performance degrades.
+
+#### hashCount: 1, size: 28755000
+
+Trial 1
+
+| Count  | Unique     | Bloom      |
+| ------ | ---------- | ---------- |
+| 100000 | n/a        | 128660.39  |
+
+Trial 2
+
+| Count  | Unique     | Bloom      |
+| ------ | ---------- | ---------- |
+| 100000 | n/a        | 127663.77  |
+
+Using a larger bit array requires more memory: ~3.5Mb in this case, still extremely memory-efficient for what it's doing. It seems to yield something like 5% clock time improvement over a smaller array, possibly due to decreased false positives, leading to less invocations of `fast-deep-equal` for deep comparison.

package/README.md

@@ -2,7 +2,7 @@
 
 Extends the native `Set` class to deeply compare using [fast-deep-equal](https://www.npmjs.com/package/fast-deep-equal), with optional Bloom filter optimization.
 
-Supports ESM and CommonJS.
+Supports ESM and CommonJS. Thanks [@sakgoyal](https://github.com/sakgoyal) for contributing to and instigating ESM support.
 
 ```js
 import { BloomSet, UniqueSet } from '@sepiariver/unique-set';
@@ -137,10 +137,19 @@ console.log(bloom2.size); // 6
 2. `npm install`
 3. `npm run test`
 
+## Issues
+
+Issue reporting is encouraged: [https://github.com/sepiariver/unique-set/issues]
+
 ## Contributing
 
 Submit pull requests to [https://github.com/sepiariver/unique-set/pulls]
 
-## Issues
+## Contributors
 
-Issue reporting is encouraged: [https://github.com/sepiariver/unique-set/issues]
+- @sepiariver
+- @sakgoyal
+
+## License
+
+MIT

package/dist/index.mjs

@@ -1,5 +1,48 @@
 // index.ts
 import equal from "fast-deep-equal/es6/index.js";
+var serialize = (item) => {
+  if (typeof item === "number" && isNaN(item)) {
+    return "NaN";
+  }
+  if (item && typeof item === "object") {
+    if (Array.isArray(item)) {
+      return `[${item.map(serialize).join("")}]`;
+    } else {
+      return `{${Object.entries(item).sort(([a], [b]) => a.localeCompare(b)).map(([k, v]) => `${k}:${serialize(v)}`).join("")}}`;
+    }
+  }
+  return String(item);
+};
+var fnv1a = (str) => {
+  if (typeof str !== "string") {
+    str = String(str);
+  }
+  let hash = 2166136261;
+  for (let i = 0; i < str.length; i++) {
+    hash ^= str.charCodeAt(i);
+    hash = hash * 16777619 >>> 0;
+  }
+  return hash >>> 0;
+};
+var findNextPrime = (num) => {
+  if (num < 2) return 2;
+  if ((num & 1) === 0) num++;
+  while (!isPrime(num)) {
+    num += 2;
+  }
+  return num;
+};
+var isPrime = (num) => {
+  if (num < 2) return false;
+  if (num === 2 || num === 3) return true;
+  if ((num & 1) === 0) return false;
+  if (num % 3 === 0) return false;
+  const sqrt = Math.sqrt(num);
+  for (let i = 5; i <= sqrt; i += 6) {
+    if (num % i === 0 || num % (i + 2) === 0) return false;
+  }
+  return true;
+};
 var UniqueSet = class extends Set {
   /*** @throws TypeError If the input is not iterable. */
   constructor(iterable = []) {
@@ -62,7 +105,7 @@ var BloomSet = class extends Set {
     if (typeof size !== "number" || size <= 0) {
       size = 6553577;
     }
-    this.#aSize = this.#findNextPrime(size);
+    this.#aSize = findNextPrime(size);
     if (typeof hashCount !== "number" || hashCount <= 0) {
       hashCount = 7;
     }
@@ -73,45 +116,10 @@ var BloomSet = class extends Set {
     }
   }
   /** @internal */
-  #findNextPrime(num) {
-    if (num < 2) return 2;
-    if (num % 2 === 0) num++;
-    while (!this.#isPrime(num)) {
-      num += 2;
-    }
-    return num;
-  }
-  /** @internal */
-  #isPrime(num) {
-    if (num < 2) return false;
-    if (num === 2 || num === 3) return true;
-    if (num % 2 === 0 || num % 3 === 0) return false;
-    const sqrt = Math.floor(Math.sqrt(num));
-    for (let i = 5; i <= sqrt; i += 6) {
-      if (num % i === 0 || num % (i + 2) === 0) return false;
-    }
-    return true;
-  }
-  /** @internal */
-  #serialize(item) {
-    if (typeof item === "number" && isNaN(item)) {
-      return "NaN";
-    }
-    if (item && typeof item === "object") {
-      const serialize = this.#serialize.bind(this);
-      if (Array.isArray(item)) {
-        return `[${item.map(serialize).join(",")}]`;
-      } else {
-        return `{${Object.entries(item).sort(([a], [b]) => a.localeCompare(b)).map(([k, v]) => `${k}:${serialize(v)}`).join(",")}}`;
-      }
-    }
-    return String(item);
-  }
-  /** @internal */
   #hashes(item) {
     const hashes = [];
-    const str = this.#serialize(item);
-    let hash = this.#fnv1a(str);
+    const str = serialize(item);
+    let hash = fnv1a(str);
     for (let i = 0; i < this.#hashCount; i++) {
       hash %= this.#aSize;
       hashes.push(hash);
@@ -121,18 +129,6 @@ var BloomSet = class extends Set {
     return hashes;
   }
   /** @internal */
-  #fnv1a(str) {
-    if (typeof str !== "string") {
-      str = String(str);
-    }
-    let hash = 2166136261;
-    for (let i = 0; i < str.length; i++) {
-      hash ^= str.charCodeAt(i);
-      hash = hash * 16777619 >>> 0;
-    }
-    return hash >>> 0;
-  }
-  /** @internal */
   #setBits(hashes) {
     for (const hash of hashes) {
       const index = Math.floor(hash / 8);

package/index.ts

@@ -1,5 +1,63 @@
 import equal from "fast-deep-equal/es6/index.js";
 
+/** Utility functions */
+
+const serialize = (item: any | number | object): string => {
+  if (typeof item === "number" && isNaN(item)) {
+    return "NaN";
+  }
+
+  if (item && typeof item === "object") {
+    if (Array.isArray(item)) {
+      return `[${item.map(serialize).join("")}]`;
+    } else {
+      return `{${Object.entries(item)
+        .sort(([a], [b]) => a.localeCompare(b))
+        .map(([k, v]) => `${k}:${serialize(v)}`)
+        .join("")}}`;
+    }
+  }
+
+  return String(item);
+};
+
+const fnv1a = (str: string) => {
+  if (typeof str !== "string") {
+    str = String(str);
+  }
+  let hash = 2166136261; // FNV offset basis for 32-bit
+  for (let i = 0; i < str.length; i++) {
+    hash ^= str.charCodeAt(i);
+    hash = (hash * 16777619) >>> 0; // Multiply by the FNV prime and ensure 32-bit unsigned
+  }
+  return hash >>> 0;
+};
+
+const findNextPrime = (num: number) => {
+  if (num < 2) return 2;
+  if ((num & 1) === 0) num++; // Odd numbers only
+
+  while (!isPrime(num)) {
+    num += 2; // Odd numbers only
+  }
+
+  return num;
+};
+
+const isPrime = (num: number): boolean => {
+  if (num < 2) return false;
+  if (num === 2 || num === 3) return true;
+  if ((num & 1) === 0) return false;
+  if (num % 3 === 0) return false;
+
+  const sqrt = Math.sqrt(num);
+  for (let i = 5; i <= sqrt; i += 6) {
+    if (num % i === 0 || num % (i + 2) === 0) return false;
+  }
+
+  return true;
+};
+
 /** A `Set` extension that ensures uniqueness of items using deep equality checks. */
 export class UniqueSet<T> extends Set<T> {
   /*** @throws TypeError If the input is not iterable. */
@@ -72,7 +130,7 @@ export class BloomSet<T> extends Set<T> {
     if (typeof size !== "number" || size <= 0) {
       size = 6553577; // Targeting < 1 collision per 100,000 elements, ~819 KB memory, needs 7 hashes
     }
-    this.#aSize = this.#findNextPrime(size);
+    this.#aSize = findNextPrime(size);
 
     if (typeof hashCount !== "number" || hashCount <= 0) {
       hashCount = 7;
@@ -85,58 +143,11 @@ export class BloomSet<T> extends Set<T> {
     }
   }
 
-  /** @internal */
-  #findNextPrime(num: number) {
-    if (num < 2) return 2;
-    if (num % 2 === 0) num++; // Odd numbers only
-
-    while (!this.#isPrime(num)) {
-      num += 2; // Odd numbers only
-    }
-
-    return num;
-  }
-
-  /** @internal */
-  #isPrime(num: number) {
-    if (num < 2) return false;
-    if (num === 2 || num === 3) return true;
-    if (num % 2 === 0 || num % 3 === 0) return false;
-
-    const sqrt = Math.floor(Math.sqrt(num));
-    for (let i = 5; i <= sqrt; i += 6) {
-      if (num % i === 0 || num % (i + 2) === 0) return false;
-    }
-
-    return true;
-  }
-
-  /** @internal */
-  #serialize(item: T | number | object): string {
-    if (typeof item === "number" && isNaN(item)) {
-      return "NaN";
-    }
-
-    if (item && typeof item === "object") {
-      const serialize = this.#serialize.bind(this);
-      if (Array.isArray(item)) {
-        return `[${item.map(serialize).join(",")}]`;
-      } else {
-        return `{${Object.entries(item)
-          .sort(([a], [b]) => a.localeCompare(b))
-          .map(([k, v]) => `${k}:${serialize(v)}`)
-          .join(",")}}`;
-      }
-    }
-
-    return String(item);
-  }
-
   /** @internal */
   #hashes(item: T) {
     const hashes: number[] = [];
-    const str = this.#serialize(item);
-    let hash = this.#fnv1a(str); // Base hash
+    const str = serialize(item);
+    let hash = fnv1a(str); // Base hash
 
     // Bloom into hashCount hash values
     for (let i = 0; i < this.#hashCount; i++) {
@@ -151,19 +162,6 @@ export class BloomSet<T> extends Set<T> {
     return hashes;
   }
 
-  /** @internal */
-  #fnv1a(str: string) {
-    if (typeof str !== "string") {
-      str = String(str);
-    }
-    let hash = 2166136261; // FNV offset basis for 32-bit
-    for (let i = 0; i < str.length; i++) {
-      hash ^= str.charCodeAt(i);
-      hash = (hash * 16777619) >>> 0; // Multiply by the FNV prime and ensure 32-bit unsigned
-    }
-    return hash >>> 0;
-  }
-
   /** @internal */
   #setBits(hashes: number[]): void {
     for (const hash of hashes) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@sepiariver/unique-set",
-  "version": "2.0.1",
+  "version": "2.0.3",
   "description": "Extends the native Set class to deeply compare using fast-deep-equal, with optional Bloom filter optimization. This version exports 2 classes instead of a default, breaking b/c with version 1.",
   "main": "dist/index.js",
   "module": "dist/index.mjs",
@@ -10,7 +10,8 @@
     "import": "./dist/index.mjs"
   },
   "scripts": {
-    "test": "npm run build && vitest",
+    "test": "npm run build && vitest basic.spec.ts",
+    "bench": "npm run build && vitest bench.spec.ts bench-nested.spec.ts",
     "lint": "tsc",
     "build": "tsup index.ts --format esm --dts"
   },

package/temp.cjs

@@ -1,10 +0,0 @@
-const { BloomSet, UniqueSet } = require("./dist/index.js");
-
-const bloom = new BloomSet();
-bloom.add("foo");
-console.log(bloom.has("foo")); // true
-
-const unique = new UniqueSet();
-unique.add("foo");
-unique.add("foo");
-console.log(unique.size); // 1

package/temp.mjs

@@ -1,10 +0,0 @@
-import { BloomSet, UniqueSet } from "./dist/index.js";
-
-const bloom = new BloomSet();
-bloom.add("foo");
-console.log(bloom.has("foo")); // true
-
-const unique = new UniqueSet();
-unique.add("foo");
-unique.add("foo");
-console.log(unique.size); // 1