@sepiariver/unique-set 2.0.2 → 3.1.0

package/PERF.md

@@ -1,153 +1,43 @@
 # Performance
 
-Comparison against the native Set is irrelevant. Not only does it lack value deduplication, it's so much faster it makes comparison meaningless.
+Benchmarks run with `npm run bench` on Node.js v20.18.1, Apple M2 Pro.
 
-BloomSet was initialized with the default sized bit array `6553577` and either `1` or `7` hash iterations.
+## Flat Data (`bench.spec.ts`)
 
-All timing is expressed in milliseconds.
+Mixed strings, flat objects (2 keys), and 2-element arrays with ~10-15% duplicate rate.
 
-## Shallow Data
+|   Items | MapSet `add()` | Native Set | Overhead |
+| ------: | -------------: | ---------: | -------: |
+|     400 |        1.23 ms |    0.04 ms |     ~31x |
+|   1,000 |        1.46 ms |    0.18 ms |      ~8x |
+|  20,000 |       13.09 ms |    1.46 ms |      ~9x |
+| 100,000 |       54.08 ms |    6.44 ms |      ~8x |
 
-Shallow plain objects and Arrays, with 5 - 10% duplicates
+## Nested Data (`bench-nested.spec.ts`)
 
-### BloomSet hashCount: 1
+Deeply nested objects (3-4 levels), nested arrays with objects, and mixed structures.
 
-Trial 1
+### `add()` - insert all items
 
-| Count | Unique   | Bloom   |
-| ----- | -------- | ------- |
-| 400   | 10.17    | 8.79    |
-| 1000  | 49.37    | 10.46   |
-| 20000 | 19812.43 | 2530.15 |
+|   Items |   MapSet | Native Set | Overhead |
+| ------: | -------: | ---------: | -------: |
+|     400 |  1.80 ms |    0.03 ms |     ~60x |
+|   1,000 |  3.01 ms |    0.05 ms |     ~60x |
+|  20,000 | 14.23 ms |    0.89 ms |     ~16x |
+| 100,000 | 80.60 ms |    3.80 ms |     ~21x |
 
-Trial 2
+### `has()` - query all items (50% hits, 50% misses)
 
-| Count | Unique   | Bloom   |
-| ----- | -------- | ------- |
-| 400   | 10.06    | 8.75    |
-| 1000  | 48.99    | 10.36   |
-| 20000 | 19663.32 | 2536.92 |
+|   Items | `has()` time | Queries |   Hits | per query |
+| ------: | -----------: | ------: | -----: | --------: |
+|     400 |      0.96 ms |     457 |    228 |   ~2.1 us |
+|   1,000 |      0.95 ms |   1,144 |    572 |   ~0.8 us |
+|  20,000 |     16.86 ms |  22,892 | 11,446 |   ~0.7 us |
+| 100,000 |     73.89 ms | 114,458 | 57,229 |   ~0.6 us |
 
-It's clear BloomSet has something to offer starting at just a few hundred elements.
+## Notes
 
-### BloomSet hashCount: 7
-
-Trial 1
-
-| Count | Unique   | Bloom   |
-| ----- | -------- | ------- |
-| 400   | 10.53    | 9.65    |
-| 1000  | 48.60    | 10.39   |
-| 20000 | 19242.54 | 2490.88 |
-
-Trial 2
-
-| Count | Unique   | Bloom   |
-| ----- | -------- | ------- |
-| 400   | 9.79     | 9.02    |
-| 1000  | 48.85    | 10.49   |
-| 20000 | 19255.17 | 2489.50 |
-
-Performance is fairly stable and predictable with small datasets of shallow objects, regardless of hashCount.
-
-## Nested Data
-
-Plain objects and Arrays nested 1 or 2 levels deep, with 10-20% duplicates.
-
-### BloomSet hashCount: 1
-
-Trial 1
-
-| Count | Unique   | Bloom   |
-| ----- | -------- | ------- |
-| 400   | 26.32    | 12.78   |
-| 1000  | 91.30    | 16.86   |
-| 20000 | 37671.41 | 5116.11 |
-
-Trial 2
-
-| Count | Unique   | Bloom   |
-| ----- | -------- | ------- |
-| 400   | 21.15    | 12.65   |
-| 1000  | 115.2    | 16.33   |
-| 20000 | 37169.66 | 5031.50 |
-
-UniqueSet starts to suffer on > 1000 elements. It gets exponentially worse, especially with nested objects. Whereas BloomSet's optimizations keep it in the realm of usable at 20k elements. Subjectively I feel I'm willing to spend 5 seconds processing 20k elements if I need guaranteed uniqueness-by-value (but not 30 seconds).
-
-### BloomSet hashCount: 7
-
-Trial 1
-
-| Count | Unique   | Bloom   |
-| ----- | -------- | ------- |
-| 400   | 20.58    | 13.57   |
-| 1000  | 91.23    | 16.81   |
-| 20000 | 37639.03 | 5151.90 |
-
-Running 7 hashes doesn't add a lot of clock time for BloomSet, even with nested objects. Rather than recalculating the hash over the entire serialized value, BloomSet does some bit-mixing to distribute the value's representation across the bit array.
-
-Trial 2
-
-| Count | Unique   | Bloom   |
-| ----- | -------- | ------- |
-| 400   | 22.86    | 13.48   |
-| 1000  | 94.64    | 17.80   |
-| 20000 | 37673.08 | 5276.09 |
-
-## Large (relatively)
-
-Still using the nested dataset. Very roughly 15% duplicates, distributed in a contrived manner using modulo.
-
-### hashCount: 7, size: 6553577
-
-Trial 1
-
-| Count  | Unique     | Bloom      |
-| ------ | ---------- | ---------- |
-| 100000 | 982,727.79 | 142,716.46 |
-
-```txt
-UniqueSet size: 100000 Expected size: 100000
-BloomSet size: 100000 Expected size: 100000
-Native Set size: 114458 Expected size: 114458
-```
-
-With a (relatively) large dataset, UniqueSet is slow enough to make me not want to test it again. It might be possible to squeeze extra performance from BloomSet by tweaking the config options.
-
-Trial 2
-
-| Count  | Unique     | Bloom      |
-| ------ | ---------- | ---------- |
-| 100000 | n/a        | 149600.27  |
-
-### hashCount: 1, size: 6553577
-
-Trial 1
-
-| Count  | Unique     | Bloom      |
-| ------ | ---------- | ---------- |
-| 100000 | n/a        | 135919.53  |
-
-Trial 2
-
-| Count  | Unique     | Bloom      |
-| ------ | ---------- | ---------- |
-| 100000 | n/a        | 135913.43  |
-
-Reducing the hashCount predictably improves performance by 5-10%. Collisions fallback to `fast-deep-equal`, so we can tolerate false positives unless performance degrades.
-
-#### hashCount: 1, size: 28755000
-
-Trial 1
-
-| Count  | Unique     | Bloom      |
-| ------ | ---------- | ---------- |
-| 100000 | n/a        | 128660.39  |
-
-Trial 2
-
-| Count  | Unique     | Bloom      |
-| ------ | ---------- | ---------- |
-| 100000 | n/a        | 127663.77  |
-
-Using a larger bit array requires more memory: ~3.5Mb in this case, still extremely memory-efficient for what it's doing. It seems to yield something like 5% clock time improvement over a smaller array, possibly due to decreased false positives, leading to less invocations of `fast-deep-equal` for deep comparison.
+- Native `Set` uses reference equality and cannot deduplicate objects/arrays by value. The overhead shown is the cost of deep value comparison.
+- MapSet uses a streaming 32-bit FNV-1a structural hash with `fast-deep-equal` fallback for hash collisions.
+- At 20,000 items, ~47 hash collisions are expected (birthday paradox). These are handled correctly with no impact on results.
+- The `has()` cost per query decreases at larger sizes due to V8 JIT warmup.

package/README.md

@@ -1,86 +1,71 @@
 # @sepiariver/unique-set
 
-Extends the native `Set` class to deeply compare using [fast-deep-equal](https://www.npmjs.com/package/fast-deep-equal), with optional Bloom filter optimization.
+Uses a streaming structural hash to optimize deep equality checks in a `Set`-like class. Falls back to deeply compare using [fast-deep-equal](https://www.npmjs.com/package/fast-deep-equal) when hash collisions occur.
 
-Supports ESM and CommonJS. Thanks @sakgoyal for contributing to and instigating ESM support.
+Supports ESM and CommonJS. Thanks [@sakgoyal](https://github.com/sakgoyal) for contributing to and instigating ESM support.
 
 ```js
-import { BloomSet, UniqueSet } from '@sepiariver/unique-set';
+import { MapSet, UniqueSet } from '@sepiariver/unique-set';
 ```
 
 ```js
-const { BloomSet, UniqueSet } = require('@sepiariver/unique-set');
+const { MapSet, UniqueSet } = require('@sepiariver/unique-set');
 ```
 
-WARNING: This version exports 2 classes instead of a single default class, breaking b/c with version 1.
+WARNING: Version 3 includes breaking changes. Older versions are deprecated.
 
-The overridden methods iterate through the elements of the `UniqueSet` deeply comparing equality until existence is found. If no elements match, the entire `UniqueSet` would have been iterated. However fast `fast-deep-equal` is [reported to be](https://github.com/epoberezkin/fast-deep-equal?tab=readme-ov-file#performance-benchmark), its time complexity is dependent on the depth of objects being compared. Calling it in a loop makes performance many, many times worse than the native `Set`.
+Configuration options from previous versions are no longer supported. Usage is identical to the native `Set` class.
 
-_For datasets greater than a thousand elements, there is probably a better way to achieve what you're trying to do._ Otherwise, `UniqueSet` is convenient.
+IMPORTANT: `MapSet` and `UniqueSet` are the same class (`UniqueSet` is an alias). The `delete` method uses deep equality, so `delete({a: 1})` will remove a previously added `{a: 1}` even if it's a different reference.
 
-**UPDATE:** Version 2 ships with `BloomSet`, which uses a Bloom filter to greatly optimize absence checks, falling back to `fast-deep-equal` to validate potential false positives. This class is useful for larger datasets, up to the tens of thousands or even 100k depending largely on configuration. It performs about 3-10 times faster than `UniqueSet` for datasets greater than 1000 elements. Less than a few hundred (~400) elements, `UniqueSet` can be faster—it all depens on your dataset and config options. In all scenarios except the absolute best case, BloomSet is still orders of magnitude slower than the native `Set`, but if deep equality is required, this is a decent option.
+## API
 
-Highly recommended: experiment with config options to find the best performance for your use case.
+### Constructor
 
-IMPORTANT: The `delete` method is unmodified in both classes. In the case of duplicate objects that are equivalent but have different references, the results of `delete` operations may be unexpected.
-
-## Config Options
-
-### Constructor Signature
-
-`new BloomSet(iterable = [], options = { size, hashCount });`
-
-### Options
-
-The options object allows you to customize the behavior and performance of the BloomSet. The following properties can be configured:
-
-#### 1. size (number)
-
-Description: Specifies the size of the bit array used internally by the Bloom filter. This directly impacts the memory usage and false positive probability.
-
-Default: 6,553,577 (a prime number using roughly 800 KB of memory).
-
-Recommendations:
-
-For datasets with ~100,000 elements, this default size provides excellent performance (compared against `UniqueSet`) with minimal (< 1) false positives.
-
-Larger datasets may require increasing the size for lower false positive rates. Remember though, false positives are mitigated by a fallback to `fast-deep-equal`, so you may be able to squeeze more performance from a higher tolerance for false positives, depending on your dataset.
+```js
+new MapSet(iterable?)
+new UniqueSet(iterable?)
+```
 
-#### 2. hashCount (number)
+Accepts any iterable (array, Set, generator, etc.). Duplicates by value are discarded during construction.
 
-Description: Specifies the number of hash functions used by the Bloom filter. This impacts both the false positive probability and the computational cost of adding/checking elements.
+### Methods
 
-Default: 7
+| Method | Description |
+|---|---|
+| `add(value)` | Adds `value` if no deeply-equal value exists. Returns `this`. |
+| `has(value)` | Returns `true` if a deeply-equal value is in the set. |
+| `delete(value)` | Removes the first deeply-equal value. Returns `true` if found. |
+| `clear()` | Removes all values. |
+| `forEach(cb, thisArg?)` | Calls `cb(value, value, set)` for each value. |
+| `values()` | Returns an iterator over all values. |
+| `[Symbol.iterator]()` | Makes the set iterable (e.g., `for...of`, spread). |
+| `size` | The number of unique values in the set. |
 
 ### Examples
 
-Default Configuration:
-
 ```js
-const bloomSet = new BloomSet();
-bloomSet.add("example");
-console.log(bloomSet.has("example")); // true
-```
+const set = new UniqueSet();
 
-Custom Configuration for Larger Datasets:
+set.add({ a: 1, b: 2 });
+set.add({ b: 2, a: 1 }); // same value, different key order: not added
+set.size; // 1
 
-Example 28,755,000 bit array size uses roughly 3.5 MB of memory, but this configuration is robust against datasets of something like 1M elements. The practicality of using BloomSet with that many elements is low, due to the performance hit of deep equality checks.
+set.has({ a: 1, b: 2 }); // true (deep equality, not reference)
 
-```js
-const bloomSet = new BloomSet([], { size: 28755000, hashCount: 20 });
-bloomSet.add("custom");
-console.log(bloomSet.has("custom")); // true
+set.add([1, [2, 3]]);
+set.add([1, [2, 3]]); // duplicate nested array: not added
+set.size; // 2
+
+set.delete({ a: 1, b: 2 }); // true
+set.size; // 1
 ```
 
 ### Considerations
 
-- Memory Usage: The bit array uses size / 8 bytes of memory. Even at 800 KB, initializing 1250 BloomSets in the same scope would use a gigabyte of memory.
-- False Positive Rate: The probability of a false positive is influenced by size, hashCount, and the number of elements. Adjust these values to balance performance and accuracy for your dataset.
-
-#### Further Tuning
-
-- Use a larger size for datasets exceeding 100,000 elements.
-- Reduce hashCount if performance is critical and your dataset contains very few duplicates.
+- **Memory**: Each unique value is stored once, bucketed by a 32-bit structural hash. Overhead is minimal: one `Map` entry plus a small array per hash bucket, with >99% of buckets containing exactly one item at typical sizes.
+- **Collisions**: At 20,000 items, roughly 47 hash collisions are expected (birthday paradox on 32-bit). Collisions are handled correctly via `fast-deep-equal`. They add a small cost but never affect correctness.
+- **Equality semantics**: Object key order is ignored. Array element order matters. `NaN === NaN`. `0` and `-0` are treated as equal. Functions and symbols are compared by reference.
 
 ## Installation
 
@@ -91,7 +76,7 @@ npm install @sepiariver/unique-set
 ## Usage
 
 ```js
-const { BloomSet, UniqueSet } = require('@sepiariver/unique-set');
+import { MapSet, UniqueSet } from "./dist/index.mjs";
 
 const data = [
   "string",
@@ -114,21 +99,24 @@ const data = [
   [1, 2, 3],
 ];
 
+const norm = new Set();
 const unique1 = new UniqueSet();
 data.forEach((el) => {
   unique1.add(el);
+  norm.add(el);
 });
 const unique2 = new UniqueSet(data);
 console.log(unique1.size); // 6 instead of 8 with Set
 console.log(unique2.size); // 6
+console.log(norm.size); // 8 with Set
 
-const bloom1 = new BloomSet();
+const map1 = new MapSet();
 data.forEach((el) => {
-  bloom1.add(el);
+  map1.add(el);
 });
-const bloom2 = new BloomSet(data);
-console.log(bloom1.size); // 6 instead of 8 with Set
-console.log(bloom2.size); // 6
+const map2 = new MapSet(data);
+console.log(map1.size); // 6 instead of 8 with Set
+console.log(map2.size); // 6
 ```
 
 ## Testing

package/dist/index.d.mts

@@ -1,41 +1,15 @@
-/** A `Set` extension that ensures uniqueness of items using deep equality checks. */
-declare class UniqueSet<T> extends Set<T> {
-    /*** @throws TypeError If the input is not iterable. */
-    constructor(iterable?: Iterable<T>);
-    /**
-     * Determines whether an object is in the UniqueSet using deep equality.
-     * @param o The object to check for presence in the UniqueSet.
-     * @returns `true` if the object is found, `false` otherwise.
-     */
-    has(o: T): boolean;
-    /**
-     * Adds a new object to the UniqueSet if it is not already present.
-     * @param o The object to add to the UniqueSet.
-     * @returns The `UniqueSet` instance, allowing for chaining.
-     */
-    add(o: T): this;
-}
-/** A `Set` extension that uses a Bloom filter for fast existence checks combined with deep equality for accuracy. */
-declare class BloomSet<T> extends Set<T> {
+declare const structuralHash: (value: unknown) => number;
+declare class MapSet<T> {
     #private;
-    /**
-     * Creates a new `BloomSet` instance.
-     * @param iterable Optional: an iterable object with which to initialize the BloomSet.
-     * @param options Bloom filter configuration options.
-     * @param options.size The size of the Bloom filter's bit array. Defaults to 6553577.
-     * @param options.hashCount The number of hash functions to use. Defaults to 7.
-     * @throws TypeError If the input is not iterable.
-     */
-    constructor(iterable?: Iterable<T>, options?: {
-        size?: number;
-        hashCount?: number;
-    });
-    /** Determines existence of an object in the BloomSet using the Bloom filter and deep equality */
-    has(o: T): boolean;
-    /** Adds a new object to the BloomSet if it is not already present.
-     * @returns The `BloomSet` instance, allowing for chaining.
-     */
-    add(o: T): this;
+    constructor(iterable?: Iterable<T>);
+    add(value: T): this;
+    has(value: T): boolean;
+    delete(value: T): boolean;
+    get size(): number;
+    clear(): void;
+    forEach(callback: (value: T, valueAgain: T, set: this) => void, thisArg?: any): void;
+    values(): IterableIterator<T>;
+    [Symbol.iterator](): IterableIterator<T>;
 }
 
-export { BloomSet, UniqueSet };
+export { MapSet, MapSet as UniqueSet, structuralHash };

package/dist/index.d.ts

@@ -0,0 +1,15 @@
+declare const structuralHash: (value: unknown) => number;
+declare class MapSet<T> {
+    #private;
+    constructor(iterable?: Iterable<T>);
+    add(value: T): this;
+    has(value: T): boolean;
+    delete(value: T): boolean;
+    get size(): number;
+    clear(): void;
+    forEach(callback: (value: T, valueAgain: T, set: this) => void, thisArg?: any): void;
+    values(): IterableIterator<T>;
+    [Symbol.iterator](): IterableIterator<T>;
+}
+
+export { MapSet, MapSet as UniqueSet, structuralHash };