@oschwald/maxminddb 0.2.1 → 0.3.1

package/CHANGELOG.md

@@ -5,6 +5,89 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.1] - 2026-07-19
+
+### Fixed
+
+- Updated the packed-package smoke test to accept the npm 12
+  `npm pack --json` output format, allowing trusted publishing validation to
+  complete with the current npm release.
+
+## [0.3.0] - 2026-07-19
+
+### Added
+
+- Added `reloadAsync()` to open and validate replacement databases without
+  blocking the JavaScript event loop.
+- Added `getPaths()` and `getManyPaths()` for decoding multiple selected values
+  with one database lookup per IP.
+- Added `networksPath()` and `withinPath()` for selectively decoding one value
+  while iterating database networks.
+
+### Changed
+
+- Numeric path components now reject fractional, non-finite, and unsafe
+  JavaScript numbers instead of relying on native integer coercion.
+- Updated the Rust `maxminddb` dependency to 0.30.0, inheriting safer corrupt
+  database traversal and faster path and record decoding.
+- Invalid UTF-8 in decoded MMDB string values and map keys now follows Node's
+  native conversion behavior and is represented with replacement characters.
+- `MODE_BUFFER` now uses the same native owned-memory reader as `MODE_MEMORY`;
+  the constant remains available as a compatibility alias.
+- Network iterators now close their native cursor when a `for...of` loop exits
+  early.
+- Compiled `PathLookup` instances can be closed explicitly, release their
+  native path automatically after collection, and survive watched reader swaps.
+- Consolidated mutable native reader state behind one runtime borrow, keeping
+  reentrant JavaScript access safe without per-field borrow checks.
+- Stopped exposing undocumented native reader constructors and helpers from the
+  JavaScript entry point, leaving `nativeVersion()` as the supported diagnostic.
+
+### Fixed
+
+- Preserved complete invalid IP values in errors when an overlong string crosses
+  the native UTF-8 buffer boundary at a multibyte character.
+
+### Performance
+
+- Moved asynchronous reload work, including memory-mapped reloads triggered by
+  file watchers, off the JavaScript event loop.
+- Added multi-field projections that reuse one search-tree lookup per IP,
+  improving batch throughput when several fields are needed without decoding
+  the full record.
+- Moved gzip input detection into the native readers, checking owned-file
+  prefixes before allocating their buffers and eliminating an extra
+  asynchronous file open, read, and close during path-based opens.
+- Decode MMDB strings and map keys directly from raw bytes into JavaScript
+  strings, avoiding redundant UTF-8 decoding in Rust and Node.
+- Reduced lookup and batch allocation by reading IP strings directly from V8
+  and constructing batch result arrays in place.
+- Reuse records with shared data offsets within cached network cursor pages,
+  avoiding repeated decoding during large network walks.
+- Format network strings on the stack and construct network-result pairs in
+  place, avoiding two heap allocations per iterated network.
+- Construct decoded arrays and objects in a single pass and carry JavaScript
+  decode context through nested values, reducing allocations and TLS access.
+- Use a protected LRU segment for caches up to 10,000 records, retaining
+  frequently reused records during broad scans without increasing capacity.
+- Open owned-memory file modes on a worker thread and read directly into Rust
+  memory, avoiding a full intermediate Node `Buffer` and event-loop blocking.
+- Store common compiled paths inline, avoiding a heap allocation on each hot
+  path lookup.
+- Return native network pages directly when no buffered records remain,
+  avoiding an extra JavaScript array allocation and copy per page.
+- Enable thin link-time optimization for release builds, reducing the native
+  binary size and allowing optimization across crate boundaries.
+
+### Development
+
+- Corrected projection benchmark found counts to exclude rows containing only
+  missing values.
+- Added focused unit coverage for the custom IPv4 parser, IPv4 prefix
+  translation, signed path indexes, and inline path storage boundary.
+- Removed the unused eager native network collector; the public API continues
+  to use the lower-memory native cursor implementation.
+
 ## [0.2.1] - 2026-07-03
 
 * Test fix. No other changes.
@@ -83,5 +166,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added Rust formatting, `cargo check`, clippy, TypeScript, Node test, npm pack,
   and packed-package smoke-test validation.
 
+[0.3.1]: https://github.com/oschwald/maxminddb-node/releases/tag/v0.3.1
+[0.3.0]: https://github.com/oschwald/maxminddb-node/releases/tag/v0.3.0
+[0.2.1]: https://github.com/oschwald/maxminddb-node/releases/tag/v0.2.1
 [0.2.0]: https://github.com/oschwald/maxminddb-node/releases/tag/v0.2.0
 [0.1.0]: https://github.com/oschwald/maxminddb-node/releases/tag/v0.1.0

package/README.md

@@ -61,9 +61,9 @@ cache sizes can therefore retain significant heap memory when the database
 records are large or lookups touch many distinct data offsets. Cached records
 are returned by reference, so mutating a cached record can affect later lookups
 for the same data offset until that entry is evicted, `reader.clearCache()` is
-called, or the reader is closed. `getPath()`, `getManyPath()`, and compiled
-`reader.path()` lookups decode only the requested path and do not populate the
-full-record cache.
+called, or the reader is closed. `getPath()`, `getPaths()`, `getManyPath()`,
+`getManyPaths()`, and compiled `reader.path()` lookups decode only the requested
+paths and do not populate the full-record cache.
 
 When `watchForUpdates` is enabled, file-change reloads run serially. A failed
 watched reload leaves the existing reader active, stores the failure on
@@ -71,12 +71,24 @@ watched reload leaves the existing reader active, stores the failure on
 reload clears `lastReloadError` and calls the hook. Close watched readers with
 `reader.close()` to remove the file watcher.
 
+Call `await reader.reloadAsync()` to open and validate a replacement database
+on a worker thread before atomically swapping it into the reader. The existing
+`reader.reload()` remains available when synchronous replacement is preferred.
+
 ## Extensions
 
 ```js
 reader.getPath('8.8.8.8', ['country', 'iso_code']);
 reader.getMany(['8.8.8.8', '1.1.1.1']);
 reader.getManyPath(['8.8.8.8', '1.1.1.1'], ['country', 'iso_code']);
+reader.getPaths('8.8.8.8', [
+  ['country', 'iso_code'],
+  ['continent', 'code'],
+]);
+reader.getManyPaths(['8.8.8.8', '1.1.1.1'], [
+  ['country', 'iso_code'],
+  ['continent', 'code'],
+]);
 
 const countryCode = reader.path(['country', 'iso_code']);
 countryCode.get('8.8.8.8');
@@ -91,20 +103,32 @@ for (const page of reader.withinPages('81.2.69.0/24', { pageSize: 100 })) {
     console.log(network, record);
   }
 }
+
+for (const [network, country] of reader.withinPath(
+  '81.2.69.0/24',
+  ['country', 'iso_code']
+)) {
+  console.log(network, country);
+}
 ```
 
-Path elements are strings for map keys and numbers for array indexes. Negative
-indexes count from the end of an array.
+Path elements are strings for map keys and finite, safe-integer numbers for
+array indexes. Negative indexes count from the end of an array.
 
 Create compiled path lookups once and reuse them in hot paths. `reader.path()`
 parses and stores the path, and the returned `PathLookup` avoids reparsing the
 path array on each lookup.
 
-For high-volume lookup workloads, prefer `getMany()` or `getManyPath()` when
-you can batch IPs. They cross the native boundary once for the whole batch and
-are significantly faster than calling `get()` in a JavaScript loop.
+For high-volume lookup workloads, prefer `getMany()`, `getManyPath()`, or
+`getManyPaths()` when you can batch IPs. They cross the native boundary once
+for the whole batch and are significantly faster than calling `get()` in a
+JavaScript loop. `getPaths()` and `getManyPaths()` perform one database lookup
+per IP and return projected values in the same order as the requested paths;
+missing paths are returned as `null`.
 
 `networks()` and `within()` return lazy iterators backed by native cursors.
+Use `networksPath()` or `withinPath()` to decode only a selected field while
+walking networks.
 For large network walks, use `networkPages()`, `withinPages()`, or
 `NetworkIterator#nextPage()` to cross the native boundary once per page rather
 than once per network.
@@ -118,8 +142,8 @@ Path-based `open()` defaults to memory-mapped reads:
 - `MODE_MEMORY`
 - `MODE_BUFFER`
 
-Use `MODE_BUFFER` if you want `open()` to read the file into a Node `Buffer`
-before constructing the reader.
+`MODE_MEMORY` and `MODE_BUFFER` asynchronously read the file once into
+Rust-owned memory. `MODE_BUFFER` is retained as a compatibility alias.
 
 ```js
 import maxmind from '@oschwald/maxminddb';
@@ -133,11 +157,9 @@ Mode tradeoffs:
 
 - `MODE_MMAP`/`MODE_AUTO` opens quickly and keeps RSS low by mapping the
   database file. Replace database files atomically when using watched reloads.
-- `MODE_MEMORY` reads the database into Rust-owned memory. It costs more memory
-  at open time but is independent of the source file after open.
-- `MODE_BUFFER` reads the database into a Node `Buffer` before constructing the
-  native reader. Use it when you need Node-side file loading behavior or when
-  tests need to mutate a watched temporary file safely.
+- `MODE_MEMORY`/`MODE_BUFFER` asynchronously read the database into Rust-owned
+  memory. They cost more memory at open time but are independent of the source
+  file after open.
 
 ## Performance Notes
 

package/index.d.ts

@@ -193,15 +193,20 @@ export interface CacheStats {
   readonly evictions: number;
 }
 
+export type LookupPath = ReadonlyArray<string | number>;
+
 export declare class PathLookup<TValue = unknown> {
   readonly path: ReadonlyArray<string | number>;
   get(ipAddress: string): TValue | null;
   getMany(ipAddresses: ReadonlyArray<string>): Array<TValue | null>;
+  close(): void;
 }
 
-export declare class NetworkIterator<T extends Response = Response>
+export declare class NetworkIterator<T = Response>
   implements IterableIterator<[string, T | null]> {
+  readonly path: ReadonlyArray<string | number> | null;
   next(): IteratorResult<[string, T | null]>;
+  return(): IteratorReturnResult<undefined>;
   nextPage(pageSize?: number): Array<[string, T | null]>;
   pages(pageSize?: number): IterableIterator<Array<[string, T | null]>>;
   close(): void;
@@ -220,22 +225,40 @@ export declare class Reader<T extends Response = Response> {
   metadata: Metadata;
   load(database: Buffer): void;
   reload(): void;
+  reloadAsync(): Promise<void>;
   close(): void;
   clearCache(): void;
   cacheStats(): CacheStats;
   get(ipAddress: string): T | null;
-  getPath(ipAddress: string, path: ReadonlyArray<string | number>): unknown;
+  getPath(ipAddress: string, path: LookupPath): unknown;
+  getPaths<TValues extends ReadonlyArray<unknown> = unknown[]>(
+    ipAddress: string,
+    paths: ReadonlyArray<LookupPath>,
+  ): TValues;
   path<TValue = unknown>(
-    path: ReadonlyArray<string | number>,
+    path: LookupPath,
   ): PathLookup<TValue>;
   getWithPrefixLength(ipAddress: string): [T | null, number];
   getMany(ipAddresses: ReadonlyArray<string>): Array<T | null>;
   getManyPath(
     ipAddresses: ReadonlyArray<string>,
-    path: ReadonlyArray<string | number>,
+    path: LookupPath,
   ): unknown[];
+  getManyPaths<TValues extends ReadonlyArray<unknown> = unknown[]>(
+    ipAddresses: ReadonlyArray<string>,
+    paths: ReadonlyArray<LookupPath>,
+  ): TValues[];
   networks(options?: NetworkPagesOptions): NetworkIterator<T>;
   within(cidr: string, options?: NetworkPagesOptions): NetworkIterator<T>;
+  networksPath<TValue = unknown>(
+    path: ReadonlyArray<string | number>,
+    options?: NetworkPagesOptions,
+  ): NetworkIterator<TValue>;
+  withinPath<TValue = unknown>(
+    cidr: string,
+    path: ReadonlyArray<string | number>,
+    options?: NetworkPagesOptions,
+  ): NetworkIterator<TValue>;
   networkPages(
     options?: NetworkPagesOptions,
   ): IterableIterator<Array<[string, T | null]>>;

package/index.js

@@ -54,12 +54,14 @@ function loadNativeBinding() {
 }
 
 const native = loadNativeBinding();
+const pathFinalizer = new FinalizationRegistry(({ nativeReader, pathId }) => {
+  try {
+    nativeReader.releasePath(pathId);
+  } catch {
+    // Reader shutdown may have already released all compiled paths.
+  }
+});
 
-const DEFAULT_LARGE_FILE_THRESHOLD = 512 * 1024 * 1024;
-const LARGE_FILE_THRESHOLD = normalizeLargeFileThreshold(
-  process.env.MAXMINDDB_LARGE_FILE_THRESHOLD_BYTES
-);
-const STREAM_WATERMARK = 8 * 1024 * 1024;
 const DEFAULT_CACHE_MAX = 10_000;
 const MAX_CACHE_MAX = 0xffffffff;
 const legacyErrorMessage = `Maxmind v2 module has changed API.
@@ -74,36 +76,6 @@ const MODE_MMAP = 'mmap';
 const MODE_MEMORY = 'memory';
 const MODE_BUFFER = 'buffer';
 
-function normalizeLargeFileThreshold(value) {
-  if (value == null || value === '') {
-    return DEFAULT_LARGE_FILE_THRESHOLD;
-  }
-
-  const threshold = Number(value);
-  return Number.isSafeInteger(threshold) && threshold >= 0
-    ? threshold
-    : DEFAULT_LARGE_FILE_THRESHOLD;
-}
-
-function isGzipBuffer(buffer) {
-  return buffer.length >= 2 && buffer[0] === 0x1f && buffer[1] === 0x8b;
-}
-
-async function assertNotGzipFile(filepath) {
-  const handle = await fs.promises.open(filepath, 'r');
-  try {
-    const buffer = Buffer.alloc(2);
-    const { bytesRead } = await handle.read(buffer, 0, 2, 0);
-    if (bytesRead === 2 && isGzipBuffer(buffer)) {
-      throw new Error(
-        'Looks like you are passing in a file in gzip format, please use mmdb database instead.'
-      );
-    }
-  } finally {
-    await handle.close();
-  }
-}
-
 function normalizeMode(mode) {
   if (mode == null || mode === MODE_AUTO) {
     return MODE_MMAP;
@@ -178,93 +150,74 @@ function waitForFile(filepath) {
   });
 }
 
-async function readLargeFile(filepath, size) {
-  return new Promise((resolve, reject) => {
-    const buffer = Buffer.allocUnsafe(size);
-    let offset = 0;
-    let settled = false;
-    const stream = fs.createReadStream(filepath, {
-      highWaterMark: STREAM_WATERMARK,
-    });
-
-    const finish = (error, value) => {
-      if (settled) {
-        return;
-      }
-      settled = true;
-      if (error) {
-        stream.destroy();
-        reject(error);
-      } else {
-        resolve(value);
-      }
-    };
-
-    stream.on('data', (chunk) => {
-      const bufferChunk = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
-      if (offset + bufferChunk.length > size) {
-        finish(
-          new Error(
-            `File changed while reading ${filepath}: expected ${size} bytes but read more`
-          )
-        );
-        return;
-      }
-      bufferChunk.copy(buffer, offset);
-      offset += bufferChunk.length;
-    });
-    stream.on('end', () => {
-      if (offset !== size) {
-        finish(
-          new Error(
-            `File changed while reading ${filepath}: expected ${size} bytes but read ${offset}`
-          )
-        );
-        return;
-      }
-      finish(null, buffer);
-    });
-    stream.on('error', (error) => finish(error));
-  });
-}
-
-async function readFile(filepath) {
-  const stat = await fs.promises.stat(filepath);
-  return stat.size < LARGE_FILE_THRESHOLD
-    ? fs.promises.readFile(filepath)
-    : readLargeFile(filepath, stat.size);
-}
-
 class PathLookup {
   constructor(reader, path) {
     this._reader = reader;
-    this._pathId = reader._reader.compilePath(path);
     this.path = Object.freeze([...path]);
+    this._closed = false;
+    this._compileFor(reader._reader);
   }
 
   get(ipAddress) {
-    return this._reader._reader.getCompiledPath(ipAddress, this._pathId);
+    this._ensureCompiled();
+    return this._nativeReader.getCompiledPath(ipAddress, this._pathId);
   }
 
   getMany(ipAddresses) {
-    return this._reader._reader.getManyCompiledPath(ipAddresses, this._pathId);
+    this._ensureCompiled();
+    return this._nativeReader.getManyCompiledPath(ipAddresses, this._pathId);
+  }
+
+  close() {
+    if (!this._closed) {
+      pathFinalizer.unregister(this);
+      this._nativeReader.releasePath(this._pathId);
+      this._closed = true;
+    }
+  }
+
+  _ensureCompiled() {
+    if (this._closed) {
+      throw new Error('Path lookup is closed.');
+    }
+    if (this._nativeReader !== this._reader._reader) {
+      pathFinalizer.unregister(this);
+      this._nativeReader.releasePath(this._pathId);
+      this._compileFor(this._reader._reader);
+    }
+  }
+
+  _compileFor(nativeReader) {
+    this._nativeReader = nativeReader;
+    this._pathId = nativeReader.compilePath(this.path);
+    pathFinalizer.register(
+      this,
+      { nativeReader: this._nativeReader, pathId: this._pathId },
+      this
+    );
   }
 }
 
 class NetworkIterator {
-  constructor(reader, cidr, options = {}) {
+  constructor(reader, cidr, options = {}, path = null) {
     const [networkOptions, pageSize] = normalizeNetworkIteratorOptions(options);
-    this._cursor = reader._reader.networkCursor(cidr, ...networkOptions);
+    this._cursor = reader._reader.networkCursor(cidr, ...networkOptions, path);
     this._pageSize = pageSize;
     this._page = [];
     this._index = 0;
     this._done = false;
+    this.path = path == null ? null : Object.freeze([...path]);
   }
 
   [Symbol.iterator]() {
     return this;
   }
 
+  return() {
+    this.close();
+    return { done: true, value: undefined };
+  }
+
   next() {
     if (this._done) {
       return { done: true, value: undefined };
@@ -290,6 +243,16 @@ class NetworkIterator {
       return [];
     }
 
+    if (this._index >= this._page.length) {
+      this._page = [];
+      this._index = 0;
+      const page = this._cursor.nextPage(pageSize);
+      if (page.length === 0) {
+        this.close();
+      }
+      return page;
+    }
+
     const page = [];
     while (page.length < pageSize && this._index < this._page.length) {
       page.push(this._page[this._index]);
@@ -298,7 +261,9 @@ class NetworkIterator {
 
     if (page.length < pageSize) {
       const nativePage = this._cursor.nextPage(pageSize - page.length);
-      page.push(...nativePage);
+      for (const record of nativePage) {
+        page.push(record);
+      }
       if (nativePage.length === 0) {
         this.close();
       }
@@ -349,6 +314,28 @@ class Reader {
 
   static open(filepath, options = {}) {
     const mode = normalizeMode(options.mode);
+    return Reader._fromNative(
+      filepath,
+      mode,
+      options,
+      native.openReader(filepath, mode, normalizeCacheCapacity(options))
+    );
+  }
+
+  static async openOwned(filepath, mode, options = {}) {
+    return Reader._fromNative(
+      filepath,
+      mode,
+      options,
+      await native.openReaderAsync(
+        filepath,
+        mode,
+        normalizeCacheCapacity(options)
+      )
+    );
+  }
+
+  static _fromNative(filepath, mode, options, nativeReader) {
     const reader = Object.create(Reader.prototype);
     reader._mode = mode;
     reader._filepath = filepath;
@@ -359,7 +346,7 @@ class Reader {
     reader._watchReloadPromise = Promise.resolve();
     reader._lastReloadError = null;
     reader._cacheCapacity = normalizeCacheCapacity(options);
-    reader._reader = native.openReader(filepath, mode, reader._cacheCapacity);
+    reader._reader = nativeReader;
     reader.metadata = normalizeMetadata(reader._reader.metadata());
     reader.options = options;
     return reader;
@@ -398,6 +385,36 @@ class Reader {
     }
   }
 
+  async reloadAsync() {
+    if (!this._filepath) {
+      throw new Error('Cannot reload a buffer-backed Reader');
+    }
+    try {
+      const replacement = await native.openReaderAsync(
+        this._filepath,
+        this._mode,
+        this._cacheCapacity
+      );
+      if (this.closed) {
+        replacement.close();
+        throw new Error('Cannot reload a closed Reader');
+      }
+      this._replaceNativeReader(replacement);
+    } catch (error) {
+      this._lastReloadError = error;
+      throw error;
+    }
+  }
+
+  _replaceNativeReader(replacement) {
+    const metadata = normalizeMetadata(replacement.metadata());
+    const previous = this._reader;
+    this._reader = replacement;
+    this.metadata = metadata;
+    this._lastReloadError = null;
+    previous.close();
+  }
+
   close() {
     if (this._watchFilepath && this._watchListener) {
       fs.unwatchFile(this._watchFilepath, this._watchListener);
@@ -444,15 +461,16 @@ class Reader {
     }
 
     try {
-      if (mode === MODE_BUFFER) {
-        const database = await readFile(filepath);
-        if (this.closed || this._watchFilepath !== filepath) {
-          return;
-        }
-        this.load(database);
-      } else {
-        this.reload();
+      const replacement = await native.openReaderAsync(
+        filepath,
+        mode,
+        this._cacheCapacity
+      );
+      if (this.closed || this._watchFilepath !== filepath) {
+        replacement.close();
+        return;
       }
+      this._replaceNativeReader(replacement);
       if (!this.closed && this._watchFilepath === filepath && hook) {
         hook();
       }
@@ -477,6 +495,10 @@ class Reader {
     return this._reader.getPath(ipAddress, path);
   }
 
+  getPaths(ipAddress, paths) {
+    return this._reader.getPaths(ipAddress, paths);
+  }
+
   path(path) {
     return new PathLookup(this, path);
   }
@@ -493,6 +515,10 @@ class Reader {
     return this._reader.getManyPath(ipAddresses, path);
   }
 
+  getManyPaths(ipAddresses, paths) {
+    return this._reader.getManyPaths(ipAddresses, paths);
+  }
+
   networks(options = {}) {
     return new NetworkIterator(this, null, options);
   }
@@ -501,6 +527,14 @@ class Reader {
     return new NetworkIterator(this, cidr, options);
   }
 
+  networksPath(path, options = {}) {
+    return new NetworkIterator(this, null, options, path);
+  }
+
+  withinPath(cidr, path, options = {}) {
+    return new NetworkIterator(this, cidr, options, path);
+  }
+
   *networkPages(options = {}) {
     yield* this.networks(options).pages(options.pageSize ?? 1000);
   }
@@ -513,12 +547,11 @@ class Reader {
 async function open(filepath, opts, cb) {
   assert(!cb, legacyErrorMessage);
   const options = opts || {};
-  await assertNotGzipFile(filepath);
 
   const mode = normalizeMode(options.mode);
   const reader =
-    mode === MODE_BUFFER
-      ? new Reader(await readFile(filepath), options)
+    mode === MODE_MEMORY || mode === MODE_BUFFER
+      ? await Reader.openOwned(filepath, mode, options)
       : Reader.open(filepath, options);
 
   if (options.watchForUpdates) {
@@ -559,7 +592,7 @@ function validate(ipAddress) {
 }
 
 module.exports = {
-  ...native,
+  nativeVersion: native.nativeVersion,
   NetworkIterator,
   PathLookup,
   Reader,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@oschwald/maxminddb",
-  "version": "0.2.1",
+  "version": "0.3.1",
   "description": "Rust-backed Node.js reader for MaxMind DB files",
   "main": "index.js",
   "types": "index.d.ts",
@@ -53,7 +53,7 @@
   "devDependencies": {
     "@napi-rs/cli": "^3.7.2",
     "@types/node": "^26.1.0",
-    "typescript": "^6.0.3"
+    "typescript": "^7.0.2"
   },
   "engines": {
     "node": ">=20"