@ohos-ports/lmdb 2.5.3-beta.0 → 2.8.5-beta.0

package/LICENSE

@@ -1,21 +1,21 @@
-The MIT License (MIT)
-
-This project contains is based on the code from the node-lmdb project ([Copyright (c) 2014 Timur Kristóf](https://github.com/venemo/node-lmdb/)) and LMDB, which has a specific [OpenLDAP license](dependencies/lmdb/libraries/liblmdb/LICENSE),
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+The MIT License (MIT)
+
+This project contains is based on the code from the node-lmdb project ([Copyright (c) 2014 Timur Kristóf](https://github.com/venemo/node-lmdb/)) and LMDB, which has a specific [OpenLDAP license](dependencies/lmdb/libraries/liblmdb/LICENSE),
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.

package/README.md

@@ -4,7 +4,7 @@
 [![get](https://img.shields.io/badge/get-8.5%20MOPS-yellow)](README.md)
 [![put](https://img.shields.io/badge/put-1.7%20MOPS-yellow)](README.md)
 
-This is an ultra-fast NodeJS and Deno interface to LMDB; probably the fastest and most efficient key-value/database interface that exists for storage and retrieval of structured JS data (objects, arrays, etc.) in a true persisted, scalable, [ACID compliant](https://en.wikipedia.org/wiki/ACID) database. It provides a simple interface for interacting with LMDB, as a key-value db, that makes it easy to fully leverage the power, crash-proof design, and efficiency of LMDB using intuitive JavaScript, and is designed to scale across multiple processes or threads. Several key features that make it idiomatic, highly performant, and easy to use LMDB efficiently:
+This is an ultra-fast NodeJS, Bun, and Deno interface to LMDB; probably the fastest and most efficient key-value/database interface that exists for storage and retrieval of structured JS data (objects, arrays, etc.) in a true persisted, scalable, [ACID compliant](https://en.wikipedia.org/wiki/ACID) database. It provides a simple interface for interacting with LMDB, as a key-value db, that makes it easy to fully leverage the power, crash-proof design, and efficiency of LMDB using intuitive JavaScript, and is designed to scale across multiple processes or threads. Several key features that make it idiomatic, highly performant, and easy to use LMDB efficiently:
 * High-performance translation of JS values and data structures to/from binary key/value data
 * Queueing asynchronous off-thread write operations with promise-based API
 * Simple transaction management
@@ -13,7 +13,7 @@ This is an ultra-fast NodeJS and Deno interface to LMDB; probably the fastest an
 * Optional native off-main-thread compression with high-performance LZ4 compression <a href="https://github.com/kriszyp/db-benchmark"><img align="right" src="./assets/performance.png" width="380"/></a>
 * And ridiculously fast and efficient, with integrated (de)serialization, data retrieval can be several times faster than `JSON` alone
 
-`lmdb-js` is used in many heavy-use production applications, including as a high-performance cache for builds in [Parcel](https://parceljs.org/) and [Elasticsearch's Kibana](https://www.elastic.co/kibana/), as the storage layer for [HarperDB](https://harperdb.io/) and [Gatsby](https://www.gatsbyjs.com/)'s database, and for search and analytical engine for our [clinical medical research](https://drevidence.com).
+`lmdb-js` is used in many heavy-use production applications, including as a high-performance cache for builds in [Parcel](https://parceljs.org/) and [Elasticsearch's Kibana](https://www.elastic.co/kibana/), as the storage layer for [HarperDB](https://harperdb.io/) and [Gatsby](https://www.gatsbyjs.com/)'s database, and for search and analytical engine for [clinical medical research](https://drevidence.com).
 
 <a href="https://www.elastic.co/kibana/"><img src="https://static-www.elastic.co/v3/assets/bltefdd0b53724fa2ce/blt4466841eed0bf232/5d082a5e97f2babb5af907ee/logo-kibana-32-color.svg" width="40" align="right"></a>
 <a href="https://parceljs.org/"><img src="https://parceljs.org/avatar.633bb25a.avif" width="56" align="right"></a>
@@ -25,9 +25,13 @@ This library is published to the NPM package `lmdb` (the 1.x versions were publi
 ```
 npm install lmdb
 ```
-`lmdb-js` is based on the Node-API for maximum compatility across all supported Node versions and futue Deno versions. It also includes accelerated, high-speed functions for direct V8 interaction that are compiled for, and (automatically) loaded in Node v16. The standard Node-API based functions are used in all other versions and still provide excellent performance, but for absolute maximum performance on older versions of Node, you can use `npm install --build-from-source`.
+`lmdb-js` is based on the Node-API for maximum compatibility across all supported Node versions and future Deno versions. It also includes accelerated, high-speed functions for direct V8 interaction that are compiled for, and (automatically) loaded in Node v16. The standard Node-API based functions are used in all other versions and still provide excellent performance, but for absolute maximum performance on older versions of Node, you can use `npm install --build-from-source`.
 
-In Deno, this package could be directly used from the [deno.land `lmdb` module](https://deno.land/x/lmdb/mod.ts), but Node-API support is currently in-progress, so probably will require Deno v1.24+ (for older versions of Deno, you can use `lmdb-js` v2.2.x).
+In Deno, this package should be loaded using the NPM module identifier (which will download the package):
+```js
+import { open } from 'npm:lmdb';
+```
+Note that Deno and Bun's support for NAPI is not very stable yet, and currently asynchronous transactions (`transaction` method) are not supported (Bun issue [here](https://github.com/oven-sh/bun/issues/158#issuecomment-1126624085)).
 
 This library has minimal, tightly-controlled, and maintained dependencies to ensure stability, security, and efficiency. It supports both native ESM and CJS usage.
 
@@ -35,13 +39,13 @@ This library has minimal, tightly-controlled, and maintained dependencies to ens
 
 This library handles translation of JavaScript values, primitives, arrays, and objects, to and from the binary storage of LMDB keys and values with highly optimized native C++ code for breakneck performance. It supports multiple types of JS values for keys and values, making it easy to use idiomatic JS for storing and retrieving data in LMDB.
 
-`lmdb-js` is designed for synchronous reads, and asynchronous writes. In idiomatic NodeJS and Deno code, I/O operations are performed asynchronously. LMDB is a memory-mapped database, reading and writing within a transaction does not use any I/O (other than the slight possibility of a page fault), and can usually be performed faster than the event queue callbacks can even execute, and it is easier to write code for instant synchronous values from reads. On the otherhand, commiting transactions does involve I/O, and vastly higher throughput can be achieved by batching operations and executing on a separate thread. Consequently, `lmdb-js` is designed for transactions to go through this asynchronous batching process and return a simple promise that resolves once data is written and flushed to disk.
+`lmdb-js` is designed for synchronous reads, and asynchronous writes. In idiomatic JavaScript code, I/O operations are performed asynchronously. LMDB is a memory-mapped database, reading and writing within a transaction does not use any I/O (other than the slight possibility of a page fault), and can usually be performed faster than the event queue callbacks can even execute, and it is easier to write code for instant synchronous values from reads. On the otherhand, commiting transactions does involve I/O, and vastly higher throughput can be achieved by batching operations and executing on a separate thread. Consequently, `lmdb-js` is designed for transactions to go through this asynchronous batching process and return a simple promise that resolves once data is written and flushed to disk.
 
-With the default sync'ing configuration, LMDB has a crash-proof design; a machine can be turned off at any point, and data can not be corrupted unless the written data is actually changed or tampered. Writing data and waiting for confirmation that has been writted to the physical medium is critical for data integrity, but is well known to have latency (although not necessarily less efficient). However, by batching writes, when a database is under load, slower transactions enable more writes per transaction, and this library is able to drive LMDB to achieve the maximum levels of throughput with fully sync'ed operations,  preserving both the durability/safety of the transactions and unparalled performance.
+With the default syncing configuration, LMDB has a crash-proof design; a machine can be turned off at any point, and data can not be corrupted unless the written data is actually changed or tampered. Writing data and waiting for confirmation that it has been written to the physical medium is critical for data integrity, but is well known to have latency (although not necessarily less efficient). However, by batching writes, when a database is under load, slower transactions enable more writes per transaction, and this library is able to drive LMDB to achieve the maximum levels of throughput with fully synced operations,  preserving both the durability/safety of the transactions and unparalleled performance.
 
-This library supports and encourages the use of conditional writes; this allows for atomic operations that are dependendent on previously read data, and most transactional types of operations can be written with an optimistic-locking based, atomic-conditional-write pattern. This allows this library to delegate writes to off-thread execution, and scale to handle concurrent execution across many processes or threads while maintaining data integrity.
+This library supports and encourages the use of conditional writes; this allows for atomic operations that are dependent on previously read data, and most transactional types of operations can be written with an optimistic-locking based, atomic-conditional-write pattern. This allows this library to delegate writes to off-thread execution, and scale to handle concurrent execution across many processes or threads while maintaining data integrity.
 
-This library automatically handles automatically database growth, expanding file size with a smart heuristic that minimizes file fragmentation (as you would expect from a database).
+This library automatically handles database growth, expanding file size with a smart heuristic that minimizes file fragmentation (as you would expect from a database).
 
 This library provides optional compression using LZ4 that works in conjunction with the asynchronous writes by performing the compression in the same thread (off the main thread) that performs the writes in a transaction. LZ4 is extremely fast, and decompression can be performed at roughly 5GB/s, so excellent storage efficiency can be achieved with almost negligible performance impact.
 
@@ -69,7 +73,7 @@ myDB.transaction(() => {
 Once you have opened a database, you can store and retrieve values using keys:
 
 ### Values
-You can store a wide variety of JavaScript values and data structures in this library, including objects (with arbitrary complexity), arrays, buffers, strings, numbers, etc. in your database. Even full structural cloning (with cycles) is an optionally supported. Values are stored and retrieved according the database encoding, which can be set using the `encoding` property on the database options. By default, data is stored using MessagePack, but there are several supported encodings:
+You can store a wide variety of JavaScript values and data structures in this library, including objects (with arbitrary complexity), arrays, buffers, strings, numbers, etc. in your database. Even full structural cloning (with cycles) is optionally supported. Values are stored and retrieved according to the database encoding, which can be set using the `encoding` property on the database options. By default, data is stored using MessagePack, but there are several supported encodings:
 
 * `msgpack` (default) - All values are stored by serializing the value as MessagePack (using the [msgpackr](https://github.com/kriszyp/msgpackr) package). Values are decoded and parsed on retrieval, so `get` and `getRange` will return the object, array, or other value that you have stored. The msgpackr package is extremely fast (usually faster than native JSON), and provides the most flexibility in storing different value types. See the Shared Structures section for how to achieve maximum efficiency with this.
 * `cbor` - This specifies all values use the CBOR format, which requires that the [cbor-x](https://github.com/kriszyp/cbor-x) package be installed. This package is based on [msgpackr](https://github.com/kriszyp/msgpackr) and supports all the same options.
@@ -81,11 +85,13 @@ You can store a wide variety of JavaScript values and data structures in this li
 In addition, you can use `asBinary` to directly store a buffer or Uint8Array as a value, bypassing any encoding.
 
 ### Keys
-When using the various APIs, keys can be any JS primitive (string, number, boolean, symbol), an array of primitives, or a Buffer. Using the default `ordered-binary` conversion, primitives are translated to binary keys used by LMDB in such a way that consistent ordering is preserved. Numbers are ordered naturally, which come before strings, which are ordered lexically. The keys are stored with type information preserved. The `getRange`operations that return a set of entries will return entries with the original JS primitive values for the keys. If arrays are used as keys, they are ordering by first value in the array, with each subsequent element being a tie-breaker. Numbers are stored as doubles, with reversal of sign bit for proper ordering plus type information, so any JS number can be used as a key. For example, here are the order of some different keys:
+When using the various APIs, keys can be any JS primitive (string, number, boolean, symbol), an array of primitives, or a Buffer. Using the default `ordered-binary` conversion, primitives are translated to binary keys used by LMDB in such a way that consistent ordering is preserved. Numbers are ordered naturally, which come before strings, which are ordered lexically. The keys are stored with type information preserved. The `getRange`operations that return a set of entries will return entries with the original JS primitive values for the keys. If arrays are used as keys, they are ordered by first value in the array, with each subsequent element being a tie-breaker. Numbers are stored as doubles, with reversal of sign bit for proper ordering plus type information, so any JS number can be used as a key. For example, here is the order of some different keys:
 
 ```js
 null // lowest possible value
 Symbol.for('even symbols')
+false
+true
 -10 // negative supported
 -1.1 // decimals supported
 400
@@ -99,15 +105,19 @@ Symbol.for('even symbols')
 Buffer.from([255]) // buffers can be used directly, 255 is higher than any byte produced by primitives
 ```
 
-You can override the default encoding of keys, and cause keys to be returned as binary arrays (`Buffer`s in NodeJS) using the `keyEncoding: 'binary'` database option (generally slower), use `keyEncoding: 'uint32'` for keys that are strictly 32-bit unsigned integers, or provide a custom key encoder/decoder with `keyEncoder` (see custom key encoding).
+Keys use 0/null bytes as delimiters for arrays, and so strings currently can not have '\x00' (null char) in them.
+
+By default, the maximum key size is 1978 bytes. If you explicitly set the `pageSize` to 8192 or higher, the maximum key size will be 4026, but this is the largest key size supported.
+
+You can override the default encoding of keys, and cause keys to be returned as binary arrays (`Buffer`s in NodeJS) using the `keyEncoding: 'binary'` database option (generally slower). Use `keyEncoding: 'uint32'` for keys that are strictly 32-bit unsigned integers, or provide a custom key encoder/decoder with `keyEncoder` (see custom key encoding).
 
 Once you created have a db, the following methods are available:
 
-### `db.get(key): any`
-This will retrieve the value at the specified key. The `key` must be a JS value/primitive as described above, and the return value will be the stored data (dependent on the encoding), or `undefined` if the entry does not exist.
+### `db.get(key, options?): any`
+This will retrieve the value at the specified key. The `key` must be a JS value/primitive as described above, and the return value will be the stored data (dependent on the encoding), or `undefined` if the entry does not exist. The `options` argument may be used to specify an explicit read transaction.
 
-### `db.getEntry(key): any`
-This will retrieve the the entry at the specified key. The `key` must be a JS value/primitive as described above, and the return value will be the stored entry, or `undefined` if the entry does not exist. An entry is object with a `value` property for the value in the database (as returned by `db.get`), and a `version` property for the version number of the entry in the database (if `useVersions` is enabled for the database).
+### `db.getEntry(key, options?): any`
+This will retrieve the entry at the specified key. The `key` must be a JS value/primitive as described above, and the return value will be the stored entry, or `undefined` if the entry does not exist. An entry is object with a `value` property for the value in the database (as returned by `db.get`), and a `version` property for the version number of the entry in the database (if `useVersions` is enabled for the database). The `options` argument may be used to specify an explicit read transaction.
 
 ### `db.put(key, value, version?: number, ifVersion?: number): Promise<boolean>`
 This will store the provided value/data at the specified key. If the database is using versioning (see options below), the `version` parameter will be used to set the version number of the entry. If the `ifVersion` parameter is set, the put will only occur if the existing entry at the provided key has the version specified by `ifVersion` at the instance the commit occurs (LMDB commits are atomic by default). If the `ifVersion` parameter is not set, the put will occur regardless of the previous value.
@@ -117,7 +127,7 @@ This operation will be enqueued to be written in a batch transaction. Any other
 If `put` is called inside a transaction, the put will be executed immediately in the current transaction.
 
 ### `db.remove(key, IfVersion?: number): Promise<boolean>`
-This will delete the entry at the specified key. This functions like `put`, with the same optional conditional version. This is batched along with put operations, and returns a promise indicating the success of the operation.
+This will delete the entry at the specified key. This functions is similar to `put`, with the same optional conditional version. This is batched along with put operations, and returns a promise indicating the success of the operation.
 
 Again, if this is performed inside a transation, the removal will be performed in the current transaction.
 
@@ -151,7 +161,7 @@ function buyShoe() {
 
 Note that `db.transaction(() => db.put(...))` is functionally the same as calling `db.put(...)`, queuing the put for asynchronously being committed in transaction, except that `put` executes the database's write operation entirely in separate worker thread, whereas `transaction` must also synchronize the callback function in the main JS thread to execute (so it is a little bit less efficient, although still quite fast).
 
- Also, the callback function can be an async function (or return a promise), but this is not recommended. If the function returns a promise, this will delay/defer the commit until the callback's promise is resolved. However, while waiting for the callback to finish, other code may execute operations that would end up in the current transaction and may result in a surprising order of operations, and long running transactions are generally discouraged since they extend the single write lock.
+Also, the callback function can be an async function (or return a promise), but this is not recommended. If the function returns a promise, this will delay/defer the commit until the callback's promise is resolved. However, while waiting for the callback to finish, other code may execute operations that would end up in the current transaction and may result in a surprising order of operations, and long running transactions are generally discouraged since they extend the single write lock.
 
 ### `db.childTransaction(callback: Function): Promise`
 This will run the provided callback in a transaction much like `transaction` except an explicit child transaction will be used specifically for this callback. This makes it possible for the operations to be aborted and rolled back. The callback may return the exported `ABORT` constant to abort the child transaction for this callback. Also, if the callback function throws an error (or returns a reject promise), this will also abort the child transaction. This childTransaction function is not available if caching or `useWritemap` is enabled.
@@ -214,7 +224,7 @@ db.getRange({ start, end, offset: 10, limit: 10 }) // skip first 10 and get next
 If you want to get a true array from the range results, the `asArray` property will return the results as an array.
 
 #### Snapshots
-By default a range iterator will use a database snapshot, using a single read transaction that remains open and gives a consistent view of the database at the time it was started, for the duration of iterating through the range. However, if the iteration will take place over a long period of time, keeping a read transaction open for a long time can interfere with LMDB's free space collection and reuse and increase the database size. If you will be using a long duration iterator, you can specify `snapshot: false` flag in the range options to indicate that it snapshotting is not necessary, and it can reset and renew read transactions while iterating, to allow LMDB to collect any space that was freed during iteration.
+By default, a range iterator will use a database snapshot, using a single read transaction that remains open and gives a consistent view of the database at the time it was started, for the duration of iterating through the range. However, if the iteration will take place over a long period of time, keeping a read transaction open for a long time can interfere with LMDB's free space collection and reuse and increase the database size. If you will be using a long duration iterator, you can specify `snapshot: false` flag in the range options to indicate that it snapshotting is not necessary, and it can reset and renew read transactions while iterating, to allow LMDB to collect any space that was freed during iteration.
 
 ### `db.getValues(key, options?: RangeOptions): Iterable<any>`
 When using a database with duplicate entries per key (with `dupSort` flag), you can use this to retrieve all the values for a given key. This will return an iterator just like `getRange`, except each entry will be the value from the database:
@@ -235,7 +245,7 @@ for (let value of db.getValues('key1')) {
 }
 ```
 
-You can optionally provide a second argument with the same `options` that `getRange` handles. You can provide a `start` and/or `end` values, which will be define the starting value and ending value for the range of values to return for the key:
+You can optionally provide a second argument with the same `options` that `getRange` handles. You can provide a `start` and/or `end` values, which will define the starting value and ending value for the range of values to return for the key:
 
 ```js
 for (let value of db.getValues('key1', { start: 'value1', end: 'value3'})) ...
@@ -244,7 +254,7 @@ for (let value of db.getValues('key1', { start: 'value1', end: 'value3'})) ...
 Using `start`/`end` is only supported if using the `ordered-binary` encoding.
 
 ### `db.getKeys(options: RangeOptions): Iterable<any>`
-This behaves like `getRange`, but only returns the keys. If this is duplicate key database, each key is only returned once (even if it has multiple values/entries).
+This behaves like `getRange`, but only returns the keys. If this is a duplicate key database, each key is only returned once (even if it has multiple values/entries).
 
 ### `RangeOptions`
 Here are the options that can be provided to the range methods (all are optional):
@@ -295,7 +305,19 @@ let buffer = encode(myValue) // if we have already serialized a value, perhaps t
 db.put(key, asBinary(buffer)) // we can directly store the encoded value
 ```
 
-### `close(): Promise`
+### `db.useReadTransaction(): Transaction`
+This allows you to explicitly start a read transaction, which holds a consistent snapshot of the database, and use it for subsequent retrieval operations. This will mark the read transaction as in use until `transaction.done()` is called. For example:
+```javascript
+let transaction = myDb.useReadTransaction();
+let data = myDb.get('my-key', { transaction });
+await doSomethingElse();
+// the same read transaction is still being used and this will return the same record even if the data has been changed elsewhere: 
+data = myDb.get('my-key', { transaction });
+transaction.done(); // make sure you mark the transaction as done
+```
+It is critical that you mark read transactions as done when you no longer need it or you will exhaust the read transactions that are available. Long-lived read transaction also prevent free space reclamation. This can be used with `get`, `getEntry` and range/query methods.
+
+### `db.close(): Promise`
 This will close the current db. This closes the underlying LMDB database, and if this is the root database (opened with `open` as opposed to `db.openDB`), it will close the environment (and child databases will no longer be able to interact with the database). This is asynchronous, waiting for any outstanding transactions to finish before closing the database.
 
 ### `db.doesExist(key, valueOrVersion): boolean`
@@ -375,6 +397,8 @@ let myDB = open('my-db', {
 
 Compression is recommended for large databases that may be close to or larger than available RAM, to improve caching and reduce page faults. If you enable compression for a database, you must ensure that the data is always opened with the same compression setting, so that the data will be properly decompressed.
 
+By default, opening a database from a root database will inherited the compression settings from the root database.
+
 ## Caching
 This library supports caching of entries from databases, and uses a [LRU/LFU (LRFU) and weak-referencing caching mechanism](https://github.com/kriszyp/weak-lru-cache) for highly optimized caching and object tracking. There are several key potential benefits to using caching, including performance, key correlation with object identity, and immediate/synchronous access to saved data. Enabling caching will cache `get`s and `put`s, which can make frequent `get`s much faster. Caching is enabled by providing a truthy value for the `cache` property on the database `options`.
 
@@ -405,7 +429,7 @@ Additional databases can be opened within the main database environment with:
 `db.openDB(name, options)` or `db.openDB(options)`
 If the `path` has an `.` in it, it is treated as a file name, otherwise it is treated as a directory name, where the data will be stored. The path can be omitted to create a temporary database, which will be created in the system temp directory and deleted on close. The `options` argument to either of the functions should be an object, and supports the following properties, all of which are optional (except `name` if not otherwise specified):
 * `name` - This is the name of the database. This defaults to null (which is the root database) when opening the database environment (`open`). When an opening a database within an environment (`openDB`), this is required, if not specified in first parameter.
-* `encoding` - Sets the encoding for the database values, which can be `'msgpack'`, `'json'`, `'cbor'`, `'string'`, `'ordered-binary'`or `'binary'`.
+* `encoding` - Sets the encoding for the database values, which can be `'msgpack'`, `'json'`, `'cbor'`, `'string'`, `'ordered-binary'`or `'binary'`. Child databases will inherit this from the root database, it is specified.
 * `encoder` - Directly set the encoder to use or provide the settings for an encoder. This can be an object with settings to pass to the encoder or can be an object with `encode` and `decode` methods. It can also be an object with an `Encoder` that will be called to create the encoder instance. This allows you explicitly set the encoder with an import:
 
 ```js
@@ -426,9 +450,9 @@ The following additional option properties are only available when creating the
 * `path` - This is the file path to the database environment file you will use.
 * `maxDbs` - The maximum number of databases to be able to open within one root database/environment ([there is some extra overhead if this is set very high](http://www.lmdb.tech/doc/group__mdb.html#gaa2fc2f1f37cb1115e733b62cab2fcdbc)). This defaults to 12.
 * `maxReaders` - The maximum number of concurrent read transactions (readers) to be able to open ([more information](http://www.lmdb.tech/doc/group__mdb.html#gae687966c24b790630be2a41573fe40e2)).
-* `overlappingSync` - This enables committing transactions where LMDB waits for a transaction to be fully flushed to disk _after_ the transaction has been committed and defaults to being enabled. This option is discussed in more detail below.
+* `overlappingSync` - This enables committing transactions where LMDB waits for a transaction to be fully flushed to disk _after_ the transaction has been committed and defaults to being enabled on non-Windows OSes. This option is discussed in more detail below.
 * `separateFlushed` - Resolve asynchronous operations when commits are finished and visible and include a separate promise for when a commit is flushed to disk, as a `flushed` property on the commit promise. Note that you can alternately use the `flushed` property on the database.
-* `pageSize` - This defines the page size of the database. This is 4,096 by default. You may want to consider setting this to 8,192 for databases larger than available memory (and moreso if you have range queries) or 4,096 for databases that can mostly cache in memory. Note that this only effects the page size of new databases (does not affect existing databases).
+* `pageSize` - This defines the page size of the database. This defaults to the default page size of the OS (usually 4,096, except on MacOS with M-series, which is 16,384 bytes). You may want to consider setting this to 8,192 for databases larger than available memory (and moreso if you have range queries) or 4,096 for databases that can mostly cache in memory. Note that this only effects the page size of new databases (does not affect existing databases).
 * `eventTurnBatching` - This is enabled by default and will ensure that all asynchronous write operations performed in the same event turn will be batched together into the same transaction. Disabling this allows lmdb-js to commit a transaction at any time, and asynchronous operations will only be guaranteed to be in the same transaction if explicitly batched together (with `transaction`, `batch`, `ifVersion`). If this is disabled (set to `false`), you can control how many writes can occur before starting a transaction with `txnStartThreshold` (allow a transaction will still be started at the next event turn if the threshold is not met). Disabling event turn batching (and using lower `txnStartThreshold` values) can facilitate a faster response time to write operations. `txnStartThreshold` defaults to 5.
 * `encryptionKey` - This enables encryption, and the provided value is the key that is used for encryption. This may be a buffer or string, but must be 32 bytes/characters long. This uses the Chacha8 cipher for fast and secure on-disk encryption of data.
 * `commitDelay` - This is the amount of time to wait (in milliseconds) for batching write operations before committing the writes (in a transaction). This defaults to 0. A delay of 0 means more immediate commits with less latency (uses `setImmediate`), but a longer delay (which uses `setTimeout`) can be more efficient at collecting more writes into a single transaction and reducing I/O load. Note that NodeJS timers only have an effective resolution of about 10ms, so a `commitDelay` of 1ms will generally wait about 10ms.
@@ -460,7 +484,7 @@ let v = db.get(key) // this value now be retrieved from the db
 await db.flushed // wait for last commit to be fully flushed to disk
 ```
 
-Enabling `overlappingSync` option is generally not recommended on Windows, as Window's disk flushing operation tends to have very poor performance characteristics on larger databases (whereas Windows tends to perform well with standard transactions). This option may be enabled by default in the future, for non-Windows platforms.
+Enabling `overlappingSync` option is generally not recommended on Windows, as Window's disk flushing operation tends to have very poor performance characteristics on larger databases (whereas Windows tends to perform well with standard transactions). This option is enabled by default for non-Windows platforms.
 
 #### Serialization options
 If you are using the default encoding of `'msgpack'`, the [msgpackr](https://github.com/kriszyp/msgpackr) package is used for serialization and deserialization. You can provide encoder options that are passed to msgpackr or cbor, as well, by including them in the `encoder` property object. For example, these options can be potentially useful:
@@ -497,13 +521,33 @@ dbLevel.get(id).then(...)
 
 Benchmarking on Node 14.9, with 3.4Ghz i7-4770 Windows, a get operation, using JS numbers as a key, retrieving data from the database (random access), and decoding the data into a structured object with 10 properties (using default [MessagePack encoding](https://github.com/kriszyp/msgpackr)), can be done in about half a microsecond, or about 1,900,000/sec on a single thread. This is almost three times as fast as a single native `JSON.parse` call with the same object without any DB interaction! LMDB scales effortlessly across multiple processes or threads; over 6,000,000 operations/sec on the same 4/8 core computer by running across multiple threads (or 18,000,000 operations/sec with raw binary data). By running writes on a separate transactional thread, writing is extremely fast as well. With encoding the same objects, full encoding and writes can be performed at about 500,000 puts/second or 1,700,000 puts/second on multiple threads.
 
+
+### Full Prebuild Script
+This package includes an NPM executable to download all the prebuilds for all OS/architectures. This can be useful if
+ you are building a full set of files/artifacts to be run on different machines. This requires installing the `prebuildify-ci
+ ` package (globally is recommended) and adding something like this to your package.json:
+```
+{
+    "dependencies": {
+        "lmdb": "2.6.0"
+    },
+    "scripts": {
+        "download-lmdb-prebuilds": "download-lmdb-prebuilds"
+    }
+}
+```
+
 #### Build Options
 A few LMDB options are available at build time, and can be specified with options with `npm install` (which can be specified in your package.json install script):
-`npm install --use_robust=true`: This will enable LMDB's MDB_USE_ROBUST option, which uses robust semaphores/mutexes so that if you are using multiple processes, and one process dies in the middle of transaction, the OS will cleanup the semaphore/mutex, aborting the transaction and allowing other processes to run without hanging. There is a slight performance overhead, but this is recommended if you will be using multiple processes.
+`npm install lmdb --build-from-source --use_robust=false`: This will disable LMDB's MDB_USE_ROBUST option, which uses robust semaphores/mutexes so that if you are using multiple processes, and one process dies in the middle of transaction, the OS will cleanup the semaphore/mutex, aborting the transaction and allowing other processes to run without hanging. There is a slight performance overhead to robust mutexes, but keeping this enabled is recommended if you will be using multiple processes.
 
-On MacOS, there is a default limit of 10 robust locked semaphores, which imposes a limit on the number of open write transactions (if you have over 10 database environments with a write transaction). If you need more concurrent write transactions, you can increase your maximum undoable semaphore count by setting kern.sysv.semmnu on your local computer. Otherwise don't use the robust mutex option. You can also try to minimize overlapping transactions and/or reduce the number of database environments (and use more databases within each environment).
+On MacOS, there is a default limit of 10 robust locked semaphores, which imposes a limit on the number of open write transactions (if you have over 10 database environments with a write transaction). If you need more concurrent write transactions, you can increase your maximum undoable semaphore count with:
+```
+sudo sysctl kern.sysv.semume=50
+```
+Otherwise you may need to disable the robust mutex option. You can also try to minimize overlapping transactions and/or reduce the number of database environments (and use more databases within each environment).
 
-`npm install --use_data_v1=true`: This will build from an older version of LMDB that uses the legacy data format version 1 (the latest LMDB uses data format version 2). For portability of the data format, this may be preferable since many libraries still use older versions of LMDB. Since this is an older version of LMDB, some features may not be available, including encryption and remapping.
+`npm install lmdb --build-from-source --use_data_v1=true`: This will build from an older version of LMDB that uses the legacy data format version 1 (the latest LMDB uses data format version 2). For portability of the data format, this may be preferable since many libraries still use older versions of LMDB. Since this is an older version of LMDB, some features may not be available, including encryption and remapping.
 
 #### Turbo Mode
 On Node V16+, lmdb-js will automatically enable V8's turbo fast-api calls (the `--turbo-fast-api-calls` V8 flag) to accelerate `lmdb-js`'s turbo-enabled functions. If you do not want this flag enabled, set the env variable `DISABLE_TURBO_CALLS=true` for your node process, or build from source with `--enable_fast_api_calls=false`.

package/SECURITY.md

@@ -1,12 +1,12 @@
-# Security Policy
-
-## Supported Versions
-
-| Version | Supported          |
-| ------- | ------------------ |
-| 1.6.x   | :white_check_mark: |
-| 2.0.x   | :white_check_mark: |
-
-## Reporting a Vulnerability
-
-Please report security vulnerabilities to kriszyp@gmail.com.
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 1.6.x   | :white_check_mark: |
+| 2.0.x   | :white_check_mark: |
+
+## Reporting a Vulnerability
+
+Please report security vulnerabilities to kriszyp@gmail.com.

package/bin/download-prebuilds.js

@@ -0,0 +1,15 @@
+#!/usr/bin/env node
+
+import { dirname } from 'path';
+import { fileURLToPath } from 'url';
+import { exec } from 'child_process';
+
+exec('download-msgpackr-prebuilds', propagateOutput);
+process.chdir(fileURLToPath(dirname(dirname(import.meta.url))));
+exec('prebuildify-ci download', propagateOutput);
+function propagateOutput(error, stdout, stderr) {
+	console.error(stderr);
+	console.log(stdout);
+	if (error?.code)
+		process.exit(error.code);
+}

package/binding.gyp

@@ -1,111 +1,115 @@
-{
-  "variables": {
-      "os_linux_compiler%": "gcc",
-      "use_robust%": "false",
-      "use_data_v1%": "false",
-      "enable_v8%": "true",
-      "enable_fast_api_calls%": "true",
-      "enable_pointer_compression%": "false",
-      "openssl_fips": "X",
-      "target%": "",
-      "build_v8_with_gn": "false",
-      "runtime%": "node"
-  },
-  "conditions": [
-    ['OS=="win"', {
-      "variables": {
-        "enable_fast_api_calls%": "<!(echo %ENABLE_FAST_API_CALLS%)",
-        "enable_v8%": "<!(echo %ENABLE_V8_FUNCTIONS%)",
-      }
-    }],
-    ['OS!="win"', {
-      "variables": {
-        "enable_fast_api_calls%": "<!(echo $ENABLE_FAST_API_CALLS)",
-        "enable_v8%": "<!(echo $ENABLE_V8_FUNCTIONS)",
-      }
-    }]
-  ],
-  "targets": [
-    {
-      "target_name": "lmdb",
-      "sources": [
-        "src/lmdb-js.cpp",
-        "dependencies/lmdb/libraries/liblmdb/midl.c",
-        "dependencies/lmdb/libraries/liblmdb/chacha8.c",
-        "dependencies/lz4/lib/lz4.h",
-        "dependencies/lz4/lib/lz4.c",
-        "src/writer.cpp",
-        "src/env.cpp",
-        "src/compression.cpp",
-        "src/ordered-binary.cpp",
-        "src/misc.cpp",
-        "src/txn.cpp",
-        "src/dbi.cpp",
-        "src/cursor.cpp",
-        "src/v8-functions.cpp"
-      ],
-      "include_dirs": [
-        "<!(node -p \"require('node-addon-api').include_dir\")",
-        "dependencies/lz4/lib"
-      ],
-      "defines": ["MDB_MAXKEYSIZE=0", "NAPI_DISABLE_CPP_EXCEPTIONS", "MDB_USE_ROBUST=0" ],
-      "conditions": [
-        ["OS=='linux'", {
-          "variables": {
-            "gcc_version" : "<!(<(os_linux_compiler) -dumpversion | cut -d '.' -f 1)",
-          },
-          "cflags_cc": [
-            "-fPIC",
-            "-Wno-strict-aliasing",
-            "-Wno-unused-result",
-            "-Wno-cast-function-type",
-            "-fvisibility=hidden",
-            "-fvisibility-inlines-hidden",
-          ],
-          "ldflags": [
-            "-fPIC",
-            "-fvisibility=hidden"
-          ],
-          "cflags": [
-            "-fPIC",
-            "-fvisibility=hidden",
-            "-O3"
-          ],
-        }],
-        ["OS=='win'", {
-            "libraries": ["ntdll.lib", "synchronization.lib"]
-        }],
-        ["use_data_v1=='true'", {
-          "sources": [
-            "dependencies/lmdb-data-v1/libraries/liblmdb/mdb.c"
-          ],
-          "include_dirs": [
-            "dependencies/lmdb-data-v1/libraries/liblmdb",
-          ],
-        }, {
-          "sources": [
-            "dependencies/lmdb/libraries/liblmdb/mdb.c"
-          ],
-          "include_dirs": [
-            "dependencies/lmdb/libraries/liblmdb",
-          ],
-        }],
-        ["enable_pointer_compression=='true'", {
-          "defines": ["V8_COMPRESS_POINTERS", "V8_COMPRESS_POINTERS_IN_ISOLATE_CAGE"],
-        }],
-        ['runtime=="electron"', {
-          "defines": ["NODE_RUNTIME_ELECTRON=1"]
-        }],
-        ["enable_v8!='false'", {
-          "defines": ["ENABLE_V8_API=1"],
-        }],
-        ["enable_fast_api_calls=='true'", {
-          "defines": ["ENABLE_FAST_API_CALLS=1"],
-        }],
-        ["use_robust=='true'", {
-          "defines": ["MDB_USE_ROBUST"],
-        }],
-      ],
-    }
-  ]
-}
+{
+  "variables": {
+      "os_linux_compiler%": "gcc",
+      "use_robust%": "false",
+      "use_data_v1%": "false",
+      "enable_v8%": "true",
+      "enable_fast_api_calls%": "true",
+      "enable_pointer_compression%": "false",
+      "openssl_fips": "X",
+      "target%": "",
+      "build_v8_with_gn": "false",
+      "runtime%": "node"
+  },
+  "conditions": [
+    ['OS=="win"', {
+      "variables": {
+        "enable_fast_api_calls%": "<!(echo %ENABLE_FAST_API_CALLS%)",
+        "enable_v8%": "<!(echo %ENABLE_V8_FUNCTIONS%)",
+        "use_data_v1%": "<!(echo %LMDB_DATA_V1%)",
+      }
+    }],
+    ['OS!="win"', {
+      "variables": {
+        "enable_fast_api_calls%": "<!(echo $ENABLE_FAST_API_CALLS)",
+        "enable_v8%": "<!(echo $ENABLE_V8_FUNCTIONS)",
+        "use_data_v1%": "<!(echo $LMDB_DATA_V1)",
+      }
+    }]
+  ],
+  "targets": [
+    {
+      "target_name": "lmdb",
+      "sources": [
+        "src/lmdb-js.cpp",
+        "dependencies/lmdb/libraries/liblmdb/midl.c",
+        "dependencies/lmdb/libraries/liblmdb/chacha8.c",
+        "dependencies/lz4/lib/lz4.h",
+        "dependencies/lz4/lib/lz4.c",
+        "src/writer.cpp",
+        "src/env.cpp",
+        "src/compression.cpp",
+        "src/ordered-binary.cpp",
+        "src/misc.cpp",
+        "src/txn.cpp",
+        "src/dbi.cpp",
+        "src/cursor.cpp",
+        "src/v8-functions.cpp"
+      ],
+      "include_dirs": [
+        "<!(node -p \"require('node-addon-api').include_dir\")",
+        "dependencies/lz4/lib"
+      ],
+      "defines": ["MDB_MAXKEYSIZE=0", "NAPI_DISABLE_CPP_EXCEPTIONS" ],
+      "conditions": [
+        ["OS=='linux'", {
+          "variables": {
+            "gcc_version" : "<!(<(os_linux_compiler) -dumpversion | cut -d '.' -f 1)",
+          },
+          "cflags_cc": [
+            "-fPIC",
+            "-Wno-strict-aliasing",
+            "-Wno-unused-result",
+            "-Wno-cast-function-type",
+            "-fvisibility=hidden",
+            "-fvisibility-inlines-hidden",
+          ],
+          "ldflags": [
+            "-fPIC",
+            "-fvisibility=hidden"
+          ],
+          "cflags": [
+            "-fPIC",
+            "-fvisibility=hidden",
+            "-O3"
+          ],
+        }],
+        ["OS=='win'", {
+            "libraries": ["ntdll.lib", "synchronization.lib"]
+        }],
+        ["use_data_v1=='true'", {
+          "sources": [
+            "dependencies/lmdb-data-v1/libraries/liblmdb/mdb.c"
+          ],
+          "include_dirs": [
+            "dependencies/lmdb-data-v1/libraries/liblmdb",
+          ],
+        }, {
+          "sources": [
+            "dependencies/lmdb/libraries/liblmdb/mdb.c"
+          ],
+          "include_dirs": [
+            "dependencies/lmdb/libraries/liblmdb",
+          ],
+        }],
+        ["enable_pointer_compression=='true'", {
+          "defines": ["V8_COMPRESS_POINTERS", "V8_COMPRESS_POINTERS_IN_ISOLATE_CAGE"],
+        }],
+        ['runtime=="electron"', {
+          "defines": ["NODE_RUNTIME_ELECTRON=1"]
+        }],
+        ["enable_v8!='false'", {
+          "defines": ["ENABLE_V8_API=1"],
+        }],
+        ["enable_fast_api_calls=='true'", {
+          "defines": ["ENABLE_FAST_API_CALLS=1"],
+        }],
+        ["use_robust=='true'", {
+          "defines": ["MDB_USE_ROBUST"],
+        }, {
+          "defines": ["MDB_USE_ROBUST=0"],
+        }],
+      ],
+    }
+  ]
+}