cbor-x 0.9.1 → 1.0.0

package/README.md

@@ -5,38 +5,41 @@
 [![decode](https://img.shields.io/badge/decode-2GB%2Fs-yellow)](benchmark.md)
 [![types](https://img.shields.io/npm/types/cbor-x)](README.md)
 [![module](https://img.shields.io/badge/module-ESM%2FCJS-blue)](README.md)
+[![license](https://img.shields.io/badge/license-MIT-brightgreen)](LICENSE)
 
-The cbor-x package is an extremely fast CBOR NodeJS/JavaScript implementation. Currently, it is significantly faster than any other known implementations, faster than Avro (for JS), and generally faster than native V8 JSON.stringify/parse, on NodeJS. It implements the CBOR format as specificed in [RFC-8949](https://www.rfc-editor.org/rfc/rfc8949.html), numerous [registered IANA tag extensions](https://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml) (the `x` in cbor-x), [RFC-8746](https://tools.ietf.org/html/rfc8746) and proposed optional [record extension](https://github.com/kriszyp/cbor-records), for defining record structures that makes CBOR even faster and more compact, often over twice as fast as even native JSON functions, several times faster than other JS implementations, and 15-50% more compact. See the performance section for more details. Structured cloning (with support for cyclical references) is supported through these tag extensions.
 
-## Basic Usage
+The cbor-x package is an extremely fast and conformant CBOR NodeJS/JavaScript implementation. Currently, it is over 3-10x faster than any other CBOR JS implementation (including cbor-js and cborg) and faster than most MessagePack encoders, Avro, and generally faster than native V8 JSON.stringify/parse, on NodeJS. It implements the CBOR format as specificed in [RFC-8949](https://www.rfc-editor.org/rfc/rfc8949.html), [RFC-8746](https://tools.ietf.org/html/rfc8746), [RFC-8742](https://datatracker.ietf.org/doc/html/rfc8742), [Packed CBOR](https://www.ietf.org/id/draft-ietf-cbor-packed-03.html), numerous [registered IANA tag extensions](https://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml) (the `x` in cbor-x),  and proposed optional [record extension](https://github.com/kriszyp/cbor-records), for defining record structures that makes CBOR even faster and more compact, often over twice as fast as even native JSON functions, and 15-50% more compact. See the performance section for more details. Structured cloning (with support for cyclical references) is supported through these tag extensions.
+<img align="right" src="./assets/performance.png" width="380"/>
+
 
-Install with:
+## Basic Usage
+Install on NodeJS with:
 
 ```
 npm i cbor-x
 ```
 And `import` or `require` it for basic standard serialization/encoding (`encode`) and deserialization/decoding (`decode`) functions:
-```
+```JavaScript
 import { decode, encode } from 'cbor-x';
 let serializedAsBuffer = encode(value);
 let data = decode(serializedAsBuffer);
 ```
-This `encode` function will generate standard CBOR without any extensions that should be compatible with any standard CBOR parser/decoder. It will serialize JavaScript objects as CBOR `map`s by default. The `decode` function will deserialize CBOR `map`s as an `Object` with the properties from the map.
+This `encode` function will generate standard CBOR without any extensions that should be compatible with any standard CBOR parser/decoder. It will serialize JavaScript objects as CBOR `map`s by default. The `decode` function will deserialize CBOR `map`s as an `Object` with the properties from the map. The cbor-x package runs on any modern JS platform, but does have additional optimizations for NodeJS usage (and will use a node addon for performance boost as an optional dependency).
 
-## Node Usage
-The cbor-x package runs on any modern JS platform, but is optimized for NodeJS usage (and will use a node addon for performance boost as an optional dependency).
+## Deno Usage
+Cbor-x modules are standard ESM modules and can be loaded directly from the [deno.land registry for cbor](https://deno.land/x/cbor) for use in Deno. The standard encode and decode functionality is available on Deno, like other platforms.
 
 ### Streams
 We can use the including streaming functionality (which further improves performance). The `EncoderStream` is a NodeJS transform stream that can be used to serialize objects to a binary stream (writing to network/socket, IPC, etc.), and the `DecoderStream` can be used to deserialize objects from a binary sream (reading from network/socket, etc.):
 
-```
+```JavaScript
 import { EncoderStream } from 'cbor-x';
 let stream = new EncoderStream();
 stream.write(myData);
 
 ```
 Or for a full example of sending and receiving data on a stream:
-```
+```JavaScript
 import { EncoderStream } from 'cbor-x';
 let sendingStream = new EncoderStream();
 let receivingStream = new DecoderStream();
@@ -51,24 +54,24 @@ receivingStream.on('data', (data) => {
 The `EncoderStream` and `DecoderStream` instances  will have also the record structure extension enabled by default (see below).
 
 ## Deno Usage
-CBOR modules are standard ESM modules and can be loaded directly from github (https://raw.githubusercontent.com/kriszyp/cbor-x/master/index.js) or downloaded and used directly in Deno. The standard encode and decode functionality is available on Deno, like other platforms.
+Cbor-x modules are standard ESM modules and can be loaded directly from the [deno.land registry for cbor](https://deno.land/x/cbor) for use in Deno. The standard pack/encode and unpack/decode functionality is available on Deno, like other platforms.
 
 ## Browser Usage
 Cbor-x  works as standalone JavaScript as well, and runs on modern browsers. It includes a bundled script, at `dist/index.js` for ease of direct loading:
-```
+```HTML
 <script src="node_modules/cbor-x/dist/index.js"></script>
 ```
 
 This is UMD based, and will register as a module if possible, or create a `CBOR` global with all the exported functions.
 
 For module-based development, it is recommended that you directly import the module of interest, to minimize dependencies that get pulled into your application:
-```
+```JavaScript
 import { decode } from 'cbor-x/decode' // if you only need to decode
 ```
 
 ## Structured Cloning
 You can also use cbor-x for [structured cloning](https://html.spec.whatwg.org/multipage/structured-data.html). By enabling the `structuredClone` option, you can include references to other objects or cyclic references, and object identity will be preserved. Structured cloning also enables preserving certain typed objects like `Error`, `Set`, `RegExp` and TypedArray instances, using [registered CBOR tag extensions](https://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml). For example:
-```
+```JavaScript
 let obj = {
 	set: new Set(['a', 'b']),
 	regular: /a\spattern/
@@ -86,7 +89,7 @@ This option is disabled by default because reference checking degrades performan
 
 ## Record / Object Structures
 There is a critical difference between maps (or dictionaries) that hold an arbitrary set of keys and values (JavaScript `Map` is designed for these), and records or object structures that have a well-defined set of fields. Typical JS objects/records may have many instances re(use) the same structure. By using the record extension, this distinction is preserved in CBOR and the encoding can reuse structures and not only provides better type preservation, but yield much more compact encodings and increase decoding performance by 2-3x. Cbor-x automatically generates record definitions that are reused and referenced by objects with the same structure. Records use CBOR's tags to align well CBOR's tag/extension mechanism. There are a number of ways to use this to our advantage. For large object structures with repeating nested objects with similar structures, simply serializing with the record extension can yield significant benefits. To use the record structures extension, we create a new `Encoder` instance. By default a new `Encoder` instance will have the record extension enabled:
-```
+```JavaScript
 import { Encoder } from 'cbor-x';
 let encoder = new Encoder();
 encoder.encode(myBigData);
@@ -101,14 +104,14 @@ Streaming with record structures works by encoding a structure the first time it
 ### Shared Record Structures
 Another useful way of using cbor-x, and the record extension, is for storing data in a databases, files, or other storage systems. If a number of objects with common data structures are being stored, a shared structure can be used to greatly improve data storage and deserialization efficiency. In the simplest form, provide a `structures` array, which is updated if any new object structure is encountered:
 
-```
+```JavaScript
 import { Encoder } from 'cbor-x';
 let encoder = new Encoder({
 	structures: [... structures that were last generated ...]
 });
 ```
 If you are working with persisted data, you will need to persist the `structures` data when it is updated. Cbor-x provides an API for loading and saving the `structures` on demand (which is robust and can be used in multiple-process situations where other processes may be updating this same `structures` array), we just need to provide a way to store the generated shared structure so it is available to deserialize stored data in the future:
-```
+```JavaScript
 import { Encoder } from 'cbor-x';
 let encoder = new Encoder({
 	getStructures() {
@@ -136,16 +139,45 @@ decodeMultiple(data, (value) => {
 	// called for each value
 	// return false if you wish to end the parsing
 })
+```
+### KeyMaps for Senml
+KeyMaps can be used to remap properties of source Objects and Maps to numerical equivalents for more efficient encoding. 
+The principle driver for this feature is to support `application/senml+cbor`content-encoding as defined in https://datatracker.ietf.org/doc/html/rfc8428#section-6 for use in LWM2M application (see http://www.openmobilealliance.org/release/LightweightM2M/V1_2-20201110-A/HTML-Version/OMA-TS-LightweightM2M_Core-V1_2-20201110-A.html#7-4-7-0-747-SenML-CBOR)
+
+Records are also supported in conjunction with keyMaps, but these are disabled by default when keyMaps are specified as use of the two features does not introduce any additional compression efficiency unless that the data arrays are quite large (> 10 items).
+
+```JavaScript
+import { Encoder } from 'cbor-x'
+const data = [ 
+	{ bn: '/3303/0/5700', bt: 1278887, v: 35.5 },
+	{ t: 10, v: 34 },
+	{ t: 20, v: 33 },
+	{ t: 30, v: 32 },
+	{ t: 40, v: 31 },
+	{ t: 50, v: 30 } 
+]
+
+let senmlKeys = { bs: -6, bv: -5, bu: -4, bt: -3, bn: -2, bver: -1, n: 0, u: 1, v: 2, vs: 3, vb: 4, s: 5, t: 6, ut: 7, vd: 8}}
+let senmlCbor = new Encoder({ keyMap: senmlKeys })
+let basicCbor = new Encoder()
+let senmlBuff = senmlCbor.encode(data)
+let basicBuff = basicCbor.encode(data)
+console.log('Senml CBOR size:', senmlBuff.length) // 77
+console.log('Basic CBOR size:', basicBuff.length) // 90
+assert.deepEqual(senmlEncoder.decode(senmlBuff), data)
+
 ```
 
-CBOR Packing
+### CBOR Packing
 [Packed CBOR](https://datatracker.ietf.org/doc/html/draft-ietf-cbor-packed) is additional specification for CBOR which allows for compact encoding of data that has repeated values. Cbor-x supports decoding packed CBOR, no or flags/options needed. Cbor-x can also optionally generate packed CBOR (with the `pack` option), which will cause the encoder to look for repeated strings in a data structure that is being encoded, and store the strings in a packed table that can be referenced, to reduce encoding size. This involves extra overhead and reduces encoding performance, and generally does not yield as much compaction as standard compression tools. However, this is can be much faster than encoding plus compression, while still providing some level of reduction in encoding size. In addition to size reduction, packed CBOR is also usually faster to decode (assuming that some repetitive values could be found/packed).
 
 Cbor-x also has in-progress effort to support shared packed tables.
 
+
 ## Options
 The following options properties can be provided to the Encoder or Decoder constructor:
 
+* `keyMap` - This can be set to an object which will be used to map keys in the source Object or Map to other keys including integers. This allows for more efficient encoding, and enables support for numeric cbar tag encodings such as used by `application/senml+cbor` (https://datatracker.ietf.org/doc/html/rfc8428#section-6)
 * `useRecords` - Setting this to `false` disables the record extension and stores JavaScript objects as CBOR maps (with tag 259), and decodes maps as JavaScript `Object`s, which ensures compatibilty with other decoders.
 * `structures` - Provides the array of structures that is to be used for record extension, if you want the structures saved and used again. This array will be modified in place with new record structures that are serialized (if less than 64 structures are in the array).
 * `structuredClone` - This enables the structured cloning extensions that will encode object/cyclic references and additional built-in types/classes.
@@ -154,14 +186,16 @@ The following options properties can be provided to the Encoder or Decoder const
 * `pack` - This will enable [CBOR packing](https://datatracker.ietf.org/doc/html/draft-ietf-cbor-packed) for encoding, as described above.
 * `variableMapSize` - This will use varying map size definition (from single-byte to full 32-bit representation) based on the number of keys when encoding objects, which yields slightly more compact encodings (for small objects), but is typically 5-10% slower during encoding. This is only relevant when record extension is disabled.
 * `copyBuffers` - When decoding a CBOR message with binary data (Buffers are encoded as binary data), copy the buffer rather than providing a slice/view of the buffer. If you want your input data to be collected or modified while the decoded embedded buffer continues to live on, you can use this option (there is extra overhead to copying).
+* `bundleStrings` - If `true` this uses a custom extension that bundles strings together, so that they can be decoded more quickly on browsers and Deno that do not have access to the NodeJS addon. This a custom extension, so both encoder and decoder need to support this. This can yield significant decoding performance increases on browsers (30%-50%).
 * `useTimestamp32` - Encode JS `Date`s in 32-bit format when possible by dropping the milliseconds. This is a more efficient encoding of dates. You can also cause dates to use 32-bit format by manually setting the milliseconds to zero (`date.setMilliseconds(0)`).
 * `sequential` - Encode structures in serialized data, and reference previously encoded structures with expectation that decoder will read the encoded structures in the same order as encoded, with `unpackMultiple`.
 * `largeBigIntToFloat` - If a bigint needs to be encoded that is larger than will fit in 64-bit integers, it will be encoded as a float-64 (otherwise will throw a RangeError).
 * `useTag259ForMaps` - This flag indicates if [tag 259 (explicit maps)](https://github.com/shanewholloway/js-cbor-codec/blob/master/docs/CBOR-259-spec--explicit-maps.md) should be used to encode JS `Map`s. When using records is enabled, this is disabled by default, since plain objects are encoded with record structures and unambigiously differentiated from `Map`s, which are encoded as CBOR maps. Without using records, this enabled by default and is necessary to distinguish plain objects from `Map`s (but can be disabled by setting this to `false`).
+* `int64AsNumber` - This will decode uint64 and int64 numbers as standard JS numbers rather than as bigint numbers.
 
 ### 32-bit Float Options
 By default all non-integer numbers are serialized as 64-bit float (double). This is fast, and ensures maximum precision. However, often real-world data doesn't not need 64-bits of precision, and using 32-bit encoding can be much more space efficient. There are several options that provide more efficient encodings. Using the decimal rounding options for encoding and decoding provides lossless storage of common decimal representations like 7.99, in more efficient 32-bit format (rather than 64-bit). The `useFloat32` property has several possible options, available from the module as constants:
-```
+```JavaScript
 import { ALWAYS, DECIMAL_ROUND, DECIMAL_FIT } from 'cbor-x'
 ```
 
@@ -176,6 +210,15 @@ In addition, msgpackr exports a `roundFloat32(number)` function that can be used
 ## Performance
 Cbor-x is fast. Really fast. Here is comparison with the next fastest JS projects using the benchmark tool from `msgpack-lite` (and the sample data is from some clinical research data we use that has a good mix of different value types and structures). It also includes comparison to V8 native JSON functionality, and JavaScript Avro (`avsc`, a very optimized Avro implementation):
 
+### Native Acceleration
+Cbor-x employs an optional native node-addon to accelerate the parsing of strings. This should be automatically installed and utilized on NodeJS. However, you can verify this by checking the `isNativeAccelerationEnabled` property that is exported from cbor-x. If this is `false`, the `cbor-extract` package may not have been properly installed, and you may want to verify that it is installed correctly:
+```js
+import { isNativeAccelerationEnabled } from 'cbor-x'
+if (!isNativeAccelerationEnabled)
+	console.warn('Native acceleration not enabled, verify that install finished properly')
+```
+
+
 operation                                                  |   op   |   ms  |  op/s
 ---------------------------------------------------------- | ------: | ----: | -----:
 buf = Buffer(JSON.stringify(obj));                         |   78200 |  5004 |  15627
@@ -217,7 +260,7 @@ See the [benchmark.md](benchmark.md) for more benchmarks and information about b
 
 ## Custom Extensions
 You can add your own custom extensions, which can be used to encode specific types/classes in certain ways. This is done by using the `addExtension` function, and specifying the class, extension type code (custom extensions should be a number greater than 40500, all others are reserved for  CBOR or cbor-x), and your encode and decode functions (or just the one you need). You can use cbor-x encoding and decoding within your extensions:
-```
+```JavaScript
 import { addExtension, Encoder } from 'cbor-x';
 
 class MyCustomClass {...}
@@ -267,7 +310,10 @@ Here is a list of CBOR tags that are supported for decoding:
 * 1 - Numeric Date
 * 2 - BigInt
 * 3 - Negative BigInt
+* 6 - Packed string reference
 * 27 - Generic named objects (used for Error, RegExp)
+* 28, 29 - Value sharing/object referencing
+* 51 - Packed table
 * 64 - Uint8Array
 * 68 - Uint8ClampedArray
 * 69 - Uint16Array
@@ -282,11 +328,11 @@ Here is a list of CBOR tags that are supported for decoding:
 * 105 - Records
 * 258 - Set
 * 259 - Map
-* 40009, 40010 - Pointers for cycles
+* 57344 - 57599 - Records
 
 ## Alternate Encoding/Package
 The high-performance serialization and deserialization algorithms in this package are also available in the [msgpackr](https://github.com/kriszyp/msgpackr) for the MessagePack format, with the same API and design. A quick summary of the pros and cons of using MessagePack vs CBOR are:
-* MessagePack has wider adoption, and, at least with this implementation is slightly more efficient (by roughly 1%, but YMMV).
+* MessagePack has wider adoption and msgpackr has broader usage.
 * CBOR has an [official IETF standardization track](https://www.rfc-editor.org/rfc/rfc8949.html), and the record extensions is conceptually/philosophically a better fit for CBOR tags.
 
 ## License

package/decode.d.ts

@@ -13,6 +13,7 @@ export interface Options {
 	mapsAsObjects?: boolean
 	variableMapSize?: boolean
 	copyBuffers?: boolean
+	bundleStrings?: boolean
 	useTimestamp32?: boolean
 	largeBigIntToFloat?: boolean
 	encodeUndefinedAsNil?: boolean
@@ -21,6 +22,7 @@ export interface Options {
 	shouldShareStructure?: (keys: string[]) => boolean
 	getStructures?(): {}[]
 	saveStructures?(structures: {}[]): boolean | void
+	onInvalidDate?: () => any
 }
 interface Extension {
 	Class: Function
@@ -37,3 +39,5 @@ export function decode(messagePack: Buffer | Uint8Array): any
 export function decodeMultiple(messagePack: Buffer | Uint8Array, forEach?: (value: any) => any): [] | void
 export function addExtension(extension: Extension): void
 export function roundFloat32(float32Number: number): number
+export let isNativeAccelerationEnabled: boolean
+