cborg 4.0.9 → 4.1.1

package/CHANGELOG.md

@@ -1,3 +1,22 @@
+## [4.1.1](https://github.com/rvagg/cborg/compare/v4.1.0...v4.1.1) (2024-03-11)
+
+
+### Trivial Changes
+
+* add byte strings example using tags to retain typedarray types ([5bf989b](https://github.com/rvagg/cborg/commit/5bf989bf7dde0a6124fb1605f3caf899273c6d4e))
+
+## [4.1.0](https://github.com/rvagg/cborg/compare/v4.0.9...v4.1.0) (2024-02-29)
+
+
+### Features
+
+* export `Tokenizer`, document how it can be used ([fbba395](https://github.com/rvagg/cborg/commit/fbba395b6848bd8dcb66a1d18c36e1033581b5ef))
+
+
+### Bug Fixes
+
+* update tyupes for new exports ([986035d](https://github.com/rvagg/cborg/commit/986035dbffbc682b06d7bfecb8d2c3cc02048429))
+
 ## [4.0.9](https://github.com/rvagg/cborg/compare/v4.0.8...v4.0.9) (2024-02-07)
 
 

package/README.md

@@ -31,10 +31,12 @@
   * [`encodedLength(data[, options])`](#encodedlengthdata-options)
   * [Type encoders](#type-encoders)
   * [Tag decoders](#tag-decoders)
+* [Decoding with a custom tokeniser](#decoding-with-a-custom-tokeniser)
 * [Deterministic encoding recommendations](#deterministic-encoding-recommendations)
   * [Round-trip consistency](#round-trip-consistency)
 * [JSON mode](#json-mode)
   * [Example](#example-1)
+* [Advanced types and tags](#advanced-types-and-tags)
 * [License and Copyright](#license-and-copyright)
 
 ## Example
@@ -243,7 +245,7 @@ Decode valid CBOR bytes from a `Uint8Array` (or `Buffer`) and return a JavaScrip
 * `rejectDuplicateMapKeys` (boolean, default `false`): when the decoder encounters duplicate keys for the same map, an error will be thrown when this option is set. This is an additional _strictness_ option, disallowing data-hiding and reducing the number of same-data different-bytes possibilities where it matters.
 * `retainStringBytes` (boolean, default `false`): when decoding strings, retain the original bytes on the `Token` object as `byteValue`. Since it is possible to encode non-UTF-8 characters in strings in CBOR, and JavaScript doesn't properly handle non-UTF-8 in its conversion from bytes (`TextEncoder` or `Buffer`), this can result in a loss of data (and an inability to round-trip). Where this is important, a token stream should be consumed instead of a plain `decode()` and the `byteValue` property on string tokens can be inspected (see [lib/diagnostic.js](lib/diagnostic.js) for an example of its use.)
 * `tags` (array): a mapping of tag number to tag decoder function. By default no tags are supported. See [Tag decoders](#tag-decoders).
-* `tokenizer` (object): an object with two methods, `next()` which returns a `Token` and `done()` which returns a `boolean`. Can be used to implement custom input decoding. See the source code for examples.
+* `tokenizer` (object): an object with two methods, `next()` which returns a `Token`, `done()` which returns a `boolean` and `pos()` which returns the current byte position being decoded. Can be used to implement custom input decoding. See the source code for examples. (Note en-US spelling "tokenizer" used throughout exported methods and types, which may be confused with "tokeniser" used in these docs).
 
 ### `decodeFirst(data[, options])`
 
@@ -383,6 +385,45 @@ function bigNegIntDecoder (bytes) {
 }
 ```
 
+## Decoding with a custom tokeniser
+
+`decode()` allows overriding the `tokenizer` option to provide a custom tokeniser. This object can be described with the following interface:
+
+```typescript
+export interface DecodeTokenizer {
+  next(): Token,
+  done(): boolean,
+  pos(): number,
+}
+```
+
+`next()` should return the next token in the stream, `done()` should return `true` when the stream is finished, and `pos()` should return the current byte position in the stream.
+
+Overriding the default tokeniser can be useful for changing the rules of decode. For example, it is used to turn cborg into a JSON decoder by changing parsing rules on how to turn bytes into tokens. See the source code for how this works.
+
+The default `Tokenizer` class is available from the default export. Providing `options.tokenizer = new Tokenizer(bytes, options)` would result in the same decode path using this tokeniser. However, this can also be used to override or modify default decode paths by intercepting the token stream. For example, to perform a decode that disallows bytes, the following code would work:
+
+```js
+import { decode, Tokenizer, Type } from 'cborg'
+
+class CustomTokeniser extends Tokenizer {
+  next () {
+    const nextToken = super.next()
+    if (nextToken.type === Type.bytes) {
+      throw new Error('Unsupported type: bytes')
+    }
+    return nextToken
+  }
+}
+
+function customDecode (data, options) {
+  options = Object.assign({}, options, {
+    tokenizer: new CustomTokeniser(data, options)
+  })
+  return decode(data, options)
+}
+```
+
 ## Deterministic encoding recommendations
 
 cborg is designed with deterministic encoding forms as a primary feature. It is suitable for use with content addressed systems or other systems where convergence of binary forms is important. The ideal is to have strictly _one way_ of mapping a set of data into a binary form. Unfortunately CBOR has many opportunities for flexibility, including:
@@ -428,7 +469,7 @@ There are a number of forms where an object will not round-trip precisely, if th
 
 Use `import { encode, decode, decodeFirst } from 'cborg/json'` to access the JSON handling encoder and decoder.
 
-Many of the same encode and decode options available for CBOR can be used to manage JSON handling. These include strictness requirements for decode and custom tag encoders for encode. Tag encoders can't create new tags as there are no tags in JSON, but they can replace JavaScript object forms with custom JSON forms (e.g. convert a `Uint8Array` to a valid JSON form rather than having the encoder throw an error). The inverse is also possible, turning specific JSON forms into JavaScript forms, by using a custom tokenizer on decode.
+Many of the same encode and decode options available for CBOR can be used to manage JSON handling. These include strictness requirements for decode and custom tag encoders for encode. Tag encoders can't create new tags as there are no tags in JSON, but they can replace JavaScript object forms with custom JSON forms (e.g. convert a `Uint8Array` to a valid JSON form rather than having the encoder throw an error). The inverse is also possible, turning specific JSON forms into JavaScript forms, by using a custom tokeniser on decode.
 
 Special notes on options specific to the JSON:
 
@@ -461,6 +502,10 @@ encoded: Uint8Array(34) [
 encoded (string): {"this":{"is":"JSON!","yay":true}}
 ```
 
+## Advanced types and tags
+
+As demonstrated above, the ability to provide custom `typeEncoders` to `encode()`, `tags` and even a custom `tokenizer` to `decode()` allow for quite a bit of flexibility in manipulating both the encode and decode process. An advanced example that uses all of these features can be found in [example-bytestrings.js](./example-bytestrings.js) which demonstrates how one might implement [RFC 8746](https://www.rfc-editor.org/rfc/rfc8746.html) to allow typed arrays to round-trip through CBOR and retain their original types. Since cborg is designed to speak purely in terms of `Uint8Array`s, its default behaviour will squash all typed arrays down to their byte array forms and materialise them as plain `Uint8Arrays`. Where round-trip fidelity is important and CBOR tags are an option, this form of usage is an option.
+
 ## License and Copyright
 
 Copyright 2020 Rod Vagg

package/cborg.js

@@ -1,5 +1,5 @@
 import { encode } from './lib/encode.js'
-import { decode, decodeFirst } from './lib/decode.js'
+import { decode, decodeFirst, Tokeniser, tokensToObject } from './lib/decode.js'
 import { Token, Type } from './lib/token.js'
 
 /**
@@ -14,6 +14,8 @@ import { Token, Type } from './lib/token.js'
 export {
   decode,
   decodeFirst,
+  Tokeniser as Tokenizer,
+  tokensToObject,
   encode,
   Token,
   Type

package/example-bytestrings.js

@@ -0,0 +1,180 @@
+/*
+RFC 8746 defines a set of tags to use for typed arrays. Out of the box, cborg doesn't care about
+tags and just squashes all concerns around byte arrays to Uint8Array with major type 2. This is
+fine for most use cases, but it is lossy, you can't round-trip and retain your original type.
+
+This example shows how to use cborg to round-trip a typed array with tags.
+
+https://www.rfc-editor.org/rfc/rfc8746.html
+*/
+
+import { encode, decode, Token, Tokenizer, Type } from 'cborg.js'
+
+const tagUint8Array = 64
+const tagUint64Array = 71
+// etc... see https://www.rfc-editor.org/rfc/rfc8746.html#name-iana-considerations
+
+/* ENCODERS */
+
+/**
+ * @param {any} obj
+ * @returns {[Token]}
+ */
+function uint8ArrayEncoder (obj) {
+  if (!(obj instanceof Uint8Array)) {
+    throw new Error('expected Uint8Array')
+  }
+  return [
+    new Token(Type.tag, tagUint8Array),
+    new Token(Type.bytes, obj)
+  ]
+}
+
+/**
+ * @param {any} obj
+ * @returns {[Token]}
+ */
+function uint64ArrayEncoder (obj) {
+  if (!(obj instanceof BigUint64Array)) {
+    throw new Error('expected BigUint64Array')
+  }
+  return [
+    new Token(Type.tag, tagUint64Array),
+    // BigUint64Array to a Uint8Array, but we have to pay attention to the possibility of it being
+    // a view of a larger ArrayBuffer.
+    new Token(Type.bytes, new Uint8Array(obj.buffer, obj.byteOffset, obj.byteLength))
+  ]
+}
+
+// etc...
+
+const typeEncoders = {
+  Uint8Array: uint8ArrayEncoder,
+  BigUint64Array: uint64ArrayEncoder
+}
+
+/* DECODERS */
+
+/**
+ * @param {ArrayBuffer} bytes
+ * @returns {any}
+ */
+function uint8ArrayDecoder (bytes) {
+  if (!(bytes instanceof ArrayBuffer)) {
+    throw new Error('expected ArrayBuffer')
+  }
+  return new Uint8Array(bytes)
+}
+
+/**
+ * @param {ArrayBuffer} bytes
+ * @returns {any}
+ */
+function uint64ArrayDecoder (bytes) {
+  if (!(bytes instanceof ArrayBuffer)) {
+    throw new Error('expected ArrayBuffer')
+  }
+  return new BigUint64Array(bytes)
+}
+
+// etc...
+
+const tags = []
+tags[tagUint8Array] = uint8ArrayDecoder
+tags[tagUint64Array] = uint64ArrayDecoder
+
+/* TOKENIZER */
+
+// We have to deal with the fact that cborg talks in Uint8Arrays but we now want it to treat major 2
+// as ArrayBuffers, so we have to transform the token stream to replace the Uint8Array with an
+// ArrayBuffer.
+
+class ArrayBufferTransformingTokeniser extends Tokenizer {
+  next () {
+    const nextToken = super.next()
+    if (nextToken.type === Type.bytes) {
+      // Transform the (assumed) Uint8Array value to an ArrayBuffer of the same bytes, note though
+      // that all tags we care about are going to be <tag><bytes>, so we're also transforming those
+      // into ArrayBuffers, so our tag decoders need to also assume they are getting ArrayBuffers
+      // now. An alternative would be to watch the token stream for <tag> and not transform the next
+      // token if it's <bytes>, but that's a bit more complicated for demo purposes.
+      nextToken.value = nextToken.value.buffer
+    }
+    return nextToken
+  }
+}
+
+// Optional: a new decode() wrapper, mainly so we don't have to deal with the complications of\
+// instantiating a Tokenizer which needs both data and the options.
+function byteStringDecoder (data, options) {
+  options = Object.assign({}, options, {
+    tags,
+    tokenizer: new ArrayBufferTransformingTokeniser(data, options)
+  })
+  return decode(data, options)
+}
+
+/* ROUND-TRIP */
+
+const original = {
+  u8: new Uint8Array([1, 2, 3, 4, 5]),
+  u64: new BigUint64Array([10000000000000000n, 20000000000000000n, 30000000000000000n, 40000000000000000n, 50000000000000000n]),
+  ab: new Uint8Array([6, 7, 8, 9, 10]).buffer
+}
+
+const encoded = encode(original, { typeEncoders })
+
+const decoded = byteStringDecoder(encoded)
+
+console.log('Original:', original)
+console.log('Encoded:', Buffer.from(encoded).toString('hex')) // excuse the Buffer, sorry browser peeps
+console.log('Decoded:', decoded)
+
+/* Output:
+
+Original: {
+  u8: Uint8Array(5) [ 1, 2, 3, 4, 5 ],
+  u64: BigUint64Array(5) [
+    10000000000000000n,
+    20000000000000000n,
+    30000000000000000n,
+    40000000000000000n,
+    50000000000000000n
+  ],
+  ab: ArrayBuffer { [Uint8Contents]: <06 07 08 09 0a>, byteLength: 5 }
+}
+Encoded: a362616245060708090a627538d84045010203040563753634d84758280000c16ff2862300000082dfe40d47000000434fd7946a00000004bfc91b8e000000c52ebca2b100
+Decoded: {
+  ab: ArrayBuffer { [Uint8Contents]: <06 07 08 09 0a>, byteLength: 5 },
+  u8: Uint8Array(5) [ 1, 2, 3, 4, 5 ],
+  u64: BigUint64Array(5) [
+    10000000000000000n,
+    20000000000000000n,
+    30000000000000000n,
+    40000000000000000n,
+    50000000000000000n
+  ]
+}
+
+*/
+
+/* Diagnostic:
+
+$ cborg hex2diag a362616245060708090a627538d84045010203040563753634d84758280000c16ff2862300000082dfe40d47000000434fd7946a00000004bfc91b8e000000c52ebca2b100
+a3                                                # map(3)
+  62                                              #   string(2)
+    6162                                          #     "ab"
+  45                                              #   bytes(5)
+    060708090a                                    #     "\x06\x07\x08\x09\x0a"
+  62                                              #   string(2)
+    7538                                          #     "u8"
+  d8 40                                           #   tag(64)
+    45                                            #     bytes(5)
+      0102030405                                  #       "\x01\x02\x03\x04\x05"
+  63                                              #   string(3)
+    753634                                        #     "u64"
+  d8 47                                           #   tag(71)
+    58 28                                         #     bytes(40)
+      0000c16ff2862300000082dfe40d47000000434fd7  #       "\x00\x00Áoò\x86#\x00\x00\x00\x82ßä\x0dG\x00\x00\x00CO×"
+      946a00000004bfc91b8e000000c52ebca2b100      #       "\x94j\x00\x00\x00\x04¿É\x1b\x8e\x00\x00\x00Å.¼¢±\x00
+*/

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "cborg",
-  "version": "4.0.9",
+  "version": "4.1.1",
   "description": "Fast CBOR with a focus on strictness",
   "main": "cborg.js",
   "type": "module",

package/types/cborg.d.ts

@@ -16,8 +16,10 @@ export type DecodeOptions = import('./interface').DecodeOptions;
 export type EncodeOptions = import('./interface').EncodeOptions;
 import { decode } from './lib/decode.js';
 import { decodeFirst } from './lib/decode.js';
+import { Tokeniser } from './lib/decode.js';
+import { tokensToObject } from './lib/decode.js';
 import { encode } from './lib/encode.js';
 import { Token } from './lib/token.js';
 import { Type } from './lib/token.js';
-export { decode, decodeFirst, encode, Token, Type };
+export { decode, decodeFirst, Tokeniser as Tokenizer, tokensToObject, encode, Token, Type };
 //# sourceMappingURL=cborg.d.ts.map

package/types/cborg.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"cborg.d.ts","sourceRoot":"","sources":["../cborg.js"],"names":[],"mappings":";;;yBAMa,OAAO,aAAa,EAAE,UAAU;;;;0BAEhC,OAAO,aAAa,EAAE,mBAAmB;;;;4BACzC,OAAO,aAAa,EAAE,aAAa;;;;4BACnC,OAAO,aAAa,EAAE,aAAa;uBATZ,iBAAiB;4BAAjB,iBAAiB;uBAD9B,iBAAiB;sBAEZ,gBAAgB;qBAAhB,gBAAgB"}
+{"version":3,"file":"cborg.d.ts","sourceRoot":"","sources":["../cborg.js"],"names":[],"mappings":";;;yBAMa,OAAO,aAAa,EAAE,UAAU;;;;0BAEhC,OAAO,aAAa,EAAE,mBAAmB;;;;4BACzC,OAAO,aAAa,EAAE,aAAa;;;;4BACnC,OAAO,aAAa,EAAE,aAAa;uBATe,iBAAiB;4BAAjB,iBAAiB;0BAAjB,iBAAiB;+BAAjB,iBAAiB;uBADzD,iBAAiB;sBAEZ,gBAAgB;qBAAhB,gBAAgB"}