@ipld/car 3.2.4 → 4.1.1

package/README.md

@@ -240,6 +240,21 @@ be directly fed to a
  * [`async CarWriter.createAppender()`](#CarWriter__createAppender)
  * [`async CarWriter.updateRootsInBytes(bytes, roots)`](#CarWriter__updateRootsInBytes)
  * [`async CarWriter.updateRootsInFile(fd, roots)`](#CarWriter__updateRootsInFile)
+ * [`class CarBufferWriter`](#CarBufferWriter)
+ * [`CarBufferWriter#addRoot(root, options)`](#CarBufferWriter_addRoot)
+ * [`CarBufferWriter#write(block)`](#CarBufferWriter_write)
+ * [`CarBufferWriter#close([options])`](#CarBufferWriter_close)
+ * [`CarBufferWriter.blockLength(Block)`](#CarBufferWriter__blockLength__Block__)
+ * [`CarBufferWriter.calculateHeaderLength(rootLengths)`](#CarBufferWriter__calculateHeaderLength__rootLengths__)
+ * [`CarBufferWriter.headerLength({ roots })`](#CarBufferWriter__headerLength______roots______)
+ * [`CarBufferWriter.estimateHeaderLength(rootCount[, rootByteLength])`](#CarBufferWriter__estimateHeaderLength__rootCount______rootByteLength____)
+ * [`CarBufferWriter.createWriter(buffer[, options])`](#CarBufferWriter__createWriter__buffer______options____)
+ * [`async decoder.readHeader(reader)`](#async__decoder__readHeader__reader__)
+ * [`async decoder.readBlockHead(reader)`](#async__decoder__readBlockHead__reader__)
+ * [`decoder.createDecoder(reader)`](#decoder__createDecoder__reader__)
+ * [`decoder.bytesReader(bytes)`](#decoder__bytesReader__bytes__)
+ * [`decoder.asyncIterableReader(asyncIterable)`](#decoder__asyncIterableReader__asyncIterable__)
+ * [`decoder.limitReader(reader, byteLimit)`](#decoder__limitReader__reader____byteLimit__)
 
 <a name="CarReader"></a>
 ### `class CarReader`
@@ -247,7 +262,7 @@ be directly fed to a
 Properties:
 
 * `version` `(number)`: The version number of the CAR referenced by this
-  reader (should be `1`).
+  reader (should be `1` or `2`).
 
 Provides blockstore-like access to a CAR.
 
@@ -748,7 +763,8 @@ upon successful modification.
 
 Update the list of roots in the header of an existing CAR file. The first
 argument must be a file descriptor for CAR file that is open in read and
-write mode (not append).
+write mode (not append), e.g. `fs.open` or `fs.promises.open` with `'r+'`
+mode.
 
 This operation is an _overwrite_, the total length of the CAR will not be
 modified. A rejection will occur if the new header will not be the same
@@ -759,6 +775,171 @@ replaced encode as the same length as the new roots.
 This function is **only available in Node.js** and not a browser
 environment.
 
+<a name="CarBufferWriter"></a>
+### `class CarBufferWriter`
+
+A simple CAR writer that writes to a pre-allocated buffer.
+
+<a name="CarBufferWriter_addRoot"></a>
+### `CarBufferWriter#addRoot(root, options)`
+
+* `root` `(CID)`
+* `options`
+
+* Returns:  `CarBufferWriter`
+
+Add a root to this writer, to be used to create a header when the CAR is
+finalized with [`close()`](#CarBufferWriter__close)
+
+<a name="CarBufferWriter_write"></a>
+### `CarBufferWriter#write(block)`
+
+* `block` `(Block)`: A `{ cid:CID, bytes:Uint8Array }` pair.
+
+* Returns:  `CarBufferWriter`
+
+Write a `Block` (a `{ cid:CID, bytes:Uint8Array }` pair) to the archive.
+Throws if there is not enough capacity.
+
+<a name="CarBufferWriter_close"></a>
+### `CarBufferWriter#close([options])`
+
+* `options` `(object, optional)`
+  * `options.resize` `(boolean, optional)`
+
+* Returns:  `Uint8Array`
+
+Finalize the CAR and return it as a `Uint8Array`.
+
+<a name="CarBufferWriter__blockLength__Block__"></a>
+### `CarBufferWriter.blockLength(Block)`
+
+* `block` `(Block)`
+
+* Returns:  `number`
+
+Calculates number of bytes required for storing given block in CAR. Useful in
+estimating size of an `ArrayBuffer` for the `CarBufferWriter`.
+
+<a name="CarBufferWriter__calculateHeaderLength__rootLengths__"></a>
+### `CarBufferWriter.calculateHeaderLength(rootLengths)`
+
+* `rootLengths` `(number[])`
+
+* Returns:  `number`
+
+Calculates header size given the array of byteLength for roots.
+
+<a name="CarBufferWriter__headerLength______roots______"></a>
+### `CarBufferWriter.headerLength({ roots })`
+
+* `options` `(object)`
+  * `options.roots` `(CID[])`
+
+* Returns:  `number`
+
+Calculates header size given the array of roots.
+
+<a name="CarBufferWriter__estimateHeaderLength__rootCount______rootByteLength____"></a>
+### `CarBufferWriter.estimateHeaderLength(rootCount[, rootByteLength])`
+
+* `rootCount` `(number)`
+* `rootByteLength` `(number, optional)`
+
+* Returns:  `number`
+
+Estimates header size given a count of the roots and the expected byte length
+of the root CIDs. The default length works for a standard CIDv1 with a
+single-byte multihash code, such as SHA2-256 (i.e. the most common CIDv1).
+
+<a name="CarBufferWriter__createWriter__buffer______options____"></a>
+### `CarBufferWriter.createWriter(buffer[, options])`
+
+* `buffer` `(ArrayBuffer)`
+* `options` `(object, optional)`
+  * `options.roots` `(CID[], optional)`
+  * `options.byteOffset` `(number, optional)`
+  * `options.byteLength` `(number, optional)`
+  * `options.headerSize` `(number, optional)`
+
+* Returns:  `CarBufferWriter`
+
+Creates synchronous CAR writer that can be used to encode blocks into a given
+buffer. Optionally you could pass `byteOffset` and `byteLength` to specify a
+range inside buffer to write into. If car file is going to have `roots` you
+need to either pass them under `options.roots` (from which header size will
+be calculated) or provide `options.headerSize` to allocate required space
+in the buffer. You may also provide known `roots` and `headerSize` to
+allocate space for the roots that may not be known ahead of time.
+
+Note: Incorrect `headerSize` may lead to copying bytes inside a buffer
+which will have a negative impact on performance.
+
+<a name="async__decoder__readHeader__reader__"></a>
+### `async decoder.readHeader(reader)`
+
+* `reader` `(BytesReader)`
+* `strictVersion` `(number, optional)`
+
+* Returns:  `Promise<(CarHeader|CarV2Header)>`
+
+Reads header data from a `BytesReader`. The header may either be in the form
+of a `CarHeader` or `CarV2Header` depending on the CAR being read.
+
+<a name="async__decoder__readBlockHead__reader__"></a>
+### `async decoder.readBlockHead(reader)`
+
+* `reader` `(BytesReader)`
+
+* Returns:  `Promise<BlockHeader>`
+
+Reads the leading data of an individual block from CAR data from a
+`BytesReader`. Returns a `BlockHeader` object which contains
+`{ cid, length, blockLength }` which can be used to either index the block
+or read the block binary data.
+
+<a name="decoder__createDecoder__reader__"></a>
+### `decoder.createDecoder(reader)`
+
+* `reader` `(BytesReader)`
+
+* Returns:  `CarDecoder`
+
+Creates a `CarDecoder` from a `BytesReader`. The `CarDecoder` is as async
+interface that will consume the bytes from the `BytesReader` to yield a
+`header()` and either `blocks()` or `blocksIndex()` data.
+
+<a name="decoder__bytesReader__bytes__"></a>
+### `decoder.bytesReader(bytes)`
+
+* `bytes` `(Uint8Array)`
+
+* Returns:  `BytesReader`
+
+Creates a `BytesReader` from a `Uint8Array`.
+
+<a name="decoder__asyncIterableReader__asyncIterable__"></a>
+### `decoder.asyncIterableReader(asyncIterable)`
+
+* `asyncIterable` `(AsyncIterable<Uint8Array>)`
+
+* Returns:  `BytesReader`
+
+Creates a `BytesReader` from an `AsyncIterable<Uint8Array>`, which allows for
+consumption of CAR data from a streaming source.
+
+<a name="decoder__limitReader__reader____byteLimit__"></a>
+### `decoder.limitReader(reader, byteLimit)`
+
+* `reader` `(BytesReader)`
+* `byteLimit` `(number)`
+
+* Returns:  `BytesReader`
+
+Wraps a `BytesReader` in a limiting `BytesReader` which limits maximum read
+to `byteLimit` bytes. It _does not_ update `pos` of the original
+`BytesReader`.
+
 ## License
 
 Licensed under either of

package/api.ts

@@ -1,17 +1,21 @@
 import { CID } from 'multiformats/cid'
 
+export type { CID }
 /* Generic types for interfacing with block storage */
 
-export type Block = { cid: CID, bytes: Uint8Array }
+export type Block = {
+  cid: CID
+  bytes: Uint8Array
+}
 
 export type BlockHeader = {
-  cid: CID,
-  length: number,
+  cid: CID
+  length: number
   blockLength: number
 }
 
 export type BlockIndex = BlockHeader & {
-  offset: number,
+  offset: number
   blockOffset: number
 }
 
@@ -36,6 +40,20 @@ export interface BlockWriter {
   close(): Promise<void>
 }
 
+export interface CarBufferWriter {
+  addRoot(root:CID, options?:{ resize?: boolean }):CarBufferWriter
+  write(block: Block): CarBufferWriter
+  close(options?:{ resize?: boolean }): Uint8Array
+}
+
+export interface CarBufferWriterOptions {
+  roots?: CID[] // defaults to []
+  byteOffset?: number // defaults to 0
+  byteLength?: number // defaults to buffer.byteLength
+
+  headerSize?: number // defaults to size needed for provided roots
+}
+
 export interface WriterChannel {
   writer: BlockWriter
   out: AsyncIterable<Uint8Array>

package/buffer-writer

@@ -0,0 +1 @@
+module.exports = require('./cjs/lib/buffer-writer.js')

package/cjs/browser-test/common.js

@@ -5,7 +5,7 @@ Object.defineProperty(exports, '__esModule', { value: true });
 var multiformats = require('multiformats');
 var sha2 = require('multiformats/hashes/sha2');
 var raw = require('multiformats/codecs/raw');
-var dagCbor = require('@ipld/dag-cbor');
+var CBOR = require('@ipld/dag-cbor');
 var dagPb = require('@ipld/dag-pb');
 var chai = require('chai');
 var chaiAsPromised = require('chai-as-promised');
@@ -31,7 +31,7 @@ function _interopNamespace(e) {
 }
 
 var raw__namespace = /*#__PURE__*/_interopNamespace(raw);
-var dagCbor__namespace = /*#__PURE__*/_interopNamespace(dagCbor);
+var CBOR__namespace = /*#__PURE__*/_interopNamespace(CBOR);
 var dagPb__namespace = /*#__PURE__*/_interopNamespace(dagPb);
 var chai__default = /*#__PURE__*/_interopDefaultLegacy(chai);
 var chaiAsPromised__default = /*#__PURE__*/_interopDefaultLegacy(chaiAsPromised);
@@ -101,7 +101,7 @@ async function makeData() {
       cborBlocks.push(await toBlock({
         name: b[0],
         link: b[1]
-      }, dagCbor__namespace));
+      }, CBOR__namespace));
     }
     allBlocks = [
       [
@@ -214,12 +214,87 @@ const goCarIndex = [
     blockLength: 18
   }
 ];
+const goCarV2Bytes = multiformats.bytes.fromHex('0aa16776657273696f6e02000000000000000000000000000000003300000000000000c001000000000000f30100000000000038a265726f6f747381d82a5823001220fb16f5083412ef1371d031ed4aa239903d84efdadf1ba3cd678e6475b1a232f86776657273696f6e01511220fb16f5083412ef1371d031ed4aa239903d84efdadf1ba3cd678e6475b1a232f8122d0a221220d9c0d5376d26f1931f7ad52d7acc00fc1090d2edb0808bf61eeb0a152826f6261204f09f8da418a40185011220d9c0d5376d26f1931f7ad52d7acc00fc1090d2edb0808bf61eeb0a152826f62612310a221220d745b7757f5b4593eeab7820306c7bc64eb496a7410a0d07df7a34ffec4b97f1120962617272656c657965183a122e0a2401551220a2e1c40da1ae335d4dffe729eb4d5ca23b74b9e51fc535f4a804a261080c294d1204f09f90a11807581220d745b7757f5b4593eeab7820306c7bc64eb496a7410a0d07df7a34ffec4b97f112340a2401551220b474a99a2705e23cf905a484ec6d14ef58b56bbe62e9292783466ec363b5072d120a666973686d6f6e67657218042801551220b474a99a2705e23cf905a484ec6d14ef58b56bbe62e9292783466ec363b5072d666973682b01551220a2e1c40da1ae335d4dffe729eb4d5ca23b74b9e51fc535f4a804a261080c294d6c6f62737465720100000028000000c800000000000000a2e1c40da1ae335d4dffe729eb4d5ca23b74b9e51fc535f4a804a261080c294d9401000000000000b474a99a2705e23cf905a484ec6d14ef58b56bbe62e9292783466ec363b5072d6b01000000000000d745b7757f5b4593eeab7820306c7bc64eb496a7410a0d07df7a34ffec4b97f11201000000000000d9c0d5376d26f1931f7ad52d7acc00fc1090d2edb0808bf61eeb0a152826f6268b00000000000000fb16f5083412ef1371d031ed4aa239903d84efdadf1ba3cd678e6475b1a232f83900000000000000');
+const goCarV2Roots = [multiformats.CID.parse('QmfEoLyB5NndqeKieExd1rtJzTduQUPEV8TwAYcUiy3H5Z')];
+const goCarV2Index = [
+  {
+    blockLength: 47,
+    blockOffset: 143,
+    cid: multiformats.CID.parse('QmfEoLyB5NndqeKieExd1rtJzTduQUPEV8TwAYcUiy3H5Z'),
+    length: 82,
+    offset: 108
+  },
+  {
+    blockLength: 99,
+    blockOffset: 226,
+    cid: multiformats.CID.parse('QmczfirA7VEH7YVvKPTPoU69XM3qY4DC39nnTsWd4K3SkM'),
+    length: 135,
+    offset: 190
+  },
+  {
+    blockLength: 54,
+    blockOffset: 360,
+    cid: multiformats.CID.parse('Qmcpz2FHJD7VAhg1fxFXdYJKePtkx1BsHuCrAgWVnaHMTE'),
+    length: 89,
+    offset: 325
+  },
+  {
+    blockLength: 4,
+    blockOffset: 451,
+    cid: multiformats.CID.parse('bafkreifuosuzujyf4i6psbneqtwg2fhplc2wxptc5euspa2gn3bwhnihfu'),
+    length: 41,
+    offset: 414
+  },
+  {
+    blockLength: 7,
+    blockOffset: 492,
+    cid: multiformats.CID.parse('bafkreifc4hca3inognou377hfhvu2xfchn2ltzi7yu27jkaeujqqqdbjju'),
+    length: 44,
+    offset: 455
+  }
+];
+const goCarV2Contents = {
+  QmfEoLyB5NndqeKieExd1rtJzTduQUPEV8TwAYcUiy3H5Z: {
+    Links: [{
+        Hash: multiformats.CID.parse('QmczfirA7VEH7YVvKPTPoU69XM3qY4DC39nnTsWd4K3SkM'),
+        Name: '\uD83C\uDF64',
+        Tsize: 164
+      }]
+  },
+  QmczfirA7VEH7YVvKPTPoU69XM3qY4DC39nnTsWd4K3SkM: {
+    Links: [
+      {
+        Hash: multiformats.CID.parse('Qmcpz2FHJD7VAhg1fxFXdYJKePtkx1BsHuCrAgWVnaHMTE'),
+        Name: 'barreleye',
+        Tsize: 58
+      },
+      {
+        Hash: multiformats.CID.parse('bafkreifc4hca3inognou377hfhvu2xfchn2ltzi7yu27jkaeujqqqdbjju'),
+        Name: '\uD83D\uDC21',
+        Tsize: 7
+      }
+    ]
+  },
+  Qmcpz2FHJD7VAhg1fxFXdYJKePtkx1BsHuCrAgWVnaHMTE: {
+    Links: [{
+        Hash: multiformats.CID.parse('bafkreifuosuzujyf4i6psbneqtwg2fhplc2wxptc5euspa2gn3bwhnihfu'),
+        Name: 'fishmonger',
+        Tsize: 4
+      }]
+  },
+  bafkreifuosuzujyf4i6psbneqtwg2fhplc2wxptc5euspa2gn3bwhnihfu: 'fish',
+  bafkreifc4hca3inognou377hfhvu2xfchn2ltzi7yu27jkaeujqqqdbjju: 'lobster'
+};
 
 exports.assert = assert;
 exports.carBytes = carBytes;
 exports.goCarBytes = goCarBytes;
 exports.goCarIndex = goCarIndex;
 exports.goCarRoots = goCarRoots;
+exports.goCarV2Bytes = goCarV2Bytes;
+exports.goCarV2Contents = goCarV2Contents;
+exports.goCarV2Index = goCarV2Index;
+exports.goCarV2Roots = goCarV2Roots;
 exports.makeData = makeData;
 exports.makeIterable = makeIterable;
 exports.rndCid = rndCid;

package/cjs/browser-test/node-test-large.js

@@ -4,7 +4,7 @@ var fs = require('fs');
 var stream = require('stream');
 var ipldGarbage = require('ipld-garbage');
 var varint = require('varint');
-var dagCbor = require('@ipld/dag-cbor');
+var CBOR = require('@ipld/dag-cbor');
 var sha2 = require('multiformats/hashes/sha2');
 var cid = require('multiformats/cid');
 require('../car-browser.js');
@@ -36,7 +36,7 @@ function _interopNamespace(e) {
 
 var fs__default = /*#__PURE__*/_interopDefaultLegacy(fs);
 var varint__default = /*#__PURE__*/_interopDefaultLegacy(varint);
-var dagCbor__namespace = /*#__PURE__*/_interopNamespace(dagCbor);
+var CBOR__namespace = /*#__PURE__*/_interopNamespace(CBOR);
 
 describe('Large CAR', () => {
   const objects = [];
@@ -45,7 +45,7 @@ describe('Large CAR', () => {
   it('create, no roots', async () => {
     const {writer, out} = writerBrowser.CarWriter.create([]);
     stream.Readable.from(out).pipe(fs__default["default"].createWriteStream('./test.car'));
-    let offset = dagCbor__namespace.encode({
+    let offset = CBOR__namespace.encode({
       version: 1,
       roots: []
     }).length;
@@ -53,9 +53,9 @@ describe('Large CAR', () => {
     for (let i = 0; i < 500; i++) {
       const obj = ipldGarbage.garbage(1000);
       objects.push(obj);
-      const bytes = dagCbor__namespace.encode(obj);
+      const bytes = CBOR__namespace.encode(obj);
       const hash = await sha2.sha256.digest(bytes);
-      const cid$1 = cid.CID.create(1, dagCbor__namespace.code, hash);
+      const cid$1 = cid.CID.create(1, CBOR__namespace.code, hash);
       cids.push(cid$1.toString());
       const blockLength = bytes.length;
       let length = cid$1.bytes.length + blockLength;
@@ -101,7 +101,7 @@ describe('Large CAR', () => {
     let i = 0;
     for await (const {cid, bytes} of reader.blocks()) {
       common.assert.strictEqual(cid.toString(), cids[i], `cid #${ i } ${ cid } <> ${ cids[i] }`);
-      const obj = dagCbor__namespace.decode(bytes);
+      const obj = CBOR__namespace.decode(bytes);
       common.assert.deepStrictEqual(obj, objects[i], `object #${ i }`);
       i++;
     }
@@ -112,7 +112,7 @@ describe('Large CAR', () => {
     let i = 0;
     for await (const {cid, bytes} of reader.blocks()) {
       common.assert.strictEqual(cid.toString(), cids[i], `cid #${ i } ${ cid } <> ${ cids[i] }`);
-      const obj = dagCbor__namespace.decode(bytes);
+      const obj = CBOR__namespace.decode(bytes);
       common.assert.deepStrictEqual(obj, objects[i], `object #${ i }`);
       i++;
     }
@@ -123,7 +123,7 @@ describe('Large CAR', () => {
     let i = 0;
     for await (const {cid, bytes} of reader.blocks()) {
       common.assert.strictEqual(cid.toString(), cids[i], `cid #${ i } ${ cid } <> ${ cids[i] }`);
-      const obj = dagCbor__namespace.decode(bytes);
+      const obj = CBOR__namespace.decode(bytes);
       common.assert.deepStrictEqual(obj, objects[i], `object #${ i }`);
       i++;
     }