hypercore 10.0.0-alpha.9 → 10.2.0

package/README.md

@@ -1,21 +1,24 @@
-# Hypercore 10
+# Hypercore
 
-NOTE: This is the _ALPHA_ version of the upcoming [Hypercore](https://github.com/hypercore-protocol/hypercore) 10 protocol upgrade.
+Hypercore is a secure, distributed append-only log.
 
-Features all the power of Hypercore combined with
+Built for sharing large datasets and streams of real time data
 
-* Multiwriter support
-* Fork recovery
-* Promises
-* Simplications and performance/scaling improvements
-* Internal oplog design
+## Features
 
-## Install
+* **Sparse replication.** Only download the data you are interested in.
+* **Realtime.** Get the latest updates to the log fast and securely.
+* **Performant.** Uses a simple flat file structure to maximize I/O performance.
+* **Secure.** Uses signed merkle trees to verify log integrity in real time.
+* **Modular.** Hypercore aims to do one thing and one thing well - distributing a stream of data.
+
+Note that the latest release is Hypercore 10, which adds support for truncate and many other things.
+Version 10 is not compatible with earlier versions (9 and earlier), but is considered LTS, meaning the storage format and wire protocol is forward compatible with future versions.
 
-Install from NPM using the next tag
+## Install
 
 ```sh
-npm install hypercore@next
+npm install hypercore
 ```
 
 ## API
@@ -33,13 +36,13 @@ const core = new Hypercore('./directory') // store data in ./directory
 Alternatively you can pass a function instead that is called with every filename Hypercore needs to function and return your own [abstract-random-access](https://github.com/random-access-storage/abstract-random-access) instance that is used to store the data.
 
 ``` js
-const ram = require('random-access-memory')
+const RAM = require('random-access-memory')
 const core = new Hypercore((filename) => {
   // filename will be one of: data, bitfield, tree, signatures, key, secret_key
   // the data file will contain all your data concatenated.
 
   // just store all files in ram by returning a random-access-memory instance
-  return ram()
+  return new RAM()
 })
 ```
 
@@ -62,35 +65,126 @@ Note that `tree`, `data`, and `bitfield` are normally heavily sparse files.
 {
   createIfMissing: true, // create a new Hypercore key pair if none was present in storage
   overwrite: false, // overwrite any old Hypercore that might already exist
+  sparse: true, // enable sparse mode, counting unavailable blocks towards core.length and core.byteLength
   valueEncoding: 'json' | 'utf-8' | 'binary', // defaults to binary
+  encodeBatch: batch => { ... }, // optionally apply an encoding to complete batches
   keyPair: kp, // optionally pass the public key and secret key as a key pair
-  encryptionKey: k // optionally pass an encryption key to enable block encryption
+  encryptionKey: k, // optionally pass an encryption key to enable block encryption
+  onwait: () => {} // hook that is called if gets are waiting for download
 }
 ```
 
 You can also set valueEncoding to any [abstract-encoding](https://github.com/mafintosh/abstract-encoding) or [compact-encoding](https://github.com/compact-encoding) instance.
 
-#### `const seq = await core.append(block)`
+valueEncodings will be applied to individually blocks, even if you append batches. If you want to control encoding at the batch-level, you can use the `encodeBatch` option, which is a function that takes a batch and returns a binary-encoded batch. If you provide a custom valueEncoding, it will not be applied prior to `encodeBatch`.
+
+#### `const { length, byteLength } = await core.append(block)`
 
 Append a block of data (or an array of blocks) to the core.
-Returns the seq the first block was stored at.
+Returns the new length and byte length of the core.
+
+``` js
+// simple call append with a new block of data
+await core.append(Buffer.from('I am a block of data'))
+
+// pass an array to append multiple blocks as a batch
+await core.append([Buffer.from('batch block 1'), Buffer.from('batch block 2')])
+```
 
 #### `const block = await core.get(index, [options])`
 
 Get a block of data.
 If the data is not available locally this method will prioritize and wait for the data to be downloaded.
 
-Options include
+``` js
+// get block #42
+const block = await core.get(42)
+
+// get block #43, but only wait 5s
+const blockIfFast = await core.get(43, { timeout: 5000 })
+
+// get block #44, but only if we have it locally
+const blockLocal = await core.get(44, { wait: false })
+```
+
+Additional options include
 
 ``` js
 {
-  wait: true, // wait for index to be downloaded
+  wait: true, // wait for block to be downloaded
   onwait: () => {}, // hook that is called if the get is waiting for download
   timeout: 0, // wait at max some milliseconds (0 means no timeout)
   valueEncoding: 'json' | 'utf-8' | 'binary' // defaults to the core's valueEncoding
 }
 ```
 
+#### `const updated = await core.update()`
+
+Wait for the core to try and find a signed update to it's length.
+Does not download any data from peers except for a proof of the new core length.
+
+``` js
+const updated = await core.update()
+
+console.log('core was updated?', updated, 'length is', core.length)
+```
+
+#### `const [index, relativeOffset] = await core.seek(byteOffset)`
+
+Seek to a byte offset.
+
+Returns `[index, relativeOffset]`, where `index` is the data block the byteOffset is contained in and `relativeOffset` is
+the relative byte offset in the data block.
+
+``` js
+await core.append([Buffer.from('abc'), Buffer.from('d'), Buffer.from('efg')])
+
+const first = await core.seek(1) // returns [0, 1]
+const second = await core.seek(3) // returns [1, 0]
+const third = await core.seek(5) // returns [2, 1]
+```
+
+#### `const stream = core.createReadStream([options])`
+
+Make a read stream to read a range of data out at once.
+
+``` js
+// read the full core
+const fullStream = core.createReadStream()
+
+// read from block 10-15
+const partialStream = core.createReadStream({ start: 10, end: 15 })
+
+// pipe the stream somewhere using the .pipe method on Node.js or consume it as
+// an async iterator
+
+for await (const data of fullStream) {
+  console.log('data:', data)
+}
+```
+
+Additional options include:
+
+``` js
+{
+  start: 0,
+  end: core.length,
+  live: false,
+  snapshot: true // auto set end to core.length on open or update it on every read
+}
+```
+
+#### `await core.clear(start, [end])`
+
+Clear stored blocks between `start` and `end`, reclaiming storage when possible.
+
+``` js
+await core.clear(4) // clear block 4 from your local cache
+await core.clear(0, 10) // clear block 0-10 from your local cache
+```
+
+The core will also gossip to peers it is connected to, that is no longer has these blocks.
+
 #### `await core.truncate(newLength, [forkId])`
 
 Truncate the core to a smaller length.
@@ -98,6 +192,10 @@ Truncate the core to a smaller length.
 Per default this will update the fork id of the core to `+ 1`, but you can set the fork id you prefer with the option.
 Note that the fork id should be monotonely incrementing.
 
+#### `const hash = await core.treeHash([length])`
+
+Get the Merkle Tree hash of the core at a given length, defaulting to the current length of the core.
+
 #### `const range = core.download([range])`
 
 Download a range of data.
@@ -105,7 +203,7 @@ Download a range of data.
 You can await when the range has been fully downloaded by doing:
 
 ```js
-await range.downloaded()
+await range.done()
 ```
 
 A range can have the following properties:
@@ -130,7 +228,7 @@ core.download({ start: 0, end: -1 })
 To downloaded a discrete range of blocks pass a list of indices.
 
 ```js
-core.download({ blocks: [4, 9, 7] });
+core.download({ blocks: [4, 9, 7] })
 ```
 
 To cancel downloading a range simply destroy the range instance.
@@ -140,21 +238,22 @@ To cancel downloading a range simply destroy the range instance.
 range.destroy()
 ```
 
-#### `const [index, relativeOffset] = await core.seek(byteOffset)`
-
-Seek to a byte offset.
-
-Returns `(index, relativeOffset)`, where `index` is the data block the byteOffset is contained in and `relativeOffset` is
-the relative byte offset in the data block.
+#### `const info = await core.info()`
 
-#### `const updated = await core.update()`
+Get information about this core, such as its total size in bytes.
 
-Wait for the core to try and find a signed update to it's length.
-Does not download any data from peers except for a proof of the new core length.
+The object will look like this:
 
-``` js
-const updated = await core.update()
-console.log('core was updated?', updated, 'length is', core.length)
+```js
+Info {
+  key: Buffer(...),
+  discoveryKey: Buffer(...),
+  length: 18,
+  contiguousLength: 16,
+  byteLength: 742,
+  fork: 0,
+  padding: 8
+}
 ```
 
 #### `await core.close()`
@@ -190,12 +289,24 @@ Can we read from this core? After closing the core this will be false.
 
 Populated after `ready` has been emitted. Will be `false` before the event.
 
+#### `core.id`
+
+String containing the id (z-base-32 of the public key) identifying this core.
+
+Populated after `ready` has been emitted. Will be `null` before the event.
+
 #### `core.key`
 
 Buffer containing the public key identifying this core.
 
 Populated after `ready` has been emitted. Will be `null` before the event.
 
+#### `core.keyPair`
+
+Object containing buffers of the core's public and secret key
+
+Populated after `ready` has been emitted. Will be `null` before the event.
+
 #### `core.discoveryKey`
 
 Buffer containing a key derived from the core's public key.
@@ -209,13 +320,13 @@ Buffer containing the optional block encryption key of this core. Will be `null`
 
 #### `core.length`
 
-How many blocks of data are available on this core?
+How many blocks of data are available on this core? If `sparse: false`, this will equal `core.contiguousLength`.
 
 Populated after `ready` has been emitted. Will be `0` before the event.
 
-#### `core.byteLength`
+#### `core.contiguousLength`
 
-How much data is available on this core in bytes?
+How many blocks are contiguously available starting from the first block of this core?
 
 Populated after `ready` has been emitted. Will be `0` before the event.
 
@@ -254,10 +365,17 @@ const socket = net.connect(...)
 socket.pipe(localCore.replicate(true)).pipe(socket)
 ```
 
+#### `const done = core.findingPeers()`
+
+Create a hook that tells Hypercore you are finding peers for this core in the background. Call `done` when your current discovery iteration is done.
+If you're using Hyperswarm, you'd normally call this after a `swarm.flush()` finishes.
+
+This allows `core.update` to wait for either the `findingPeers` hook to finish or one peer to appear before deciding whether it should wait for a merkle tree update before returning.
+
 #### `core.on('append')`
 
 Emitted when the core has been appended to (i.e. has a new length / byteLength), either locally or remotely.
 
-#### `core.on('truncate')`
+#### `core.on('truncate', ancestors, forkId)`
 
 Emitted when the core has been truncated, either locally or remotely.