hypercore 10.0.0-alpha.9 → 10.0.0

package/README.md

@@ -1,21 +1,24 @@
-# Hypercore 10
+# Hypercore
 
-NOTE: This is the _ALPHA_ version of the upcoming [Hypercore](https://github.com/hypercore-protocol/hypercore) 10 protocol upgrade.
+Hypercore is a secure, distributed append-only log.
 
-Features all the power of Hypercore combined with
+Built for sharing large datasets and streams of real time data
 
-* Multiwriter support
-* Fork recovery
-* Promises
-* Simplications and performance/scaling improvements
-* Internal oplog design
+## Features
 
-## Install
+* **Sparse replication.** Only download the data you are interested in.
+* **Realtime.** Get the latest updates to the log fast and securely.
+* **Performant.** Uses a simple flat file structure to maximize I/O performance.
+* **Secure.** Uses signed merkle trees to verify log integrity in real time.
+* **Modular.** Hypercore aims to do one thing and one thing well - distributing a stream of data.
+
+Note that the latest release is Hypercore 10, which adds support for truncate and many other things.
+Version 10 is not compatible with earlier versions (9 and earlier), but is considered LTS, meaning the storage format and wire protocol is forward compatible with future versions.
 
-Install from NPM using the next tag
+## Install
 
 ```sh
-npm install hypercore@next
+npm install hypercore
 ```
 
 ## API
@@ -33,13 +36,13 @@ const core = new Hypercore('./directory') // store data in ./directory
 Alternatively you can pass a function instead that is called with every filename Hypercore needs to function and return your own [abstract-random-access](https://github.com/random-access-storage/abstract-random-access) instance that is used to store the data.
 
 ``` js
-const ram = require('random-access-memory')
+const RAM = require('random-access-memory')
 const core = new Hypercore((filename) => {
   // filename will be one of: data, bitfield, tree, signatures, key, secret_key
   // the data file will contain all your data concatenated.
 
   // just store all files in ram by returning a random-access-memory instance
-  return ram()
+  return new RAM()
 })
 ```
 
@@ -62,18 +65,23 @@ Note that `tree`, `data`, and `bitfield` are normally heavily sparse files.
 {
   createIfMissing: true, // create a new Hypercore key pair if none was present in storage
   overwrite: false, // overwrite any old Hypercore that might already exist
+  sparse: true, // enable sparse mode, counting unavailable blocks towards core.length and core.byteLength
   valueEncoding: 'json' | 'utf-8' | 'binary', // defaults to binary
+  encodeBatch: batch => { ... }, // optionally apply an encoding to complete batches
   keyPair: kp, // optionally pass the public key and secret key as a key pair
-  encryptionKey: k // optionally pass an encryption key to enable block encryption
+  encryptionKey: k, // optionally pass an encryption key to enable block encryption
+  onwait: () => {} // hook that is called if gets are waiting for download
 }
 ```
 
 You can also set valueEncoding to any [abstract-encoding](https://github.com/mafintosh/abstract-encoding) or [compact-encoding](https://github.com/compact-encoding) instance.
 
-#### `const seq = await core.append(block)`
+valueEncodings will be applied to individually blocks, even if you append batches. If you want to control encoding at the batch-level, you can use the `encodeBatch` option, which is a function that takes a batch and returns a binary-encoded batch. If you provide a custom valueEncoding, it will not be applied prior to `encodeBatch`.
+
+#### `const { length, byteLength } = await core.append(block)`
 
 Append a block of data (or an array of blocks) to the core.
-Returns the seq the first block was stored at.
+Returns the new length and byte length of the core.
 
 #### `const block = await core.get(index, [options])`
 
@@ -84,7 +92,7 @@ Options include
 
 ``` js
 {
-  wait: true, // wait for index to be downloaded
+  wait: true, // wait for block to be downloaded
   onwait: () => {}, // hook that is called if the get is waiting for download
   timeout: 0, // wait at max some milliseconds (0 means no timeout)
   valueEncoding: 'json' | 'utf-8' | 'binary' // defaults to the core's valueEncoding
@@ -98,6 +106,23 @@ Truncate the core to a smaller length.
 Per default this will update the fork id of the core to `+ 1`, but you can set the fork id you prefer with the option.
 Note that the fork id should be monotonely incrementing.
 
+#### `const hash = await core.treeHash([length])`
+
+Get the Merkle Tree hash of the core at a given length, defaulting to the current length of the core.
+
+#### `const stream = core.createReadStream([options])`
+
+Make a read stream. Options include:
+
+``` js
+{
+  start: 0,
+  end: core.length,
+  live: false,
+  snapshot: true // auto set end to core.length on open or update it on every read
+}
+```
+
 #### `const range = core.download([range])`
 
 Download a range of data.
@@ -130,7 +155,7 @@ core.download({ start: 0, end: -1 })
 To downloaded a discrete range of blocks pass a list of indices.
 
 ```js
-core.download({ blocks: [4, 9, 7] });
+core.download({ blocks: [4, 9, 7] })
 ```
 
 To cancel downloading a range simply destroy the range instance.
@@ -157,6 +182,23 @@ const updated = await core.update()
 console.log('core was updated?', updated, 'length is', core.length)
 ```
 
+#### `const info = await core.info()`
+
+Get information about this core, such as its total size in bytes.
+
+The object will look like this:
+
+```js
+Info {
+  key: Buffer(...),
+  length: 18,
+  contiguousLength: 16,
+  byteLength: 742,
+  fork: 0,
+  padding: 8
+}
+```
+
 #### `await core.close()`
 
 Fully close this core.
@@ -196,6 +238,12 @@ Buffer containing the public key identifying this core.
 
 Populated after `ready` has been emitted. Will be `null` before the event.
 
+#### `core.keyPair`
+
+Object containing buffers of the core's public and secret key
+
+Populated after `ready` has been emitted. Will be `null` before the event.
+
 #### `core.discoveryKey`
 
 Buffer containing a key derived from the core's public key.
@@ -209,13 +257,19 @@ Buffer containing the optional block encryption key of this core. Will be `null`
 
 #### `core.length`
 
-How many blocks of data are available on this core?
+How many blocks of data are available on this core? If `sparse: false`, this will equal `core.contiguousLength`.
 
 Populated after `ready` has been emitted. Will be `0` before the event.
 
-#### `core.byteLength`
+#### `core.contiguousLength`
 
-How much data is available on this core in bytes?
+How many blocks are contiguously available starting from the first block of this core?
+
+Populated after `ready` has been emitted. Will be `0` before the event.
+
+#### `core.contiguousByteLength`
+
+How much data is contiguously available starting from the first block of this core?
 
 Populated after `ready` has been emitted. Will be `0` before the event.
 
@@ -254,10 +308,17 @@ const socket = net.connect(...)
 socket.pipe(localCore.replicate(true)).pipe(socket)
 ```
 
+#### `const done = core.findingPeers()`
+
+Create a hook that tells Hypercore you are finding peers for this core in the background. Call `done` when your current discovery iteration is done.
+If you're using Hyperswarm, you'd normally call this after a `swarm.flush()` finishes.
+
+This allows `core.update` to wait for either the `findingPeers` hook to finish or one peer to appear before deciding whether it should wait for a merkle tree update before returning.
+
 #### `core.on('append')`
 
 Emitted when the core has been appended to (i.e. has a new length / byteLength), either locally or remotely.
 
-#### `core.on('truncate')`
+#### `core.on('truncate', ancestors, forkId)`
 
 Emitted when the core has been truncated, either locally or remotely.