@libp2p/peer-store 0.0.0

package/src/README.md

@@ -0,0 +1,145 @@
+# PeerStore
+
+Libp2p's PeerStore is responsible for keeping an updated register with the relevant information of the known peers. It should be the single source of truth for all peer data, where a subsystem can learn about peers' data and where someone can listen for updates. The PeerStore comprises four main components: `addressBook`, `keyBook`, `protocolBook` and `metadataBook`.
+
+The PeerStore manages the high level operations on its inner books. Moreover, the PeerStore should be responsible for notifying interested parties of relevant events, through its Event Emitter.
+
+## Submitting records to the PeerStore
+
+Several libp2p subsystems will perform operations that might gather relevant information about peers.
+
+### Identify
+- The Identify protocol automatically runs on every connection when multiplexing is enabled. The protocol will put the multiaddrs and protocols provided by the peer to the PeerStore.
+- In the background, the Identify Service is also waiting for protocol change notifications of peers via the IdentifyPush protocol. Peers may leverage the `identify-push` message to communicate protocol changes to all connected peers, so that their PeerStore can be updated with the updated protocols.
+- While it is currently not supported in js-libp2p, future iterations may also support the [IdentifyDelta protocol](https://github.com/libp2p/specs/pull/176).
+- Taking into account that the Identify protocol records are directly from the peer, they should be considered the source of truth and weighted accordingly.
+
+### Peer Discovery
+- Libp2p discovery protocols aim to discover new peers in the network. In a typical discovery protocol, addresses of the peer are discovered along with its peer id. Once this happens, a libp2p discovery protocol should emit a `peer` event with the information of the discovered peer and this information will be added to the PeerStore by libp2p.
+
+### Dialer
+- Libp2p API supports dialing a peer given a `multiaddr`, and no prior knowledge of the peer. If the node is able to establish a connection with the peer, it and its multiaddr is added to the PeerStore.
+- When a connection is being upgraded, more precisely after its encryption, or even in a discovery protocol, a libp2p node can get to know other parties public keys. In this scenario, libp2p will add the peer's public key to its `KeyBook`.
+
+### DHT
+- On some DHT operations, such as finding providers for a given CID, nodes may exchange peer data as part of the query. This passive peer discovery should result in the DHT emitting the `peer` event in the same way [Peer Discovery](#peerdiscovery) does.
+
+## Retrieving records from the PeerStore
+
+When data in the PeerStore is updated the PeerStore will emit events based on the changes, to allow applications and other subsystems to take action on those changes. Any subsystem interested in these notifications should subscribe the [`PeerStore events`][peer-store-events].
+
+### Peer
+- Each time a new peer is discovered, the PeerStore should emit a [`peer` event][peer-store-events], so that interested parties can leverage this peer and establish a connection with it.
+
+### Protocols
+- When the known protocols of a peer change, the PeerStore emits a [`change:protocols` event][peer-store-events].
+
+### Multiaddrs
+- When the known listening `multiaddrs` of a peer change, the PeerStore emits a [`change:multiaddrs` event][peer-store-events].
+
+## PeerStore implementation
+
+The PeerStore wraps four main components: `addressBook`, `keyBook`, `protocolBook` and `metadataBook`. Moreover, it provides a high level API for those components, as well as data events.
+
+### Components
+
+#### Address Book
+
+The `addressBook` keeps the known multiaddrs of a peer. The multiaddrs of each peer may change over time and the Address Book must account for this.
+
+`Map<string, Address>`
+
+A `peerId.toString(base58btc)` identifier mapping to a `Address` object, which should have the following structure:
+
+```js
+{
+  multiaddr: <Multiaddr>
+}
+```
+
+#### Key Book
+
+The `keyBook` tracks the public keys of the peers by keeping their [`PeerId`][peer-id].
+
+`Map<string, PeerId`
+
+A `peerId.toString(base58btc)` identifier mapping to a `PeerId` of the peer. This instance contains the peer public key.
+
+#### Protocol Book
+
+The `protoBook` holds the identifiers of the protocols supported by each peer. The protocols supported by each peer are dynamic and will change over time.
+
+`Map<string, Set<string>>`
+
+A `peerId.toString(base58btc)` identifier mapping to a `Set` of protocol identifier strings.
+
+#### Metadata Book
+
+The `metadataBook` keeps track of the known metadata of a peer. Its metadata is stored in a key value fashion, where a key identifier (`string`) represents a metadata value (`Uint8Array`).
+
+`Map<string, Map<string, Uint8Array>>`
+
+A `peerId.toString(base58btc)` identifier mapping to the peer metadata Map.
+
+### API
+
+For the complete API documentation, you should check the [API.md](../../doc/API.md).
+
+Access to its underlying books:
+
+- `peerStore.addressBook.*`
+- `peerStore.keyBook.*`
+- `peerStore.metadataBook.*`
+- `peerStore.protoBook.*`
+
+### Events
+
+- `peer` - emitted when a new peer is added.
+- `change:multiaadrs` - emitted when a known peer has a different set of multiaddrs.
+- `change:protocols` - emitted when a known peer supports a different set of protocols.
+- `change:pubkey` - emitted when a peer's public key is known.
+- `change:metadata` - emitted when known metadata of a peer changes.
+
+## Data Persistence
+
+The data stored in the PeerStore can be persisted if configured appropriately. Keeping a record of the peers already discovered by the peer, as well as their known data aims to improve the efficiency of peers joining the network after being offline.
+
+The libp2p node will need to receive a [datastore](https://github.com/ipfs/interface-datastore), in order to persist this data across restarts. A [datastore](https://github.com/ipfs/interface-datastore) stores its data in a key-value fashion. As a result, we need coherent keys so that we do not overwrite data.
+
+The PeerStore should not continuously update the datastore whenever data is changed. Instead, it should only store new data after reaching a certain threshold of "dirty" peers, as well as when the node is stopped, in order to batch writes to the datastore.
+
+The peer id will be appended to the datastore key for each data namespace. The namespaces were defined as follows:
+
+**AddressBook**
+
+All the known peer addresses are stored with a key pattern as follows:
+
+`/peers/addrs/<b32 peer id no padding>`
+
+**ProtoBook**
+
+All the known peer protocols are stored with a key pattern as follows:
+
+`/peers/protos/<b32 peer id no padding>`
+
+**KeyBook**
+
+All public keys are stored under the following pattern:
+
+` /peers/keys/<b32 peer id no padding>`
+
+**MetadataBook**
+
+Metadata is stored under the following key pattern:
+
+`/peers/metadata/<b32 peer id no padding>/<key>`
+
+## Future Considerations
+
+- If multiaddr TTLs are added, the PeerStore may schedule jobs to delete all addresses that exceed the TTL to prevent AddressBook bloating
+- Further API methods will probably need to be added in the context of multiaddr validity and confidence.
+- When improving libp2p configuration for specific runtimes, we should take into account the PeerStore recommended datastore.
+- When improving libp2p configuration, we should think about a possible way of allowing the configuration of Bootstrap to be influenced by the persisted peers, as a way to decrease the load on Bootstrap nodes.
+
+[peer-id]: https://github.com/libp2p/js-peer-id
+[peer-store-events]: ../../doc/API.md#libp2ppeerstore

package/src/address-book.ts

@@ -0,0 +1,330 @@
+import { logger } from '@libp2p/logger'
+import errcode from 'err-code'
+import { Multiaddr } from '@multiformats/multiaddr'
+import { codes } from './errors.js'
+import { PeerRecord, RecordEnvelope } from '@libp2p/peer-record'
+import { pipe } from 'it-pipe'
+import all from 'it-all'
+import filter from 'it-filter'
+import map from 'it-map'
+import each from 'it-foreach'
+import { base58btc } from 'multiformats/bases/base58'
+import { PeerId } from '@libp2p/peer-id'
+import type { PeerStore } from '@libp2p/interfaces/peer-store'
+import type { Store } from './store.js'
+import type { AddressFilter, AddressSorter } from './index.js'
+import type { Envelope } from '@libp2p/interfaces/record'
+
+const log = logger('libp2p:peer-store:address-book')
+const EVENT_NAME = 'change:multiaddrs'
+
+async function allowAll () {
+  return true
+}
+
+export class PeerStoreAddressBook {
+  private emit: PeerStore["emit"]
+  private store: Store
+  private addressFilter: AddressFilter
+
+  constructor (emit: PeerStore["emit"], store: Store, addressFilter?: AddressFilter) {
+    this.emit = emit
+    this.store = store
+    this.addressFilter = addressFilter ?? allowAll
+  }
+
+  /**
+   * ConsumePeerRecord adds addresses from a signed peer record contained in a record envelope.
+   * This will return a boolean that indicates if the record was successfully processed and added
+   * into the AddressBook.
+   */
+  async consumePeerRecord (envelope: Envelope) {
+    log('consumePeerRecord await write lock')
+    const release = await this.store.lock.writeLock()
+    log('consumePeerRecord got write lock')
+
+    let peerId
+    let updatedPeer
+
+    try {
+      let peerRecord
+      try {
+        peerRecord = PeerRecord.createFromProtobuf(envelope.payload)
+      } catch (err: any) {
+        log.error('invalid peer record received')
+        return false
+      }
+
+      peerId = peerRecord.peerId
+      const multiaddrs = peerRecord.multiaddrs
+
+      // Verify peerId
+      if (!peerId.equals(envelope.peerId)) {
+        log('signing key does not match PeerId in the PeerRecord')
+        return false
+      }
+
+      // ensure the record has multiaddrs
+      if (!multiaddrs || !multiaddrs.length) {
+        return false
+      }
+
+      if (await this.store.has(peerId)) {
+        const peer = await this.store.load(peerId)
+
+        if (peer.peerRecordEnvelope) {
+          const storedEnvelope = await RecordEnvelope.createFromProtobuf(peer.peerRecordEnvelope)
+          const storedRecord = PeerRecord.createFromProtobuf(storedEnvelope.payload)
+
+          // ensure seq is greater than, or equal to, the last received
+          if (storedRecord.seqNumber >= peerRecord.seqNumber) {
+            return false
+          }
+        }
+      }
+
+      // Replace unsigned addresses by the new ones from the record
+      // TODO: Once we have ttls for the addresses, we should merge these in
+      updatedPeer = await this.store.patchOrCreate(peerId, {
+        addresses: await filterMultiaddrs(peerId, multiaddrs, this.addressFilter, true),
+        peerRecordEnvelope: envelope.marshal()
+      })
+
+      log(`stored provided peer record for ${peerRecord.peerId.toString(base58btc)}`)
+    } finally {
+      log('consumePeerRecord release write lock')
+      release()
+    }
+
+    this.emit(EVENT_NAME, { peerId, multiaddrs: updatedPeer.addresses.map(({ multiaddr }) => multiaddr) })
+
+    return true
+  }
+
+  async getRawEnvelope (peerId: PeerId) {
+    log('getRawEnvelope await read lock')
+    const release = await this.store.lock.readLock()
+    log('getRawEnvelope got read lock')
+
+    try {
+      const peer = await this.store.load(peerId)
+
+      return peer.peerRecordEnvelope
+    } catch (err: any) {
+      if (err.code !== codes.ERR_NOT_FOUND) {
+        throw err
+      }
+    } finally {
+      log('getRawEnvelope release read lock')
+      release()
+    }
+  }
+
+  /**
+   * Get an Envelope containing a PeerRecord for the given peer.
+   * Returns undefined if no record exists.
+   */
+  async getPeerRecord (peerId: PeerId) {
+    const raw = await this.getRawEnvelope(peerId)
+
+    if (!raw) {
+      return undefined
+    }
+
+    return RecordEnvelope.createFromProtobuf(raw)
+  }
+
+  async get (peerId: PeerId) {
+    peerId = PeerId.fromPeerId(peerId)
+
+    log('get wait for read lock')
+    const release = await this.store.lock.readLock()
+    log('get got read lock')
+
+    try {
+      const peer = await this.store.load(peerId)
+
+      return peer.addresses
+    } catch (err: any) {
+      if (err.code !== codes.ERR_NOT_FOUND) {
+        throw err
+      }
+    } finally {
+      log('get release read lock')
+      release()
+    }
+
+    return []
+  }
+
+  async set (peerId: PeerId, multiaddrs: Multiaddr[]) {
+    peerId = PeerId.fromPeerId(peerId)
+
+    if (!Array.isArray(multiaddrs)) {
+      log.error('multiaddrs must be an array of Multiaddrs')
+      throw errcode(new Error('multiaddrs must be an array of Multiaddrs'), codes.ERR_INVALID_PARAMETERS)
+    }
+
+    log('set await write lock')
+    const release = await this.store.lock.writeLock()
+    log('set got write lock')
+
+    let hasPeer = false
+    let updatedPeer
+
+    try {
+      const addresses = await filterMultiaddrs(peerId, multiaddrs, this.addressFilter)
+
+      // No valid addresses found
+      if (!addresses.length) {
+        return
+      }
+
+      try {
+        const peer = await this.store.load(peerId)
+        hasPeer = true
+
+        if (new Set([
+          ...addresses.map(({ multiaddr }) => multiaddr.toString()),
+          ...peer.addresses.map(({ multiaddr }) => multiaddr.toString())
+        ]).size === peer.addresses.length && addresses.length === peer.addresses.length) {
+          // not changing anything, no need to update
+          return
+        }
+      } catch (err: any) {
+        if (err.code !== codes.ERR_NOT_FOUND) {
+          throw err
+        }
+      }
+
+      updatedPeer = await this.store.patchOrCreate(peerId, { addresses })
+
+      log(`set multiaddrs for ${peerId.toString(base58btc)}`)
+    } finally {
+      log('set release write lock')
+      release()
+    }
+
+    this.emit(EVENT_NAME, { peerId, multiaddrs: updatedPeer.addresses.map(addr => addr.multiaddr) })
+
+    // Notify the existence of a new peer
+    if (!hasPeer) {
+      this.emit('peer', peerId)
+    }
+  }
+
+  async add (peerId: PeerId, multiaddrs: Multiaddr[]) {
+    peerId = PeerId.fromPeerId(peerId)
+
+    if (!Array.isArray(multiaddrs)) {
+      log.error('multiaddrs must be an array of Multiaddrs')
+      throw errcode(new Error('multiaddrs must be an array of Multiaddrs'), codes.ERR_INVALID_PARAMETERS)
+    }
+
+    log('add await write lock')
+    const release = await this.store.lock.writeLock()
+    log('add got write lock')
+
+    let hasPeer
+    let updatedPeer
+
+    try {
+      const addresses = await filterMultiaddrs(peerId, multiaddrs, this.addressFilter)
+
+      // No valid addresses found
+      if (!addresses.length) {
+        return
+      }
+
+      try {
+        const peer = await this.store.load(peerId)
+        hasPeer = true
+
+        if (new Set([
+          ...addresses.map(({ multiaddr }) => multiaddr.toString()),
+          ...peer.addresses.map(({ multiaddr }) => multiaddr.toString())
+        ]).size === peer.addresses.length) {
+          return
+        }
+      } catch (err: any) {
+        if (err.code !== codes.ERR_NOT_FOUND) {
+          throw err
+        }
+      }
+
+      updatedPeer = await this.store.mergeOrCreate(peerId, { addresses })
+
+      log(`added multiaddrs for ${peerId.toString(base58btc)}`)
+    } finally {
+      log('set release write lock')
+      release()
+    }
+
+    this.emit(EVENT_NAME, { peerId, multiaddrs: updatedPeer.addresses.map(addr => addr.multiaddr) })
+
+    // Notify the existence of a new peer
+    if (!hasPeer) {
+      this.emit('peer', peerId)
+    }
+  }
+
+  async delete (peerId: PeerId) {
+    peerId = PeerId.fromPeerId(peerId)
+
+    log('delete await write lock')
+    const release = await this.store.lock.writeLock()
+    log('delete got write lock')
+
+    let has
+
+    try {
+      has = await this.store.has(peerId)
+
+      await this.store.patchOrCreate(peerId, {
+        addresses: []
+      })
+    } finally {
+      log('delete release write lock')
+      release()
+    }
+
+    if (has) {
+      this.emit(EVENT_NAME, { peerId, multiaddrs: [] })
+    }
+  }
+
+  async getMultiaddrsForPeer (peerId: PeerId, addressSorter: AddressSorter = (mas) => mas) {
+    const addresses = await this.get(peerId)
+
+    return addressSorter(
+      addresses
+    ).map((address) => {
+      const multiaddr = address.multiaddr
+
+      const idString = multiaddr.getPeerId()
+      if (idString && idString === peerId.toString()) return multiaddr
+
+      return multiaddr.encapsulate(`/p2p/${peerId.toString(base58btc)}`)
+    })
+  }
+}
+
+function filterMultiaddrs (peerId: PeerId, multiaddrs: Multiaddr[], addressFilter: AddressFilter, isCertified: boolean = false) {
+  return pipe(
+    multiaddrs,
+    (source) => each(source, (multiaddr) => {
+      if (!Multiaddr.isMultiaddr(multiaddr)) {
+        log.error('multiaddr must be an instance of Multiaddr')
+        throw errcode(new Error('multiaddr must be an instance of Multiaddr'), codes.ERR_INVALID_PARAMETERS)
+      }
+    }),
+    (source) => filter(source, (multiaddr) => addressFilter(peerId, multiaddr)),
+    (source) => map(source, (multiaddr) => {
+      return {
+        multiaddr: new Multiaddr(multiaddr.toString()),
+        isCertified
+      }
+    }),
+    (source) => all(source)
+  )
+}

package/src/errors.ts

@@ -0,0 +1,5 @@
+
+export const codes = {
+  ERR_INVALID_PARAMETERS: 'ERR_INVALID_PARAMETERS',
+  ERR_NOT_FOUND: 'ERR_NOT_FOUND'
+}

package/src/index.ts

@@ -0,0 +1,123 @@
+import { logger } from '@libp2p/logger'
+import { EventEmitter } from 'events'
+import { PeerStoreAddressBook } from './address-book.js'
+import { PeerStoreKeyBook } from './key-book.js'
+import { PeerStoreMetadataBook } from './metadata-book.js'
+import { PeerStoreProtoBook } from './proto-book.js'
+import { PersistentStore, Store } from './store.js'
+import type { PeerStore, Address, AddressBook, KeyBook, MetadataBook, ProtoBook } from '@libp2p/interfaces/peer-store'
+import type { PeerId } from '@libp2p/interfaces/peer-id'
+import type { Multiaddr } from '@multiformats/multiaddr'
+import type { Datastore } from 'interface-datastore'
+import { base58btc } from 'multiformats/bases/base58'
+
+const log = logger('libp2p:peer-store')
+
+export interface AddressFilter {
+  (peerId: PeerId, multiaddr: Multiaddr): Promise<boolean>
+}
+
+export interface AddressSorter {
+  (addresses: Address[]): Address[]
+}
+
+export interface PeerStoreOptions {
+  peerId: PeerId
+  datastore: Datastore
+  addressFilter?: AddressFilter
+}
+
+/**
+ * An implementation of PeerStore that stores data in a Datastore
+ */
+export class DefaultPeerStore extends EventEmitter implements PeerStore {
+  public addressBook: AddressBook
+  public keyBook: KeyBook
+  public metadataBook: MetadataBook
+  public protoBook: ProtoBook
+
+  private peerId: PeerId
+  private store: Store
+
+  constructor (options: PeerStoreOptions) {
+    super()
+
+    const { peerId, datastore, addressFilter } = options
+
+    this.peerId = peerId
+    this.store = new PersistentStore(datastore)
+
+    this.addressBook = new PeerStoreAddressBook(this.emit.bind(this), this.store, addressFilter)
+    this.keyBook = new PeerStoreKeyBook(this.emit.bind(this), this.store)
+    this.metadataBook = new PeerStoreMetadataBook(this.emit.bind(this), this.store)
+    this.protoBook = new PeerStoreProtoBook(this.emit.bind(this), this.store)
+  }
+
+  async * getPeers () {
+    log('getPeers await read lock')
+    const release = await this.store.lock.readLock()
+    log('getPeers got read lock')
+
+    try {
+      for await (const peer of this.store.all()) {
+        if (peer.id.toString(base58btc) === this.peerId.toString(base58btc)) {
+          // Remove self peer if present
+          continue
+        }
+
+        yield peer
+      }
+    } finally {
+      log('getPeers release read lock')
+      release()
+    }
+  }
+
+  /**
+   * Delete the information of the given peer in every book
+   */
+  async delete (peerId: PeerId) {
+    log('delete await write lock')
+    const release = await this.store.lock.writeLock()
+    log('delete got write lock')
+
+    try {
+      await this.store.delete(peerId)
+    } finally {
+      log('delete release write lock')
+      release()
+    }
+  }
+
+  /**
+   * Get the stored information of a given peer
+   */
+  async get (peerId: PeerId) {
+    log('get await read lock')
+    const release = await this.store.lock.readLock()
+    log('get got read lock')
+
+    try {
+      return this.store.load(peerId)
+    } finally {
+      log('get release read lock')
+      release()
+    }
+  }
+
+  /**
+   * Returns true if we have a record of the peer
+   */
+  async has (peerId: PeerId) {
+    log('has await read lock')
+    const release = await this.store.lock.readLock()
+    log('has got read lock')
+
+    try {
+      return this.store.has(peerId)
+    } finally {
+      log('has release read lock')
+      release()
+    }
+  }
+}