tool-db 2.2.0 → 2.5.0

package/README.md

@@ -1,201 +1 @@
-# Tool Db
-
-ToolDb is a peer-to-peer model for a decentralized database, inspired by Gun (Mark Nadal): https://gun.eco/
-
-
-It is the mix of a lot of cool concepts into one;
-- Cryptographically secure.
-- Offline first.
-- Fully decentralized.
-- Capable of providing realtime updates.
-- Key-value/document storage.
-- Works in the Browser and Nodejs seamlessly.
-
-For a more detailed documentation head here;
-
-[https://github.com/Manwe-777/tool-db-docs](https://github.com/Manwe-777/tool-db-docs)
-
-_(No live version just yet!, until the protocol is more finalized)_
-
-We push towards of the concept of federated servers, where we set up a p2p mesh network between any servers who want to join the swarm, and have those servers manage the connection and data sharing between them, while allowing client peers to connect to them to push updates to the database.
-This setup IS NOT a requirement! It is just what I think is the best and most reliable way of creating a p2p network for dApps using both desktop and web compatible technologies. Any peer has the capability of connecting to any other peer trough websockets, and modifiying the code to allow webRtc connections between web peers is absolutely possible.
-
-Since anyone can join a federated server swarm to help growing the network we use cryptography (basic public and private key authenthication and signature validation) to ensure all messages are coming from the real authors of the data that they intend to modify; this way by joining a swarm all peers in it can help validate without even having full data; even a new server peer can help, because all information is stored on each message and we dont rely on any centralized database to fetch users data.
-
-Please check the [chain-swarm](https://github.com/Manwe-777/chain-swarm) repository to see how a federated server swarm would look like, Tool Db only handles the connection and messaging between peers, but it does not have any logic for peer discovery.
-We do that using [discovery-channel](https://www.npmjs.com/package/discovery-channel), but you can use any DHT/networking solution you want! even WebRTC between browsers works using the `toolDbWebrtc` transport.
-
-# Future work
-
-While the database is currently functioning as expected, there are many things that could be added or improved to make it even better!
-
-- Use a common web3 format for users identity, or a more standarized key pair.
-- Allow messages encryption, add methods for Elliptic Curves, shared keys, etc.
-- Allow adding Noise/encryption to connections (probably based on the ECC?)
-
-# Install
-
-You can install ToolDb via npm or yarn;
-
-```
-npm install tool-db
-```
-
-or use it via script in your html;
-
-```
-<script src="https://unpkg.com/tool-db/bundle.js"></script>
-```
-
-That will include all of ToolDb exports in `tooldb`, then you can use it like;
-
-```
-const { ToolDb, sha256 } = tooldb;
-```
-
-## Base usage
-Connect to the selected toolDb peers;
-```
-const client = new ToolDb(options);
-```
-
-These are the options you can pass to the constructor:
-
-```
-{
-  // Database name to use (default "tooldb")
-  db: string;
-
-  // Show debug console logs (default false)
-  debug: boolean;
-
-  // Array of peers to connect to, each one in the form of { host: "127.0.0.1", port: 9000 }
-  peers: { host: "127.0.0.1", port: 9000 }[];
-
-  // Max number of tries when a connection fails (default 5)
-  maxRetries: number;
-
-  // How long to wait (max) for a debounced key listener recv (default 100)
-  triggerDebouce: number;
-
-  // How long to wait between retries (default 2000)
-  wait: number;
-
-  // If you want to force a Proof of Work on all messages, set how much (zero is no POW, default 0)
-  pow: number;
-
-  // Whether we are a server or not (default false)
-  server: boolean;
-
-  // Our hostname (server only)
-  host: string
-
-  // Port to listen incoming connections (server only, default is 8080)
-  port: number;
-
-  // A server instance like Express (server only)
-  httpServer: HTTPServer | HTTPSServer | undefined;
-
-  // Our storage namespace (default is "tooldb")
-  storageName: string;
-
-  // A custom network adapter class
-  networkAdapter: typeof ToolDbNetworkAdapter;
-
-  // A custom storage adapter function
-  storageAdapter: ToolDbStorageAdapter;
-
-  // The namespace/topic of our app (default is "tool-db-default")
-  topic: string;
-}
-```
-Notice you can use your own network and storage modules if you would like to implement custom solutions for peers discovery, connections, storage, etc, the default adapters will work both on nodejs and the browser.
-
-
-You can create a user, sign in or create a random set of keys for anonymous usage;
-```
-client.signUp(user, pass).then();
-client.signIn(user, pass).then(keys);
-client.anonSignIn().then();
-```
-
-To retrieve your address; (only if logged in)
-```
-client.getAddress();
-```
-
-You can check if you are correctly logged in by checking if the user field exists or not;
-```
-if (client.user) {
-  // Ok!
-}
-```
-
-## Putting and getting data
-
-Core methods are very straighforward:
-```
-client.getData("key", userNamespaced?, timeout?).then();
-client.putData("key", value, userNamespaced?).then();
-client.putCrdt("key", documentChanges, userNamespaced?).then();
-```
-UserNamespaced will check for the user namespace, so it will transform the key used to ":user.key", This way all peers reading this entry will understand the key belongs to this user and will enforce the verifications required to namespaced entries.
-
-You can use `client.getUserNamespacedKey(key)` to convert any key to a private namespaced key of the current logged in user.
-
-Note the third function (putCrdt); For p2p networks some times you want to have conflict resolution on certain documents, to do this we use [automerge](https://github.com/automerge/automerge). Please take a look at it to know how it works in detail! The only thing you need to know on Tool Db is that you have to send an `Automerge.BinaryChange[]` (you get it using `Automerge.change()`). The recieving peer will process the changes, compare it to its stored documents and generate a new document with JSON CRDT applied to it, then relay the final document back to you.
-
-To listen for a value changes you can set up a listener on a key. Beware the listener will check for all keys *starting with* the supplied key, so for example, if you use "value." as your listener if will execute on every key that starts with "value.". This is useful for checking against an address or namespace.
-```
-const listenerId = client.addKeyListener("value", console.log);
-client.removeKeyListener(listenerId);
-```
-
-Similarly, for using a custom verification on a key (or subset of keys) you can create a new function that returns a Promise boolean;
-```
-const validateFn = (msg) => {
-  return new Promise((resolve) => {
-    console.log("Custom verification: ", msg);
-    if (typeof msg.value === "string") resolve(true);
-    else resolve(false);
-  });
-}
-
-const validatorId = client.addCustomVerification("value", validateFn);
-
-client.removeCustomVerification(validatorId);
-```
-
-Keep in mind custom validators should run on all client and server nodes, and even though the nodes not running your validator will be able to store and relay invalid messages its up to each peer to check these messages on arrival. ToolDb does this automatically when using the custom validation, but its important to make sure every peer runs the same code to avoid tampered messages flowing in the network.
-
-## Query
-
-To make queries or just create indexes you can make a query, it simply asks all server or peers connected a list of all keys starting with a prefix. You can later use these keys to get the data itself.
-
-```
-client.queryKeys(keyPrefix, userNamespaced?, timeout?).then()
-```
-
-Just like previous methods you can configure the namespace and timeout and returns a promise, but in this case the value is always an array with the found keys.
-Using for example ":" + address as our prefix would return all keys stored for that specific user namespace; you can also create keys for very specific use cases like "post-1" and query again "post-" to get a list of all available posts.
-
-Keep in mind this can be an intensive thing to do if your indexes are too big, and not recommended to be used very frequently.
-
-
-## Listen for changes
-
-Some times you want to subscribe to the changes made on a certain key, this is possible via subscribeData;
-```
-client.subscribeData("key", userNamespaced?);
-```
-
-This will relay back to you all Put and CrdtPut messages on that key.
-
-## Events
-
-If you need to check when you are connected to a server peer or not you can use the following method replacements;
-
-```
-client.onConnect = (remotePeerId: string) => { /* Your code here */ };
-client.onDisconnect = () => { /* Your code here */ };
-```
+# `tool-db`

package/lib/adapters-base/networkAdapter.ts

@@ -0,0 +1,215 @@
+import {
+  ToolDb,
+  PingMessage,
+  ToolDbMessage,
+  Peer,
+  getPeerSignature,
+  textRandom,
+  uniq,
+} from "..";
+
+export default class ToolDbNetworkAdapter {
+  private _clientToSend: Record<string, (message: string) => void> = {};
+
+  private _isClientConnected: Record<string, () => boolean> = {};
+
+  private _clientIsServer: Record<string, boolean> = {};
+
+  private _tooldb: ToolDb;
+
+  constructor(db: ToolDb) {
+    this._tooldb = db;
+
+    if (this.tooldb.options.server) {
+      this.getMeAsPeer().then((meAsPeer) => {
+        this.tooldb.serverPeers.push(meAsPeer);
+      });
+    }
+  }
+
+  get clientToSend() {
+    return this._clientToSend;
+  }
+
+  get isClientConnected() {
+    return this._isClientConnected;
+  }
+
+  get tooldb() {
+    return this._tooldb;
+  }
+
+  public getMeAsPeer() {
+    const timestamp = new Date().getTime();
+    return getPeerSignature(
+      this.tooldb.peerAccount,
+      this.tooldb.options.topic,
+      timestamp,
+      this.tooldb.options.host,
+      this.tooldb.options.port
+    ).then((signature) => {
+      return {
+        topic: this.tooldb.options.topic,
+        timestamp: timestamp,
+        host: this.tooldb.options.host,
+        port: this.tooldb.options.port,
+        address: this.tooldb.peerAccount.getAddress() || "",
+        sig: signature,
+      } as Peer;
+    });
+  }
+
+  /**
+   * Check if the specified client is connected or not
+   * @param clientId Client ID
+   * @returns boolean
+   */
+  public isConnected(clientId: string) {
+    return this._isClientConnected[clientId]
+      ? this._isClientConnected[clientId]()
+      : false;
+  }
+
+  /**
+   * Check if the specified client is a server/relay
+   * @param clientId Client ID
+   * @returns boolean
+   */
+  public isServer(clientId: string) {
+    return this._clientIsServer[clientId] || false;
+  }
+
+  public craftPingMessage() {
+    return this.getMeAsPeer().then((meAsPeer) => {
+      return JSON.stringify({
+        type: "ping",
+        clientId: this.getClientAddress(),
+        to: [this.getClientAddress()],
+        isServer: this.tooldb.options.server,
+        id: textRandom(10),
+        peer: meAsPeer,
+      } as PingMessage);
+    });
+  }
+
+  /**
+   * Execute the function to send a message to the specified client ID
+   * @param clientId Client ID
+   * @param message Message
+   */
+  private executeSendToClient(clientId: string, message: string) {
+    if (this._clientToSend[clientId]) {
+      this._clientToSend[clientId](message);
+    }
+  }
+
+  public getClientAddress() {
+    // This is not a good idea to use on all adapters, so it should be replaced
+    // if its causing issues. The only reason we use the last 20 chars is to
+    // muse the same peer address as the webrtc adapter.
+    return (this.tooldb.peerAccount.getAddress() || "").slice(-20);
+  }
+
+  public onClientDisconnect(clientId: string) {
+    delete this._clientToSend[clientId];
+    delete this._clientIsServer[clientId];
+  }
+
+  /**
+   * Should be called as a message payload handler
+   * This function will take care of processing the messages and also making sure
+   * we use the correct response methods troughout the adapter.
+   * @param message message payload
+   * @param clientId Client ID (can be null for ping/pong)
+   * @param setClientId Callback to set the client id on the parent class
+   */
+  public onClientMessage(
+    message: string,
+    clientId: string | null,
+    setClientId: (clientId: string) => void
+  ) {
+    // this.tooldb.logger("onClientMessage", clientId);
+
+    if (clientId && !this.tooldb.processedOutHashes[clientId]) {
+      this.tooldb.processedOutHashes[clientId] = [];
+    }
+
+    try {
+      const parsedMessage = JSON.parse(message) as ToolDbMessage;
+      // We assume the first messages to arrive will always be ping or pong.
+      // Only after that we can get the client id for this socket.
+      if (parsedMessage.type === "ping" || parsedMessage.type === "pong") {
+        const cid = parsedMessage.clientId;
+        setClientId(cid);
+        this.tooldb.onPeerConnect(cid);
+        this._clientIsServer[cid] = parsedMessage.isServer;
+        this.tooldb.processedOutHashes[cid] = [];
+        this.tooldb.clientOnMessage(parsedMessage, cid);
+      } else if (clientId) {
+        this.tooldb.clientOnMessage(parsedMessage, clientId);
+      }
+    } catch (e) {
+      this.tooldb.logger("Got message ERR", message);
+      this.tooldb.logger(e);
+    }
+  }
+
+  /**
+   * Sends a message to all peers connected to us
+   * This function takes care of the message deduplication, making sure we dont send
+   * the same message twice to the same peer.
+   * @param msg message data
+   * @param crossServerOnly If this message should be send to server peers only
+   * @param isRelay if we should relay this message
+   */
+  public sendToAll(msg: ToolDbMessage, crossServerOnly = false) {
+    const to = uniq([...msg.to, this.getClientAddress()]);
+
+    const finalMessage = JSON.stringify({ ...msg, to });
+
+    const filteredConns = Object.keys(this.clientToSend)
+      .filter((id) => !to.includes(id))
+      .filter((clientId) => this.isConnected(clientId));
+
+    filteredConns.forEach((clientId) => {
+      if ((crossServerOnly && this.isServer(clientId)) || !crossServerOnly) {
+        this.tooldb.logger(to, "Sent out to:", clientId);
+
+        if (msg.type === "put" || msg.type === "crdtPut") {
+          if (!this.tooldb.processedOutHashes[clientId].includes(msg.data.h)) {
+            this.executeSendToClient(clientId, finalMessage);
+            this.tooldb.processedOutHashes[clientId].push(msg.data.h);
+          }
+        } else {
+          this.executeSendToClient(clientId, finalMessage);
+        }
+      }
+      // } else {
+      //   this.tooldb.logger("Fitlered out;", clientId);
+      // }
+    });
+  }
+
+  /**
+   * Sends a message to a single peer.
+   * This function also takes care of the message deduplication.
+   * @param clientId Peer/Client id we want to send to.
+   * @param msg message data
+   */
+  public sendToClientId(clientId: string, msg: ToolDbMessage) {
+    const to = uniq([...msg.to, this.getClientAddress()]);
+    const finalMessage = JSON.stringify({ ...msg, to });
+
+    if (msg.type === "put" || msg.type === "crdtPut") {
+      if (
+        clientId &&
+        !this.tooldb.processedOutHashes[clientId].includes(msg.data.h)
+      ) {
+        this.executeSendToClient(clientId, finalMessage);
+        this.tooldb.processedOutHashes[clientId].push(msg.data.h);
+      }
+    } else {
+      this.executeSendToClient(clientId, finalMessage);
+    }
+  }
+}

package/lib/adapters-base/storageAdapter.ts

@@ -0,0 +1,35 @@
+import { ToolDb } from "..";
+
+export default class ToolDbStorageAdapter {
+  private _tooldb: ToolDb;
+
+  private _forceStorageName: string | undefined;
+
+  get tooldb() {
+    return this._tooldb;
+  }
+
+  get storageName() {
+    return this._forceStorageName || this._tooldb.options.storageName;
+  }
+
+  constructor(db: ToolDb, forceStorageName?: string) {
+    this._tooldb = db;
+
+    if (forceStorageName) {
+      this._forceStorageName = forceStorageName;
+    }
+  }
+
+  put(key: string, data: string): Promise<unknown> {
+    return Promise.resolve();
+  }
+
+  get(key: string): Promise<string> {
+    return Promise.resolve("");
+  }
+
+  query(key: string): Promise<string[]> {
+    return Promise.resolve([]);
+  }
+}

package/lib/adapters-base/userAdapter.ts

@@ -0,0 +1,49 @@
+import { ToolDb, VerificationData } from "..";
+
+export default class ToolDbUserAdapter {
+  private _tooldb: ToolDb;
+
+  get tooldb() {
+    return this._tooldb;
+  }
+
+  constructor(db: ToolDb) {
+    this._tooldb = db;
+  }
+
+  public anonUser() {
+    return;
+  }
+
+  public setUser(account: unknown | undefined, name: string): void {}
+
+  public signData(data: string): Promise<string> {
+    return Promise.resolve("");
+  }
+
+  public verifySignature(
+    message: Partial<VerificationData<any>>
+  ): Promise<boolean> {
+    return Promise.resolve(false);
+  }
+
+  public getAccountFromPrivate(privateKey: string): Promise<unknown> {
+    return Promise.resolve(undefined);
+  }
+
+  public encryptAccount(password: string): Promise<unknown> {
+    return Promise.resolve(undefined);
+  }
+
+  public decryptAccount(acc: unknown, password: string): Promise<any> {
+    return Promise.resolve(undefined);
+  }
+
+  public getAddress(): string | undefined {
+    return "";
+  }
+
+  public getUsername(): string | undefined {
+    return "";
+  }
+}

package/lib/crdt/baseCrdt.ts

@@ -0,0 +1,21 @@
+export const CRDT_MAP = "MAP";
+
+export const CRDT_LIST = "LIST";
+
+export const CRDT_COUNTER = "COUNTER";
+
+export class BaseCrdt<T = any, Changes = any, Value = any> {
+  public type: string = "";
+
+  public mergeChanges(changes: Changes[]) {
+    //
+  }
+
+  public getChanges(): Changes[] {
+    return [];
+  }
+
+  get value(): Value {
+    return "" as any;
+  }
+}

package/lib/crdt/counterCrdt.ts

@@ -0,0 +1,111 @@
+import { BaseCrdt, CRDT_LIST } from "..";
+
+export type CounterOperations = "ADD" | "SUB";
+
+export interface ChangeCounterBase {
+  t: CounterOperations; // Operation type
+  v: number; // Value
+  a: string;
+  i: number;
+}
+
+export interface AddCounterChange extends ChangeCounterBase {
+  t: "ADD";
+}
+
+export interface SubCounterChange extends ChangeCounterBase {
+  t: "SUB";
+}
+
+export type CounterChanges = AddCounterChange | SubCounterChange;
+
+export class CounterCrdt<T> extends BaseCrdt<number, CounterChanges, number> {
+  public type = CRDT_LIST;
+
+  public _changes: CounterChanges[] = [];
+
+  private _author = "";
+
+  private _value: number = 0;
+
+  private _lastUpdateSize: number = 0;
+
+  constructor(author: string, changes?: CounterChanges[]) {
+    super();
+    this._author = author;
+    if (changes) {
+      this.mergeChanges(changes);
+    }
+
+    this.calculate();
+  }
+
+  changesSort(a: CounterChanges, b: CounterChanges) {
+    if (a.i > b.i) return 1;
+    if (a.i < b.i) return -1;
+    return 0; // Should never be equal!
+  }
+
+  calculate() {
+    let temp: number = 0;
+    // Only update if we have new changes
+    if (Object.values(this._changes).length !== this._lastUpdateSize) {
+      this._changes.sort(this.changesSort).forEach((change) => {
+        if (change.t === "ADD") {
+          temp += change.v;
+        } else if (change.t === "SUB") {
+          temp -= change.v;
+        }
+      });
+
+      this._value = temp;
+    }
+  }
+
+  get value(): number {
+    this.calculate();
+    return this._value;
+  }
+
+  public mergeChanges(newChanges: CounterChanges[]) {
+    newChanges.forEach((change) => {
+      // Filter by author and index
+      const filtered = this._changes.filter(
+        (c) => c.i === change.i && c.a === change.a && c.t === change.t
+      );
+      // Only add if there are not matches
+      if (filtered.length === 0) {
+        this._changes.push(change);
+      }
+    });
+    this.calculate();
+  }
+
+  public getChanges(): CounterChanges[] {
+    return this._changes;
+  }
+
+  public ADD(value: number) {
+    const ourChanges = this._changes.filter((c) => c.a === this._author);
+
+    this._changes.push({
+      t: "ADD",
+      v: value,
+      a: this._author,
+      i: ourChanges.length,
+    });
+    this.calculate();
+  }
+
+  public SUB(value: number) {
+    const ourChanges = this._changes.filter((c) => c.a === this._author);
+
+    this._changes.push({
+      t: "SUB",
+      v: value,
+      a: this._author,
+      i: ourChanges.length,
+    });
+    this.calculate();
+  }
+}