@fireproof/core 0.0.7 → 0.0.9

package/README.md

@@ -0,0 +1,148 @@
+# 🔥 Fireproof
+
+Fireproof is a realtime database for today's interactive applications. It uses immutable data and distributed protocols 
+to offer a new kind of database that:
+- can be embedded in any page or app, with a flexible data ownership model
+- scales without incurring developer costs, thanks to Filecoin
+- uses cryptographically verifiable protocols (what plants crave)
+
+Learn more about the concepts and architecture behind Fireproof [in our plan,](https://hackmd.io/@j-chris/SyoE-Plpj) or jump to the [quick start](#quick-start) for React and server-side examples.
+
+### Status
+
+Fireproof is alpha software, you should only use it if you are planning to contribute. For now, [check out our React TodoMVC implementation running in browser-local mode.](https://main--lucky-naiad-5aa507.netlify.app/) It demonstrates document persistence, index queries, and event subscriptions, and uses the [`useFireproof()` React hook.](https://github.com/fireproof-storage/fireproof/blob/main/packages/fireproof/hooks/use-fireproof.tsx)
+
+[![Test](https://github.com/jchris/fireproof/actions/workflows/test.yml/badge.svg)](https://github.com/jchris/fireproof/actions/workflows/test.yml)
+[![JavaScript Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://standardjs.com)
+
+## Usage
+
+```js
+import Fireproof from 'fireproof';
+
+async function main() {
+  const database = Fireproof.storage('my-db');
+  const ok = await database.put({
+    name: 'alice',
+    age: 42
+  });
+  
+  const doc = await database.get(ok.id);
+  console.log(doc.name); // 'alice'
+}
+
+main();
+```
+
+## Features
+
+### Document Store
+
+A simple put, get, and delete interface for keeping track of all your JSON documents. Once your data is in Fireproof you can access it from any app or website. Fireproof document store uses MVCC versioning and Merkle clocks so you can always recover the version you are looking for.
+
+```js
+const { id, ref } = await database.put({
+    _id: 'three-thousand'
+    name: 'André',
+    age: 47
+});
+const doc = await database.get('three-thousand', {mvcc : true}) // mvcc is optional
+// {
+//    _id  : 'three-thousand'
+//    _clock : CID(bafy84...agfw7)
+//    name : 'André',
+//    age  : 47
+// }
+```
+
+The `_clock` allows you to query a stable snapshot of that version of the database. Fireproof uses immutable data structures under the hood, so you can always rollback to old data. Files can be embedded anywhere in your document using IPFS links like `{"/":"bafybeih3e3zdiehbqfpxzpppxrb6kaaw4xkbqzyr2f5pwr5refq2te2ape"}`, with API sugar coming soon.
+
+### Flexible Indexes
+
+Fireproof indexes are defined by custom JavaScript functions that you write, allowing you to easily index and search your data in the way that works best for your application. Easily handle data variety and schema drift by normalizing any data to the desired index.
+
+```js
+const index = new Index(database, function (doc, map) {
+  map(doc.age, doc.name)
+})
+const { rows, ref } = await index.query({ range: [40, 52] })
+// [ { key: 42, value: 'alice', id: 'a1s3b32a-3c3a-4b5e-9c1c-8c5c0c5c0c5c' },
+//   { key: 47, value: 'André', id: 'three-thousand' } ]
+```
+
+### Realtime Updates
+
+Subscribe to query changes in your application, so your UI updates automatically. Use the supplied React hooks, our Redux connector, or simple function calls to be notified of relevant changes.
+
+```js
+const listener = new Listener(database, function(doc, emit) {
+  if (doc.type == 'member') {
+    emit('member')
+  }
+})
+listener.on('member', (id) => {
+  const doc = await db.get(id)
+  alert(`Member update ${doc.name}`)
+})
+```
+
+### Self-sovereign Identity
+
+Fireproof is so easy to integrate with any site or app because you can get started right away, and set up an account later. By default users write to their own database copy, so you can get pretty far before you even have to think about API keys. [Authorization is via non-extractable keypair](https://ucan.xyz), like TouchID / FaceID.
+
+### Automatic Replication
+
+Documents changes are persisted to [Filecoin](https://filecoin.io) via [web3.storage](https://web3.storage), and made available over [IPFS] and on a global content delivery network. All you need to do to sync state is send a link to the latest database head, and Fireproof will take care of the rest. [Learn how to enable replication.](#status)
+
+### Cryptographic Proofs
+
+The [UCAN protocol](https://ucan.xyz) verifably links Fireproof updates to authorized agents via cryptographic proof chains. These proofs are portable like bearer tokens, but because invocations are signed by end-user device keys, UCAN proofs don't need to be hidden to be secure, allowing for delegation of service capabilities across devices and parties. Additionally, Fireproof's Merkle clocks and hash trees are immutable and self-validating, making merging changes safe and efficient. Fireproof makes cryptographic proofs available for all of it's operations, making it an ideal verfiable document database for smart contracts and other applications running in trustless environments. [Proof chains provide performance benefits as well](https://purrfect-tracker-45c.notion.site/Data-Routing-23c37b269b4c4c3dacb60d0077113bcb), by allowing recipients to skip costly I/O operations and instead cryptographically verify that changes contain all of the required context.
+
+## Limitations 💣
+
+### Security
+
+Until encryption support is enabled, all data written to Fireproof is public. There are no big hurdles for this feature but it's not ready yet.
+
+### Replication
+
+Currently Fireproof writes transactions and proofs to [CAR files](https://ipld.io/specs/transport/car/carv2/) which are well suited for peer and cloud replication. They are stored in IndexedDB locally, with cloud replication coming very soon.
+
+### Pre-beta Software
+
+While the underlying data structures and libraries Fireproof uses are trusted with billions of dollars worth of data, Fireproof started in February of 2023. Results may vary.
+
+## Thanks 🙏
+
+Fireproof is a synthesis of work done by people in the web community over the years. I couldn't even begin to name all the folks who made pivotal contributions. Without npm, React, and VS Code all this would have taken so much longer. Thanks to everyone who supported me getting into database development via Apache CouchDB, one of the original document databases. The distinguishing work on immutable datastructures comes from the years of consideration [IPFS](https://ipfs.tech), [IPLD](https://ipld.io), and the [Filecoin APIs](https://docs.filecoin.io) have enjoyed.
+
+Thanks to Alan Shaw and Mikeal Rogers without whom this project would have never got started. The core Merkle hash-tree clock is based on [Alan's Pail](https://github.com/alanshaw/pail), and you can see the repository history goes all the way back to work begun as a branch of that repo. Mikeal wrote [the prolly trees implementation](https://github.com/mikeal/prolly-trees).
+
+## Quick Start
+
+Look in the `examples/` directory for projects using the database. It's not picky how you use it, but we want to provide convenient jumping off places. Think of the examples as great to fork when starting your next project.
+
+If are adding Fireproof to an existing page, just install it and try some operations.
+
+```sh
+npm install @fireproof/core
+```
+
+In your `app.js` or `app.tsx` file:
+
+```js
+import { Fireproof } from '@fireproof/core'
+const fireproof = Fireproof.storage()
+const ok = await fireproof.put({ hello: 'world' })
+const doc = await fireproof.get(ok.id)
+```
+
+🤫 I like to drop a `window.fireproof = fireproof` in there as a development aid.
+
+# Contributing
+
+Feel free to join in. All welcome. [Open an issue](https://github.com/jchris/fireproof/issues)!
+
+# License
+
+Dual-licensed under [MIT or Apache 2.0](https://github.com/jchris/fireproof/blob/main/LICENSE.md)

package/hooks/use-fireproof.ts

@@ -53,9 +53,13 @@ export function useFireproof(defineDatabaseFn: Function, setupDatabaseFn: Functi
         console.log("Loading previous database clock. (localStorage.removeItem('fireproof') to reset)")
         await database.setClock(clock)
         try {
-          await database.changesSince()
+          const changes = await database.changesSince()
+          if (changes.rows.length < 2) {
+            console.log('Resetting database')
+            throw new Error('Resetting database')
+          }
         } catch (e) {
-          console.error('Error loading previous database clock.', e)
+          console.error(`Error loading previous database clock. ${fp} Resetting.`, e)
           await database.setClock([])
           await setupDatabaseFn(database)
           localSet('fireproof', JSON.stringify(database))
@@ -80,10 +84,9 @@ export function useFireproof(defineDatabaseFn: Function, setupDatabaseFn: Functi
 const husherMap = new Map()
 const husher = (id: string, workFn: { (): Promise<any> }, ms: number) => {
   if (!husherMap.has(id)) {
-    husherMap.set(
-      id,
-      workFn().finally(() => setTimeout(() => husherMap.delete(id), ms))
-    )
+    const start: number = Date.now()
+    husherMap.set(id, workFn().finally(() => 
+      setTimeout(() => husherMap.delete(id), ms - (Date.now() - start))))
   }
   return husherMap.get(id)
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@fireproof/core",
-  "version": "0.0.7",
+  "version": "0.0.9",
   "description": "Realtime database for IPFS",
   "main": "index.js",
   "type": "module",
@@ -37,7 +37,7 @@
     "cli-color": "^2.0.3",
     "idb": "^7.1.1",
     "multiformats": "^11.0.1",
-    "prolly-trees": "1.0.2",
+    "prolly-trees": "1.0.3",
     "sade": "^1.8.1"
   },
   "devDependencies": {

package/src/blockstore.js

@@ -55,7 +55,7 @@ export default class TransactionBlockstore {
     const key = cid.toString()
     // it is safe to read from the in-flight transactions becauase they are immutable
     const bytes = await Promise.any([this.#transactionsGet(key), this.commitedGet(key)]).catch((e) => {
-      console.log('networkGet', cid.toString(), e)
+      // console.log('networkGet', cid.toString(), e)
       return this.networkGet(key)
     })
     if (!bytes) throw new Error('Missing block: ' + key)
@@ -83,15 +83,13 @@ export default class TransactionBlockstore {
       const value = await husher(key, async () => await this.valet.remoteBlockFunction(key))
       if (value) {
         // console.log('networkGot: ' + key, value.length)
-        // dont turn this on until the Nan thing is fixed
-        // it keep the network blocks in indexedb but lets get the basics solid first
         doTransaction('networkGot: ' + key, this, async (innerBlockstore) => {
           await innerBlockstore.put(CID.parse(key), value)
         })
         return value
       }
     } else {
-      throw new Error('No remoteBlockFunction')
+      return false
     }
   }
 

package/src/clock.js

@@ -94,6 +94,7 @@ export class EventFetcher {
     /** @private */
     this._blocks = blocks
     this._cids = new CIDCounter()
+    this._cache = new Map()
   }
 
   /**
@@ -101,10 +102,15 @@ export class EventFetcher {
    * @returns {Promise<EventBlockView<T>>}
    */
   async get (link) {
+    const slink = link.toString()
+    // console.log('get', link.toString())
+    if (this._cache.has(slink)) return this._cache.get(slink)
     const block = await this._blocks.get(link)
     this._cids.add({ address: link })
     if (!block) throw new Error(`missing block: ${link}`)
-    return decodeEventBlock(block.bytes)
+    const got = decodeEventBlock(block.bytes)
+    this._cache.set(slink, got)
+    return got
   }
 
   async all () {
@@ -200,22 +206,34 @@ export async function * vis (blocks, head, options = {}) {
 }
 
 export async function findEventsToSync (blocks, head) {
+  // const callTag = Math.random().toString(36).substring(7)
   const events = new EventFetcher(blocks)
+  // console.time(callTag + '.findCommonAncestorWithSortedEvents')
   const { ancestor, sorted } = await findCommonAncestorWithSortedEvents(events, head)
+  // console.timeEnd(callTag + '.findCommonAncestorWithSortedEvents')
+  // console.log('sorted', sorted.length)
+  // console.time(callTag + '.contains')
   const toSync = await asyncFilter(sorted, async (uks) => !(await contains(events, ancestor, uks.cid)))
+  // console.timeEnd(callTag + '.contains')
+
   return { cids: events.cids, events: toSync }
 }
 
 const asyncFilter = async (arr, predicate) =>
+
   Promise.all(arr.map(predicate)).then((results) => arr.filter((_v, index) => results[index]))
 
 export async function findCommonAncestorWithSortedEvents (events, children) {
+  // const callTag = Math.random().toString(36).substring(7)
+  // console.time(callTag + '.findCommonAncestor')
   const ancestor = await findCommonAncestor(events, children)
+  // console.timeEnd(callTag + '.findCommonAncestor')
   if (!ancestor) {
     throw new Error('failed to find common ancestor event')
   }
-  // Sort the events by their sequence number
+  // console.time(callTag + '.findSortedEvents')
   const sorted = await findSortedEvents(events, children, ancestor)
+  // console.timeEnd(callTag + '.findSortedEvents')
   return { ancestor, sorted }
 }
 
@@ -279,6 +297,7 @@ function findCommonString (arrays) {
  * @param {import('./clock').EventLink<EventData>} tail
  */
 async function findSortedEvents (events, head, tail) {
+  // const callTag = Math.random().toString(36).substring(7)
   // get weighted events - heavier events happened first
   /** @type {Map<string, { event: import('./clock').EventBlockView<EventData>, weight: number }>} */
   const weights = new Map()
@@ -312,6 +331,7 @@ async function findSortedEvents (events, head, tail) {
     .sort((a, b) => b[0] - a[0])
     .flatMap(([, es]) => es.sort((a, b) => (String(a.cid) < String(b.cid) ? -1 : 1)))
   // console.log('sorted', sorted.map(s => s.value.data.value))
+
   return sorted
 }
 
@@ -322,6 +342,7 @@ async function findSortedEvents (events, head, tail) {
  * @returns {Promise<Array<{ event: import('./clock').EventBlockView<EventData>, depth: number }>>}
  */
 async function findEvents (events, start, end, depth = 0) {
+  // console.log('findEvents', start)
   const event = await events.get(start)
   const acc = [{ event, depth }]
   const { parents } = event.value

package/src/db-index.js

@@ -1,15 +1,19 @@
 import { create, load } from 'prolly-trees/db-index'
+// import { create, load } from '../../../../prolly-trees/src/db-index.js'
+
 import { sha256 as hasher } from 'multiformats/hashes/sha2'
 import { nocache as cache } from 'prolly-trees/cache'
 import { bf, simpleCompare } from 'prolly-trees/utils'
 import { makeGetBlock } from './prolly.js'
 import { cidsToProof } from './fireproof.js'
+import { CID } from 'multiformats'
+
 import * as codec from '@ipld/dag-cbor'
 // import { create as createBlock } from 'multiformats/block'
 import { doTransaction } from './blockstore.js'
 import charwise from 'charwise'
 
-const ALWAYS_REBUILD = true // todo: remove this
+const ALWAYS_REBUILD = false // todo: make false
 
 // const arrayCompare = (a, b) => {
 //   if (Array.isArray(a) && Array.isArray(b)) {
@@ -37,12 +41,13 @@ const compare = (a, b) => {
 const refCompare = (aRef, bRef) => {
   if (Number.isNaN(aRef)) return -1
   if (Number.isNaN(bRef)) throw new Error('ref may not be Infinity or NaN')
-  if (!Number.isFinite(aRef)) return 1
+  if (aRef === Infinity) return 1 // need to test this on equal docids!
   // if (!Number.isFinite(bRef)) throw new Error('ref may not be Infinity or NaN')
   return simpleCompare(aRef, bRef)
 }
 
-const opts = { cache, chunker: bf(3), codec, hasher, compare }
+const dbIndexOpts = { cache, chunker: bf(3), codec, hasher, compare }
+const idIndexOpts = { cache, chunker: bf(3), codec, hasher, compare: simpleCompare }
 
 const makeDoc = ({ key, value }) => ({ _id: key, ...value })
 
@@ -87,14 +92,6 @@ const indexEntriesForChanges = (changes, mapFun) => {
   return indexEntries
 }
 
-const indexEntriesForOldChanges = async (blocks, byIDindexRoot, ids, mapFun) => {
-  const { getBlock } = makeGetBlock(blocks)
-  const byIDindex = await load({ cid: byIDindexRoot.cid, get: getBlock, ...opts })
-
-  const result = await byIDindex.getMany(ids)
-  return result
-}
-
 /**
  * Represents an DbIndex for a Fireproof database.
  *
@@ -118,11 +115,31 @@ export default class DbIndex {
      */
     this.mapFun = mapFun
 
-    this.dbIndexRoot = null
-    this.dbIndex = null
+    this.database.indexes.set(mapFun.toString(), this)
+
+    this.indexById = { root: null, cid: null }
+    this.indexByKey = { root: null, cid: null }
 
-    this.byIDindexRoot = null
     this.dbHead = null
+
+    this.instanceId = this.database.instanceId + `.DbIndex.${Math.random().toString(36).substring(2, 7)}`
+
+    this.updateIndexPromise = null
+  }
+
+  toJSON () {
+    return { code: this.mapFun?.toString(), clock: { db: this.dbHead?.map(cid => cid.toString()), byId: this.indexById.cid?.toString(), byKey: this.indexByKey.cid?.toString() } }
+  }
+
+  static fromJSON (database, { code, clock: { byId, byKey, db } }) {
+    let mapFun
+    // eslint-disable-next-line
+    eval("mapFun = "+ code)
+    const index = new DbIndex(database, mapFun)
+    index.indexById.cid = CID.parse(byId)
+    index.indexByKey.cid = CID.parse(byKey)
+    index.dbHead = db.map(cid => CID.parse(cid))
+    return index
   }
 
   /**
@@ -140,13 +157,18 @@ export default class DbIndex {
    * @instance
    */
   async query (query) {
+    // const callId = Math.random().toString(36).substring(2, 7)
     // if (!root) {
     // pass a root to query a snapshot
-    await doTransaction('#updateIndex', this.database.blocks, async (blocks) => {
-      await this.#updateIndex(blocks)
-    })
+    // console.time(callId + '.#updateIndex')
+    await this.#updateIndex(this.database.blocks)
+    // console.timeEnd(callId + '.#updateIndex')
+
     // }
-    const response = await doIndexQuery(this.database.blocks, this.dbIndexRoot, this.dbIndex, query)
+    // console.time(callId + '.doIndexQuery')
+    const response = await doIndexQuery(this.database.blocks, this.indexByKey, query)
+    // console.timeEnd(callId + '.doIndexQuery')
+
     return {
       proof: { index: await cidsToProof(response.cids) },
       // TODO fix this naming upstream in prolly/db-DbIndex?
@@ -162,50 +184,54 @@ export default class DbIndex {
    * @private
    * @returns {Promise<void>}
    */
+
   async #updateIndex (blocks) {
+    // todo this could enqueue the request and give fresh ones to all second comers -- right now it gives out stale promises while working
+    // what would it do in a world where all indexes provide a database snapshot to query?
+    if (this.updateIndexPromise) return this.updateIndexPromise
+    this.updateIndexPromise = this.#innerUpdateIndex(blocks)
+    this.updateIndexPromise.finally(() => { this.updateIndexPromise = null })
+    return this.updateIndexPromise
+  }
+
+  async #innerUpdateIndex (inBlocks) {
+    // const callTag = Math.random().toString(36).substring(4)
+    // console.log(`#updateIndex ${callTag} >`, this.instanceId, this.dbHead?.toString(), this.dbIndexRoot?.cid.toString(), this.indexByIdRoot?.cid.toString())
     // todo remove this hack
     if (ALWAYS_REBUILD) {
       this.dbHead = null // hack
-      this.dbIndex = null // hack
+      this.indexByKey = null // hack
       this.dbIndexRoot = null
     }
+    // console.log('dbHead', this.dbHead)
+    // console.time(callTag + '.changesSince')
     const result = await this.database.changesSince(this.dbHead) // {key, value, del}
-    if (this.dbHead) {
-      const oldChangeEntries = await indexEntriesForOldChanges(
-        blocks,
-        this.byIDindexRoot,
-        result.rows.map(({ key }) => key),
-        this.mapFun
-      )
-      const oldIndexEntries = oldChangeEntries.result.map((key) => ({ key, del: true }))
-      const removalResult = await bulkIndex(blocks, this.dbIndexRoot, this.dbIndex, oldIndexEntries, opts)
-      this.dbIndexRoot = removalResult.root
-      this.dbIndex = removalResult.dbIndex
+    // console.timeEnd(callTag + '.changesSince')
+    // console.log('result.rows.length', result.rows.length)
 
-      const removeByIdIndexEntries = oldIndexEntries.map(({ key }) => ({ key: key[1], del: true }))
-      const purgedRemovalResults = await bulkIndex(
-        blocks,
-        this.byIDindexRoot,
-        this.byIDIndex,
-        removeByIdIndexEntries,
-        opts
-      )
-      this.byIDindexRoot = purgedRemovalResults.root
-      this.byIDIndex = purgedRemovalResults.dbIndex
-    }
-    const indexEntries = indexEntriesForChanges(result.rows, this.mapFun)
-    const byIdIndexEntries = indexEntries.map(({ key }) => ({ key: key[1], value: key }))
-    const addFutureRemovalsResult = await bulkIndex(blocks, this.byIDindexRoot, this.byIDIndex, byIdIndexEntries, opts)
-    this.byIDindexRoot = addFutureRemovalsResult.root
-    this.byIDIndex = addFutureRemovalsResult.dbIndex
+    // console.time(callTag + '.doTransaction#updateIndex')
 
-    // console.log('indexEntries', indexEntries)
-
-    const updateIndexResult = await bulkIndex(blocks, this.dbIndexRoot, this.dbIndex, indexEntries, opts)
-    this.dbIndexRoot = updateIndexResult.root
-    this.dbIndex = updateIndexResult.dbIndex
-
-    this.dbHead = result.clock
+    if (result.rows.length === 0) {
+      // console.log('#updateIndex < no changes')
+      this.dbHead = result.clock
+      return
+    }
+    await doTransaction('#updateIndex', inBlocks, async (blocks) => {
+      let oldIndexEntries = []
+      let removeByIdIndexEntries = []
+      if (this.dbHead) { // need a maybe load
+        const oldChangeEntries = await this.indexById.root.getMany(result.rows.map(({ key }) => key))
+        oldIndexEntries = oldChangeEntries.result.map((key) => ({ key, del: true }))
+        removeByIdIndexEntries = oldIndexEntries.map(({ key }) => ({ key: key[1], del: true }))
+      }
+      const indexEntries = indexEntriesForChanges(result.rows, this.mapFun)
+      const byIdIndexEntries = indexEntries.map(({ key }) => ({ key: key[1], value: key }))
+      this.indexById = await bulkIndex(blocks, this.indexById, removeByIdIndexEntries.concat(byIdIndexEntries), idIndexOpts)
+      this.indexByKey = await bulkIndex(blocks, this.indexByKey, oldIndexEntries.concat(indexEntries), dbIndexOpts)
+      this.dbHead = result.clock
+    })
+    // console.timeEnd(callTag + '.doTransaction#updateIndex')
+    // console.log(`#updateIndex ${callTag} <`, this.instanceId, this.dbHead?.toString(), this.dbIndexRoot?.cid.toString(), this.indexByIdRoot?.cid.toString())
   }
 }
 
@@ -216,45 +242,47 @@ export default class DbIndex {
  * @param {DbIndexEntry[]} indexEntries
  * @private
  */
-async function bulkIndex (blocks, inRoot, inDBindex, indexEntries) {
-  if (!indexEntries.length) return { dbIndex: inDBindex, root: inRoot }
+async function bulkIndex (blocks, inIndex, indexEntries, opts) {
+  if (!indexEntries.length) return inIndex
   const putBlock = blocks.put.bind(blocks)
   const { getBlock } = makeGetBlock(blocks)
   let returnRootBlock
   let returnNode
-  if (!inDBindex) {
-    for await (const node of await create({ get: getBlock, list: indexEntries, ...opts })) {
-      const block = await node.block
-      await putBlock(block.cid, block.bytes)
-      returnRootBlock = block
-      returnNode = node
-    }
-  } else {
-    // const dbIndex = await load({ cid: inRoot.cid, get: getBlock, ...opts }) // todo load from root on refresh
-    const { root, blocks } = await inDBindex.bulk(indexEntries)
-    returnRootBlock = await root.block
-    returnNode = root
-    for await (const block of blocks) {
-      await putBlock(block.cid, block.bytes)
+  if (!inIndex.root) {
+    const cid = inIndex.cid
+    if (!cid) {
+      for await (const node of await create({ get: getBlock, list: indexEntries, ...opts })) {
+        const block = await node.block
+        await putBlock(block.cid, block.bytes)
+        returnRootBlock = block
+        returnNode = node
+      }
+      return { root: returnNode, cid: returnRootBlock.cid }
     }
-    await putBlock(returnRootBlock.cid, returnRootBlock.bytes)
+    inIndex.root = await load({ cid, get: getBlock, ...dbIndexOpts })
+  }
+  const { root, blocks: newBlocks } = await inIndex.root.bulk(indexEntries)
+  returnRootBlock = await root.block
+  returnNode = root
+  for await (const block of newBlocks) {
+    await putBlock(block.cid, block.bytes)
   }
-  return { dbIndex: returnNode, root: returnRootBlock }
+  await putBlock(returnRootBlock.cid, returnRootBlock.bytes)
+  return { root: returnNode, cid: returnRootBlock.cid }
 }
 
-async function doIndexQuery (blocks, dbIndexRoot, dbIndex, query) {
-  if (!dbIndex) {
-    const cid = dbIndexRoot && dbIndexRoot.cid
+async function doIndexQuery (blocks, indexByKey, query) {
+  if (!indexByKey.root) {
+    const cid = indexByKey.cid
     if (!cid) return { result: [] }
     const { getBlock } = makeGetBlock(blocks)
-    dbIndex = await load({ cid, get: getBlock, ...opts })
+    indexByKey.root = await load({ cid, get: getBlock, ...dbIndexOpts })
   }
   if (query.range) {
     const encodedRange = query.range.map((key) => charwise.encode(key))
-    return dbIndex.range(...encodedRange)
+    return indexByKey.root.range(...encodedRange)
   } else if (query.key) {
     const encodedKey = charwise.encode(query.key)
-    console.log('getting key', encodedKey)
-    return dbIndex.get(encodedKey)
+    return indexByKey.root.get(encodedKey)
   }
 }

package/src/fireproof.js

@@ -38,6 +38,7 @@ export default class Fireproof {
     this.clock = clock
     this.config = config
     this.authCtx = authCtx
+    this.indexes = new Map()
   }
 
   /**
@@ -78,7 +79,11 @@ export default class Fireproof {
    */
   toJSON () {
     // todo this also needs to return the index roots...
-    return { clock: this.clock }
+    return {
+      clock: this.clock.map(cid => cid.toString()),
+      name: this.name,
+      indexes: [...this.indexes.values()].map((index) => index.toJSON())
+    }
   }
 
   /**
@@ -133,7 +138,7 @@ export default class Fireproof {
   }
 
   async #notifyListeners (changes) {
-    // await sleep(0)
+    // await sleep(10)
     for (const listener of this.#listeners) {
       await listener(changes)
     }
@@ -221,6 +226,7 @@ export default class Fireproof {
       console.error('failed', event)
       throw new Error('failed to put at storage layer')
     }
+    // console.log('new clock head', this.instanceId, result.head.toString())
     this.clock = result.head // do we want to do this as a finally block
     await this.#notifyListeners([decodedEvent]) // this type is odd
     return {

package/src/hydrator.js

@@ -0,0 +1,10 @@
+import Fireproof from './fireproof.js'
+import DbIndex from './db-index.js'
+
+export function fromJSON (json, blocks) {
+  const fp = new Fireproof(blocks, json.clock, { name: json.name })
+  for (const index of json.indexes) {
+    DbIndex.fromJSON(fp, index)
+  }
+  return fp
+}

package/src/prolly.js

@@ -164,7 +164,8 @@ export async function put (inBlocks, head, event, options) {
 
   // Otherwise, we find the common ancestor and update the root and other blocks
   const events = new EventFetcher(blocks)
-  // this is returning more events than necessary
+  // todo this is returning more events than necessary, lets define the desired semantics from the top down
+  // good semantics mean we can cache the results of this call
   const { ancestor, sorted } = await findCommonAncestorWithSortedEvents(events, head)
   // console.log('sorted', JSON.stringify(sorted.map(({ value: { data: { key, value } } }) => ({ key, value }))))
   const prollyRootNode = await prollyRootFromAncestor(events, ancestor, getBlock)

package/test/db-index.test.js

@@ -113,7 +113,7 @@ describe('DbIndex query', () => {
     await index.query({ range: [51, 54] })
 
     console.x('--- make Xander 53')
-    const DOCID = 'xxxx-3c3a-4b5e-9c1c-8c5c0c5c0c5c'
+    const DOCID = 'xander-doc'
     const r1 = await database.put({ _id: DOCID, name: 'Xander', age: 53 })
     assert(r1.id, 'should have id')