xitdb 0.14.0 → 0.15.0

package/README.md

@@ -26,6 +26,7 @@ This database was originally made for the [xit version control system](https://g
 * [Initializing a Database](#initializing-a-database)
 * [Types](#types)
 * [Cloning and Undoing](#cloning-and-undoing)
+* [Sorting and Paginating](#sorting-and-paginating)
 * [Large Byte Arrays](#large-byte-arrays)
 * [Iterators](#iterators)
 * [Hashing](#hashing)
@@ -287,6 +288,116 @@ const bigCities = new ReadArrayList(bigCitiesCursor!);
 assert.strictEqual(bigCities.count(), 2);
 ```
 
+## Sorting and Paginating
+
+The `Hash`-based structures are great for looking data up by key, but they store their contents in hash order, which is meaningless to a human. You may need to display data in a sensible order (like newest posts first or users by signup date) and show it one page at a time. Relational databases like SQLite have this built-in: you declare a `CREATE INDEX`, write `ORDER BY created_ts LIMIT 20 OFFSET 40`, and the query planner maintains the index and seeks into it for you.
+
+In xitdb there are no built-in indexes, so you build and maintain them yourself. That's a little more code, but the index is just another data structure: a `SortedMap` whose keys are crafted to sort the way you want. You keep it in sync by writing to it in the same transaction that writes the primary data.
+
+Let's model the storage a basic blog would need: a collection of posts we look up by id, plus a secondary index that lets us list them oldest-first with pagination. The primary store is a `HashMap` from post id to the post's fields (like a row keyed by its primary key). The secondary index is a `SortedMap` keyed by creation time, whose value is the post id to look up.
+
+The trick is the key. A `SortedMap` orders its keys lexicographically by their raw bytes, so we encode the timestamp as a big-endian integer (so byte order matches chronological order) and append the post id to break ties between posts created in the same second and keep every key unique:
+
+```typescript
+// build a SortedMap key that sorts by creation time. the big-endian
+// timestamp makes byte order match chronological order; the post id is
+// appended so two posts with the same timestamp still get distinct keys.
+function orderKey(timestamp: number, postId: Uint8Array): Uint8Array {
+  const key = new Uint8Array(8 + postId.length);
+  new DataView(key.buffer).setBigUint64(0, BigInt(timestamp), false); // false = big-endian
+  key.set(postId, 8);
+  return key;
+}
+```
+
+Now we write some posts. On each insert we write the post into the primary map and add an entry to the secondary index (keeping both in sync is your job, not the database's):
+
+```typescript
+interface Post {
+  id: string;
+  title: string;
+  createdTs: number;
+}
+
+// post ids are fixed-length so the timestamp tie-breaker stays aligned
+const newPosts: Post[] = [
+  { id: 'post000000000001', title: 'Hello, world', createdTs: 1_700_000_000 },
+  { id: 'post000000000002', title: 'Second post', createdTs: 1_700_000_500 },
+  { id: 'post000000000003', title: 'Third post', createdTs: 1_700_001_000 },
+];
+
+history.appendContext(history.getSlot(-1), (cursor) => {
+  const moment = new WriteHashMap(cursor);
+
+  // the primary store: a HashMap from post id to the post's fields
+  const idToPostCursor = moment.putCursor('id->post');
+  const idToPost = new WriteHashMap(idToPostCursor);
+
+  // the secondary index: a SortedMap ordered by creation time. there's
+  // no CREATE INDEX here, so we maintain it ourselves on every write.
+  const createdTsToPostIdCursor = moment.putCursor('created-ts->post-id');
+  const createdTsToPostId = new WriteSortedMap(createdTsToPostIdCursor);
+
+  for (const post of newPosts) {
+    // write the post into the primary map under its id
+    const postCursor = idToPost.putCursor(post.id);
+    const postMap = new WriteHashMap(postCursor);
+    postMap.put('title', new Bytes(post.title));
+    postMap.put('created-ts', new Uint(post.createdTs));
+
+    // add an entry to the secondary index. the key sorts by time,
+    // and the value is the post id we'll use to look the post back up.
+    const orderKeyBytes = orderKey(post.createdTs, new TextEncoder().encode(post.id));
+    createdTsToPostId.put(orderKeyBytes, new Bytes(post.id));
+  }
+});
+```
+
+To display a page, we walk the `SortedMap` instead of the `HashMap`. A web app would take a `pageSize` and an `after` offset from the request (something like `/posts?after=20`), so this is the xitdb equivalent of `ORDER BY created_ts LIMIT pageSize OFFSET after`:
+
+```typescript
+const momentCursor = history.getCursor(-1);
+const moment = new ReadHashMap(momentCursor!);
+
+const idToPostCursor = moment.getCursor('id->post');
+const idToPost = new ReadHashMap(idToPostCursor!);
+
+const createdTsToPostIdCursor = moment.getCursor('created-ts->post-id');
+const createdTsToPostId = new ReadSortedMap(createdTsToPostIdCursor!);
+
+// a web request would supply these; here we just grab the first page
+const pageSize = 2;
+const after = 0;
+
+const count = createdTsToPostId.count();
+const end = Math.min(after + pageSize, count);
+
+// seek straight to the start of the page, then walk forward one entry at a
+// time. because SortedMap is a count-augmented B+tree, iteratorFromIndex
+// finds rank `after` in O(log n) without scanning the entries it skips, so
+// jumping to page 500 is just as cheap as page 1.
+const decoder = new TextDecoder();
+const iter = createdTsToPostId.iteratorFromIndex(after);
+for (let i = after; i < end && iter.hasNext(); i++) {
+  const idCursor = iter.next()!;
+  const idKv = idCursor.readKeyValuePair();
+
+  // the index entry's value is the post id; use it to read the
+  // full post out of the primary map
+  const postId = decoder.decode(idKv.valueCursor.readBytes(MAX_READ_BYTES));
+
+  const postCursor = idToPost.getCursor(postId);
+  const postMap = new ReadHashMap(postCursor!);
+  const titleCursor = postMap.getCursor('title');
+  const title = decoder.decode(titleCursor!.readBytes(MAX_READ_BYTES));
+
+  // a real app would render this into the page's HTML
+  console.log(title);
+}
+```
+
+This works for any ordering you need: sort by a username with a string key, by score with a big-endian integer key, or build several `SortedMap` indexes over the same primary `HashMap` to offer the data in different orders. With xitdb you "bring your own index". It takes a bit more effort than the declarative convenience of SQL databases, but it gives you more explicit control, and avoids the common problem in SQL where queries silently become inefficient due to not using indexes. In xitdb, inefficiency is hard to miss because you are always writing your queries as imperative code and the indexes are always explicit.
+
 ## Large Byte Arrays
 
 When reading and writing large byte arrays, you probably don't want to have all of their contents in memory at once. To incrementally write to a byte array, just get a writer from a cursor:
@@ -361,6 +472,8 @@ The above code iterates over `people`, which is an `ArrayList`, and for each per
 
 The iteration of the `HashMap` looks the same with `HashSet`, `CountedHashMap`, and `CountedHashSet`. When iterating, you call `readKeyValuePair` on the cursor and can read the `keyCursor` and `valueCursor` from it. In maps, `put` sets the key and value. In sets, `put` only sets the key; the value will always have a tag type of `NONE`.
 
+`ArrayList` and `LinkedArrayList` also have an `iteratorFrom` method, which starts the iterator from the given index. `SortedMap` and `SortedSet` have `iteratorFrom` and `iteratorFromIndex` to start the iterator from a key or index respectively. This is especially useful for pagination: you can seek straight to the start of a page and walk forward only as far as you need. See the [Sorting and Paginating](#sorting-and-paginating) section for an example.
+
 ## Hashing
 
 The hashing data structures will create the hash for you when you call methods like `put` or `getCursor` and provide the key as a string or `Bytes`. If you want to do the hashing yourself, there is an overload of those methods that take a `Uint8Array` as the key, which should be the hash that you computed.

package/dist/read-array-list.d.ts

@@ -8,6 +8,7 @@ export declare class ReadArrayList implements Slotted {
     slot(): Slot;
     count(): number;
     iterator(): CursorIterator;
+    iteratorFrom(index: number): CursorIterator;
     [Symbol.iterator](): Iterator<ReadCursor>;
     getCursor(index: number): ReadCursor | null;
     getSlot(index: number): Slot | null;

package/dist/read-array-list.js

@@ -1,3 +1,4 @@
+import { CursorIterator } from './read-cursor.js';
 import { ArrayListGet } from './database.js';
 import { UnexpectedTagException } from './exceptions.js';
 export class ReadArrayList {
@@ -23,6 +24,11 @@ export class ReadArrayList {
     iterator() {
         return this.cursor.iterator();
     }
+    // iterate starting at the given index, seeking straight to it instead of
+    // walking from the front. negative indexes count from the end.
+    iteratorFrom(index) {
+        return CursorIterator.initArrayListFromIndex(this.cursor, index);
+    }
     *[Symbol.iterator]() {
         yield* this.cursor;
     }

package/dist/read-cursor.d.ts

@@ -48,13 +48,18 @@ export declare class CursorIterator {
     private stack;
     private nextCursorMaybe;
     constructor(cursor: ReadCursor);
+    private static resolveStartIndex;
     static initSortedFromIndex(cursor: ReadCursor, startIndex: number): CursorIterator;
+    static initArrayListFromIndex(cursor: ReadCursor, startIndex: number): CursorIterator;
+    static initLinkedArrayListFromIndex(cursor: ReadCursor, startIndex: number): CursorIterator;
     static initSortedFromKey(cursor: ReadCursor, startKey: Uint8Array): CursorIterator;
     private static sortedRootPtr;
     private static sortedStackFromIndex;
+    private static arrayListStackFromIndex;
+    private static btreeStackFromIndex;
     private static sortedStackFromKey;
     init(): void;
-    private readSlotBlock;
+    static readSlotBlock(cursor: ReadCursor, position: number): Slot[];
     private initStack;
     hasNext(): boolean;
     next(): ReadCursor | null;

package/dist/read-cursor.js

@@ -1,6 +1,6 @@
 import { Slot } from './slot.js';
 import { SlotPointer } from './slot-pointer.js';
-import { WriteMode, ArrayListHeader, BTreeHeader, KeyValuePair, INDEX_BLOCK_SIZE, BTREE_NODE_HEADER_SIZE, BTreeNodeKind, SLOT_COUNT, } from './database.js';
+import { WriteMode, ArrayListHeader, BTreeHeader, KeyValuePair, INDEX_BLOCK_SIZE, BTREE_NODE_HEADER_SIZE, BTreeNodeKind, SLOT_COUNT, BIT_COUNT, } from './database.js';
 import { UnexpectedTagException, StreamTooLongException, EndOfStreamException, InvalidOffsetException, KeyNotFoundException, ExpectedUnsignedLongException, } from './exceptions.js';
 import { Bytes } from './writeable-data.js';
 export class KeyValuePairCursor {
@@ -321,19 +321,77 @@ export class CursorIterator {
     constructor(cursor) {
         this.cursor = cursor;
     }
+    // resolve a possibly-negative start index against `size` to a 0-based rank.
+    // negatives count from the end (-1 is the last entry); anything out of range
+    // returns -1 (yield nothing).
+    static resolveStartIndex(index, size) {
+        const resolved = index < 0 ? index + size : index;
+        if (resolved < 0 || resolved >= size)
+            return -1;
+        return resolved;
+    }
     // start a sorted-map iterator at the entry with rank startIndex (the count descent),
-    // iterating in key order from there
+    // iterating in key order from there. negative indexes count from the end.
     static initSortedFromIndex(cursor, startIndex) {
-        const total = cursor.count();
         const it = new CursorIterator(cursor);
         // an unwritten map is NONE (like iterator()): yield nothing
-        if (cursor.slotPtr.slot.tag === 0 /* Tag.NONE */ || startIndex >= total) {
+        if (cursor.slotPtr.slot.tag === 0 /* Tag.NONE */) {
+            return it;
+        }
+        const total = cursor.count();
+        const idx = CursorIterator.resolveStartIndex(startIndex, total);
+        if (idx < 0) {
             return it;
         }
         const rootPtr = CursorIterator.sortedRootPtr(cursor);
         it.size = total;
-        it.index = startIndex;
-        it.stack = CursorIterator.sortedStackFromIndex(cursor, rootPtr, startIndex);
+        it.index = idx;
+        it.stack = CursorIterator.sortedStackFromIndex(cursor, rootPtr, idx);
+        return it;
+    }
+    // start an array-list iterator at startIndex, descending the radix trie
+    // straight to that index. negatives count from the end; out of range (or an
+    // unwritten list) yields nothing.
+    static initArrayListFromIndex(cursor, startIndex) {
+        const it = new CursorIterator(cursor);
+        if (cursor.slotPtr.slot.tag !== 2 /* Tag.ARRAY_LIST */) {
+            return it;
+        }
+        cursor.db.core.seek(Number(cursor.slotPtr.slot.value));
+        const reader = cursor.db.core.reader();
+        const headerBytes = new Uint8Array(ArrayListHeader.LENGTH);
+        reader.readFully(headerBytes);
+        const header = ArrayListHeader.fromBytes(headerBytes);
+        const idx = CursorIterator.resolveStartIndex(startIndex, header.size);
+        if (idx < 0) {
+            return it;
+        }
+        const lastKey = header.size - 1;
+        const shift = lastKey < SLOT_COUNT ? 0 : Math.floor(Math.log(lastKey) / Math.log(SLOT_COUNT));
+        it.size = header.size;
+        it.index = idx;
+        it.stack = CursorIterator.arrayListStackFromIndex(cursor, header.ptr, idx, shift);
+        return it;
+    }
+    // start a linked-array-list iterator at startIndex, descending the
+    // count-augmented b-tree straight to that index. negatives count from the end.
+    static initLinkedArrayListFromIndex(cursor, startIndex) {
+        const it = new CursorIterator(cursor);
+        if (cursor.slotPtr.slot.tag !== 3 /* Tag.LINKED_ARRAY_LIST */) {
+            return it;
+        }
+        cursor.db.core.seek(Number(cursor.slotPtr.slot.value));
+        const reader = cursor.db.core.reader();
+        const headerBytes = new Uint8Array(BTreeHeader.LENGTH);
+        reader.readFully(headerBytes);
+        const header = BTreeHeader.fromBytes(headerBytes);
+        const idx = CursorIterator.resolveStartIndex(startIndex, header.size);
+        if (idx < 0) {
+            return it;
+        }
+        it.size = header.size;
+        it.index = idx;
+        it.stack = CursorIterator.btreeStackFromIndex(cursor, header.rootPtr, idx);
         return it;
     }
     // start a sorted-map iterator at the first entry with key >= startKey
@@ -387,6 +445,49 @@ export class CursorIterator {
             }
         }
     }
+    // descend the array-list radix trie to startIndex, pushing one IteratorLevel
+    // per tier with its index set to that tier's child slot. nextInternal then
+    // walks forward from there.
+    static arrayListStackFromIndex(cursor, rootPtr, startIndex, shift) {
+        const stack = [];
+        let pos = rootPtr;
+        let sh = shift;
+        while (true) {
+            const block = CursorIterator.readSlotBlock(cursor, pos);
+            const i = (startIndex >>> (sh * BIT_COUNT)) & (SLOT_COUNT - 1);
+            stack.push(new IteratorLevel(pos, block, i));
+            if (sh === 0)
+                return stack;
+            // every tier above the leaf is a populated INDEX slot for any
+            // startIndex < size, so this child always exists
+            pos = Number(block[i].value);
+            sh -= 1;
+        }
+    }
+    // descend the linked-array-list count b-tree to startIndex; the positional
+    // analog of sortedStackFromIndex (no separator keys).
+    static btreeStackFromIndex(cursor, rootPtr, startIndex) {
+        const stack = [];
+        let nodePtr = rootPtr;
+        let rem = startIndex;
+        while (true) {
+            const node = cursor.db.readBTreeNode(nodePtr);
+            const position = nodePtr + BTREE_NODE_HEADER_SIZE;
+            if (node.kind === BTreeNodeKind.LEAF) {
+                stack.push(new IteratorLevel(position, node.values, rem));
+                return stack;
+            }
+            else {
+                let i = 0;
+                while (i + 1 < node.num && rem >= node.counts[i]) {
+                    rem -= node.counts[i];
+                    i++;
+                }
+                stack.push(new IteratorLevel(position, node.children, i));
+                nodePtr = Number(node.children[i].value);
+            }
+        }
+    }
     static sortedStackFromKey(cursor, rootPtr, key) {
         const stack = [];
         let nodePtr = rootPtr;
@@ -471,7 +572,7 @@ export class CursorIterator {
     }
     // read a 16-slot index block (the iterable structures all use 9-byte slots in
     // their index/node blocks)
-    readSlotBlock(cursor, position) {
+    static readSlotBlock(cursor, position) {
         cursor.db.core.seek(position);
         const reader = cursor.db.core.reader();
         const indexBlockBytes = new Uint8Array(INDEX_BLOCK_SIZE);
@@ -484,7 +585,7 @@ export class CursorIterator {
         return indexBlock;
     }
     initStack(cursor, position) {
-        return [new IteratorLevel(position, this.readSlotBlock(cursor, position), 0)];
+        return [new IteratorLevel(position, CursorIterator.readSlotBlock(cursor, position), 0)];
     }
     hasNext() {
         switch (this.cursor.slotPtr.slot.tag) {
@@ -557,7 +658,7 @@ export class CursorIterator {
                 if (nextSlot.tag === 1 /* Tag.INDEX */) {
                     // nodeOffset skips a b-tree node's kind+num header
                     const nextPos = Number(nextSlot.value) + nodeOffset;
-                    this.stack.push(new IteratorLevel(nextPos, this.readSlotBlock(this.cursor, nextPos), 0));
+                    this.stack.push(new IteratorLevel(nextPos, CursorIterator.readSlotBlock(this.cursor, nextPos), 0));
                     continue;
                 }
                 else {

package/dist/read-linked-array-list.d.ts

@@ -8,6 +8,7 @@ export declare class ReadLinkedArrayList implements Slotted {
     slot(): Slot;
     count(): number;
     iterator(): CursorIterator;
+    iteratorFrom(index: number): CursorIterator;
     [Symbol.iterator](): Iterator<ReadCursor>;
     getCursor(index: number): ReadCursor | null;
     getSlot(index: number): Slot | null;

package/dist/read-linked-array-list.js

@@ -1,3 +1,4 @@
+import { CursorIterator } from './read-cursor.js';
 import { LinkedArrayListGet } from './database.js';
 import { UnexpectedTagException } from './exceptions.js';
 export class ReadLinkedArrayList {
@@ -23,6 +24,11 @@ export class ReadLinkedArrayList {
     iterator() {
         return this.cursor.iterator();
     }
+    // iterate starting at the given index, seeking straight to it instead of
+    // walking from the front. negative indexes count from the end.
+    iteratorFrom(index) {
+        return CursorIterator.initLinkedArrayListFromIndex(this.cursor, index);
+    }
     *[Symbol.iterator]() {
         yield* this.cursor;
     }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "xitdb",
-  "version": "0.14.0",
+  "version": "0.15.0",
   "description": "An immutable database",
   "license": "MIT",
   "repository": {