xitdb 0.13.0 → 0.15.0

package/README.md

@@ -26,6 +26,7 @@ This database was originally made for the [xit version control system](https://g
 * [Initializing a Database](#initializing-a-database)
 * [Types](#types)
 * [Cloning and Undoing](#cloning-and-undoing)
+* [Sorting and Paginating](#sorting-and-paginating)
 * [Large Byte Arrays](#large-byte-arrays)
 * [Iterators](#iterators)
 * [Hashing](#hashing)
@@ -97,18 +98,18 @@ const moment = new ReadHashMap(momentCursor!);
 // the cursor to "foo" and then calling readBytes on it
 const fooCursor = moment.getCursor('foo');
 const fooValue = fooCursor!.readBytes(MAX_READ_BYTES);
-expect(new TextDecoder().decode(fooValue)).toBe('foo');
+assert.strictEqual(new TextDecoder().decode(fooValue), 'foo');
 
 // to get the "fruits" list, we get the cursor to it and
 // then pass it to the ReadArrayList constructor
 const fruitsCursor = moment.getCursor('fruits');
 const fruits = new ReadArrayList(fruitsCursor!);
-expect(fruits.count()).toBe(3);
+assert.strictEqual(fruits.count(), 3);
 
 // now we can get the first item from the fruits list and read it
 const appleCursor = fruits.getCursor(0);
 const appleValue = appleCursor!.readBytes(MAX_READ_BYTES);
-expect(new TextDecoder().decode(appleValue)).toBe('apple');
+assert.strictEqual(new TextDecoder().decode(appleValue), 'apple');
 ```
 
 ## Initializing a Database
@@ -132,8 +133,9 @@ In xitdb there are a variety of immutable data structures that you can nest arbi
 * `CountedHashMap` and `CountedHashSet` are just a `HashMap` and `HashSet` that maintain a count of their contents
 * `ArrayList` is a growable array
 * `LinkedArrayList` is like an `ArrayList` that can also be efficiently sliced and concatenated
+* `SortedMap` and `SortedSet` are like a `HashMap` and `HashSet` where the keys are byte arrays kept in lexicographic order
 
-All data structures use the hash array mapped trie, invented by Phil Bagwell. The `LinkedArrayList` is based on his later work on RRB trees. These data structures were originally made immutable and widely available by Rich Hickey in Clojure. To my knowledge, they haven't been available in any open source database until xitdb.
+The `Hash`-based data structures and the `Arraylist` use the hash array mapped trie, invented by Phil Bagwell (originally made immutable and widely available by Rich Hickey in Clojure). The `LinkedArrayList`, `SortedMap`, and `SortedSet` are based on a B-tree.
 
 There are also scalar types you can store in the above-mentioned data structures:
 
@@ -157,7 +159,7 @@ Then, you can read it like this:
 ```typescript
 const randomNumberCursor = moment.getCursor('random-number');
 const randomNumber = randomNumberCursor!.readBytesObject(MAX_READ_BYTES);
-expect(new TextDecoder().decode(randomNumber.formatTag!)).toBe('bi');
+assert.strictEqual(new TextDecoder().decode(randomNumber.formatTag!), 'bi');
 const randomBigInt = randomNumber.value;
 ```
 
@@ -191,12 +193,12 @@ const moment = new ReadHashMap(momentCursor!);
 // the food list includes the fruits
 const foodCursor = moment.getCursor('food');
 const food = new ReadArrayList(foodCursor!);
-expect(food.count()).toBe(6);
+assert.strictEqual(food.count(), 6);
 
 // ...but the fruits list hasn't been changed
 const fruitsCursor = moment.getCursor('fruits');
 const fruits = new ReadArrayList(fruitsCursor!);
-expect(fruits.count()).toBe(3);
+assert.strictEqual(fruits.count(), 3);
 ```
 
 Before we continue, let's save the latest history index, so we can revert back to this moment of the database later:
@@ -232,12 +234,12 @@ const moment = new ReadHashMap(momentCursor!);
 // the cities list contains all four
 const citiesCursor = moment.getCursor('cities');
 const cities = new ReadArrayList(citiesCursor!);
-expect(cities.count()).toBe(4);
+assert.strictEqual(cities.count(), 4);
 
 // ..but so does big-cities! we did not intend to mutate this
 const bigCitiesCursor = moment.getCursor('big-cities');
 const bigCities = new ReadArrayList(bigCitiesCursor!);
-expect(bigCities.count()).toBe(4);
+assert.strictEqual(bigCities.count(), 4);
 ```
 
 The reason that `big-cities` was mutated is because all data in a given transaction is temporarily mutable. This is a very important optimization, but in this case, it's not what we want.
@@ -278,14 +280,124 @@ const moment = new ReadHashMap(momentCursor!);
 // the cities list contains all four
 const citiesCursor = moment.getCursor('cities');
 const cities = new ReadArrayList(citiesCursor!);
-expect(cities.count()).toBe(4);
+assert.strictEqual(cities.count(), 4);
 
 // and big-cities only contains the original two
 const bigCitiesCursor = moment.getCursor('big-cities');
 const bigCities = new ReadArrayList(bigCitiesCursor!);
-expect(bigCities.count()).toBe(2);
+assert.strictEqual(bigCities.count(), 2);
 ```
 
+## Sorting and Paginating
+
+The `Hash`-based structures are great for looking data up by key, but they store their contents in hash order, which is meaningless to a human. You may need to display data in a sensible order (like newest posts first or users by signup date) and show it one page at a time. Relational databases like SQLite have this built-in: you declare a `CREATE INDEX`, write `ORDER BY created_ts LIMIT 20 OFFSET 40`, and the query planner maintains the index and seeks into it for you.
+
+In xitdb there are no built-in indexes, so you build and maintain them yourself. That's a little more code, but the index is just another data structure: a `SortedMap` whose keys are crafted to sort the way you want. You keep it in sync by writing to it in the same transaction that writes the primary data.
+
+Let's model the storage a basic blog would need: a collection of posts we look up by id, plus a secondary index that lets us list them oldest-first with pagination. The primary store is a `HashMap` from post id to the post's fields (like a row keyed by its primary key). The secondary index is a `SortedMap` keyed by creation time, whose value is the post id to look up.
+
+The trick is the key. A `SortedMap` orders its keys lexicographically by their raw bytes, so we encode the timestamp as a big-endian integer (so byte order matches chronological order) and append the post id to break ties between posts created in the same second and keep every key unique:
+
+```typescript
+// build a SortedMap key that sorts by creation time. the big-endian
+// timestamp makes byte order match chronological order; the post id is
+// appended so two posts with the same timestamp still get distinct keys.
+function orderKey(timestamp: number, postId: Uint8Array): Uint8Array {
+  const key = new Uint8Array(8 + postId.length);
+  new DataView(key.buffer).setBigUint64(0, BigInt(timestamp), false); // false = big-endian
+  key.set(postId, 8);
+  return key;
+}
+```
+
+Now we write some posts. On each insert we write the post into the primary map and add an entry to the secondary index (keeping both in sync is your job, not the database's):
+
+```typescript
+interface Post {
+  id: string;
+  title: string;
+  createdTs: number;
+}
+
+// post ids are fixed-length so the timestamp tie-breaker stays aligned
+const newPosts: Post[] = [
+  { id: 'post000000000001', title: 'Hello, world', createdTs: 1_700_000_000 },
+  { id: 'post000000000002', title: 'Second post', createdTs: 1_700_000_500 },
+  { id: 'post000000000003', title: 'Third post', createdTs: 1_700_001_000 },
+];
+
+history.appendContext(history.getSlot(-1), (cursor) => {
+  const moment = new WriteHashMap(cursor);
+
+  // the primary store: a HashMap from post id to the post's fields
+  const idToPostCursor = moment.putCursor('id->post');
+  const idToPost = new WriteHashMap(idToPostCursor);
+
+  // the secondary index: a SortedMap ordered by creation time. there's
+  // no CREATE INDEX here, so we maintain it ourselves on every write.
+  const createdTsToPostIdCursor = moment.putCursor('created-ts->post-id');
+  const createdTsToPostId = new WriteSortedMap(createdTsToPostIdCursor);
+
+  for (const post of newPosts) {
+    // write the post into the primary map under its id
+    const postCursor = idToPost.putCursor(post.id);
+    const postMap = new WriteHashMap(postCursor);
+    postMap.put('title', new Bytes(post.title));
+    postMap.put('created-ts', new Uint(post.createdTs));
+
+    // add an entry to the secondary index. the key sorts by time,
+    // and the value is the post id we'll use to look the post back up.
+    const orderKeyBytes = orderKey(post.createdTs, new TextEncoder().encode(post.id));
+    createdTsToPostId.put(orderKeyBytes, new Bytes(post.id));
+  }
+});
+```
+
+To display a page, we walk the `SortedMap` instead of the `HashMap`. A web app would take a `pageSize` and an `after` offset from the request (something like `/posts?after=20`), so this is the xitdb equivalent of `ORDER BY created_ts LIMIT pageSize OFFSET after`:
+
+```typescript
+const momentCursor = history.getCursor(-1);
+const moment = new ReadHashMap(momentCursor!);
+
+const idToPostCursor = moment.getCursor('id->post');
+const idToPost = new ReadHashMap(idToPostCursor!);
+
+const createdTsToPostIdCursor = moment.getCursor('created-ts->post-id');
+const createdTsToPostId = new ReadSortedMap(createdTsToPostIdCursor!);
+
+// a web request would supply these; here we just grab the first page
+const pageSize = 2;
+const after = 0;
+
+const count = createdTsToPostId.count();
+const end = Math.min(after + pageSize, count);
+
+// seek straight to the start of the page, then walk forward one entry at a
+// time. because SortedMap is a count-augmented B+tree, iteratorFromIndex
+// finds rank `after` in O(log n) without scanning the entries it skips, so
+// jumping to page 500 is just as cheap as page 1.
+const decoder = new TextDecoder();
+const iter = createdTsToPostId.iteratorFromIndex(after);
+for (let i = after; i < end && iter.hasNext(); i++) {
+  const idCursor = iter.next()!;
+  const idKv = idCursor.readKeyValuePair();
+
+  // the index entry's value is the post id; use it to read the
+  // full post out of the primary map
+  const postId = decoder.decode(idKv.valueCursor.readBytes(MAX_READ_BYTES));
+
+  const postCursor = idToPost.getCursor(postId);
+  const postMap = new ReadHashMap(postCursor!);
+  const titleCursor = postMap.getCursor('title');
+  const title = decoder.decode(titleCursor!.readBytes(MAX_READ_BYTES));
+
+  // a real app would render this into the page's HTML
+  console.log(title);
+}
+```
+
+This works for any ordering you need: sort by a username with a string key, by score with a big-endian integer key, or build several `SortedMap` indexes over the same primary `HashMap` to offer the data in different orders. With xitdb you "bring your own index". It takes a bit more effort than the declarative convenience of SQL databases, but it gives you more explicit control, and avoids the common problem in SQL where queries silently become inefficient due to not using indexes. In xitdb, inefficiency is hard to miss because you are always writing your queries as imperative code and the indexes are always explicit.
+
 ## Large Byte Arrays
 
 When reading and writing large byte arrays, you probably don't want to have all of their contents in memory at once. To incrementally write to a byte array, just get a writer from a cursor:
@@ -315,7 +427,7 @@ for (let n; (n = cursorReader.read(buf)) > 0; ) {
   }
 }
 if (line.length > 0) lineCount++;
-expect(lineCount).toBe(50);
+assert.strictEqual(lineCount, 50);
 ```
 
 ## Iterators
@@ -360,6 +472,8 @@ The above code iterates over `people`, which is an `ArrayList`, and for each per
 
 The iteration of the `HashMap` looks the same with `HashSet`, `CountedHashMap`, and `CountedHashSet`. When iterating, you call `readKeyValuePair` on the cursor and can read the `keyCursor` and `valueCursor` from it. In maps, `put` sets the key and value. In sets, `put` only sets the key; the value will always have a tag type of `NONE`.
 
+`ArrayList` and `LinkedArrayList` also have an `iteratorFrom` method, which starts the iterator from the given index. `SortedMap` and `SortedSet` have `iteratorFrom` and `iteratorFromIndex` to start the iterator from a key or index respectively. This is especially useful for pagination: you can seek straight to the start of a page and walk forward only as far as you need. See the [Sorting and Paginating](#sorting-and-paginating) section for an example.
+
 ## Hashing
 
 The hashing data structures will create the hash for you when you call methods like `put` or `getCursor` and provide the key as a string or `Bytes`. If you want to do the hashing yourself, there is an overload of those methods that take a `Uint8Array` as the key, which should be the hash that you computed.
@@ -377,7 +491,7 @@ The size of the hash in bytes will be stored in the database's header. If you tr
 ```typescript
 core.seek(0);
 const header = Header.read(core);
-expect(header.hashSize).toBe(20);
+assert.strictEqual(header.hashSize, 20);
 ```
 
 The hash size alone does not disambiguate hashing algorithms, though. In addition, xitdb reserves four bytes in the header that you can use to put the name of the algorithm. You must provide it in the `Hasher` constructor:
@@ -391,7 +505,7 @@ The hash id is only written to the database header when it is first initialized.
 ```typescript
 core.seek(0);
 const header = Header.read(core);
-expect(Hasher.idToString(header.hashId)).toBe("sha1");
+assert.strictEqual(Hasher.idToString(header.hashId), "sha1");
 ```
 
 If you want to use SHA-256, I recommend using `sha2` as the hash id. You can then distinguish between SHA-256 and SHA-512 using the hash size, like this:
@@ -431,7 +545,7 @@ const compactDb = db.compact(compactCore);
 
 // read from the new compacted db
 const history = new ReadArrayList(compactDb.rootCursor());
-expect(history.count()).toBe(1);
+assert.strictEqual(history.count(), 1);
 ```
 
 This compacted database will be in a separate file. If you want to delete the original database and replace it with this one, you'll need to do that yourself. It is not possible to compact a database in-place (using the same file as the target database); doing so would fail and would render your original database unreadable.

package/dist/core-buffered-file.d.ts

@@ -1,5 +1,5 @@
-import type { Core, DataReader, DataWriter } from './core';
-import { CoreFile } from './core-file';
+import type { Core, DataReader, DataWriter } from './core.js';
+import { CoreFile } from './core-file.js';
 export declare class CoreBufferedFile implements Core {
     file: RandomAccessBufferedFile;
     constructor(filePath: string, bufferSize?: number);

package/dist/core-buffered-file.js

@@ -0,0 +1,182 @@
+import { CoreFile } from './core-file.js';
+import { CoreMemory } from './core-memory.js';
+export class CoreBufferedFile {
+    file;
+    constructor(filePath, bufferSize) {
+        this.file = new RandomAccessBufferedFile(filePath, bufferSize);
+    }
+    reader() {
+        return this.file;
+    }
+    writer() {
+        return this.file;
+    }
+    length() {
+        return this.file.length();
+    }
+    seek(pos) {
+        this.file.seek(pos);
+    }
+    position() {
+        return this.file.position();
+    }
+    setLength(len) {
+        this.file.setLength(len);
+    }
+    flush() {
+        this.file.flush();
+    }
+    sync() {
+        this.file.sync();
+    }
+    [Symbol.dispose]() {
+        this.file.file[Symbol.dispose]();
+    }
+}
+const DEFAULT_BUFFER_SIZE = 8 * 1024 * 1024; // 8MB
+class RandomAccessBufferedFile {
+    file;
+    memory;
+    bufferSize; // flushes when the memory is >= this size
+    filePos;
+    memoryPos;
+    constructor(filePath, bufferSize = DEFAULT_BUFFER_SIZE) {
+        this.file = new CoreFile(filePath);
+        this.memory = new CoreMemory();
+        this.bufferSize = bufferSize;
+        this.filePos = 0;
+        this.memoryPos = 0;
+    }
+    seek(pos) {
+        // flush if we are going past the end of the in-memory buffer
+        if (pos > this.memoryPos + this.memory.length()) {
+            this.flush();
+        }
+        this.filePos = pos;
+        // if the buffer is empty, set its position to this offset as well
+        if (this.memory.length() === 0) {
+            this.memoryPos = pos;
+        }
+    }
+    length() {
+        return Math.max(this.memoryPos + this.memory.length(), this.file.length());
+    }
+    position() {
+        return this.filePos;
+    }
+    setLength(len) {
+        this.flush();
+        this.file.setLength(len);
+        this.filePos = Math.min(len, this.filePos);
+    }
+    flush() {
+        if (this.memory.length() > 0) {
+            this.file.seek(this.memoryPos);
+            this.file.writer().write(this.memory.memory.toByteArray());
+            this.memoryPos = 0;
+            this.memory.memory.reset();
+        }
+    }
+    sync() {
+        this.flush();
+        this.file.sync();
+    }
+    // DataWriter interface
+    write(buffer) {
+        if (this.memory.length() + buffer.length > this.bufferSize) {
+            this.flush();
+        }
+        if (this.filePos >= this.memoryPos && this.filePos <= this.memoryPos + this.memory.length()) {
+            this.memory.seek(this.filePos - this.memoryPos);
+            this.memory.memory.write(buffer);
+        }
+        else {
+            // a direct disk write that overlaps the buffered region would be
+            // clobbered by a later flush of stale buffer bytes, so flush first
+            if (this.filePos < this.memoryPos + this.memory.length() && this.filePos + buffer.length > this.memoryPos) {
+                this.flush();
+            }
+            // Write directly to file
+            this.file.seek(this.filePos);
+            this.file.writer().write(buffer);
+        }
+        this.filePos += buffer.length;
+    }
+    writeByte(v) {
+        this.write(new Uint8Array([v & 0xff]));
+    }
+    writeShort(v) {
+        const buffer = new ArrayBuffer(2);
+        const view = new DataView(buffer);
+        view.setInt16(0, v & 0xffff, false); // big-endian
+        this.write(new Uint8Array(buffer));
+    }
+    writeLong(v) {
+        const buffer = new ArrayBuffer(8);
+        const view = new DataView(buffer);
+        view.setBigInt64(0, BigInt(v), false);
+        this.write(new Uint8Array(buffer));
+    }
+    // DataReader interface
+    readFully(buffer) {
+        let pos = 0;
+        // read from the disk -- before the in-memory buffer
+        if (this.filePos < this.memoryPos) {
+            const sizeBeforeMem = Math.min(this.memoryPos - this.filePos, buffer.length);
+            const tempBuffer = new Uint8Array(sizeBeforeMem);
+            this.file.seek(this.filePos);
+            this.file.reader().readFully(tempBuffer);
+            buffer.set(tempBuffer, pos);
+            pos += sizeBeforeMem;
+            this.filePos += sizeBeforeMem;
+        }
+        if (pos === buffer.length)
+            return;
+        // read from the in-memory buffer
+        if (this.filePos >= this.memoryPos && this.filePos < this.memoryPos + this.memory.length()) {
+            const memPos = this.filePos - this.memoryPos;
+            const sizeInMem = Math.min(this.memory.length() - memPos, buffer.length - pos);
+            this.memory.seek(memPos);
+            const memBuffer = new Uint8Array(sizeInMem);
+            this.memory.memory.readFully(memBuffer);
+            buffer.set(memBuffer, pos);
+            pos += sizeInMem;
+            this.filePos += sizeInMem;
+        }
+        if (pos === buffer.length)
+            return;
+        // read from the disk -- after the in-memory buffer
+        if (this.filePos >= this.memoryPos + this.memory.length()) {
+            const sizeAfterMem = buffer.length - pos;
+            const tempBuffer = new Uint8Array(sizeAfterMem);
+            this.file.seek(this.filePos);
+            this.file.reader().readFully(tempBuffer);
+            buffer.set(tempBuffer, pos);
+            pos += sizeAfterMem;
+            this.filePos += sizeAfterMem;
+        }
+    }
+    readByte() {
+        const bytes = new Uint8Array(1);
+        this.readFully(bytes);
+        return bytes[0];
+    }
+    readShort() {
+        const bytes = new Uint8Array(2);
+        this.readFully(bytes);
+        const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+        return view.getInt16(0, false); // big-endian
+    }
+    readInt() {
+        const bytes = new Uint8Array(4);
+        this.readFully(bytes);
+        const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+        return view.getInt32(0, false); // big-endian
+    }
+    readLong() {
+        const bytes = new Uint8Array(8);
+        this.readFully(bytes);
+        const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+        return Number(view.getBigInt64(0, false));
+    }
+}

package/dist/core-file.d.ts

@@ -1,4 +1,4 @@
-import type { Core, DataReader, DataWriter } from './core';
+import type { Core, DataReader, DataWriter } from './core.js';
 export declare class CoreFile implements Core {
     filePath: string;
     private _position;

package/dist/core-file.js

@@ -0,0 +1,105 @@
+import * as fs from 'fs';
+export class CoreFile {
+    filePath;
+    _position = 0;
+    fd;
+    constructor(filePath) {
+        this.filePath = filePath;
+        // Create file if it doesn't exist
+        try {
+            fs.accessSync(filePath);
+        }
+        catch {
+            fs.writeFileSync(filePath, new Uint8Array(0));
+        }
+        // Open file for reading and writing
+        this.fd = fs.openSync(filePath, 'r+');
+    }
+    reader() {
+        return new FileDataReader(this);
+    }
+    writer() {
+        return new FileDataWriter(this);
+    }
+    length() {
+        const stats = fs.fstatSync(this.fd);
+        return stats.size;
+    }
+    seek(pos) {
+        this._position = pos;
+    }
+    position() {
+        return this._position;
+    }
+    setLength(len) {
+        fs.ftruncateSync(this.fd, len);
+    }
+    flush() {
+    }
+    sync() {
+        fs.fsyncSync(this.fd);
+    }
+    [Symbol.dispose]() {
+        fs.closeSync(this.fd);
+    }
+}
+class FileDataReader {
+    core;
+    constructor(core) {
+        this.core = core;
+    }
+    readFully(b) {
+        const position = this.core.position();
+        fs.readSync(this.core.fd, b, 0, b.length, position);
+        this.core.seek(position + b.length);
+    }
+    readByte() {
+        const bytes = new Uint8Array(1);
+        this.readFully(bytes);
+        return bytes[0];
+    }
+    readShort() {
+        const bytes = new Uint8Array(2);
+        this.readFully(bytes);
+        const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+        return view.getInt16(0, false);
+    }
+    readInt() {
+        const bytes = new Uint8Array(4);
+        this.readFully(bytes);
+        const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+        return view.getInt32(0, false);
+    }
+    readLong() {
+        const bytes = new Uint8Array(8);
+        this.readFully(bytes);
+        const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+        return Number(view.getBigInt64(0, false));
+    }
+}
+class FileDataWriter {
+    core;
+    constructor(core) {
+        this.core = core;
+    }
+    write(buffer) {
+        const position = this.core.position();
+        fs.writeSync(this.core.fd, buffer, 0, buffer.length, position);
+        this.core.seek(position + buffer.length);
+    }
+    writeByte(v) {
+        this.write(new Uint8Array([v & 0xff]));
+    }
+    writeShort(v) {
+        const buffer = new ArrayBuffer(2);
+        const view = new DataView(buffer);
+        view.setInt16(0, v, false);
+        this.write(new Uint8Array(buffer));
+    }
+    writeLong(v) {
+        const buffer = new ArrayBuffer(8);
+        const view = new DataView(buffer);
+        view.setBigInt64(0, BigInt(v), false);
+        this.write(new Uint8Array(buffer));
+    }
+}

package/dist/core-memory.d.ts

@@ -1,4 +1,4 @@
-import type { Core, DataReader, DataWriter } from './core';
+import type { Core, DataReader, DataWriter } from './core.js';
 export declare class CoreMemory implements Core {
     memory: RandomAccessMemory;
     constructor();