datafolder 1.0.6 → 1.0.7

package/README.md

@@ -3,6 +3,8 @@
 datafolder is a lightweight JSON database, featuring an AI-friendly API for simple and human maintainable code generation, millisecond data response, with full index and trigger features supported, and the unique ability to browse and manage data as a folder tree directly in your browser. It works perfectly for a wide range of scenarios, from game/application backends to embedded systems to IoT to edge computing.
 
 ### Changelog
+
+1.0.7 Added trigger editor and [internal.md](https://github.com/datafolderdev/datafolder/wiki/Internals-of-datafolder)
 1.0.6 Added code mirror as the query editor.
 
 ## Installation
@@ -189,4 +191,4 @@ Next time to bring up the UI:
 npm run ui
 ```
 
-Complex query examples can be found in the scripts folder source code.
+Complex query examples can be found in the scripts folder source code.

package/internal.md

@@ -0,0 +1,119 @@
+## Internals of datafolder
+
+The best mechanism to index and partition data is **by the nature of data**. Instead of forcing data to transform, it should seamlessly embrace and fully cater to the **uniqueness** of each and every data to accomplish nature-level efficiency, **mentally and mechanically**. Whatever data is, it follows like fluid.
+
+This is the **design philosophy** behind datafolder, a lightweight schemaless JSON database, featuring an AI-friendly API for simple and human maintainable code generation, millisecond data response, with full index and trigger features supported, and the unique ability to browse and manage data as a folder tree directly in your browser. It works perfectly for a wide range of scenarios, from game/application backends to embedded systems to IoT to edge computing.
+
+### User Level Data Layout
+
+Data can be organized naturally into folders, sub folders and files. For example, Orders can be stored in
+
+```
+orders/{year}/{month}/{day}/{order_id}
+```
+
+Flights can be stored as
+
+```
+flights/{from}/{to}/{flight_id}
+```
+
+Airports can be stored as
+
+```
+airports/{country}/{state}/{airport_id}
+```
+
+When you conceive the data model, you don't need to think about tables, schemas, indexes etc. All you need to know is the nature of the data and then goes with the most comfortable one for your mental mind.
+
+### In-memory Tree Structure
+
+datafolder uses a tree structure to store the folder/file relationship in memory. Each node has a subdir map and a file map. subdir map stores subdir nodes, which in turn has its subdir map. file map stores the files that are inside this folder. JSON data is stored as file content within each file node. They are all normal Javascript Objects themselves. Thanks to V8 engine's highly optimized object [property](https://v8.dev/blog/fast-properties) [access](https://v8.dev/docs/hidden-classes), getting the dir/file node by path and getting properties from JSON data are extremely fast. These dir/file node and file content can be loaded beforehand, or loaded on demand. And will be released according to visiting time/count to reduce runtime memory usage.
+
+### Lock Free, Always Sync, Immediate Programming Mode
+
+When dealing with datafolder, you write your Node.js code in a lock free, always sync, immediate programming mode.
+
+- The data are just normal Javascript objects and live within the same memory space of your code. No network traffic, no serialization/deserialization, no overhead. You code directly manipulates the data.
+- No need to worry about whether data is modified by others during your changes. No need to guard data corruption with a transaction.
+- You get the needed data, exam them, make your decisions and changes, and store them back. That is it.
+- datafolder's underlying mechanism guarantees the correctness of running your code in this mode.
+- start() and stop() are the only two APIs that need be called with await. All other APIs are called synchronously.
+
+### Accumulated Change List
+
+datafolder make changes immediately in memory. It calculates deltas of such changes and store these deltas in a log file called acl file. Since only storing deltas, these changes are compact. datafolder utilize Node.js's processes and offload the actual acl storing work to separate process. In fact, this change stream is also sent out to other processes to create snapshots, read-only replicas and to compress snapshots etc.
+These deltas can be of the following types:
+
+1. Adding a subdir or file to a dir.
+2. Deleting a subdir or file from a dir.
+3. Delete a dir or a file itself.
+4. Merging JSON changes.
+
+JSON changes are further separated into the following:
+
+1. Deleting a property
+2. Adding a property with value.
+3. Changing a property to new value.
+
+These deltas are appended to the acl log file in sequence. Once the log file reaches a certain size a new log file will be created and appended to. Storing these acl1, acl2, ...,acln files keep the change history of the whole database. The whole data can be reconstructed by replying these acl files in sequence.
+
+### Snapshots
+
+The whole data changes are stored as acl1, acl2, acl3,..., acln files. But the reconstruct process takes time to read and apply all these acl files. In order to reduce reconstruction time snapshots are introduced. They store the exact shapes and contents of the items changed after a certain acl file. For example,
+
+```
+acl1, snapshot1, acl2, snapshot2, acl3, snapshot3, acl4, acl5, snapshot5, ...
+```
+
+snapshot1 will store the whole data afte applying acl1. It will shadow acl1.
+
+snapshot2 will store the whole changes that are introduced by applying acl2 and will shadow acl2. That is, if a dir/file is changed during acl2, it will be stored as whole in snapshot2 and will shadow the one in snapshot1. If a dir/file is not changed during acl2, it will remain intact in snapshot1.
+
+The same as snapshot3, snapshot5 etc. They only contain changed items after the last snapshots. This way the whole data is constructed by various snapshots and the following acls. Since snapshots are holding the whole items, they don't need reconstruction but only need to be able to read a certain dir/file and content out.
+
+### Snapshot Indexes
+
+Each Snapshot is a big key value pair map. It is stored in two files: posMap file and valueList file.
+
+- posMap file is storing the [start, length] of the content of each key in the valueList file.
+- valueList file is storing the values of this snapshot.
+
+For example,
+posMap file contains
+
+```
+key1 [0, 28]
+key2 [28, 128]
+key3 ...
+```
+
+valueList file contains
+
+```
+content1 (from 0, length 28)
+content2 (from 28, length 128)
+content3 (...)
+```
+
+In this format, when posMap is loaded we can read arbitrary key content from valueList file. But if posMap is big, itself will be indexed using prefixPM file.
+prefixPM file is storing the [start, length] of each prefix (like the first 2 characthers of keys) to the posMap file.
+
+```
+prefix1 [0, 38]
+prefix2 [38, 120]
+```
+
+This way we only need to load prefixPM files in memory. Corresponding posmap content and valueList content can be loaded on-demand.
+
+For example, we can store 20 years of [Hacker](https://news.ycombinator.com/item?id=46435308) [News](https://github.com/DOSAYGO-STUDIO/HackerBook) which is 22GB into datafolder in the layout of
+
+```
+hackernews/{year}/{month}/{day}/{id}
+```
+
+And each file contains the full story and the nested comments in a Javascript object. Even with 22GB contents, since they are partitioned naturally and each folder only contains a few thousands files they can be read via datafolder UI directly within the browser.
+
+### Snapshot Shadowing and Compaction
+
+The same key with updated values will be stored in multiple snapshots and the latest one shadows all previous ones. At some point, thes redundant key/values will take up too much disc space. datafolder can automatically detect the threshold, and save the snapshots into new compacted ones with these redundances removed. These compactions are done in different processes and won't use the main process's time.

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "datafolder",
-	"version": "1.0.6",
+	"version": "1.0.7",
 	"author": {
 		"name": "lysh",
 		"email": "datafolderdev@outlook.com"
@@ -11,7 +11,7 @@
 		"type": "git",
 		"url": "git+https://github.com/datafolderdev/datafolder"
 	},
-	"homepage": "https://github.com/datafolderdev/datafolder",
+	"homepage": "https://github.com/datafolderdev/datafolder/wiki/Internals-of-datafolder",
 	"bugs": {
 		"url": "https://github.com/datafolderdev/datafolder/issues",
 		"email": "datafolderdev@outlook.com"

package/scripts/convertBinFiles.js

@@ -0,0 +1,49 @@
+import { join, basename } from "node:path";
+import { readFile, readdir, writeFile } from "node:fs/promises";
+
+import { iterPMBuffer, loadPrefixPMFile } from "../src/QySnapshots.js";
+import { ensureDir, folderOrFileExists } from "../src/QyUtils.js";
+
+logger.level = "info";
+
+run();
+
+async function loadPosMapFile(posMapFilePath) {
+    const buffer = await readFile(posMapFilePath);
+    const posMap = {};
+    iterPMBuffer(buffer, (key, nums) => (posMap[key] = nums));
+    return posMap;
+}
+
+async function run() {
+    const folderPath = process.argv[2];
+    let destFolderPath = process.argv[3];
+    if (!folderPath || !destFolderPath) {
+        console.log(`Usage: node ${basename(import.meta.filename)} folder/to/convert folder/to/save or inplace`);
+    } else {
+        if (destFolderPath == "inplace") destFolderPath = folderPath;
+        await convertFolder(folderPath, destFolderPath);
+    }
+}
+
+async function convertFolder(folder, destFolder) {
+    const promises = [];
+    for (const item of await readdir(folder, { withFileTypes: true })) {
+        const { name } = item;
+        if (item.isDirectory()) {
+            promises.push(convertFolder(join(folder, name), join(destFolder, name)));
+        } else {
+            const match = /^[0-9]+_(pM|prefixPM)\.txt/.exec(name);
+            if (match) {
+                const filePath = join(folder, name);
+                const destFilePath = join(destFolder, "dump_" + name + ".json");
+                if (await folderOrFileExists(destFilePath)) continue;
+                await ensureDir(destFolder);
+                console.log(`Converting ${filePath} -> ${destFilePath}`);
+                const map = await (match[1] == "pM" ? loadPosMapFile(filePath) : loadPrefixPMFile(filePath));
+                promises.push(writeFile(destFilePath, JSON.stringify(map, null, 2)));
+            }
+        }
+    }
+    await Promise.all(promises);
+}

package/scripts/fastify.js

@@ -16,7 +16,7 @@ const { port, staticRoot } = config;
 
 const fastify = Fastify({ logger: false, forceCloseConnections: true });
 
-const qyDB = new QyDB();
+const qyDB = new QyDB(join(__dirname, "../tmp/QyTrigger/1"));
 
 let sigintReceived = false;
 
@@ -45,7 +45,7 @@ async function insertRpcs() {
 
 async function handleGet(request, reply) {
   const { query, params } = request;
-  const result = await qyDB.callRpc(params.funName, JSON.parse(query.args));
+  const result = await qyDB.callRpc(params.funName, query && query.args ? JSON.parse(query.args) : undefined);
   if (Error.isError(result)) {
     logger.warn(result.message);
   }

package/src/QyCD.js

@@ -38,6 +38,15 @@ class r {
     triggerFile(r) {
         return this.qyCache.getTriggerFile(r);
     }
+    get triggerFileList() {
+        return this.qyCache.triggerFileList;
+    }
+    rpcFile(r) {
+        return this.qyCache.getRpcFile(r);
+    }
+    get rpcFileList() {
+        return this.qyCache.rpcFileList;
+    }
     view(r, e, t = this.currentDir) {
         return i(r) && (r = s(r)), this.viewP(r, e, t);
     }
@@ -50,32 +59,32 @@ class r {
     queryFilesMulti(r, e, t, i = this.currentDir) {
         return this.qyCache.queryFilesMulti(r, e, t, i);
     }
-    queryTree(r, e, t, i, s, n = this.currentDir) {
-        return u(this.qyCache.queryFiles(r, e, i, s, n), t);
+    queryTree(r, e, t, i, s, h = this.currentDir) {
+        return n(this.qyCache.queryFiles(r, e, i, s, h), t);
     }
     queryTreeMulti(r, e, t, i, s = this.currentDir) {
-        return u(this.qyCache.queryFilesMulti(r, t, i, s), e);
+        return n(this.qyCache.queryFilesMulti(r, t, i, s), e);
     }
     resetCurrentDir() {
         this.currentDir = void 0;
     }
     fileListToTree(r, e) {
-        return u(r, e);
+        return n(r, e);
     }
 }
 
-function u(r, t) {
+function n(r, t) {
     var i, s = {};
     for (i of r) {
-        var n, u = i.parentList;
+        var h, n = i.parentList;
         let e = s;
-        for (n of u) {
+        for (h of n) {
             var {
-                name: h,
+                name: u,
                 fullPathHash: c
-            } = n;
+            } = h;
             let r = e.subdirMap;
-            r = r || (e.subdirMap = {}), e = (e = r[h]) || (r[h] = {
+            r = r || (e.subdirMap = {}), e = (e = r[u]) || (r[u] = {
                 fullPathHash: c
             });
         }

package/src/QyCache.js

@@ -193,12 +193,18 @@ class t extends r {
     getOrCreateTriggerFile(r) {
         return this.triggerRootDir.getOrCreateFile(r);
     }
+    get triggerFileList() {
+        return this.triggerRootDir.fileList;
+    }
     getRpcFile(r) {
         return this.rpcRootDir.getFile(r);
     }
     getOrCreateRpcFile(r) {
         return this.rpcRootDir.getOrCreateFile(r);
     }
+    get rpcFileList() {
+        return this.rpcRootDir.fileList;
+    }
     insertTriggerToTree(r, e, t, i) {
         var {
             rootDir: o,

package/src/QyFetch.js

@@ -7,7 +7,7 @@ class e extends r {
         super(r), Object.assign(this, {
             stack: []
         });
-        for (let s of [ "dir", "indexDir", "file", "view", "queryFiles", "queryFilesMulti", "queryTree", "queryTreeMulti" ]) this[s] = (...r) => {
+        for (let s of [ "dir", "indexDir", "file", "view", "triggerFile", "triggerFiles", "rpcFile", "rpcFiles", "queryFiles", "queryFilesMulti", "queryTree", "queryTreeMulti" ]) this[s] = (...r) => {
             return e = this, r = super[s].apply(this, r), e.stack.push(r), e;
             var e;
         };

package/src/QyKVData.js

@@ -83,7 +83,11 @@ class a {
             compactInterval: s,
             unloadInterval: r
         }), n.bgLoadPosMaps ? (async a => {
-            for (var t of Object.keys(a.qySnapshots.numToSnapshotMap).sort((a, t) => t - a)) {
+            var t, {
+                qySnapshots: o,
+                options: e
+            } = a;
+            for (t of Object.keys(o.numToSnapshotMap).sort((a, t) => t - a)) {
                 if (await async function s(p, i, r, h, c, l) {
                     if (p.isStopping) return void (l && l.close());
                     let d = p.qySnapshots;
@@ -113,7 +117,7 @@ class a {
                 }(a, t), a.isStopping) break;
                 if (await c(0), a.isStopping) break;
             }
-            !a.isStopping && a.options.compactAtStart && l(a);
+            !a.isStopping && e.compactAtStart && l(a);
         })(this) : n.compactAtStart && l(this), this;
     }
     async stop() {
@@ -126,8 +130,8 @@ class a {
             unloadInterval: n,
             qyKVDataCleaner: s
         } = this;
-        clearInterval(e), clearInterval(n), await a.stop(), o && o.closeAllFDs(), 
-        await t.stop(), await s.stop();
+        clearInterval(e), clearInterval(n), this.onSnapshotSavedChangeId(await a.stop()), 
+        o && o.closeAllFDs(), await t.stop(), await s.stop();
     }
     save(a, t, o, e) {
         var n = this.qyKVDataSaver;
@@ -156,15 +160,20 @@ class a {
         o[a] = void 0;
     }
     onSnapshotSavedChangeId(a, t) {
-        if (this.snapshotSavedChangeId = a, 3 == this.compactState && a >= this.compactMaxChangeId && (K(this), 
-        this.compactState = 1), t) {
-            var o, a = t, e = this.qySnapshots, n = e.keyToSnapshotMap;
-            for (o in a.posMap) {
-                var s = n[o];
-                s && e.removeKeyFromSnapshot(s, o);
+        this.snapshotSavedChangeId = a;
+        var {
+            compactState: o,
+            compactMaxChangeId: e,
+            qyKVDataSaver: n
+        } = this;
+        if (3 == o && e <= a && (K(this), this.compactState = 1), t) {
+            var s, o = t, p = this.qySnapshots, i = p.keyToSnapshotMap;
+            for (s in o.posMap) {
+                var r = i[s];
+                r && p.removeKeyFromSnapshot(r, s);
             }
-            e.addMaxSnapshot(a), logger.info("addPMOnlySnapshot: " + e.snapshotMaxChangeId), 
-            this.qyKVDataSaver.releaseSnapshot(t.snapshotNum);
+            p.addMaxSnapshot(o), logger.info("addPMOnlySnapshot: " + p.snapshotMaxChangeId), 
+            n.releaseSnapshot(t.snapshotNum);
         }
     }
     onCompactUpdates(a) {

package/src/QyKVDataSaver_Worker.js

@@ -19,7 +19,7 @@ import {
 } from "./QySnapshotSaver.js";
 
 import {
-    delCmdKey as m
+    delCmdKey as u
 } from "./QyAcl.js";
 
 class s extends a {
@@ -46,7 +46,7 @@ class s extends a {
             maxChangeId: a,
             snapshotMaxChangeId: a,
             currentAclNum: (t || s) + 1
-        }), o.start(u(this)), n.start(a, t, e);
+        }), o.start(m(this)), n.start(a, t, e);
     }
     async _op_stop() {
         var {
@@ -57,8 +57,8 @@ class s extends a {
             maxChangeId: o,
             snapshotMaxChangeId: n
         } = this;
-        this.isStopping = !0, await a.stop(), s.saveSnapshotAtStop && n < o && (this.isSavingSnapshot && await new Promise(a => this.saveSnapshotResolve = a), 
-        logger.log(`Saving snapshot ${e} at stop.`), t.saveSnapshot(e)), await t.stop();
+        return this.isStopping = !0, await a.stop(), s.saveSnapshotAtStop && n < o && (this.isSavingSnapshot && await new Promise(a => this.saveSnapshotResolve = a), 
+        logger.log(`Saving snapshot ${e} at stop.`), t.saveSnapshot(e)), t.stop();
     }
     callSave(a, s, t) {
         return r(this, a, s, t, !0);
@@ -88,16 +88,16 @@ function r(a, s, t, e, o) {
     var l = t, c = e;
     if (c) {
         var g = c.length;
-        for (let a = 0; a < g; a += 2) m(l, c[a]);
+        for (let a = 0; a < g; a += 2) u(l, c[a]);
     }
     return r.castSave(s, t, e), a.maxChangeId = s, !S && s - i >= h.snapshotMaxChangeCount && (++a.currentAclNum, 
-    n.switch(u(a)), r.saveSnapshot(p), Object.assign(a, {
+    n.switch(m(a)), r.saveSnapshot(p), Object.assign(a, {
         snapshotMaxChangeId: s,
         isSavingSnapshot: !0
     })), v;
 }
 
-function u(a) {
+function m(a) {
     return t(a.aclFolder, a.currentAclNum + "_acl.txt");
 }
 

package/src/QySnapshotCompactor_Worker.js

@@ -3,7 +3,7 @@ import {
 } from "node:path";
 
 import {
-    open as h
+    open as f
 } from "node:fs/promises";
 
 import {
@@ -19,7 +19,8 @@ import {
 } from "./QyKVDataCleaner.js";
 
 import {
-    QySnapshots as n
+    QySnapshots as n,
+    iterPMBuffer as h
 } from "./QySnapshots.js";
 
 import {
@@ -66,7 +67,7 @@ class s extends a {
                     info: n,
                     prefixPM: i
                 } = s;
-                (null == i || r(i) || await f(a, s)) && (await t.savePosMap(s), 
+                (null == i || r(i) || await M(a, s)) && (await t.savePosMap(s), 
                 e[p] = n, o[p] = s.prefixPM);
             })(this, a, s)));
         }
@@ -77,7 +78,7 @@ class s extends a {
                     qySnapshots: e,
                     vLInfoMap: p
                 } = a;
-                await f(a, s) && (await M(a, s), await e.saveKeyValueMap(s), {
+                await M(a, s) && (await m(a, s), await e.saveKeyValueMap(s), {
                     snapshotNum: a,
                     info: e,
                     posMap: s,
@@ -97,12 +98,12 @@ class s extends a {
                 snapshotNum: r,
                 info: f,
                 keyValueMap: h
-            } = (await M(a, i), i);
+            } = (await m(a, i), i);
             Object.assign(i, {
                 posMap: void 0,
                 prefixPM: void 0
             });
-            for (t of s) t != i && (n.push(t.snapshotNum), await M(a, t), Object.assign(h, t.keyValueMap), 
+            for (t of s) t != i && (n.push(t.snapshotNum), await m(a, t), Object.assign(h, t.keyValueMap), 
             Object.assign(t, {
                 keyValueMap: void 0,
                 posMap: void 0,
@@ -142,23 +143,23 @@ class s extends a {
     }
 }
 
-async function f(a, s) {
-    var o = a.qySnapshots;
+async function M(a, s) {
+    a = a.qySnapshots;
     let {
-        prefixPM: t,
-        posMap: e
+        prefixPM: o,
+        posMap: t
     } = s;
-    var p, n = o.getPMFilePath(s), i = await h(n);
-    for (p of Object.keys(t).sort()) {
-        var [ r, f ] = t[p], r = await l(n, i, r, f);
-        if (!r) return void i.close();
-        o.iterPMBuffer(r, (a, s) => e[a] = s);
+    var e, p = a.getPMFilePath(s), n = await f(p);
+    for (e of Object.keys(o).sort()) {
+        var [ i, r ] = o[e], i = await l(p, n, i, r);
+        if (!i) return n.close(), !1;
+        h(i, (a, s) => t[a] = s);
     }
-    return i.close(), !0;
+    return n.close(), !0;
 }
 
-async function M(a, s) {
-    var o, a = a.qySnapshots, t = s.posMap, e = s.keyValueMap = {}, p = a.getVLFilePath(s), n = await h(p);
+async function m(a, s) {
+    var o, a = a.qySnapshots, t = s.posMap, e = s.keyValueMap = {}, p = a.getVLFilePath(s), n = await f(p);
     for (o of Object.keys(t).sort()) {
         var [ i, r ] = t[o];
         if (0 < r) {