datafolder 1.0.8 → 1.0.10

package/README.md

@@ -4,9 +4,11 @@ datafolder is a lightning fluid JSON database for SSD, featuring an AI-friendly
 
 ### Changelog
 
-1.0.8 Added Typescript type declarations.
+1.0.10 Added document [datafolder for high speed SSDs](https://github.com/datafolderdev/datafolder/wiki/datafolder-for-high-speed-SSDs)
 
-1.0.7 Added trigger editor and [internal.md](https://github.com/datafolderdev/datafolder/wiki/Internals-of-datafolder)
+1.0.9 Added Typescript type declarations.
+
+1.0.7 Added trigger editor and [Internals of datafolder](https://github.com/datafolderdev/datafolder/wiki/Internals-of-datafolder)
 
 1.0.6 Added code mirror as the query editor.
 

package/SSD.md

@@ -0,0 +1,60 @@
+## datafolder, a lightning fluid JSON database to smooth SSD bumps and keep the high speed
+
+### Solid-State Drives (SSDs)
+
+[SSDs](https://www.kingston.com/en/blog/pc-performance/nand-flash-technology-and-ssd) are non-volatile, high-speed solid-state drives that store data on flash memory chips without requiring power, making them ideal for computers and mobile devices. Using NAND logic gates, they provide faster read/write speeds, better durability, and lower power consumption than traditional mechanical hard drives.
+
+But they have physical constraints:
+
+- NAND flash **cannot overwrite** data;
+- it must **erase a whole block** before writing new data.
+
+**High** Speed. But with **bumps**.
+
+### Flash Translation Layer (FTL)
+
+The Flash Translation Layer (FTL) is critical firmware running on an SSD controller that maps logical block addresses (LBAs) from the operating system to physical NAND flash locations. The FTL manages the "write amplification" issue by handling random write operations efficiently, typically using a mapping table cached in DRAM to maintain high speed. It enables SSDs to act like hard drives, managing data mapping, wear leveling, garbage collection, and bad block management to ensure performance and durability.
+
+Key Functions of the FTL:
+
+- Address Mapping: Translates Logical Block Addresses (LBAs) used by the OS to Physical Block Addresses (PBAs) on the NAND flash.
+- Wear Leveling: Distributes write/erase cycles evenly across flash cells to prevent premature failure of specific sectors.
+- Garbage Collection: Cleans up invalid data (data marked for deletion) and consolidates valid data into new blocks, freeing up space.
+- Bad Block Management: Identifies and blocks memory locations that can no longer hold data reliably.
+
+### Log-structured merge-trees (LSM trees)
+
+Log-structured merge-trees (LSM trees) are write-optimized data structures designed for high-throughput, insert-heavy workloads by batching updates in memory and writing them sequentially to disk. They reduce random writes—ideal for SSDs—by avoiding in-place updates, using an in-memory memtable, and periodically merging immutable Sorted String Tables (SSTables).
+
+Key Characteristics and Benefits of LSM trees to run on SSDs
+
+- Sequential IO: LSM trees convert random write operations into sequential writes, which are much faster and more efficient on SSDs compared to traditional B-tree in-place updates, reducing SSD endurance issues.
+- Write-Optimized Structure: By using a "log-structured" approach, incoming data is buffered in a memtable in memory and, when full, flushed as an ordered, immutable file (SSTable) to the SSD, minimizing write amplification.
+- Compaction: To manage storage and improve read performance, files are periodically merged in the background, consolidating data and removing old versions, often referred to as compaction.
+
+### datafolder, a lightning fluid JSON database, is a natural fit for high speed SSDs
+
+datafolder is [based on LSM trees](https://github.com/datafolderdev/datafolder/wiki/Internals-of-datafolder), but taked to a higher level.
+
+- Complex JSON data in a single virtual file for a record
+
+The data is just a normal JSON [object](https://github.com/datafolderdev/datafolder/wiki/Internals-of-datafolder#in-memory-tree-structure). It can be as complex as keeping a customer's whole airbnb listings and comments. Such a big record greatly reduceds read/write amplifications.
+
+- Changes are stored as deltas in append-only mode.
+
+When changes are maded to each virtual file, only [the deltas](https://github.com/datafolderdev/datafolder/wiki/Internals-of-datafolder#accumulated-change-list) are stored. And the delta changes from different virtual files/folders are merged and appended to ACL log files.
+
+- Never overwrite. Always append or create new file.
+
+datafolder never overwrites. It always appends to an ACL file, or create new [snapshot files](https://github.com/datafolderdev/datafolder/wiki/Internals-of-datafolder#snapshot-indexes). Each snapshot is stored in a prefix Position Mapping file, a Position Mapping file, and a Value List file.
+
+- Multi-level Indexing makes it lightning fast.
+
+	- Memory level folder tree indexing
+    	- datafolder uses a tree structure to store the folder/file relationship in memory. Each node has a subdir map and a file map. subdir map stores subdir nodes, which in turn has its subdir map. file map stores the files that are inside this folder. JSON data is stored as file content within each file node. Thanks to V8 engine's highly optimized object [property](https://v8.dev/blog/fast-properties) [access](https://v8.dev/docs/hidden-classes), getting the dir/file node by path and getting properties from JSON data are extremely fast.
+	- memory level JSON property indexing
+    	- For example, we can create "emailAddress" property indexing on "users" folder, and use this index to query for users with a given email address. Arbitrary properties and comparison combinations are supported, like "address.city" + "department" etc.
+	- Snapshot Position Mapping maps a key to its position in the corresponding valueList file.
+	- Snapshot Prefix Position Mapping maps a key prefix to its range position in the corresponding Positing Mapping file.
+
+### Future: Make datafolder runs directly on SSDs without the FTL

package/index.js

@@ -1 +1,2 @@
+export { logger } from "./src/QyLogger.js";
 export { QyDB as default } from "./src/QyDB.js";

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "datafolder",
-	"version": "1.0.8",
+	"version": "1.0.10",
 	"author": {
 		"name": "lysh",
 		"email": "datafolderdev@outlook.com"

package/sample.js

@@ -1,4 +1,4 @@
-import DataFolder from "datafolder";
+import DataFolder, { logger } from "datafolder";
 const dataFolder = new DataFolder();
 
 run();
@@ -18,8 +18,8 @@ async function run() {
 	address.zipcode = 678;
 	dataFolder.insert("traindata/2026/1/20/chat_0001", { feedbackScore: feedbackScore + 1, address });
 	logger.log(dataFolder.view("traindata/2026/1/20/chat_0001", ["feedbackScore", "address"]));
-	logger.log(dataFolder.view("traindata/2026/1/20/chat_0001/prompt"));
-	logger.log(dataFolder.view("traindata/2026/1/20/chat_0001/answer"));
+	logger.debug(dataFolder.view("traindata/2026/1/20/chat_0001/prompt"));
+	logger.warn(dataFolder.view("traindata/2026/1/20/chat_0001/answer"));
 	await dataFolder.stop();
 	await dataFolder.start();
 	{
@@ -27,7 +27,7 @@ async function run() {
 		const { email, feedbackScore } = file.view();
 		logger.log(email, feedbackScore);
 		dataFolder.insert(file, { feedbackScore: feedbackScore + 1 });
-		logger.log(
+		logger.error(
 			file.view(),
 			dataFolder.view("traindata/2026/1/20/chat_0001/prompt"),
 			dataFolder.view("traindata/2026/1/20/chat_0001/answer"),

package/scripts/airbnb.js

@@ -2,6 +2,7 @@ import { join } from "node:path";
 import { open, rm } from "node:fs/promises";
 import { QyDB } from "../src/QyDB.js";
 import { folderOrFileExists, isSimpleType, __dirname } from "../src/QyUtils.js";
+import { logger } from "../src/QyLogger.js";
 import { runQueries } from "./airbnb_queries.js";
 import { downloadFile } from "./downloadFile.js";
 const fileName = "listingsAndReviews.json";

package/scripts/airbnb2.js

@@ -2,6 +2,7 @@ import { join } from "node:path";
 import { open, rm } from "node:fs/promises";
 import { QyDB } from "../src/QyDB.js";
 import { folderOrFileExists, isSimpleType, __dirname } from "../src/QyUtils.js";
+import { logger } from "../src/QyLogger.js";
 import { runQueries } from "./airbnb2_queries.js";
 import { downloadFile } from "./downloadFile.js";
 const fileName = "listingsAndReviews.json";

package/scripts/convertBinFiles.js

@@ -2,6 +2,7 @@ import { join, basename } from "node:path";
 import { readFile, readdir, writeFile } from "node:fs/promises";
 import { iterPMBuffer, loadPrefixPMFile } from "../src/QySnapshots.js";
 import { ensureDir, folderOrFileExists } from "../src/QyUtils.js";
+import { logger } from "../src/QyLogger.js";
 logger.level = "info";
 run();
 async function loadPosMapFile(posMapFilePath) {

package/scripts/express.js

@@ -1,7 +1,8 @@
 import { join } from "node:path";
 import { QyDB } from "../src/QyDB.js";
-import Express from "express";
 import { __dirname } from "../src/QyUtils.js";
+import { logger } from "../src/QyLogger.js";
+import Express from "express";
 const config = {
     port: 3629,
     qyDBDataFolder: "./tmp/QyExpress",

package/scripts/fastify.js

@@ -5,6 +5,7 @@ import fastifyStatic from "@fastify/static";
 import fastifyCompress from "@fastify/compress";
 import { QyDB } from "../src/QyDB.js";
 import { folderOrFileExists, __dirname } from "../src/QyUtils.js";
+import { logger } from "../src/QyLogger.js";
 import Fastify from "fastify";
 const config = {
     port: 3628,
@@ -13,8 +14,8 @@ const config = {
 };
 const { port, staticRoot } = config;
 const fastify = Fastify({ logger: false, forceCloseConnections: true });
-const qyDB = new QyDB(join(__dirname, "../tmp/simulate2"));
-// const qyDB = new QyDB();
+// const qyDB = new QyDB(join(__dirname, "../tmp/simulate2"));
+const qyDB = new QyDB();
 let sigintReceived = false;
 process.on("SIGINT", async () => {
     logger.log("SIGINT signal received: closing HTTP server");

package/scripts/flights.js

@@ -5,6 +5,7 @@ import { readFileSync } from "node:fs";
 import { rm } from "node:fs/promises";
 import { QyDB } from "../src/QyDB.js";
 import { folderOrFileExists } from "../src/QyUtils.js";
+import { logger } from "../src/QyLogger.js";
 import { runQueries } from "./flights_queries.js";
 logger.level = "info";
 const arangoJsonFolderPath = "D:\\scripts\\GraphCourse_DemoData_ArangoDB-2";

package/scripts/flights2.js

@@ -5,6 +5,7 @@ import { readFileSync } from "node:fs";
 import { rm } from "node:fs/promises";
 import { QyDB } from "../src/QyDB.js";
 import { folderOrFileExists } from "../src/QyUtils.js";
+import { logger } from "../src/QyLogger.js";
 import { runQueries } from "./flights2_queries.js";
 logger.level = "info";
 const arangoJsonFolderPath = "D:\\scripts\\GraphCourse_DemoData_ArangoDB-2";

package/scripts/flights_queries.js

@@ -1,4 +1,5 @@
 import { deepEqual } from "node:assert/strict";
+import { logger } from "../src/QyLogger.js";
 // Return the names of all airports one can reach
 // directly (1 step) from Los Angeles International
 // (LAX) following the flights edges:

package/src/QyAcl.d.ts

@@ -6,12 +6,12 @@ export declare class QyAclCmdGenerator {
     cmdCount: number;
     batchSize: number;
     constructor();
-    pushCmd(cmdName: string, cmdHashKey: string, cmdArg: any): void;
+    pushCmd(cmdName: string, cmdHashKey: string, cmdArg?: any): void;
     pushCmdArgAsListObj(cmdArgAsListObj: any): this;
     takeCmdArgAsListObj(): {};
     toAclBuffer(changeId: any): Buffer<ArrayBuffer> | undefined;
 }
-export declare function readAclFile(aclFilePath: any, autoRepairAclFile?: boolean, currentChangeId?: number): Promise<never[]>;
+export declare function readAclFile(aclFilePath: any, autoRepairAclFile?: boolean, currentChangeId?: number): Promise<any[]>;
 export declare function numsToBuffer(nums: any): Buffer<ArrayBuffer>;
 export declare function bufferToNums(buffer: any, offset: any, nums: any): any;
-export declare function readAclBuffer(aclFilePath: any, contentBuffer: any, autoRepairAclFile: any, currentChangeId: any): Promise<never[]>;
+export declare function readAclBuffer(aclFilePath: any, contentBuffer: any, autoRepairAclFile: any, currentChangeId: any): Promise<any[]>;

package/src/QyAcl.js

@@ -66,7 +66,7 @@ class s {
     cmdCount = 0;
     batchSize = 0;
     constructor() {}
-    pushCmd(r, t, e) {
+    pushCmd(r, t, e = void 0) {
         switch (++this.cmdCount, r) {
           case "A":
             o(this, [ "A", t, e ]), f(this, [ "R", t, e ]);
@@ -151,20 +151,20 @@ function f(r, t) {
 
 async function c(t, r = !1, e = 0) {
     try {
-        return await v(t, await u.readFile(t), r, e);
+        return await A(t, await u.readFile(t), r, e);
     } catch (r) {
         throw d.error(`Reading ${t} failed:`, r), r;
     }
 }
 
-let A = [ 1, 2, 4, 8 ];
+let v = [ 1, 2, 4, 8 ];
 
 function l(t) {
     var e = t.length, a = new Array(e), n = new Array(e);
     let i = e << 6, s = 1;
     for (let r = 0; r < e; ++r) {
         var o = a[r] = (o = t[r]) <= 255 ? 0 : o <= 65535 ? 1 : o <= 4294967295 ? 2 : 3, o = (i |= o << (r << 1), 
-        n[r] = A[o]);
+        n[r] = v[o]);
         s += o;
     }
     var f = Buffer.allocUnsafe(s);
@@ -197,7 +197,7 @@ function l(t) {
 function h(t, e, a) {
     var n = t.readUInt8(e), i = (++e, n >> 6 & 3);
     for (let r = 0; r < i; ++r) {
-        var s = A[n >> (r << 1) & 3];
+        var s = v[n >> (r << 1) & 3];
         a[r] = ((r, t, e = 0) => {
             switch (t) {
               case 1:
@@ -217,7 +217,7 @@ function h(t, e, a) {
     return e;
 }
 
-async function v(e, r, a, n) {
+async function A(e, r, a, n) {
     var i = [];
     for (let t = 0; t < r.length; ) {
         var s = [], o = h(r, t, s), [ s, f, c ] = s;
@@ -246,5 +246,5 @@ export {
     c as readAclFile,
     l as numsToBuffer,
     h as bufferToNums,
-    v as readAclBuffer
+    A as readAclBuffer
 };

package/src/QyAclSaver_Worker.d.ts

@@ -1,5 +1,6 @@
 import { QyMessageWorker } from "./QyMessageWorker.ts";
 export default class QyAclSaver_Worker extends QyMessageWorker {
+    qyAclWriter: any;
     constructor(options: any);
     _op_start(aclFilePath: any): Promise<void>;
     _op_stop(): Promise<void>;

package/src/QyAclSaver_Worker.js

@@ -7,6 +7,7 @@ import {
 } from "./QyAclWriter.js";
 
 export default class extends t {
+    qyAclWriter;
     constructor(t) {
         super(t), Object.assign(this, {
             options: t,

package/src/QyAclWriter.d.ts

@@ -1,5 +1,6 @@
 import { QyBinWriter } from "./QyBinWriter.ts";
 export declare class QyAclWriter extends QyBinWriter {
-    constructor(options: any, filePath: any);
+    qyAclCmdGenerator: any;
+    constructor(options: any, filePath?: undefined);
     getBinData(dataQueue: any, changeId: any): any;
 }

package/src/QyAclWriter.js

@@ -7,20 +7,21 @@ import {
 } from "./QyAcl.js";
 
 import {
-    getDefaultOptions as s
+    getDefaultOptions as o
 } from "./QyDefaultOptions.js";
 
 class t extends r {
-    constructor(r, t) {
+    qyAclCmdGenerator;
+    constructor(r, t = void 0) {
         super({
-            ...s("QyAclWriter"),
+            ...o("QyAclWriter"),
             ...r
         }, t), this.qyAclCmdGenerator = new e();
     }
     getBinData(r, t) {
-        var e, s = this.qyAclCmdGenerator;
-        for (e of r) s.pushCmdArgAsListObj(e);
-        return s.toAclBuffer(t);
+        var e, o = this.qyAclCmdGenerator;
+        for (e of r) o.pushCmdArgAsListObj(e);
+        return o.toAclBuffer(t);
     }
 }
 

package/src/QyBatch.d.ts

@@ -1,31 +1,38 @@
 import { QyCD } from "./QyCD.ts";
 export declare class QyBatch extends QyCD {
+    fileLogLevel: any;
+    actionQueue: any;
+    changeId: any;
+    changeList: any;
+    syncList: any;
+    qyAclCmdGenerator: any;
+    qyCmdObjHistory: any;
     constructor(qyCache?: any, options?: any);
-    createDir(namePath: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    createDirP(namePath: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    delDir(namePath: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    delDirP(namePath: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    createFile(namePath: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    createFileP(namePath: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    delFile(namePath: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    delFileP(namePath: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    createIndex(indexHostDirNamePath: any, indexPropPathList: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    createIndexP(indexHostDirNamePath: any, indexPropPathList: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    delIndex(indexHostDirNamePath: any, indexPropPath: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    delIndexP(indexHostDirNamePath: any, indexPropPath: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    insert(namePath: any, content: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    insertP(namePath: any, content: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    remove(namePath: any, content: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    removeP(namePath: any, content: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    replace(namePath: any, content: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    replaceP(namePath: any, content: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    delContent(namePath: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
-    delContentP(namePath: any, parentDir?: import("./QyDir.ts").QyDir | undefined): any;
+    createDir(namePath: any, parentDir?: any): any;
+    createDirP(namePath: any, parentDir?: any): any;
+    delDir(namePath: any, parentDir?: any): any;
+    delDirP(namePath: any, parentDir?: any): any;
+    createFile(namePath: any, parentDir?: any): any;
+    createFileP(namePath: any, parentDir?: any): any;
+    delFile(namePath: any, parentDir?: any): any;
+    delFileP(namePath: any, parentDir?: any): any;
+    createIndex(indexHostDirNamePath: any, indexPropPathList: any, parentDir?: any): any;
+    createIndexP(indexHostDirNamePath: any, indexPropPathList: any, parentDir?: any): any;
+    delIndex(indexHostDirNamePath: any, indexPropPath: any, parentDir?: any): any;
+    delIndexP(indexHostDirNamePath: any, indexPropPath: any, parentDir?: any): any;
+    insert(namePath: any, content: any, parentDir?: any): any;
+    insertP(namePath: any, content: any, parentDir?: any): any;
+    remove(namePath: any, content: any, parentDir?: any): any;
+    removeP(namePath: any, content: any, parentDir?: any): any;
+    replace(namePath: any, content: any, parentDir?: any): any;
+    replaceP(namePath: any, content: any, parentDir?: any): any;
+    delContent(namePath: any, parentDir?: any): any;
+    delContentP(namePath: any, parentDir?: any): any;
     insertTrigger(triggerName: any, fileNamePath: any, propNamePathList: any, modCode: any): any;
     removeTrigger(triggerName: any): any;
     insertRpc(rpcFunName: any, modCode: any): any;
     removeRpc(rpcFunName: any): any;
-    run(needPromise: any): any;
+    run(needPromise?: boolean): any;
     getTopHistoryCmdObj(): any;
     _action_insertTrigger(action: any): boolean;
     _action_removeTrigger(action: any): boolean;
@@ -33,7 +40,7 @@ export declare class QyBatch extends QyCD {
     _action_removeRpc(action: any): boolean;
     _action_createIndex(action: any): void;
     _action_delIndex(action: any): void;
-    _action_delFile(action: any, deletingParentDir: any): boolean;
+    _action_delFile(action: any, deletingParentDir?: boolean): boolean;
     _action_delDir(action: any): boolean;
     _action_delContent(action: any): boolean;
     _action_insert(action: any): boolean;