datafolder 1.0.10 → 1.0.12

package/{internal.md → Internal.md}

@@ -2,7 +2,7 @@
 
 The best mechanism to index and partition data is **by the nature of data**. Instead of forcing data to transform, it should seamlessly embrace and fully cater to the **uniqueness** of each and every data to accomplish nature-level efficiency, **mentally and mechanically**. Whatever data is, it follows like fluid.
 
-This is the **design philosophy** behind datafolder, a lightweight schemaless JSON database, featuring an AI-friendly API for simple and human maintainable code generation, millisecond data response, with full index and trigger features supported, and the unique ability to browse and manage data as a folder tree directly in your browser. It works perfectly for a wide range of scenarios, from game/application backends to embedded systems to IoT to edge computing.
+This is the **design philosophy** behind datafolder, a lightning fluid schemaless JSON database, featuring an AI-friendly API for simple and human maintainable code generation, millisecond data response, with full index and trigger features supported, and the unique ability to browse and manage data as a folder tree directly in your browser. Data can be organized naturally into folders, sub folders and files. It works perfectly for a wide range of scenarios, from game/application backends to embedded systems to IoT to edge computing.
 
 ### User Level Data Layout
 
@@ -24,6 +24,13 @@ Airports can be stored as
 airports/{country}/{state}/{airport_id}
 ```
 
+AI training data can be stored as
+
+```
+AITraining/{year}/{month}/{day}/chat_id/prompt
+                                       /answer
+```
+
 When you conceive the data model, you don't need to think about tables, schemas, indexes etc. All you need to know is the nature of the data and then go with the most comfortable one for your mental mind.
 
 ### In-memory Tree Structure

package/README.md

@@ -1,9 +1,13 @@
 # datafolder
 
-datafolder is a lightning fluid JSON database for SSD, featuring an AI-friendly API for simple and human maintainable code generation, millisecond data response, with full index and trigger features supported, and the unique ability to browse and manage data as a folder tree directly in your browser. It works perfectly for a wide range of scenarios, from game/application backends to embedded systems to IoT to edge computing.
+datafolder is a lightning fluid JSON database for high speed SSDs, featuring an AI-friendly API for simple and human maintainable code generation, millisecond data response, with full index and trigger features supported, and the unique ability to browse and manage data as a folder tree directly in your browser. It works perfectly for a wide range of scenarios, from game/application backends to embedded systems to IoT to edge computing.
 
 ### Changelog
 
+1.0.12 Added expire support to expire a file or some properties of the JSON data in future.
+
+1.0.11 Added [SKILL.md](https://github.com/datafolderdev/datafolder/blob/main/SKILL.md) for feeding to AI
+
 1.0.10 Added document [datafolder for high speed SSDs](https://github.com/datafolderdev/datafolder/wiki/datafolder-for-high-speed-SSDs)
 
 1.0.9 Added Typescript type declarations.
@@ -160,6 +164,16 @@ onPromptScoreChanged({ promptId: "student2", feedbackScore: { oldValue: undefine
 onPromptScoreChanged({ promptId: "student1", feedbackScore: { oldValue: 80, newValue: 32 } });
 ```
 
+### 5. Expire Support
+
+```javascript
+dataFolder.expireFile("aitrainingdata/tmp/prompt", 3600); // the file will be deleted after 1 hour
+dataFolder.expireContent("users/login", { sessionId: 300, token: 3600 });
+
+dataFolder.removeFileExpire("aitrainingdata/tmp/prompt", 3600);
+dataFolder.removeContentExpire("users/login", { sessionId: 1, token: 1 });
+```
+
 ## Example
 
 Once you install datafolder via

package/SKILL.md

@@ -0,0 +1,191 @@
+---
+name: datafolder-builder
+description: >
+  Use this skill when building applications on top of datafolder — a lightning fluid JSON database for high speed SSDs, featuring an AI-friendly API for simple and human maintainable code generation, millisecond data response, with full index and trigger features supported, and the unique ability to browse and manage data as a folder tree directly in your browser. Data can be organized naturally into folders, sub folders and files. It works perfectly for a wide range of scenarios, from game/application backends to embedded systems to IoT to edge computing. Covers installation, basic usages, CRUD, batch APIs, powerful query support which auto creates needed indexes, flexible trigger support to monitor any changes.
+  Trigger when a user asks to build, query, or operate with datafolder, write application code against it, or understand how to structure data and queries.
+---
+
+# datafolder Builder Skill
+
+## What Is datafolder?
+
+The best mechanism to index and partition data is **by the nature of data**. Instead of forcing data to transform, it should seamlessly embrace and fully cater to the **uniqueness** of each and every data to accomplish nature-level efficiency, **mentally and mechanically**. Whatever data is, it follows like fluid.
+
+This is the **design philosophy** behind datafolder, a lightning fluid schemaless JSON database, featuring an AI-friendly API for simple and human maintainable code generation, millisecond data response, with full index and trigger features supported, and the unique ability to browse and manage data as a folder tree directly in your browser. Data can be organized naturally into folders, sub folders and files. It works perfectly for a wide range of scenarios, from game/application backends to embedded systems to IoT to edge computing.
+
+## Features
+
+🤖 AI-Friendly, Simple API: Designed for easy code generation by AI and long-term maintainability by humans.
+
+⚡ Extreme Data Access Speed: Supports fully in-memory operation for millisecond response, with configurable persistence to high-speed SSD.
+
+🌐 Built-in Browser Management: Visually browse and manage your data as a folder hierarchy directly within the browser, no separate tools needed.
+
+🧩 Plain JavaScript: Built with plain JavaScript, offers full TypeScript support and safe atomic writes.
+
+🔧 Designed for application/game backends, IoT, and edge computing scenarios, easily deployable on servers, in browsers, and even on embedded hardware.
+
+## User Level Data Layout
+
+Data can be organized naturally into folders, sub folders and files. For example, Orders can be stored in
+
+```
+orders/{year}/{month}/{day}/{order_id}
+```
+
+Flights can be stored as
+
+```
+flights/{from}/{to}/{flight_id}
+```
+
+Airports can be stored as
+
+```
+airports/{country}/{state}/{airport_id}
+```
+
+AI training data can be stored as
+
+```
+AITraining/{year}/{month}/{day}/chat_id/prompt
+                                       /answer
+```
+
+When you conceive the data model, you don't need to think about tables, schemas, indexes etc. All you need to know is the nature of the data and then go with the most comfortable one for your mental mind.
+
+## Installation
+
+```javascript
+  npm install "datafolder"
+```
+
+## How to use
+
+```javascript
+import DataFolder from "datafolder";
+const dataFolder = new DataFolder(); //Specify where to store the data. Default is ./data/sampleDataFolder
+await dataFolder.start(); //start() and stop() are the only two async places. All other code doesn't need await.
+dataFolder.insert("traindata/2026/1/20/chat_0001", {
+  userid: "user_0001",
+  email: "a@b.com",
+  feedbackScore: 5,
+  address: { city: "a", zipcode: 123 },
+});
+dataFolder.insert("traindata/2026/1/20/chat_0001/prompt", "What is async/await?");
+dataFolder.insert("traindata/2026/1/20/chat_0001/answer", "async/await is ...");
+const { email, password, address, feedbackScore } = dataFolder.view("traindata/2026/1/20/chat_0001");
+address.zipcode = 678;
+dataFolder.insert("traindata/2026/1/20/chat_0001", { feedbackScore: feedbackScore + 1, address });
+await dataFolder.stop();
+```
+
+Or operate with dir/file objects:
+
+```javascript
+const file = dataFolder.file("traindata/2026/1/20/chat_0001");
+const { feedbackScore, address } = file.view({ feedbackScore: 1, address: 1 });
+dataFolder.insert(file, { feedbackScore: feedbackScore + 1, address: { city: "c", zipcode: 663 } });
+```
+
+### Simple CRUD APIs
+
+#### Create
+
+```javascript
+dataFolder.createFile("path/to/file");
+dataFolder.createDir("path/to/dir");
+dataFolder.insert("path/to/file", { email, address: { city, zipcode } });
+```
+
+#### Read
+
+```javascript
+const dir = dataFolder.dir("path/to/dir");
+const { subdirList, fileList, sortedSubdirList } = dir;
+const file = dataFolder.file("path/to/file");
+const { email, address } = dataFolder.view("path/to/file", { email: 1, addres: 1 });
+```
+
+#### Update
+
+```javascript
+dataFolder.insert("path/to/file", { email: newEmail, address: { zipcode: newZipcode } });
+dataFolder.insert(file, { feedbackScore: feedbackScore + 1 });
+```
+
+#### Delete
+
+```javascript
+dataFolder.delDir("path/to/dir");
+dataFolder.delFile("path/to/file");
+dataFolder.delContent("path/to/file");
+dataFolder.remove("path/to/file", { email: 1, address: { zipcode: 1 } });
+dataFolder.remove(file, { email: 1, address: { zipcode: 1 } });
+```
+
+### Batch APIs
+
+```javascript
+const [dir, file, { email, feedbackScore }] = dataFolder.fetch
+  .dir("path/to/dir")
+  .file("path/to/file")
+  .view("path/to/file")
+  .run();
+dataFolder.batch
+  .insert("path/to/chat_details", { user: "user_0001", feedbackScore: 100 })
+  .insert("path/to/prompt", "what is LLM?")
+  .insert("path/to/answer", "LLM is AI...")
+  .remove("path/to/file", { address: { city: 1 } })
+  .run();
+```
+
+### Powerful Query Support. Auto Create Needed Indexes
+
+```javascript
+// get all the training data in year 2026 with feedbackScore between 60 and 85.
+dataFolder.queryFiles("traindata/2026/**", { feedbackScore: (x) => x >= 60 && x <= 85 });
+
+// get training data of user_001 from July to Oct in year 2025, 2026 with feedbackScore higher than 90.
+dataFolder.queryFiles(["traindata", { $in: [2025, 2026] }, { $gt: 6, $lte: 10 }], {
+  feedbackScore: { $gt: 90 },
+  userId: "user_0001",
+});
+
+dataFolder.queryFiles("flights/**", { _from: "airports/LAX" });
+dataFolder.queryFiles(["flights", "*", "jan", (x) => x >= 5 && x <= 7], {
+  $or: [{ _from: "airports/BIS" }, { _to: "airports/BIS" }],
+});
+dataFolder.queryFiles(["flights", "*", "feb", { $or: [5, 6, 7] }], {
+  $or: [{ _from: "airports/BIS" }, { _to: "airports/BIS" }],
+});
+dataFolder.queryFiles(["flights", "*", "mar", { $gte: 5, lte: 7 }], {
+  $or: [{ _from: "airports/BIS" }, { _to: "airports/BIS" }],
+});
+dataFolder.queryFilesMulti([
+  [["airbnb", "Loft", "Entire home/apt", "Real Bed"], { address: { street: "Porto, Porto, Portugal" } }],
+  ["airbnb/Loft/**", { minimum_nights: (x) => x >= 60 }],
+]);
+```
+
+### Flexible Trigger Support to Monitor Any Changes
+
+```javascript
+function onPromptScoreChanged({ promptId, feedbackScore }) {
+  const { oldValue, newValue } = feedbackScore;
+  // Logic to handle prompt feedback score change.
+}
+dataFolder.insertTrigger("promptScoreChanged", "traindata/{promptId}", "feedbackScore");
+dataFolder.on("promptScoreChanged", onPromptScoreChanged);
+dataFolder.insert("traindata/prompt_1", { feedbackScore: 80 });
+dataFolder.insert("traindata/prompt_2", { feedbackScore: 63 });
+dataFolder.insert("traindata/prompt_1", { feedbackScore: 32 });
+```
+
+Will trigger the following calls:
+
+```javascript
+onPromptScoreChanged({ promptId: "student1", feedbackScore: { oldValue: undefined, newValue: 80 } });
+onPromptScoreChanged({ promptId: "student2", feedbackScore: { oldValue: undefined, newValue: 63 } });
+onPromptScoreChanged({ promptId: "student1", feedbackScore: { oldValue: 80, newValue: 32 } });
+```

package/SSD.md

@@ -34,7 +34,7 @@ Key Characteristics and Benefits of LSM trees to run on SSDs
 
 ### datafolder, a lightning fluid JSON database, is a natural fit for high speed SSDs
 
-datafolder is [based on LSM trees](https://github.com/datafolderdev/datafolder/wiki/Internals-of-datafolder), but taked to a higher level.
+datafolder is [based on LSM trees](https://github.com/datafolderdev/datafolder/wiki/Internals-of-datafolder), but taken to a higher level.
 
 - Complex JSON data in a single virtual file for a record
 
@@ -42,7 +42,7 @@ The data is just a normal JSON [object](https://github.com/datafolderdev/datafol
 
 - Changes are stored as deltas in append-only mode.
 
-When changes are maded to each virtual file, only [the deltas](https://github.com/datafolderdev/datafolder/wiki/Internals-of-datafolder#accumulated-change-list) are stored. And the delta changes from different virtual files/folders are merged and appended to ACL log files.
+When changes are made to each virtual file, only [the deltas](https://github.com/datafolderdev/datafolder/wiki/Internals-of-datafolder#accumulated-change-list) are stored. And the delta changes from different virtual files/folders are merged and appended to ACL log files.
 
 - Never overwrite. Always append or create new file.
 
@@ -57,4 +57,4 @@ datafolder never overwrites. It always appends to an ACL file, or create new [sn
 	- Snapshot Position Mapping maps a key to its position in the corresponding valueList file.
 	- Snapshot Prefix Position Mapping maps a key prefix to its range position in the corresponding Positing Mapping file.
 
-### Future: Make datafolder runs directly on SSDs without the FTL
+### Future: Make datafolder runs directly on SSDs without the FTL, or Zoned Namespaces(ZNS) SSDs (formerly Open-Channel SSDs)

package/package.json

@@ -1,11 +1,11 @@
 {
 	"name": "datafolder",
-	"version": "1.0.10",
+	"version": "1.0.12",
 	"author": {
 		"name": "lysh",
 		"email": "datafolderdev@outlook.com"
 	},
-	"description": "datafolder is a lightning fluid JSON database for SSD, featuring an AI-friendly API for simple and human maintainable code generation, millisecond data response, with full index and trigger features supported, and the unique ability to browse and manage data as a folder tree directly in your browser. It works perfectly for a wide range of scenarios, from game/application backends to embedded systems to IoT to edge computing.",
+	"description": "datafolder is a lightning fluid JSON database for high speed SSDs, featuring an AI-friendly API for simple and human maintainable code generation, millisecond data response, with full index and trigger features supported, and the unique ability to browse and manage data as a folder tree directly in your browser. It works perfectly for a wide range of scenarios, from game/application backends to embedded systems to IoT to edge computing.",
 	"license": "MIT",
 	"repository": {
 		"type": "git",

package/scripts/flights2.js

@@ -14,6 +14,7 @@ const qyDB = new QyDB(dataFolder, { preloadFoldersAtStart: false });
 run();
 function insertAirports() {
     const data = JSON.parse(readFileSync(join(arangoJsonFolderPath, "airports.json")));
+    console.log(`airports:${data.length}`);
     console.time("insertAirports");
     const { batch } = qyDB;
     for (const record of data) {
@@ -25,6 +26,7 @@ function insertAirports() {
 }
 function insertFlights() {
     const data = JSON.parse(readFileSync(join(arangoJsonFolderPath, "flights.json")));
+    console.log(`flights:${data.length}`);
     console.time("insertFlights");
     const { batch } = qyDB;
     for (let i = 0; i < data.length; ++i) {

package/src/QyAcl.d.ts

@@ -1,14 +1,20 @@
-export declare function delCmdKey(cmdArgAsListObj: any, key: string): void;
-export declare function applyAclChange(aclKeyValueMap: any, cmdArgAsListObj: any): void;
-export declare function iterCmdKeys(cmdArgAsListObj: any, fun: any): void;
+export declare const CmdNames: {
+    M: number;
+    A: number;
+    R: number;
+    D: number;
+};
+export declare function delCmdKey(cmdArgAsListChanges: any, key: string): void;
+export declare function applyAclChange(aclKeyValueMap: any, cmdArgAsListChanges: any): void;
+export declare function iterCmdKeys(cmdArgAsListChanges: any, fun: any): void;
 export declare class QyAclCmdGenerator {
-    cmdArgAsMapObj: {};
+    cmdArgAsMapChanges: any[];
     cmdCount: number;
     batchSize: number;
     constructor();
-    pushCmd(cmdName: string, cmdHashKey: string, cmdArg?: any): void;
-    pushCmdArgAsListObj(cmdArgAsListObj: any): this;
-    takeCmdArgAsListObj(): {};
+    pushCmd(cmdName: number, cmdHashKey: string, cmdArg?: any): void;
+    pushCmdArgAsListObj(cmdArgAsListChanges: any): this;
+    takeCmdArgAsListObj(): any[];
     toAclBuffer(changeId: any): Buffer<ArrayBuffer> | undefined;
 }
 export declare function readAclFile(aclFilePath: any, autoRepairAclFile?: boolean, currentChangeId?: number): Promise<any[]>;