document-dataply 0.0.2-alpha.3 → 0.0.2

package/README.md

@@ -4,23 +4,21 @@
 > **This project is currently in the Alpha stage.**
 > APIs and internal structures may change significantly between versions. Use with caution in production environments.
 
-`document-dataply` is a high-performance, document-oriented database library built on top of the [`dataply`](https://github.com/izure1/dataply) record storage engine. It provides a structured way to store, index, and query JSON-like documents with support for transactions and complex field indexing.
+`document-dataply` is a high-performance document-oriented database library built on top of the [`dataply`](https://github.com/izure1/dataply) record storage engine. It provides a structured way to store, index, and query JSON-style documents, supporting transactions and complex field indexing.
 
-## Features
+## Key Features
 
-- **Document-Oriented**: Store and retrieve JSON-like documents.
-- **B+Tree Indexing**: High-performance lookups using B+Tree indexing engine.
+- **Document-Oriented**: Store and retrieve JSON-style documents.
+- **B+Tree Indexing**: Supports high-performance lookups using a B+Tree indexing engine.
 - **Deep Indexing**: Index nested object fields and specific array elements (e.g., `user.profile.name` or `tags.0`).
-- **Flexible Indexing Policy**: Support for full re-indexing of existing data or incremental indexing for future data.
+- **Flexible Indexing Policies**: Supports full re-indexing for existing data or incremental indexing for future data.
 - **Transactions**: ACID-compliant transactions for atomic operations.
-- **Rich Querying**: Support for comparison operators (`lt`, `gt`, `equal`, etc.) and pattern matching (`like`).
+- **Rich Querying**: Supports comparison operators (`lt`, `gt`, `equal`, etc.) and pattern matching (`like`).
 
 ## Installation
 
 ```bash
 npm install document-dataply
-# or
-yarn add document-dataply
 ```
 
 ## Quick Start
@@ -28,39 +26,48 @@ yarn add document-dataply
 ```typescript
 import { DocumentDataply } from 'document-dataply';
 
+type MyDocument = {
+  name: string;
+  age: number;
+  tags: string[];
+}
+
 async function main() {
-  const db = new DocumentDataply<{
-    name: string;
-    age: number;
-    tags: string[];
-  }>('my-database.db', {
+  const db = new DocumentDataply<MyDocument>('my-database.db', {
     wal: 'my-database.wal',
     indices: {
-      name: true, // Index existing and new data
+      name: true, // Index both existing and new data
       age: false, // Index only new data
       'tags.0': true // Index the first element of the 'tags' array
     }
   });
 
-  // Initialize the database
+  // Initialize database
   await db.init();
 
-  // Insert a document
+  // Insert document
   const id = await db.insert({
     name: 'John Doe',
     age: 30,
     tags: ['admin', 'developer']
   });
 
-  // Query documents
-  const results = await db.select({
+  // Query document
+  const query = db.select({
     name: 'John Doe',
     age: { gte: 25 }
-  });
+  })
 
-  console.log(results);
+  // Get all results
+  const allResults = await query.drain();
+  // Or iterate through results
+  for await (const doc of query.stream) {
+    console.log(doc);
+  }
 
-  // Close the database
+  console.log(allResults);
+
+  // Close database
   await db.close();
 }
 
@@ -71,17 +78,17 @@ main();
 
 ### Indexing Policies
 
-When defining indices in the constructor, you can specify a boolean value:
+When defining indices in the constructor, you can specify a boolean value.
 
-- `true`: The library will index all existing documents for this field during `init()` and all subsequent insertions.
-- `false`: The library will only index documents inserted after this configuration.
+- `true`: The library indexes all existing documents for that field during `init()`, and also indexes all subsequent insertions.
+- `false`: The library only indexes documents inserted after this configuration.
 
 > [!NOTE]
-> `db.init()` automatically performs the backfilling process for fields marked as `true`.
+> `db.init()` automatically performs a backfilling process for fields marked as `true`.
 
 ### Batch Insertion
 
-For inserting multiple documents efficiently:
+To efficiently insert multiple documents, use the following:
 
 ```typescript
 const ids = await db.insertBatch([
@@ -92,7 +99,7 @@ const ids = await db.insertBatch([
 
 ### Querying
 
-`document-dataply` supports various comparison operators:
+`document-dataply` supports various comparison operators.
 
 | Operator | Description |
 | :--- | :--- |
@@ -102,8 +109,8 @@ const ids = await db.insertBatch([
 | `gte` | Greater than or equal to |
 | `equal` | Equal to |
 | `notEqual` | Not equal to |
-| `like` | SQL-like pattern matching (e.g., `Jo%`) |
-| `or` | Array of value where at least one must be met |
+| `like` | SQL-style pattern matching (e.g., `Jo%`) |
+| `or` | If any value in the array is satisfied |
 
 Example of a complex query:
 ```typescript
@@ -111,23 +118,23 @@ const users = await db.select({
   age: { gt: 18, lt: 65 },
   'address.city': 'Seoul',
   tags: { or: ['vip', 'premium'] }
-});
+}).drain();
 ```
 
 > [!IMPORTANT]
-> **Query Constraints**: You can only use query conditions (`lt`, `gt`, `equal`, etc.) on fields that have been explicitly indexed in the constructor. 
+> **Query Constraints**: Query conditions (`lt`, `gt`, `equal`, etc.) can only be used on fields explicitly indexed in the constructor.
 > 
-> **If a field in the query is not indexed, its condition will be ignored.**
+> **If a field in the query is not indexed, that condition will be ignored.**
 > 
-> If you need to filter by a non-indexed field, you must retrieve the documents first and then use JavaScript's native `.filter()` method:
-> ```typescript
-> const results = await db.select({ /* indexed fields only */ });
-> const filtered = results.filter(doc => doc.unindexedField === 'some-value');
-> ```
+> If you need to filter by unindexed fields, you should first retrieve the documents and then use JavaScript's native `.filter()` method.
+```typescript
+const results = await db.select({ /* indexed fields only */ }).drain();
+const filtered = results.filter(doc => doc.unindexedField === 'some-value');
+```
 
 ### Transactions
 
-Use transactions to ensure atomicity for multiple operations:
+To ensure the atomicity of multiple operations, use transactions.
 
 ```typescript
 const tx = db.createTransaction();
@@ -141,31 +148,45 @@ try {
 }
 ```
 
+## Tips and Advanced Features
+
+For more information on performance optimization and advanced features, see [TIPS.md](./docs/TIPS.md).
+
+- **Query Optimization**: Automatic index selection for maximum performance.
+- **Sorting and Pagination**: Detailed usage of `limit`, `orderBy`, and `sortOrder`.
+- **Memory Management**: When to use `stream` vs `drain()`.
+- **Performance**: Optimizing bulk data insertion using `insertBatch`.
+- **Indexing Policies**: Deep dive into index backfilling and configuration.
+
 ## API Reference
 
 ### `new DocumentDataply<T>(file, options)`
 Creates a new database instance. `T` defines the document structure.
+`options.indices` is an object where keys are field names and values are booleans indicating whether to index.
 
 ### `db.init()`
 Initializes the database, sets up internal metadata, and prepares indices.
 
 ### `db.insert(document, tx?)`
-Inserts a single document. Returns the document's `_id` (`number`).
+Inserts a single document. Returns the `_id` (`number`) of the document.
 
 ### `db.insertBatch(documents, tx?)`
-Inserts multiple documents efficiently. Returns an array of `_id`s (`number[]`).
+Inserts multiple documents efficiently. Returns an array of `_ids` (`number[]`).
 
-### `db.select(query, limit?, tx?)`
-Retrieves documents matching the query. `limit` defaults to `Infinity`.
+### `db.select(query, options?, tx?)`
+Searches for documents matching the query.
+Returns an object `{ stream, drain }`.
+- `stream`: An async iterator to traverse results one by one.
+- `drain()`: A promise that resolves to an array of all matching documents.
 
 ### `db.getMetadata(tx?)`
-Returns physical storage information (page count, row count, etc.).
+Returns physical storage information (number of pages, number of rows, etc.).
 
 ### `db.createTransaction()`
 Returns a new `Transaction` object.
 
 ### `db.close()`
-Flushes changes and closes the database file.
+Flushes changes and closes the database files.
 
 ## License
 

package/dist/cjs/index.js

@@ -1795,6 +1795,31 @@ var require_cjs = __commonJS({
         }
         return true;
       }
+      /**
+       * Selects the best driver key from a condition object.
+       * The driver key determines the starting point and traversal direction for queries.
+       * 
+       * @param condition The condition to analyze.
+       * @returns The best driver key or null if no valid key found.
+       */
+      getDriverKey(condition) {
+        if ("primaryEqual" in condition) return "primaryEqual";
+        if ("equal" in condition) return "equal";
+        if ("gt" in condition) return "gt";
+        if ("gte" in condition) return "gte";
+        if ("lt" in condition) return "lt";
+        if ("lte" in condition) return "lte";
+        if ("primaryGt" in condition) return "primaryGt";
+        if ("primaryGte" in condition) return "primaryGte";
+        if ("primaryLt" in condition) return "primaryLt";
+        if ("primaryLte" in condition) return "primaryLte";
+        if ("like" in condition) return "like";
+        if ("notEqual" in condition) return "notEqual";
+        if ("primaryNotEqual" in condition) return "primaryNotEqual";
+        if ("or" in condition) return "or";
+        if ("primaryOr" in condition) return "primaryOr";
+        return null;
+      }
       constructor(rootTx, mvccRoot, mvcc, strategy, comparator, option) {
         this.rootTx = rootTx === null ? this : rootTx;
         this.mvccRoot = mvccRoot;
@@ -2235,8 +2260,8 @@ var require_cjs = __commonJS({
         }
         return void 0;
       }
-      *keysStream(condition, filterValues, limit) {
-        const stream = this.whereStream(condition, limit);
+      *keysStream(condition, filterValues, limit, order = "asc") {
+        const stream = this.whereStream(condition, limit, order);
         const intersection = filterValues && filterValues.size > 0 ? filterValues : null;
         for (const [key] of stream) {
           if (intersection && !intersection.has(key)) {
@@ -2245,30 +2270,20 @@ var require_cjs = __commonJS({
           yield key;
         }
       }
-      *whereStream(condition, limit) {
-        let driverKey = null;
-        if ("primaryEqual" in condition) driverKey = "primaryEqual";
-        else if ("equal" in condition) driverKey = "equal";
-        else if ("gt" in condition) driverKey = "gt";
-        else if ("gte" in condition) driverKey = "gte";
-        else if ("lt" in condition) driverKey = "lt";
-        else if ("lte" in condition) driverKey = "lte";
-        else if ("primaryGt" in condition) driverKey = "primaryGt";
-        else if ("primaryGte" in condition) driverKey = "primaryGte";
-        else if ("primaryLt" in condition) driverKey = "primaryLt";
-        else if ("primaryLte" in condition) driverKey = "primaryLte";
-        else if ("like" in condition) driverKey = "like";
-        else if ("notEqual" in condition) driverKey = "notEqual";
-        else if ("primaryNotEqual" in condition) driverKey = "primaryNotEqual";
-        else if ("or" in condition) driverKey = "or";
-        else if ("primaryOr" in condition) driverKey = "primaryOr";
+      *whereStream(condition, limit, order = "asc") {
+        const driverKey = this.getDriverKey(condition);
         if (!driverKey) return;
         const value = condition[driverKey];
-        const startNode = this.verifierStartNode[driverKey](value);
-        const endNode = this.verifierEndNode[driverKey](value);
-        const direction = this.verifierDirection[driverKey];
+        let startNode = this.verifierStartNode[driverKey](value);
+        let endNode = this.verifierEndNode[driverKey](value);
+        let direction = this.verifierDirection[driverKey];
         const comparator = this.verifierMap[driverKey];
         const earlyTerminate = this.verifierEarlyTerminate[driverKey];
+        if (order === "desc") {
+          startNode = endNode ?? this.rightestNode();
+          endNode = null;
+          direction *= -1;
+        }
         const generator = this.getPairsGenerator(
           value,
           startNode,
@@ -2299,16 +2314,16 @@ var require_cjs = __commonJS({
           }
         }
       }
-      keys(condition, filterValues) {
+      keys(condition, filterValues, order = "asc") {
         const set = /* @__PURE__ */ new Set();
-        for (const key of this.keysStream(condition, filterValues)) {
+        for (const key of this.keysStream(condition, filterValues, void 0, order)) {
           set.add(key);
         }
         return set;
       }
-      where(condition) {
+      where(condition, order = "asc") {
         const map = /* @__PURE__ */ new Map();
-        for (const [key, value] of this.whereStream(condition)) {
+        for (const [key, value] of this.whereStream(condition, void 0, order)) {
           map.set(key, value);
         }
         return map;
@@ -3064,8 +3079,8 @@ var require_cjs = __commonJS({
         }
         return void 0;
       }
-      async *keysStream(condition, filterValues, limit) {
-        const stream = this.whereStream(condition, limit);
+      async *keysStream(condition, filterValues, limit, order = "asc") {
+        const stream = this.whereStream(condition, limit, order);
         const intersection = filterValues && filterValues.size > 0 ? filterValues : null;
         for await (const [key] of stream) {
           if (intersection && !intersection.has(key)) {
@@ -3074,30 +3089,20 @@ var require_cjs = __commonJS({
           yield key;
         }
       }
-      async *whereStream(condition, limit) {
-        let driverKey = null;
-        if ("primaryEqual" in condition) driverKey = "primaryEqual";
-        else if ("equal" in condition) driverKey = "equal";
-        else if ("gt" in condition) driverKey = "gt";
-        else if ("gte" in condition) driverKey = "gte";
-        else if ("lt" in condition) driverKey = "lt";
-        else if ("lte" in condition) driverKey = "lte";
-        else if ("primaryGt" in condition) driverKey = "primaryGt";
-        else if ("primaryGte" in condition) driverKey = "primaryGte";
-        else if ("primaryLt" in condition) driverKey = "primaryLt";
-        else if ("primaryLte" in condition) driverKey = "primaryLte";
-        else if ("like" in condition) driverKey = "like";
-        else if ("notEqual" in condition) driverKey = "notEqual";
-        else if ("primaryNotEqual" in condition) driverKey = "primaryNotEqual";
-        else if ("or" in condition) driverKey = "or";
-        else if ("primaryOr" in condition) driverKey = "primaryOr";
+      async *whereStream(condition, limit, order = "asc") {
+        const driverKey = this.getDriverKey(condition);
         if (!driverKey) return;
         const value = condition[driverKey];
-        const startNode = await this.verifierStartNode[driverKey](value);
-        const endNode = await this.verifierEndNode[driverKey](value);
-        const direction = this.verifierDirection[driverKey];
+        let startNode = await this.verifierStartNode[driverKey](value);
+        let endNode = await this.verifierEndNode[driverKey](value);
+        let direction = this.verifierDirection[driverKey];
         const comparator = this.verifierMap[driverKey];
         const earlyTerminate = this.verifierEarlyTerminate[driverKey];
+        if (order === "desc") {
+          startNode = endNode ?? await this.rightestNode();
+          endNode = null;
+          direction *= -1;
+        }
         const generator = this.getPairsGenerator(
           value,
           startNode,
@@ -3128,16 +3133,16 @@ var require_cjs = __commonJS({
           }
         }
       }
-      async keys(condition, filterValues) {
+      async keys(condition, filterValues, order = "asc") {
         const set = /* @__PURE__ */ new Set();
-        for await (const key of this.keysStream(condition, filterValues)) {
+        for await (const key of this.keysStream(condition, filterValues, void 0, order)) {
           set.add(key);
         }
         return set;
       }
-      async where(condition) {
+      async where(condition, order = "asc") {
         const map = /* @__PURE__ */ new Map();
-        for await (const [key, value] of this.whereStream(condition)) {
+        for await (const [key, value] of this.whereStream(condition, void 0, order)) {
           map.set(key, value);
         }
         return map;
@@ -8855,6 +8860,9 @@ var require_cjs = __commonJS({
       get() {
         return this.storage.getStore();
       }
+      stream(tx, callback) {
+        return this.storage.run(tx, callback);
+      }
     };
     var DataplyAPI2 = class {
       constructor(file, options) {
@@ -9072,6 +9080,38 @@ var require_cjs = __commonJS({
         }
         return result;
       }
+      /**
+       * Runs a generator callback function within a transaction context.
+       * Similar to runWithDefault but allows yielding values from an AsyncGenerator.
+       * If no transaction is provided, a new transaction is created.
+       * The transaction is committed if the generator completes successfully,
+       * or rolled back if an error occurs.
+       * @param callback The generator callback function to run within the transaction context.
+       * @param tx The transaction to use. If not provided, a new transaction is created.
+       * @returns An AsyncGenerator that yields values from the callback.
+       */
+      async *streamWithDefault(callback, tx) {
+        const isInternalTx = !tx;
+        if (!tx) {
+          tx = this.createTransaction();
+        }
+        let hasError = false;
+        try {
+          const generator = this.txContext.stream(tx, () => callback(tx));
+          for await (const value of generator) {
+            yield value;
+          }
+        } catch (error) {
+          hasError = true;
+          if (isInternalTx) {
+            await tx.rollback();
+          }
+          throw error;
+        }
+        if (!hasError && isInternalTx) {
+          await tx.commit();
+        }
+      }
       /**
        * Retrieves metadata from the dataply.
        * @returns Metadata of the dataply.
@@ -9369,12 +9409,12 @@ var DocumentSerializeStrategyAsync = class extends import_dataply.SerializeStrat
   async readHead() {
     const tx = this.txContext.get();
     const metadata = await this.api.getDocumentInnerMetadata(tx);
-    const indexInfo = metadata.indecies[this.treeKey];
+    const indexInfo = metadata.indices[this.treeKey];
     if (!indexInfo) return null;
     const headPk = indexInfo[0];
     if (headPk === -1) {
       const pk = await this.api.insertAsOverflow("__BPTREE_HEAD_PLACEHOLDER__", false, tx);
-      metadata.indecies[this.treeKey][0] = pk;
+      metadata.indices[this.treeKey][0] = pk;
       await this.api.updateDocumentInnerMetadata(metadata, tx);
       return null;
     }
@@ -9385,7 +9425,7 @@ var DocumentSerializeStrategyAsync = class extends import_dataply.SerializeStrat
   async writeHead(head) {
     const tx = this.txContext.get();
     const metadata = await this.api.getDocumentInnerMetadata(tx);
-    const indexInfo = metadata.indecies[this.treeKey];
+    const indexInfo = metadata.indices[this.treeKey];
     if (!indexInfo) {
       throw new Error(`Index info not found for tree: ${this.treeKey}. Initialization should be handled outside.`);
     }
@@ -9424,7 +9464,7 @@ async function catchPromise(promise) {
 
 // src/core/document.ts
 var DocumentDataplyAPI = class extends import_dataply3.DataplyAPI {
-  indecies = {};
+  indices = {};
   trees = /* @__PURE__ */ new Map();
   comparator = new DocumentValueComparator();
   pendingBackfillFields = [];
@@ -9441,18 +9481,18 @@ var DocumentDataplyAPI = class extends import_dataply3.DataplyAPI {
         throw new Error("Document metadata verification failed");
       }
       const metadata = await this.getDocumentInnerMetadata(tx);
-      const optionsIndecies = options.indecies ?? {};
-      const targetIndecies = {
-        ...optionsIndecies,
+      const optionsIndices = options.indices ?? {};
+      const targetIndices = {
+        ...optionsIndices,
         _id: true
       };
       const backfillTargets = [];
       let isMetadataChanged = false;
-      for (const field in targetIndecies) {
-        const isBackfillEnabled = targetIndecies[field];
-        const existingIndex = metadata.indecies[field];
+      for (const field in targetIndices) {
+        const isBackfillEnabled = targetIndices[field];
+        const existingIndex = metadata.indices[field];
         if (!existingIndex) {
-          metadata.indecies[field] = [-1, isBackfillEnabled];
+          metadata.indices[field] = [-1, isBackfillEnabled];
           isMetadataChanged = true;
           if (isBackfillEnabled && !isNewlyCreated) {
             backfillTargets.push(field);
@@ -9460,11 +9500,11 @@ var DocumentDataplyAPI = class extends import_dataply3.DataplyAPI {
         } else {
           const [_pk, isMetaBackfillEnabled] = existingIndex;
           if (isBackfillEnabled && !isMetaBackfillEnabled) {
-            metadata.indecies[field][1] = true;
+            metadata.indices[field][1] = true;
             isMetadataChanged = true;
             backfillTargets.push(field);
           } else if (!isBackfillEnabled && isMetaBackfillEnabled) {
-            metadata.indecies[field][1] = false;
+            metadata.indices[field][1] = false;
             isMetadataChanged = true;
           }
         }
@@ -9472,9 +9512,9 @@ var DocumentDataplyAPI = class extends import_dataply3.DataplyAPI {
       if (isMetadataChanged) {
         await this.updateDocumentInnerMetadata(metadata, tx);
       }
-      this.indecies = metadata.indecies;
-      for (const field in this.indecies) {
-        if (field in targetIndecies) {
+      this.indices = metadata.indices;
+      for (const field in this.indices) {
+        if (field in targetIndices) {
           const tree = new import_dataply3.BPTreeAsync(
             new DocumentSerializeStrategyAsync(
               this.rowTableEngine.order,
@@ -9576,14 +9616,14 @@ var DocumentDataplyAPI = class extends import_dataply3.DataplyAPI {
       return backfilledCount;
     }, tx);
   }
-  createDocumentInnerMetadata(indecies) {
+  createDocumentInnerMetadata(indices) {
     return {
       magicString: "document-dataply",
       version: 1,
       createdAt: Date.now(),
       updatedAt: Date.now(),
       lastId: 0,
-      indecies
+      indices
     };
   }
   async initializeDocumentFile(tx) {
@@ -9645,6 +9685,7 @@ var DocumentDataplyAPI = class extends import_dataply3.DataplyAPI {
 };
 var DocumentDataply = class {
   api;
+  indexedFields;
   operatorConverters = {
     equal: "primaryEqual",
     notEqual: "primaryNotEqual",
@@ -9657,6 +9698,12 @@ var DocumentDataply = class {
   };
   constructor(file, options) {
     this.api = new DocumentDataplyAPI(file, options ?? {});
+    this.indexedFields = /* @__PURE__ */ new Set(["_id"]);
+    if (options?.indices) {
+      for (const field of Object.keys(options.indices)) {
+        this.indexedFields.add(field);
+      }
+    }
   }
   /**
    * Initialize the document database
@@ -9706,16 +9753,29 @@ var DocumentDataply = class {
   }
   /**
    * Get the selectivity candidate for the given query
-   * @param query 
-   * @returns 
+   * @param query The query conditions
+   * @param orderByField Optional field name for orderBy optimization
+   * @returns Driver and other candidates for query execution
    */
-  async getSelectivityCandidate(query) {
+  async getSelectivityCandidate(query, orderByField) {
     const candidates = [];
     for (const field in query) {
       const tree = this.api.trees.get(field);
       if (!tree) continue;
       const condition = query[field];
-      candidates.push({ tree, condition });
+      candidates.push({ tree, condition, field });
+    }
+    if (candidates.length === 0) {
+      return null;
+    }
+    if (orderByField) {
+      const orderByCandidate = candidates.find((c) => c.field === orderByField);
+      if (orderByCandidate) {
+        return {
+          driver: orderByCandidate,
+          others: candidates.filter((c) => c.field !== orderByField)
+        };
+      }
     }
     let res = import_dataply3.BPTreeAsync.ChooseDriver(candidates);
     if (!res && candidates.length > 0) {
@@ -9723,10 +9783,7 @@ var DocumentDataply = class {
     }
     if (!res) return null;
     return {
-      driver: {
-        tree: res.tree,
-        condition: res.condition
-      },
+      driver: res,
       others: candidates.filter((c) => c.tree !== res.tree)
     };
   }
@@ -9814,43 +9871,97 @@ var DocumentDataply = class {
   }
   /**
    * Select documents from the database
-   * @param query The query to use
-   * @param limit The maximum number of documents to return
+   * @param query The query to use (only indexed fields + _id allowed)
+   * @param options The options to use
    * @param tx The transaction to use
    * @returns The documents that match the query
+   * @throws Error if query or orderBy contains non-indexed fields
    */
-  async select(query, limit = Infinity, tx) {
-    return this.api.runWithDefault(async (tx2) => {
-      const verbose = this.verboseQuery(query);
-      const selectivity = await this.getSelectivityCandidate(verbose);
-      if (!selectivity) return [];
-      const keys = /* @__PURE__ */ new Set();
+  select(query, options = {}, tx) {
+    for (const field of Object.keys(query)) {
+      if (!this.indexedFields.has(field)) {
+        throw new Error(`Query field "${field}" is not indexed. Available indexed fields: ${Array.from(this.indexedFields).join(", ")}`);
+      }
+    }
+    const orderBy = options.orderBy ?? "_id";
+    if (!this.indexedFields.has(orderBy)) {
+      throw new Error(`orderBy field "${orderBy}" is not indexed. Available indexed fields: ${Array.from(this.indexedFields).join(", ")}`);
+    }
+    const {
+      limit = Infinity,
+      sortOrder = "asc"
+    } = options;
+    const self = this;
+    const stream = this.api.streamWithDefault(async function* (tx2) {
+      const isQueryEmpty = Object.keys(query).length === 0;
+      const normalizedQuery = isQueryEmpty ? { _id: { gte: 0 } } : query;
+      const verbose = self.verboseQuery(normalizedQuery);
+      const orderByTree = self.api.trees.get(orderBy);
+      const selectivity = await self.getSelectivityCandidate(
+        verbose,
+        orderByTree ? orderBy : void 0
+      );
+      if (!selectivity) return;
       const { driver, others } = selectivity;
-      const stream = driver.tree.whereStream(driver.condition, limit);
-      for await (const [pk, val] of stream) {
-        let isMatch = true;
-        for (const { tree, condition } of others) {
-          const targetValue = await tree.get(pk);
-          if (targetValue === void 0 || !tree.verify(targetValue, condition)) {
-            isMatch = false;
-            break;
+      const isDriverOrderByField = orderByTree && driver.field === orderBy;
+      if (isDriverOrderByField) {
+        const driverStream = driver.tree.whereStream(driver.condition, limit, sortOrder);
+        let i = 0;
+        for await (const [pk, val] of driverStream) {
+          if (i >= limit) break;
+          let isMatch = true;
+          for (const { tree, condition } of others) {
+            const targetValue = await tree.get(pk);
+            if (targetValue === void 0 || !tree.verify(targetValue, condition)) {
+              isMatch = false;
+              break;
+            }
+          }
+          if (isMatch) {
+            const stringified = await self.api.select(pk, false, tx2);
+            if (!stringified) continue;
+            yield JSON.parse(stringified);
+            i++;
           }
         }
-        if (isMatch) {
-          keys.add(pk);
-          if (keys.size >= limit) break;
+      } else {
+        const results = [];
+        const driverStream = driver.tree.whereStream(driver.condition);
+        for await (const [pk, val] of driverStream) {
+          let isMatch = true;
+          for (const { tree, condition } of others) {
+            const targetValue = await tree.get(pk);
+            if (targetValue === void 0 || !tree.verify(targetValue, condition)) {
+              isMatch = false;
+              break;
+            }
+          }
+          if (isMatch) {
+            const stringified = await self.api.select(pk, false, tx2);
+            if (!stringified) continue;
+            results.push(JSON.parse(stringified));
+          }
         }
-      }
-      const documents = [];
-      for (const key of keys) {
-        const stringify = await this.api.select(key, false, tx2);
-        if (!stringify) {
-          continue;
+        results.sort((a, b) => {
+          const aVal = a[orderBy] ?? a._id;
+          const bVal = b[orderBy] ?? b._id;
+          const cmp = aVal < bVal ? -1 : aVal > bVal ? 1 : 0;
+          return sortOrder === "asc" ? cmp : -cmp;
+        });
+        const limitedResults = results.slice(0, limit === Infinity ? void 0 : limit);
+        for (const doc of limitedResults) {
+          yield doc;
         }
-        documents.push(JSON.parse(stringify));
       }
-      return documents;
     }, tx);
+    const drain = async () => {
+      const result = [];
+      for await (const document of stream) {
+        result.push(document);
+      }
+      return result;
+    };
+    return { stream, drain };
   }
   /**
    * Close the document database

package/dist/types/core/document.d.ts

@@ -1,14 +1,15 @@
-import type { DataplyTreeValue, DocumentDataplyInnerMetadata, DocumentDataplyOptions, DocumentJSON, FlattenedDocumentJSON, Primitive, DocumentDataplyQuery, FinalFlatten, DocumentDataplyCondition, DataplyDocument, DocumentDataplyMetadata } from '../types';
+import type { DataplyTreeValue, DocumentDataplyInnerMetadata, DocumentDataplyOptions, DocumentJSON, FlattenedDocumentJSON, Primitive, DocumentDataplyQuery, FinalFlatten, DocumentDataplyCondition, DataplyDocument, DocumentDataplyMetadata, DocumentDataplyQueryOptions, IndexedDocumentDataplyQuery } from '../types';
 import { DataplyAPI, Transaction, BPTreeAsync } from 'dataply';
 import { DocumentValueComparator } from './bptree/documentComparator';
 export declare class DocumentDataplyAPI<T extends DocumentJSON> extends DataplyAPI {
     runWithDefault: <T_1>(callback: (tx: Transaction) => Promise<T_1>, tx?: Transaction) => Promise<T_1>;
-    indecies: DocumentDataplyInnerMetadata['indecies'];
+    streamWithDefault: <T_1>(callback: (tx: Transaction) => AsyncGenerator<T_1>, tx?: Transaction) => AsyncGenerator<T_1>;
+    indices: DocumentDataplyInnerMetadata['indices'];
     readonly trees: Map<string, BPTreeAsync<number, DataplyTreeValue<Primitive>>>;
     readonly comparator: DocumentValueComparator<DataplyTreeValue<Primitive>, Primitive>;
     private pendingBackfillFields;
     private readonly lock;
-    constructor(file: string, options: DocumentDataplyOptions);
+    constructor(file: string, options: DocumentDataplyOptions<T>);
     readLock<T>(fn: () => T): Promise<T>;
     writeLock<T>(fn: () => T): Promise<T>;
     getDocument(pk: number, tx?: Transaction): Promise<DataplyDocument<T>>;
@@ -20,7 +21,7 @@ export declare class DocumentDataplyAPI<T extends DocumentJSON> extends DataplyA
      * @returns Number of documents that were backfilled
      */
     backfillIndices(tx?: Transaction): Promise<number>;
-    createDocumentInnerMetadata(indecies: DocumentDataplyInnerMetadata['indecies']): DocumentDataplyInnerMetadata;
+    createDocumentInnerMetadata(indices: DocumentDataplyInnerMetadata['indices']): DocumentDataplyInnerMetadata;
     initializeDocumentFile(tx: Transaction): Promise<void>;
     verifyDocumentFile(tx: Transaction): Promise<boolean>;
     /**
@@ -33,10 +34,11 @@ export declare class DocumentDataplyAPI<T extends DocumentJSON> extends DataplyA
     getDocumentInnerMetadata(tx: Transaction): Promise<DocumentDataplyInnerMetadata>;
     updateDocumentInnerMetadata(metadata: DocumentDataplyInnerMetadata, tx: Transaction): Promise<void>;
 }
-export declare class DocumentDataply<T extends DocumentJSON> {
+export declare class DocumentDataply<T extends DocumentJSON, I extends string = keyof FinalFlatten<T> & string> {
     protected readonly api: DocumentDataplyAPI<T>;
+    private readonly indexedFields;
     private readonly operatorConverters;
-    constructor(file: string, options?: DocumentDataplyOptions);
+    constructor(file: string, options?: DocumentDataplyOptions<T>);
     /**
      * Initialize the document database
      */
@@ -52,17 +54,20 @@ export declare class DocumentDataply<T extends DocumentJSON> {
     private verboseQuery;
     /**
      * Get the selectivity candidate for the given query
-     * @param query
-     * @returns
+     * @param query The query conditions
+     * @param orderByField Optional field name for orderBy optimization
+     * @returns Driver and other candidates for query execution
      */
-    getSelectivityCandidate<U extends FinalFlatten<DataplyDocument<T>>, V extends DataplyTreeValue<U>>(query: Partial<DocumentDataplyQuery<V>>): Promise<{
+    getSelectivityCandidate<U extends Partial<IndexedDocumentDataplyQuery<FinalFlatten<DataplyDocument<T>>, I>>, V extends DataplyTreeValue<U>>(query: Partial<DocumentDataplyQuery<V>>, orderByField?: string): Promise<{
         driver: {
             tree: BPTreeAsync<number, V>;
             condition: Partial<DocumentDataplyCondition<U>>;
+            field: string;
         };
         others: {
             tree: BPTreeAsync<number, V>;
             condition: Partial<DocumentDataplyCondition<U>>;
+            field: string;
         }[];
     } | null>;
     private insertDocument;
@@ -82,12 +87,16 @@ export declare class DocumentDataply<T extends DocumentJSON> {
     insertBatch(documents: T[], tx?: Transaction): Promise<number[]>;
     /**
      * Select documents from the database
-     * @param query The query to use
-     * @param limit The maximum number of documents to return
+     * @param query The query to use (only indexed fields + _id allowed)
+     * @param options The options to use
      * @param tx The transaction to use
      * @returns The documents that match the query
+     * @throws Error if query or orderBy contains non-indexed fields
      */
-    select(query: Partial<DocumentDataplyQuery<FinalFlatten<DataplyDocument<T>>>>, limit?: number, tx?: Transaction): Promise<DataplyDocument<T>[]>;
+    select(query: Partial<IndexedDocumentDataplyQuery<FinalFlatten<DataplyDocument<T>>, I>>, options?: DocumentDataplyQueryOptions<FinalFlatten<DataplyDocument<T>>, I>, tx?: Transaction): {
+        stream: AsyncIterableIterator<DataplyDocument<T>>;
+        drain: () => Promise<DataplyDocument<T>[]>;
+    };
     /**
      * Close the document database
      */

package/dist/types/types/index.d.ts

@@ -1,4 +1,4 @@
-import type { DataplyOptions } from 'dataply';
+import type { BPTreeOrder, DataplyOptions } from 'dataply';
 export type Primitive = string | number | boolean | null;
 export type JSONValue = Primitive | JSONValue[] | {
     [key: string]: JSONValue;
@@ -15,7 +15,7 @@ export interface DocumentDataplyInnerMetadata {
     createdAt: number;
     updatedAt: number;
     lastId: number;
-    indecies: {
+    indices: {
         [key: string]: [number, boolean];
     };
 }
@@ -47,11 +47,22 @@ export type DocumentDataplyCondition<V> = {
     or?: Partial<V>[];
     like?: string;
 };
+export type DocumentDataplyQueryOptions<V, I extends string = string> = {
+    limit?: number;
+    orderBy?: I | '_id';
+    sortOrder?: BPTreeOrder;
+};
 export type DocumentDataplyQuery<T> = {
     [key in keyof T]?: T[key] | DocumentDataplyCondition<T[key]>;
 } & {
     [key: string]: any;
 };
+/**
+ * Query type restricted to indexed fields only (+ _id)
+ */
+export type IndexedDocumentDataplyQuery<T, I extends string> = {
+    [key in (I | '_id')]?: key extends keyof T ? T[key] | DocumentDataplyCondition<T[key]> : never;
+};
 export interface DataplyTreeValue<T> {
     k: number;
     v: T;
@@ -66,8 +77,8 @@ type Prev = [never, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
  */
 export type DeepFlattenKeys<T, Prefix extends string = "", D extends number = 12> = [
     D
-] extends [0] ? never : T extends Primitive ? (Prefix extends `${infer P}.` ? P : never) : T extends readonly any[] ? ((Prefix extends `${infer P}.` ? P : never) | DeepFlattenKeys<T[number], `${Prefix}${number}.`, Prev[D]>) : T extends object ? {
-    [K in keyof T & string]: NonNullable<T[K]> extends Primitive ? `${Prefix}${K}` : `${Prefix}${K}` | DeepFlattenKeys<NonNullable<T[K]>, `${Prefix}${K}.`, Prev[D]>;
+] extends [0] ? never : T extends Primitive ? (Prefix extends `${infer P}.` ? P : never) : T extends readonly any[] ? (DeepFlattenKeys<T[number], `${Prefix}${number}.`, Prev[D]>) : T extends object ? {
+    [K in keyof T & string]: NonNullable<T[K]> extends Primitive ? `${Prefix}${K}` : DeepFlattenKeys<NonNullable<T[K]>, `${Prefix}${K}.`, Prev[D]>;
 }[keyof T & string] : never;
 /**
  * 경로 문자열(Path)을 기반으로 원본 객체(T)에서 타입을 찾아옵니다.
@@ -77,15 +88,15 @@ type GetTypeByPath<T, Path extends string> = T extends readonly (infer U)[] ? Pa
 export type FinalFlatten<T> = {
     [P in DeepFlattenKeys<T>]: GetTypeByPath<T, P & string>;
 };
-export interface DocumentDataplyOptions extends DataplyOptions {
+export interface DocumentDataplyOptions<T> extends DataplyOptions {
     /**
      * Indecies to create when initializing the database.
      * If not specified, no indecies will be created.
      * If the value of the index is `true`, the index will be created for the already inserted data.
      * If the value of the index is `false`, the index will not be created for the already inserted data.
      */
-    indecies?: {
-        [key: string]: boolean;
-    };
+    indices?: Partial<{
+        [key in keyof FinalFlatten<T>]: boolean;
+    }>;
 }
 export {};

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "document-dataply",
-  "version": "0.0.2-alpha.3",
-  "description": "",
+  "version": "0.0.2",
+  "description": "Simple and powerful JSON document database supporting complex queries and flexible indexing policies.",
   "license": "MIT",
   "author": "izure <admin@izure.org>",
   "type": "commonjs",
@@ -25,8 +25,21 @@
     "test": "jest -i",
     "build": "node build/index.js && tsc"
   },
+  "keywords": [
+    "database",
+    "document-database",
+    "nosql",
+    "json",
+    "indexing",
+    "bptree",
+    "transactions",
+    "acid",
+    "embedded-database",
+    "deep-indexing",
+    "dataply"
+  ],
   "dependencies": {
-    "dataply": "^0.0.18-alpha.1"
+    "dataply": "^0.0.18"
   },
   "devDependencies": {
     "@types/jest": "^30.0.0",