lmcs-db 1.0.4 → 2.0.0

package/README.md

@@ -1,125 +1,281 @@
-# lmcs-db
+# LMCS-DB v2.0
 
-**Lightweight Modular Collection Storage (LMCS)** — um micro SGBD baseado em arquivos locais, com suporte a coleções tipadas, filtros avançados e criptografia opcional.
+[![TypeScript](https://img.shields.io/badge/TypeScript-5.0-blue.svg)](https://typescriptlang.org)
+[![Node](https://img.shields.io/badge/Node-18+-green.svg)](https://nodejs.org)
+[![License](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
 
-![npm](https://img.shields.io/npm/v/lmcs-db)
+**Lightweight Modular Collection Storage** — A high-performance, file-based NoSQL database for Node.js with multiple storage engines, ACID transactions, and military-grade encryption.
 
----
+## ✨ Features
 
-## ✨ Recursos
-
-- 📦 Armazenamento em JSON ou binário  
-- 🔐 Suporte a criptografia AES opcional  
-- 🔍 Consultas com filtros e ordenação  
-- 💾 Persistência assíncrona com fila sequencial  
-- 🧩 Coleções tipadas com suporte a `_id`  
-- 🧾 Formato binário com cabeçalho, tamanho e CRC32 (container estilo SQLite)  
-- 🚀 Auto-criação de diretórios ao salvar
-
----
+- **🗄️ Multiple Storage Engines**: Memory, JSON, Binary, and Append-Only Log (AOL)
+- **🔐 Built-in Encryption**: AES-256-GCM with PBKDF2 key derivation
+- **🔄 ACID Transactions**: Multi-document transactions with rollback support
+- **⚡ High Performance**: In-memory indexes, streaming queries, and batch operations
+- **🔍 Advanced Queries**: MongoDB-like operators ($gt, $lt, $or, $and, $in)
+- **📦 Zero Dependencies**: Lightweight with minimal footprint
+- **🧪 Full TypeScript**: Type-safe collections with IntelliSense support
 
+## 🚀 Quick Start
 
 ```bash
 npm install lmcs-db
-# ou
-yarn add lmcs-db
+```
 
-🚀 Exemplo de uso
-import { DatabaseFactory, DatabaseStorageType } from 'lmcs-db';
+```typescript
+import { Database, StorageType } from "lmcs-db";
 
 interface User {
-  _id: string;
+  _id?: string;
   name: string;
   email: string;
   age: number;
-  active: boolean;
 }
 
-async function main() {
-  const db = await DatabaseFactory.create({
-    storageType: DatabaseStorageType.Binary,
-    databaseName: 'secure-db',
-    customPath: `${process.cwd()}/data`,
-    encryptionKey: 'my-secret-key-123'
-  });
+// Create database
+const db = await Database.create({
+  storageType: StorageType.Binary,
+  databaseName: "myapp",
+  encryptionKey: "your-secret-key-32-chars!!", // Optional
+});
+
+const users = db.collection<User>("users");
+
+// Insert
+await users.insert({ name: "Alice", email: "alice@test.com", age: 30 });
+
+// Query
+const adults = await users.findAll({
+  filter: { age: { $gte: 18 } },
+  sort: { name: 1 },
+  limit: 10,
+});
+
+// Transaction
+await db.transaction(async (trx) => {
+  await trx.insert("users", { name: "Bob", age: 25 });
+  await trx.update("users", "alice-id", { age: 31 });
+});
+```
 
-  const users = db.collection<User>('users');
+💾 Storage Engines
+| Engine | Persistence | Speed | Use Case | Compression |
+| ---------- | ----------- | ------------- | ------------------------------------- | -------------- |
+| **Memory** | ❌ Volatile | ⚡ Ultra-fast | Cache, testing, temporary data | N/A |
+| **JSON** | ✅ File | 🐢 Moderate | Config files, small datasets (<10MB) | None (text) |
+| **Binary** | ✅ File | 🚀 Fast | General purpose, medium datasets | Binary packing |
+| **AOL** | ✅ File | ⚡ Fast writes | Logs, event sourcing, high throughput | Compaction |
 
-  await users.insert({
-    name: 'Alice',
-    email: 'alice@example.com',
-    age: 30,
-    active: true
-  });
+Engine Details
 
-  const activeUsers = await users.findAll({
-    filter: { active: true }
-  });
+Memory Storage
 
-  console.log(activeUsers);
-}
+```typescript
+const db = await createDatabase({
+  storageType: "memory",
+  databaseName: "cache",
+});
+// Data lost on process exit. Fastest option.
+```
+
+JSON Storage
 
-main();
+```typescript
+const db = await createDatabase({
+  storageType: "json",
+  databaseName: "config",
+});
+// Human-readable, but slower than binary.
 ```
 
+Binary Storage
 
-### Encerramento
-```ts
-import { DatabaseFactory, DatabaseStorageType } from 'lmcs-db';
+```typescript
+const db = await createDatabase({
+  storageType: "binary",
+  databaseName: "data",
+  encryptionKey: "secret", // Optional encryption
+});
+// Compact binary format with CRC32 checksums
+```
 
-async function main() {
-  const db = await DatabaseFactory.create({
-    storageType: DatabaseStorageType.Binary,
-    databaseName: 'secure-db',
-    customPath: `${process.cwd()}/data`
-  });
+AOL (Append-Only Log)
+
+```typescript
+const db = await Database.create({
+  storageType: StorageType.AOL,
+  databaseName: "events",
+  bufferSize: 1000, // Buffer before fsync
+  compactionInterval: 60000, // Automatic cleanup every 60s
+});
+// O(1) writes, perfect for event sourcing
+```
 
-  await db.collection('users').insert({ _id: '1', name: 'Alice' });
+🔍 Query API
 
-  await db.flush();
+Basic Queries
+
+```typescript
+// Find one
+const user = await users.findOne({ email: "alice@test.com" });
+
+// Find all
+const all = await users.findAll();
+
+// Count
+const total = await users.count();
+```
+
+Advanced Filtering
+
+```typescript
+// Comparison operators
+const adults = await users.findAll({ filter: { age: { $gte: 18 } } });
+const rich = await users.findAll({ filter: { salary: { $gt: 100000 } } });
+
+// Logical operators
+const result = await users.findAll({
+  filter: {
+    $or: [{ age: { $lt: 18 } }, { vip: true }],
+  },
+});
+
+// Array operators (if field is array)
+const tagged = await posts.findAll({
+  filter: { tags: { $in: ["typescript", "nodejs"] } },
+});
+```
+
+Sorting and Pagination
+
+```typescript
+const page = await users.findAll({
+  filter: { active: true },
+  sort: { createdAt: -1 }, // -1 = descending, 1 = ascending
+  skip: 20, // Offset
+  limit: 10, // Page size
+});
+```
+
+Streaming (Memory Efficient)
+
+```typescript
+// Process millions of records without loading into memory
+const stream = logs.findStream({ filter: { level: "error" } });
+
+for await (const error of stream) {
+  await sendAlert(error);
 }
+```
+
+🔄 Transactions
+ACID transactions ensure data consistency across multiple operations:
+
+```typescript
+await db.transaction(async (trx) => {
+  // All operations succeed or all rollback
+  const order = await trx.insert("orders", { total: 100, status: "pending" });
+  await trx.insert("order_items", { orderId: order._id, product: "Laptop" });
+  await trx.update("inventory", "laptop-123", { stock: { $dec: 1 } });
 
-main();
+  if (somethingWrong) {
+    throw new Error("Rollback everything");
+  }
+});
 ```
 
-## 📘 API
- - DatabaseFactory.create(options): Cria uma instância do banco de dados.
+🔐 Security
+Encryption
+Algorithm: AES-256-GCM
+Key Derivation: PBKDF2 with 100,000 iterations
+Unique IV per encryption operation
+Authentication tag prevents tampering
+
+```typescript
+const db = await Database.create({
+  storageType: StorageType.Binary,
+  databaseName: "secrets",
+  encryptionKey: process.env.DB_KEY, // Load from secure source
+});
+
+// All data transparently encrypted on disk
+await secrets.insert({ password: "super-secret" });
+```
 
-Parâmetros:
- - `storageType`: `Memory` | `Json` | `Binary` — Define o formato de armazenamento
- - `databaseName`: string — Nome do arquivo base do banco
- - `encryptionKey`: string (opcional) — Chave usada para criptografia AES
- - `customPath`: string (opcional) — Diretório onde será criado o arquivo de armazenamento (criado automaticamente se não existir)
+Indexing
+Create indexes for fast queries:
 
-db.collection<T>(name)
-Obtém uma coleção tipada com suporte a:
-- insert
-- find
-- findAll
-- update
-- delete
-- count
+```typescript
+// Single field
+users.createIndex("email", { unique: true });
 
-entre outros métodos utilitários
+// Compound
+orders.createIndex(["userId", "createdAt"]);
 
-📂 Estrutura esperada
-Os dados são armazenados em um único arquivo `.db`, conforme o tipo de armazenamento escolhido.
-As escritas são enfileiradas e processadas de forma sequencial, sem bloquear as operações do banco; chame `db.save()` para solicitar flush imediato quando necessário.
-O diretório de destino é criado automaticamente durante a gravação.
+// Sparse (skip null values)
+users.createIndex("phone", { sparse: true });
+```
+📊 Performance Tips
+1. Use Memory storage for unit tests (10x faster)
+2. Batch inserts instead of individual awaits
+3. Create indexes on frequently queried fields
+4. Use streaming for large datasets (>10k records)
+5. Compact AOL periodically to reclaim space
+6. Enable checksums for critical data integrity
+
+```typescript
+// Batch insert (much faster)
+await Promise.all(
+  items.map(item => collection.insert(item))
+);
+
+// Compact AOL storage
+await db.compact();
+```
+
+🧪 Testing
+```bash
+# Run all tests
+npm test
 
-🔒 Criptografia
-O sistema utiliza o algoritmo AES-256-CBC com vetor de inicialização (IV) dinâmico.
-Se um banco for carregado com uma chave incorreta, ele será reiniciado como vazio, com um aviso exibido no console.
+# Run specific suite
+npm test -- storage.test.ts
 
-Formato binário
-O arquivo `.db` usa um contêiner com cabeçalho: `LMCSDB1` (magic), `flags`, `payloadLength` e `CRC32`. O payload (JSON, possivelmente criptografado) é ofuscado com XOR.
-Na leitura, o cabeçalho e o CRC são validados; dados inválidos retornam `'{}'` de forma segura.
+# With coverage
+npm run test:coverage
+```
+
+📁 Project Structure
+```bash
+data/
+├── myapp.bin        # Binary storage file
+├── myapp.json       # JSON storage file
+└── myapp.aol        # Append-only log
+
+src/
+├── core/
+│   ├── database.ts      # Main database class
+│   ├── collection.ts    # Collection operations
+│   ├── transaction.ts   # ACID transactions
+│   └── indexer.ts       # Index management
+├── storage/
+│   ├── base.ts          # Storage interface
+│   ├── memory.ts        # In-memory storage
+│   ├── json.ts          # JSON file storage
+│   ├── binary.ts        # Binary storage
+│   └── aol.ts           # Append-only log
+└── crypto/
+    └── manager.ts       # Encryption utilities
+```
 
-✅ Testes
-Para executar os testes de demonstração:
+🤝 Contributing
+1. Fork the repository
+2. Create your feature branch (git checkout -b feature/amazing)
+3. Commit changes (git commit -m 'Add amazing feature')
+4. Push to branch (git push origin feature/amazing)
+5. Open a Pull Request
 
-📄 Licença
-MIT
+📄 License
+[MIT License](LICENSE) - see [LICENSE](LICENSE) file.
 
-✍️ Autor
-Desenvolvido por Leandro A da Silva.
+Made with ❤️ by Leandro A. da Silva

package/dist/core/collection.d.ts

@@ -0,0 +1,33 @@
+import { BaseStorage, LogEntry } from "../storage";
+export interface QueryOptions {
+    filter?: Record<string, any>;
+    sort?: Record<string, 1 | -1>;
+    limit?: number;
+    skip?: number;
+    batchSize?: number;
+}
+export declare class Collection<T extends Record<string, any>> {
+    private name;
+    private storage;
+    private data;
+    private indexes;
+    private crypto;
+    constructor(name: string, storage: BaseStorage);
+    private loadFromStorage;
+    applyLogEntry(entry: LogEntry): void;
+    private addToIndexes;
+    private removeFromIndexes;
+    insert(doc: Omit<T, "_id"> & {
+        _id?: string;
+    }): Promise<T>;
+    update(filter: Partial<T>, updates: Partial<T>): Promise<number>;
+    delete(filter: Partial<T>): Promise<number>;
+    findOne(filter: Partial<T>): Promise<T | null>;
+    findAll(options?: QueryOptions): Promise<T[]>;
+    findStream(options?: QueryOptions): AsyncGenerator<T>;
+    createIndex(field: keyof T): void;
+    private matchesFilter;
+    private getNestedValue;
+    private queryByIndex;
+    count(): number;
+}

package/dist/core/collection.js

@@ -0,0 +1,287 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.Collection = void 0;
+const manager_1 = require("../crypto/manager");
+const uuid_1 = require("uuid");
+class Collection {
+    name;
+    storage;
+    data = new Map();
+    indexes = new Map();
+    crypto = new manager_1.CryptoManager(); // Instância sem chave para hash apenas
+    constructor(name, storage) {
+        this.name = name;
+        this.storage = storage;
+        this.loadFromStorage().catch(console.error);
+    }
+    async loadFromStorage() {
+        try {
+            if (!this.storage.readStream) {
+                throw new Error("Storage does not support streaming");
+            }
+            const stream = this.storage.readStream();
+            for await (const entry of stream) {
+                this.applyLogEntry(entry);
+            }
+        }
+        catch (err) {
+            console.error(`Failed to load collection ${this.name}:`, err);
+        }
+    }
+    applyLogEntry(entry) {
+        if (entry.collection !== this.name)
+            return;
+        const id = entry.id;
+        switch (entry.op) {
+            case "INSERT":
+            case "UPDATE":
+                const existing = this.data.get(id);
+                if (existing) {
+                    this.removeFromIndexes(id, existing);
+                }
+                this.data.set(id, entry.data);
+                this.addToIndexes(id, entry.data);
+                break;
+            case "DELETE":
+                const old = this.data.get(id);
+                if (old) {
+                    this.removeFromIndexes(id, old);
+                }
+                this.data.delete(id);
+                break;
+        }
+    }
+    addToIndexes(id, doc) {
+        for (const [field, index] of this.indexes) {
+            const value = doc[field];
+            if (value !== undefined) {
+                if (!index.has(value))
+                    index.set(value, new Set());
+                index.get(value).add(id);
+            }
+        }
+    }
+    removeFromIndexes(id, doc) {
+        for (const [field, index] of this.indexes) {
+            const value = doc[field];
+            if (value !== undefined) {
+                index.get(value)?.delete(id);
+            }
+        }
+    }
+    async insert(doc) {
+        const id = doc._id || (0, uuid_1.v7)();
+        if (this.data.has(id)) {
+            throw new Error(`Document with id ${id} already exists`);
+        }
+        const fullDoc = { ...doc, _id: id };
+        await this.storage.append({
+            op: "INSERT",
+            collection: this.name,
+            id,
+            data: fullDoc,
+            checksum: "",
+            timestamp: Date.now(),
+        });
+        this.data.set(id, fullDoc);
+        this.addToIndexes(id, fullDoc);
+        return fullDoc;
+    }
+    async update(filter, updates) {
+        let count = 0;
+        for (const [id, doc] of this.data.entries()) {
+            if (this.matchesFilter(doc, filter)) {
+                const newDoc = { ...doc, ...updates, _id: id };
+                await this.storage.append({
+                    op: "UPDATE",
+                    collection: this.name,
+                    id,
+                    data: newDoc,
+                    checksum: "",
+                    timestamp: Date.now(),
+                });
+                this.removeFromIndexes(id, doc);
+                this.data.set(id, newDoc);
+                this.addToIndexes(id, newDoc);
+                count++;
+            }
+        }
+        return count;
+    }
+    async delete(filter) {
+        let count = 0;
+        const toDelete = [];
+        for (const [id, doc] of this.data.entries()) {
+            if (this.matchesFilter(doc, filter)) {
+                toDelete.push(id);
+            }
+        }
+        for (const id of toDelete) {
+            const doc = this.data.get(id);
+            await this.storage.append({
+                op: "DELETE",
+                collection: this.name,
+                id,
+                checksum: "",
+                timestamp: Date.now(),
+            });
+            this.removeFromIndexes(id, doc);
+            this.data.delete(id);
+            count++;
+        }
+        return count;
+    }
+    async findOne(filter) {
+        // Tenta usar índice primeiro
+        const indexedId = this.queryByIndex(filter);
+        if (indexedId) {
+            return this.data.get(indexedId) || null;
+        }
+        for (const doc of this.data.values()) {
+            if (this.matchesFilter(doc, filter))
+                return doc;
+        }
+        return null;
+    }
+    async findAll(options = {}) {
+        let results = Array.from(this.data.values());
+        if (options.filter) {
+            results = results.filter((d) => this.matchesFilter(d, options.filter));
+        }
+        if (options.sort) {
+            results.sort((a, b) => {
+                for (const [field, dir] of Object.entries(options.sort)) {
+                    const aVal = a[field];
+                    const bVal = b[field];
+                    if (aVal < bVal)
+                        return dir === 1 ? -1 : 1;
+                    if (aVal > bVal)
+                        return dir === 1 ? 1 : -1;
+                }
+                return 0;
+            });
+        }
+        if (options.skip)
+            results = results.slice(options.skip);
+        if (options.limit)
+            results = results.slice(0, options.limit);
+        return results;
+    }
+    async *findStream(options = {}) {
+        if (options.sort) {
+            const all = await this.findAll(options);
+            for (const doc of all)
+                yield doc;
+            return;
+        }
+        let count = 0;
+        let skipped = 0;
+        for (const doc of this.data.values()) {
+            if (options.filter && !this.matchesFilter(doc, options.filter))
+                continue;
+            if (options.skip && skipped < options.skip) {
+                skipped++;
+                continue;
+            }
+            if (options.limit && count >= options.limit)
+                break;
+            yield doc;
+            count++;
+        }
+    }
+    createIndex(field) {
+        if (this.indexes.has(field))
+            return;
+        this.indexes.set(field, new Map());
+        // Indexa existentes
+        for (const [id, doc] of this.data.entries()) {
+            this.addToIndexes(id, doc);
+        }
+    }
+    matchesFilter(doc, filter) {
+        for (const [key, value] of Object.entries(filter)) {
+            if (key === "$or") {
+                if (!Array.isArray(value))
+                    return false;
+                if (!value.some((condition) => this.matchesFilter(doc, condition)))
+                    return false;
+                continue;
+            }
+            if (key === "$and") {
+                if (!Array.isArray(value))
+                    return false;
+                if (!value.every((condition) => this.matchesFilter(doc, condition)))
+                    return false;
+                continue;
+            }
+            // Handle dot notation for nested fields
+            const docValue = this.getNestedValue(doc, key);
+            if (typeof value === "object" && value !== null) {
+                // Handle operators like $gt, $lt, etc.
+                for (const [op, opValue] of Object.entries(value)) {
+                    switch (op) {
+                        case "$gt":
+                            if (!(docValue > opValue))
+                                return false;
+                            break;
+                        case "$gte":
+                            if (!(docValue >= opValue))
+                                return false;
+                            break;
+                        case "$lt":
+                            if (!(docValue < opValue))
+                                return false;
+                            break;
+                        case "$lte":
+                            if (!(docValue <= opValue))
+                                return false;
+                            break;
+                        case "$ne":
+                            if (docValue === opValue)
+                                return false;
+                            break;
+                        case "$in":
+                            if (!Array.isArray(opValue) || !opValue.includes(docValue))
+                                return false;
+                            break;
+                        case "$nin":
+                            if (Array.isArray(opValue) && opValue.includes(docValue))
+                                return false;
+                            break;
+                        default:
+                            // If it's not an operator, treat as equality check for object
+                            if (JSON.stringify(docValue) !== JSON.stringify(value))
+                                return false;
+                    }
+                }
+            }
+            else {
+                // Direct equality
+                if (docValue !== value)
+                    return false;
+            }
+        }
+        return true;
+    }
+    getNestedValue(obj, path) {
+        return path.split('.').reduce((o, p) => (o ? o[p] : undefined), obj);
+    }
+    queryByIndex(filter) {
+        const entries = Object.entries(filter);
+        if (entries.length !== 1)
+            return null;
+        const [field, value] = entries[0];
+        const index = this.indexes.get(field);
+        if (!index)
+            return null;
+        const ids = index.get(value);
+        if (ids && ids.size > 0) {
+            return Array.from(ids)[0];
+        }
+        return null;
+    }
+    count() {
+        return this.data.size;
+    }
+}
+exports.Collection = Collection;