s3db.js 3.3.2 → 4.0.2

package/README.md

@@ -2,907 +2,1411 @@
 
 [![license: unlicense](https://img.shields.io/badge/license-Unlicense-blue.svg)](http://unlicense.org/) [![npm version](https://img.shields.io/npm/v/s3db.js.svg?style=flat)](https://www.npmjs.com/package/s3db.js) [![Maintainability](https://api.codeclimate.com/v1/badges/26e3dc46c42367d44f18/maintainability)](https://codeclimate.com/github/forattini-dev/s3db.js/maintainability) [![Coverage Status](https://coveralls.io/repos/github/forattini-dev/s3db.js/badge.svg?branch=main)](https://coveralls.io/github/forattini-dev/s3db.js?branch=main)
 
-Another way to create a cheap document-base database with an easy ORM to handle your dataset!
+**A document-based database built on AWS S3 with a powerful ORM-like interface**
 
-<table width="100%">
-<tr>
-<td>
+Transform AWS S3 into a fully functional document database with automatic validation, encryption, caching, and streaming capabilities.
 
-1. <a href="#motivation">Motivation</a>
-1. <a href="#usage">Usage</a>
-   1. <a href="#install">Install</a>
-   1. <a href="#quick-setup">Quick Setup</a>
-   1. <a href="#insights">Insights</a>
-   1. <a href="#database">Database</a>
-   1. <a href="#create-a-resource">Create a resource</a>
-1. <a href="#resource-methods">Resource methods</a>
-   1. <a href="#insert-one">Insert one</a>
-   1. <a href="#get-one">Get one</a>
-   1. <a href="#update-one">Update one</a>
-   1. <a href="#delete-one">Delete one</a>
-   1. <a href="#count">Count</a>
-   1. <a href="#insert-many">Insert many</a>
-   1. <a href="#get-many">Get many</a>
-   1. <a href="#get-all">Get all</a>
-   1. <a href="#delete-many">Delete many</a>
-   1. <a href="#delete-all">Delete all</a>
-   1. <a href="#list-ids">List ids</a>
-1. <a href="#resource-streams">Resource streams</a>
-   1. <a href="#readable-stream">Readable stream</a>
-   1. <a href="#writable-stream">Writable stream</a>
-1. <a href="#s3-client">S3 Client</a>
-1. <a href="#events">Events</a>
-1. <a href="#plugins">Plugins</a>
-1. <a href="#cost-simulation">Cost Simulation</a>
-   1. <a href="#big-example">Big Example</a>
-   1. <a href="#small-example">Small example</a>
-1. <a href="#roadmap">Roadmap</a>
+## 🚀 Quick Start
 
-</td>
-</tr>
-</table>
+```bash
+npm i s3db.js
+```
+
+```javascript
+import { S3db } from "s3db.js";
+
+// Connect to your S3 database
+const s3db = new S3db({
+  uri: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp"
+});
+
+await s3db.connect();
 
----
+// Create a resource (collection)
+const users = await s3db.createResource({
+  name: "users",
+  attributes: {
+    name: "string|min:2|max:100",
+    email: "email|unique",
+    age: "number|integer|positive",
+    isActive: "boolean",
+    createdAt: "date"
+  }
+});
+
+// Insert data
+const user = await users.insert({
+  name: "John Doe",
+  email: "john@example.com",
+  age: 30,
+  isActive: true,
+  createdAt: new Date()
+});
 
-## Motivation
+// Query data
+const foundUser = await users.get(user.id);
+console.log(foundUser.name); // "John Doe"
+```
 
-First of all:
+## 📋 Table of Contents
 
-1. Nothing is for free, but it can be cheaper.
-2. I'm not responsible for your AWS Costs strategy, use `s3db.js` at your own risk.
-3. Please, do not use in production!
+- [🎯 What is s3db.js?](#-what-is-s3dbjs)
+- [💡 How it Works](#-how-it-works)
+- [⚡ Installation & Setup](#-installation--setup)
+- [🔧 Configuration](#-configuration)
+- [📚 Core Concepts](#-core-concepts)
+- [🛠️ API Reference](#️-api-reference)
+- [📊 Examples](#-examples)
+- [🔄 Streaming](#-streaming)
+- [🔐 Security & Encryption](#-security--encryption)
+- [💰 Cost Analysis](#-cost-analysis)
+- [🎛️ Advanced Features](#️-advanced-features)
+- [🚨 Limitations & Best Practices](#-limitations--best-practices)
+- [🧪 Testing](#-testing)
+- [📅 Version Compatibility](#-version-compatibility)
 
-**Let's go!**
+## 🎯 What is s3db.js?
 
-You might know AWS's S3 product for its high availability and its cheap pricing rules. I'll show you another clever and funny way to use S3.
+`s3db.js` is a document database that leverages AWS S3's metadata capabilities to store structured data. Instead of storing data in file bodies, it uses S3's metadata fields (up to 2KB) to store document data, making it extremely cost-effective for document storage.
 
-AWS allows you define `Metadata` to every single file you upload into your bucket. This attribute must be defined within a **2kb** limit using in `UTF-8` encoding. As this encoding [may vary the bytes width for each symbol](https://en.wikipedia.org/wiki/UTF-8) you may use [500 to 2000] chars of metadata storage. Follow the docs at [AWS S3 User Guide: Using metadata](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#object-metadata).
+### Key Features
 
-There is another management subset of data called `tags` that is used globally as [key, value] params. You can assign 10 tags with the conditions of: the key must be at most 128 unicode chars lengthy and the value up to 256 chars. With those key-values we can use more `2.5kb` of data, unicode will allow you to use up to 2500 more chars. Follow the official docs at [AWS User Guide: Object Tagging](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html).
+- **🔄 ORM-like Interface**: Familiar database operations (insert, get, update, delete)
+- **✅ Automatic Validation**: Built-in schema validation using fastest-validator
+- **🔐 Encryption**: Optional field-level encryption for sensitive data
+- **⚡ Streaming**: Handle large datasets with readable/writable streams
+- **💾 Caching**: Reduce API calls with intelligent caching
+- **📊 Cost Tracking**: Monitor AWS costs with built-in plugins
+- **🛡️ Type Safety**: Full TypeScript support
+- **🔧 Robust Serialization**: Advanced handling of arrays and objects with edge cases
+- **📝 Comprehensive Testing**: Complete test suite with journey-based scenarios
+- **🕒 Automatic Timestamps**: Optional createdAt/updatedAt fields
+- **📦 Partitions**: Organize data by fields for efficient queries
+- **🎣 Hooks**: Custom logic before/after operations
+- **🔌 Plugins**: Extensible architecture
 
-With all this set you may store objects that should be able to store up to `4.5kb` of free space **per object**.
+## 💡 How it Works
 
-Check the <a href="#cost-simulation">cost simulation</a> section below for a deep cost dive!
+### The Magic Behind s3db.js
 
-Lets give it a try! :)
+AWS S3 allows you to store metadata with each object:
+- **Metadata**: Up to 2KB of UTF-8 encoded data
 
----
+`s3db.js` cleverly uses these fields to store document data instead of file contents, making each S3 object act as a database record.
 
-## Usage
+### Data Storage Strategy
 
-You may check the snippets bellow or go straight to the <a href="#examples">Examples</a> section!
+```javascript
+// Your document
+{
+  id: "user-123",
+  name: "John Doe",
+  email: "john@example.com",
+  age: 30
+}
+
+// Stored in S3 as:
+// Key: users/user-123
+// Metadata: { "name": "John Doe", "email": "john@example.com", "age": "30", "id": "user-123" }
+```
+
+## ⚡ Installation & Setup
 
 ### Install
 
 ```bash
 npm i s3db.js
-
 # or
-
+pnpm add s3db.js
+# or
 yarn add s3db.js
 ```
 
-### Quick setup
-
-Our S3db client use connection string params.
+### Basic Setup
 
 ```javascript
 import { S3db } from "s3db.js";
 
-const {
-  AWS_BUCKET,
-  AWS_ACCESS_KEY_ID,
-  AWS_SECRET_ACCESS_KEY,
-} = process.env
-
 const s3db = new S3db({
-  uri: `s3://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}@${AWS_BUCKET}/databases/mydatabase`
+  uri: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp"
 });
 
-s3db
-  .connect()
-  .then(() => console.log('connected!')))
+await s3db.connect();
+console.log("Connected to S3 database!");
 ```
 
-If you do use `dotenv` package:
+### Environment Variables Setup
 
 ```javascript
 import * as dotenv from "dotenv";
 dotenv.config();
 
 import { S3db } from "s3db.js";
-```
 
-### Insights
+const s3db = new S3db({
+  uri: `s3://${process.env.AWS_ACCESS_KEY_ID}:${process.env.AWS_SECRET_ACCESS_KEY}@${process.env.AWS_BUCKET}/databases/${process.env.DATABASE_NAME}`
+});
+```
 
-- This implementation of ORM simulates a document repository. Due to the fact that `s3db.js` uses `aws-sdk`'s' S3 api; all requests are GET/PUT as `key=value` resources. So the best case scenario is to access like a document implementation.
+## 🔧 Configuration
 
-- For better use of the <a href="#cache">cache</a> and listing, the best ID format is to use sequential ids with leading zeros (eq: 00001, 00002, 00003) due to S3 internal keys sorting method. But you will need to manage this incremental ID by your own.
+### Connection Options
 
-### Database
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `uri` | `string` | **required** | S3 connection string |
+| `parallelism` | `number` | `10` | Concurrent operations |
+| `passphrase` | `string` | `"secret"` | Encryption key |
+| `cache` | `boolean` | `false` | Enable caching |
+| `ttl` | `number` | `86400` | Cache TTL in seconds |
+| `plugins` | `array` | `[]` | Custom plugins |
 
-Your `s3db.js` client can be initiated with options:
+### 🔐 Authentication & Connectivity
 
-|   option    | optional |                     description                     |   type    |   default   |
-| :---------: | :------: | :-------------------------------------------------: | :-------: | :---------: |
-|    cache    |   true   |  Persist searched data to reduce repeated requests  | `boolean` | `undefined` |
-| parallelism |   true   |            Number of simultaneous tasks             | `number`  |     10      |
-| passphrase  |   true   |               Your encryption secret                | `string`  | `undefined` |
-|     ttl     |   true   | (Coming soon) TTL to your cache duration in seconds | `number`  |    86400    |
-|     uri     |  false   |         A url as your S3 connection string          | `string`  | `undefined` |
+`s3db.js` supports multiple authentication methods and can connect to various S3-compatible services:
 
-Config example:
+#### Connection String Format
 
-```javascript
-const {
-  AWS_BUCKET = "my-bucket",
-  AWS_ACCESS_KEY_ID = "secret",
-  AWS_SECRET_ACCESS_KEY = "secret",
-  AWS_BUCKET_PREFIX = "databases/test-" + Date.now(),
-} = process.env;
+```
+s3://[ACCESS_KEY:SECRET_KEY@]BUCKET_NAME[/PREFIX]
+```
 
-const uri = `s3://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}@${AWS_BUCKET}/${AWS_BUCKET_PREFIX}`;
+#### 1. AWS S3 with Access Keys
 
-const options = {
-  uri,
-  parallelism: 25,
-  passphrase: fs.readFileSync("./cert.pem"),
-};
+```javascript
+const s3db = new S3db({
+  uri: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp"
+});
 ```
 
-#### s3db.connect()
+#### 2. AWS S3 with IAM Roles (EC2/EKS)
 
-This method must always be invoked before any operation take place. This will interact with AWS' S3 api and check the itens below:
+```javascript
+// No credentials needed - uses IAM role permissions
+const s3db = new S3db({
+  uri: "s3://BUCKET_NAME/databases/myapp"
+});
+```
 
-1. With current credentials:
-   - Check if client has access to the S3 bucket.
-   - Check if client has access to bucket life-cycle policies.
-1. With defined database:
-   - Check if there is already a database in this connection string.
-     - If any database is found, downloads it's medatada and loads each `Resource` definition.
-     - Else, it will generate an empty <a href="#metadata-file">`metadata`</a> file into this prefix and mark that this is a new database from scratch.
+#### 3. MinIO or S3-Compatible Services
 
-#### Metadata file
+```javascript
+const s3db = new S3db({
+  uri: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp",
+  endpoint: "http://localhost:9000" // MinIO default endpoint
+});
+```
 
-`s3db.js` will generate a file `/s3db.json` at the pre-defined prefix with this structure:
+#### 4. Environment-Based Configuration
 
 ```javascript
-{
-  // file version
-  "version": "1",
-
-  // previously defined resources
-  "resources": {
-    // definition example
-    "leads": {
-      "name": "leads",
-
-      // resource options
-      "options": {},
-
-      // resource defined schema
-      "schema": {
-        "name": "string",
-        "token": "secret"
-      },
-
-      // rules to simplify metadata usage
-      "mapper": {
-        "name": "0",
-        "token": "1"
-      },
-    }
-  }
-}
+const s3db = new S3db({
+  uri: `s3://${process.env.AWS_ACCESS_KEY_ID}:${process.env.AWS_SECRET_ACCESS_KEY}@${process.env.AWS_BUCKET}/databases/${process.env.DATABASE_NAME}`,
+  endpoint: process.env.S3_ENDPOINT
+});
 ```
 
-### Create a resource
+#### Security Best Practices
+
+- **IAM Roles**: Use IAM roles instead of access keys when possible (EC2, EKS, Lambda)
+- **Environment Variables**: Store credentials in environment variables, not in code
+- **Bucket Permissions**: Ensure your IAM role/user has the necessary S3 permissions:
+  - `s3:GetObject`, `s3:PutObject`, `s3:DeleteObject`, `s3:ListBucket`, `s3:GetBucketLocation`
 
-Resources are definitions of data collections.
+### Advanced Configuration
 
 ```javascript
-// resource
-const attributes = {
-  utm: {
-    source: "string|optional",
-    medium: "string|optional",
-    campaign: "string|optional",
-    term: "string|optional",
-  },
-  lead: {
-    fullName: "string",
-    mobileNumber: "string",
-    personalEmail: "email",
-  },
-};
+import fs from "fs";
 
-const resource = await s3db.createResource({
-  name: "leads",
-  attributes,
+const s3db = new S3db({
+  uri: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp",
+  parallelism: 25,                    // Handle 25 concurrent operations
+  passphrase: fs.readFileSync("./cert.pem"), // Custom encryption key
+  cache: true,                        // Enable caching
+  ttl: 3600,                         // 1 hour cache TTL
+  plugins: [CostsPlugin]              // Enable cost tracking
 });
 ```
 
-Resources' names **cannot** prefix each other, like: `leads` and `leads-copy`! S3's api lists keys using prefix notation, so every time you list `leads`, all keys of `leads-copy` will appear as well.
+## 📚 Core Concepts
 
-##### Attributes
+### 1. Database
 
-`s3db.js` use the [fastest-validator](https://www.npmjs.com/package/fastest-validator) package to define and validate your resource. Some few examples:
+A database is a logical container for your resources, stored in a specific S3 prefix.
 
 ```javascript
-const attributes = {
-  // few simple examples
-  name: "string|min:4|max:64|trim",
-  email: "email|nullable",
-  mobile: "string|optional",
-  count: "number|integer|positive",
-  corrency: "corrency|symbol:R$",
-  createdAt: "date",
-  website: "url",
-  id: "uuid",
-  ids: "array|items:uuid|unique",
-
-  // s3db defines a custom type "secret" that is encrypted
-  token: "secret",
-
-  // nested data works aswell
-  geo: {
-    lat: "number",
-    long: "number",
-    city: "string",
-  },
-
-  // may have multiple definitions.
-  address_number: ["string", "number"],
-};
+// This creates/connects to a database at:
+// s3://bucket/databases/myapp/
+const s3db = new S3db({
+  uri: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp"
+});
 ```
 
-##### Reference:
+### 2. Resources (Collections)
 
-You may just use the reference:
+Resources are like tables in traditional databases - they define the structure of your documents.
 
 ```javascript
-const Leads = s3db.resource("leads");
+const users = await s3db.createResource({
+  name: "users",
+  attributes: {
+    name: "string|min:2|max:100",
+    email: "email|unique",
+    age: "number|integer|positive",
+    profile: {
+      bio: "string|optional",
+      avatar: "url|optional"
+    },
+    tags: "array|items:string",
+    metadata: "object|optional"
+  }
+});
 ```
 
-##### Limitations:
-
-As we need to store the resource definition within a JSON file, to keep your definitions intact the best way is to use the [string-based shorthand definitions](https://github.com/icebob/fastest-validator#shorthand-definitions) in your resource definition.
+#### Automatic Timestamps
 
-By design, the resource definition **will will strip all functions** in attributes to avoid `eval()` calls.
+If you enable the `timestamps` option, `s3db.js` will automatically add `createdAt` and `updatedAt` fields to your resource, and keep them updated on insert and update operations.
 
-The `fastest-validator` starts with the params below:
+```js
+const users = await s3db.createResource({
+  name: "users",
+  attributes: { name: "string", email: "email" },
+  options: { timestamps: true }
+});
 
-```javascript
-// fastest-validator params
-{
-  useNewCustomCheckerFunction: true,
-  defaults: {
-    object: {
-      strict: "remove",
-    },
-  },
-}
+const user = await users.insert({ name: "John", email: "john@example.com" });
+console.log(user.createdAt); // e.g. "2024-06-27T12:34:56.789Z"
+console.log(user.updatedAt); // same as createdAt on insert
 ```
 
----
+#### Resource Behaviors
 
-## Resources methods
+`s3db.js` provides a powerful behavior system to handle how your data is managed when it approaches or exceeds S3's 2KB metadata limit. Each behavior implements different strategies for handling large documents.
 
-Consider `resource` as:
+##### Available Behaviors
 
-```javascript
-const resource = s3db.resource("leads");
-```
+| Behavior | Description | Use Case |
+|----------|-------------|----------|
+| `user-management` | **Default** - Emits warnings but allows operations | Development and testing |
+| `enforce-limits` | Throws errors when limit is exceeded | Strict data size control |
+| `data-truncate` | Truncates data to fit within limits | Preserve structure, lose data |
+| `body-overflow` | Stores excess data in S3 object body | Preserve all data |
 
-### Insert one
+##### Behavior Configuration
 
 ```javascript
-// data
-const insertedData = await resource.insert({
-  id: "mypersonal@email.com", // if not defined a id will be generated!
-  utm: {
-    source: "abc",
-  },
-  lead: {
-    fullName: "My Complex Name",
-    personalEmail: "mypersonal@email.com",
-    mobileNumber: "+5511234567890",
+const users = await s3db.createResource({
+  name: "users",
+  attributes: {
+    name: "string|min:2|max:100",
+    email: "email|unique",
+    bio: "string|optional",
+    preferences: "object|optional"
   },
-  invalidAttr: "this attribute will disappear",
+  options: {
+    behavior: "body-overflow",  // Choose behavior strategy
+    timestamps: true,           // Enable automatic timestamps
+    partitions: {               // Define data partitions
+      byRegion: {
+        fields: { region: "string" }
+      }
+    },
+    hooks: {                    // Custom operation hooks
+      preInsert: [async (data) => {
+        // Custom validation logic
+        return data;
+      }],
+      afterInsert: [async (data) => {
+        console.log("User created:", data.id);
+      }]
+    }
+  }
 });
-
-// {
-//   id: "mypersonal@email.com",
-//   utm: {
-//     source: "abc",
-//   },
-//   lead: {
-//     fullName: "My Complex Name",
-//     personalEmail: "mypersonal@email.com",
-//     mobileNumber: "+5511234567890",
-//   },
-//   invalidAttr: "this attribute will disappear",
-// }
 ```
 
-If not defined an id attribute, `s3db.js` will use [`nanoid`](https://github.com/ai/nanoid) to generate a random unique id!
+##### 1. User Management Behavior (Default)
 
-### Get one
+The default behavior that gives you full control over data size management:
 
 ```javascript
-const obj = await resource.get("mypersonal@email.com");
+const users = await s3db.createResource({
+  name: "users",
+  attributes: { name: "string", email: "email" },
+  options: { behavior: "user-management" }
+});
 
-// {
-//   id: "mypersonal@email.com",
-//   utm: {
-//     source: "abc",
-//   },
-//   lead: {
-//     fullName: "My Complex Name",
-//     personalEmail: "mypersonal@email.com",
-//     mobileNumber: "+5511234567890",
-//   },
-// }
+// Listen for limit warnings
+users.on("exceedsLimit", (info) => {
+  console.log(`Document ${info.operation} exceeds 2KB limit:`, {
+    totalSize: info.totalSize,
+    limit: info.limit,
+    excess: info.excess
+  });
+});
+
+// Operations continue normally even if limit is exceeded
+const user = await users.insert({
+  name: "John Doe",
+  email: "john@example.com",
+  largeBio: "Very long bio...".repeat(100) // Will trigger warning but succeed
+});
 ```
 
-### Update one
+##### 2. Enforce Limits Behavior
+
+Strict behavior that prevents operations when data exceeds the limit:
 
 ```javascript
-const obj = await resource.update("mypersonal@email.com", {
-  lead: {
-    fullName: "My New Name",
-    mobileNumber: "+5511999999999",
-  },
+const users = await s3db.createResource({
+  name: "users",
+  attributes: { name: "string", email: "email" },
+  options: { behavior: "enforce-limits" }
 });
 
-// {
-//   id: "mypersonal@email.com",
-//   utm: {
-//     source: "abc",
-//   },
-//   lead: {
-//     fullName: "My New Name",
-//     personalEmail: "mypersonal@email.com",
-//     mobileNumber: "+5511999999999",
-//   },
-// }
+try {
+  const user = await users.insert({
+    name: "John Doe",
+    email: "john@example.com",
+    largeBio: "Very long bio...".repeat(100)
+  });
+} catch (error) {
+  console.error("Operation failed:", error.message);
+  // Error: S3 metadata size exceeds 2KB limit. Current size: 2500 bytes, limit: 2048 bytes
+}
 ```
 
-### Delete one
+##### 3. Data Truncate Behavior
+
+Intelligently truncates data to fit within limits while preserving structure:
 
 ```javascript
-await resource.delete(id);
+const users = await s3db.createResource({
+  name: "users",
+  attributes: { name: "string", email: "email", bio: "string" },
+  options: { behavior: "data-truncate" }
+});
+
+const user = await users.insert({
+  name: "John Doe",
+  email: "john@example.com",
+  bio: "This is a very long biography that will be truncated to fit within the 2KB metadata limit..."
+});
+
+console.log(user.bio); // "This is a very long biography that will be truncated to fit within the 2KB metadata limit..."
+// Note: The bio will be truncated with "..." suffix if it exceeds available space
 ```
 
-### Count
+##### 4. Body Overflow Behavior
+
+Stores excess data in the S3 object body, preserving all information:
 
 ```javascript
-await resource.count();
+const users = await s3db.createResource({
+  name: "users",
+  attributes: { name: "string", email: "email", bio: "string" },
+  options: { behavior: "body-overflow" }
+});
 
-// 101
+const user = await users.insert({
+  name: "John Doe",
+  email: "john@example.com",
+  bio: "This is a very long biography that will be stored in the S3 object body..."
+});
+
+// All data is preserved and automatically merged when retrieved
+console.log(user.bio); // Full biography preserved
 ```
 
-### Insert many
+**How Body Overflow Works:**
+- Small attributes stay in metadata for fast access
+- Large attributes are moved to S3 object body
+- Data is automatically merged when retrieved
+- Maintains full data integrity
 
-You may bulk insert data with a friendly method that receives a list of objects.
+##### Complete Resource Configuration Reference
 
 ```javascript
-const objects = new Array(100).fill(0).map((v, k) => ({
-  id: `bulk-${k}@mymail.com`,
-  lead: {
-    fullName: "My Test Name",
-    personalEmail: `bulk-${k}@mymail.com`,
-    mobileNumber: "+55 11 1234567890",
+const resource = await s3db.createResource({
+  // Required: Resource name (unique within database)
+  name: "users",
+  
+  // Required: Schema definition
+  attributes: {
+    // Basic types
+    name: "string|min:2|max:100",
+    email: "email|unique",
+    age: "number|integer|positive",
+    isActive: "boolean",
+    
+    // Advanced types
+    website: "url",
+    uuid: "uuid",
+    createdAt: "date",
+    price: "currency|symbol:$",
+    
+    // Encrypted fields
+    password: "secret",
+    apiKey: "secret",
+    
+    // Nested objects
+    address: {
+      street: "string",
+      city: "string",
+      country: "string",
+      zipCode: "string|optional"
+    },
+    
+    // Arrays
+    tags: "array|items:string|unique",
+    scores: "array|items:number|min:1",
+    
+    // Multiple types
+    id: ["string", "number"],
+    
+    // Complex nested structures
+    metadata: {
+      settings: "object|optional",
+      preferences: "object|optional"
+    }
   },
-}));
-
-await resource.insertMany(objects);
+  
+  // Optional: Resource configuration
+  options: {
+    // Behavior strategy for handling 2KB metadata limits
+    behavior: "user-management", // "user-management" | "enforce-limits" | "data-truncate" | "body-overflow"
+    
+    // Enable automatic timestamps
+    timestamps: true, // Adds createdAt and updatedAt fields
+    
+    // Define data partitions for efficient querying
+    partitions: {
+      byRegion: {
+        fields: { region: "string" }
+      },
+      byAgeGroup: {
+        fields: { ageGroup: "string" }
+      },
+      byDate: {
+        fields: { createdAt: "date|maxlength:10" }
+      }
+    },
+    
+    // Custom operation hooks
+    hooks: {
+      // Pre-operation hooks (can modify data)
+      preInsert: [
+        async (data) => {
+          // Validate or transform data before insert
+          if (!data.email.includes("@")) {
+            throw new Error("Invalid email format");
+          }
+          return data;
+        }
+      ],
+      preUpdate: [
+        async (id, data) => {
+          // Validate or transform data before update
+          return data;
+        }
+      ],
+      preDelete: [
+        async (id) => {
+          // Validate before deletion
+          return true; // Return false to abort
+        }
+      ],
+      
+      // Post-operation hooks (cannot modify data)
+      afterInsert: [
+        async (data) => {
+          console.log("User created:", data.id);
+        }
+      ],
+      afterUpdate: [
+        async (id, data) => {
+          console.log("User updated:", id);
+        }
+      ],
+      afterDelete: [
+        async (id) => {
+          console.log("User deleted:", id);
+        }
+      ]
+    }
+  }
+});
 ```
 
-Keep in mind that we need to send a request to each object to be created. There is an option to change the amount of simultaneos connections that your client will handle.
+### 3. Schema Validation
+
+`s3db.js` uses [fastest-validator](https://github.com/icebob/fastest-validator) for schema validation with robust handling of edge cases:
 
 ```javascript
-const s3db = new S3db({
-  parallelism: 100, // default = 10
-});
+const attributes = {
+  // Basic types
+  name: "string|min:2|max:100|trim",
+  email: "email|nullable",
+  age: "number|integer|positive",
+  isActive: "boolean",
+  
+  // Advanced types
+  website: "url",
+  uuid: "uuid",
+  createdAt: "date",
+  price: "currency|symbol:$",
+  
+  // Custom s3db types
+  password: "secret",  // Encrypted field
+  
+  // Nested objects (supports empty objects and null values)
+  address: {
+    street: "string",
+    city: "string",
+    country: "string",
+    zipCode: "string|optional"
+  },
+  
+  // Arrays (robust serialization with special character handling)
+  tags: "array|items:string|unique",        // Handles empty arrays: []
+  scores: "array|items:number|min:1",       // Handles null arrays
+  categories: "array|items:string",         // Handles arrays with pipe characters: ['tag|special', 'normal']
+  
+  // Multiple types
+  id: ["string", "number"],
+  
+  // Complex nested structures
+  metadata: {
+    settings: "object|optional",     // Can be empty: {}
+    preferences: "object|optional"   // Can be null
+  }
+};
 ```
 
-This method uses [`supercharge/promise-pool`](https://github.com/supercharge/promise-pool) to organize the parallel promises.
+### Enhanced Array and Object Handling
 
-### Get many
+s3db.js now provides robust serialization for complex data structures:
 
 ```javascript
-await resource.getMany(["id1", "id2", "id3 "]);
+// ✅ Supported: Empty arrays and objects
+const user = await users.insert({
+  name: "John Doe",
+  tags: [],              // Empty array - properly serialized
+  metadata: {},          // Empty object - properly handled
+  preferences: null      // Null object - correctly preserved
+});
 
-// [
-//   obj1,
-//   obj2,
-//   obj3,
-// ]
+// ✅ Supported: Arrays with special characters
+const product = await products.insert({
+  name: "Widget",
+  categories: ["electronics|gadgets", "home|office"],  // Pipe characters escaped
+  tags: ["tag|with|pipes", "normal-tag"]               // Multiple pipes handled
+});
 ```
 
-### Get all
+## 🛠️ API Reference
 
-```javascript
-const data = await resource.getAll();
+### Database Operations
+
+#### Connect to Database
 
-// [
-//   obj1,
-//   obj2,
-//   ...
-// ]
+```javascript
+await s3db.connect();
+// Emits 'connected' event when ready
 ```
 
-### Delete many
+#### Create Resource
 
 ```javascript
-await resource.deleteMany(["id1", "id2", "id3 "]);
+const resource = await s3db.createResource({
+  name: "users",
+  attributes: {
+    name: "string",
+    email: "email"
+  }
+});
 ```
 
-### Delete all
+#### Get Resource Reference
 
 ```javascript
-await resource.deleteAll();
+const users = s3db.resource("users");
+// or 
+const users = s3db.resources.users
 ```
 
-### List ids
+### Resource Operations
+
+#### Insert Document
 
 ```javascript
-const ids = await resource.listIds();
+// With custom ID
+const user = await users.insert({
+  id: "user-123",
+  name: "John Doe",
+  email: "john@example.com"
+});
 
-// [
-//   'id1',
-//   'id2',
-//   'id3',
-// ]
+// Auto-generated ID
+const user = await users.insert({
+  name: "Jane Doe",
+  email: "jane@example.com"
+});
+// ID will be auto-generated using nanoid
 ```
 
----
+#### Get Document
 
-## Resource streams
-
-As we need to request the metadata for each id to return it's attributes, a better way to handle a huge amount off data might be using streams.
+```javascript
+const user = await users.get("user-123");
+console.log(user.name); // "John Doe"
+```
 
-### Readable stream
+#### Update Document
 
 ```javascript
-const readableStream = await resource.readable();
-
-readableStream.on("id", (id) => console.log("id =", id));
-readableStream.on("data", (lead) => console.log("lead.id =", lead.id));
-readableStream.on("end", console.log("end"));
+const updatedUser = await users.update("user-123", {
+  name: "John Smith",
+  age: 31
+});
+// Only specified fields are updated
 ```
 
-### Writable stream
+#### Upsert Document
 
 ```javascript
-const writableStream = await resource.writable();
-
-writableStream.write({
-  lead: {
-    fullName: "My Test Name",
-    personalEmail: `bulk-${k}@mymail.com`,
-    mobileNumber: "+55 11 1234567890",
-  },
+// Insert if doesn't exist, update if exists
+const user = await users.upsert("user-123", {
+  name: "John Doe",
+  email: "john@example.com",
+  age: 30
 });
 ```
 
----
+#### Delete Document
 
-## S3 Client
+```javascript
+await users.delete("user-123");
+```
 
-`s3db.js` has a S3 proxied client named [`S3Client`](https://github.com/forattini-dev/s3db.js/blob/main/src/s3-client.class.ts). It brings a few handy and less verbose functions to deal with AWS S3's api.
+#### Count Documents
 
 ```javascript
-import { S3Client } from "s3db.js";
-
-const client = new S3Client({ connectionString });
+const count = await users.count();
+console.log(`Total users: ${count}`);
 ```
 
-Each method has a **[:link:](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html) link** to the official `aws-sdk` docs.
+### Bulk Operations
 
-##### getObject [:link:](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getObject-property)
+#### Insert Many
 
 ```javascript
-const { Body, Metadata } = await client.getObject({
-  key: `my-prefixed-file.csv`,
-});
+const users = [
+  { name: "User 1", email: "user1@example.com" },
+  { name: "User 2", email: "user2@example.com" },
+  { name: "User 3", email: "user3@example.com" }
+];
 
-// AWS.Response
+await users.insertMany(users);
 ```
 
-##### putObject [:link:](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property)
+#### Get Many
 
 ```javascript
-const response = await client.putObject({
-  key: `my-prefixed-file.csv`,
-  contentType: "text/csv",
-  metadata: { a: "1", b: "2", c: "3" },
-  body: "a;b;c\n1;2;3\n4;5;6",
-});
-
-// AWS.Response
+const userList = await users.getMany(["user-1", "user-2", "user-3"]);
 ```
 
-##### headObject [:link:](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property)
+#### Delete Many
 
 ```javascript
-const { Metadata } = await client.headObject({
-  key: `my-prefixed-file.csv`,
-});
-
-// AWS.Response
+await users.deleteMany(["user-1", "user-2", "user-3"]);
 ```
 
-##### deleteObject [:link:](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#deleteObject-property)
+#### Get All
 
 ```javascript
-const response = await client.deleteObject({
-  key: `my-prefixed-file.csv`,
-});
-
-// AWS.Response
+const allUsers = await users.getAll();
+// Returns all documents in the resource
 ```
 
-##### deleteObjects [:link:](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#deleteObjects-property)
+#### List IDs
 
 ```javascript
-const response = await client.deleteObjects({
-  keys: [`my-prefixed-file.csv`, `my-other-prefixed-file.csv`],
-});
-
-// AWS.Response
+const userIds = await users.listIds();
+// Returns array of all document IDs
 ```
 
-##### listObjects [:link:](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjects-property)
+#### Delete All
 
 ```javascript
-const response = await client.listObjects({
-  prefix: `my-subdir`,
-});
-
-// AWS.Response
+await users.deleteAll();
+// ⚠️ Destructive operation - removes all documents
 ```
 
-##### count
+## 📊 Examples
 
-Custom made method to make it easier to count keys within a listObjects loop.
+### E-commerce Application
 
 ```javascript
-const count = await client.count({
-  prefix: `my-subdir`,
+// Create product resource with body-overflow behavior for long descriptions
+const products = await s3db.createResource({
+  name: "products",
+  attributes: {
+    name: "string|min:2|max:200",
+    description: "string|optional",
+    price: "number|positive",
+    category: "string",
+    tags: "array|items:string",
+    inStock: "boolean",
+    images: "array|items:url",
+    metadata: "object|optional"
+  },
+  options: {
+    behavior: "body-overflow",  // Handle long product descriptions
+    timestamps: true            // Track creation and update times
+  }
 });
 
-// 10
-```
+// Create order resource with enforce-limits for strict data control
+const orders = await s3db.createResource({
+  name: "orders",
+  attributes: {
+    customerId: "string",
+    products: "array|items:string",
+    total: "number|positive",
+    status: "string|enum:pending,paid,shipped,delivered",
+    shippingAddress: {
+      street: "string",
+      city: "string",
+      country: "string",
+      zipCode: "string"
+    },
+    createdAt: "date"
+  },
+  options: {
+    behavior: "enforce-limits",  // Strict validation for order data
+    timestamps: true
+  }
+});
 
-##### getAllKeys
+// Insert products (long descriptions will be handled by body-overflow)
+const product = await products.insert({
+  name: "Wireless Headphones",
+  description: "High-quality wireless headphones with noise cancellation, 30-hour battery life, premium comfort design, and crystal-clear audio quality. Perfect for music lovers, professionals, and gamers alike. Features include Bluetooth 5.0, active noise cancellation, touch controls, and a premium carrying case.",
+  price: 99.99,
+  category: "electronics",
+  tags: ["wireless", "bluetooth", "audio", "noise-cancellation"],
+  inStock: true,
+  images: ["https://example.com/headphones.jpg"]
+});
 
-Custom made method to make it easier to return all keys in a subpath within a listObjects loop.
+// Create order (enforce-limits ensures data integrity)
+const order = await orders.insert({
+  customerId: "customer-123",
+  products: [product.id],
+  total: 99.99,
+  status: "pending",
+  shippingAddress: {
+    street: "123 Main St",
+    city: "New York",
+    country: "USA",
+    zipCode: "10001"
+  },
+  createdAt: new Date()
+});
+```
 
-All returned keys will have the it's fullpath replaced with the current "scope" path.
+### User Authentication System
 
 ```javascript
-const keys = await client.getAllKeys({
-  prefix: `my-subdir`,
+// Create users resource with encrypted password and strict validation
+const users = await s3db.createResource({
+  name: "users",
+  attributes: {
+    username: "string|min:3|max:50|unique",
+    email: "email|unique",
+    password: "secret",  // Encrypted field
+    role: "string|enum:user,admin,moderator",
+    isActive: "boolean",
+    lastLogin: "date|optional",
+    profile: {
+      firstName: "string",
+      lastName: "string",
+      avatar: "url|optional",
+      bio: "string|optional"
+    }
+  },
+  options: {
+    behavior: "enforce-limits",  // Strict validation for user data
+    timestamps: true             // Track account creation and updates
+  }
 });
 
-// [
-//   key1,
-//   key2,
-//   ...
-// ]
+// Create sessions resource with body-overflow for session data
+const sessions = await s3db.createResource({
+  name: "sessions",
+  attributes: {
+    userId: "string",
+    token: "secret",  // Encrypted session token
+    expiresAt: "date",
+    userAgent: "string|optional",
+    ipAddress: "string|optional",
+    sessionData: "object|optional"  // Additional session metadata
+  },
+  options: {
+    behavior: "body-overflow",  // Handle large session data
+    timestamps: true
+  }
+});
+
+// Register user (enforce-limits ensures data integrity)
+const user = await users.insert({
+  username: "john_doe",
+  email: "john@example.com",
+  password: "secure_password_123",
+  role: "user",
+  isActive: true,
+  profile: {
+    firstName: "John",
+    lastName: "Doe"
+  }
+});
+
+// Create session (body-overflow preserves all session data)
+const session = await sessions.insert({
+  userId: user.id,
+  token: "jwt_token_here",
+  expiresAt: new Date(Date.now() + 24 * 60 * 60 * 1000), // 24 hours
+  userAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
+  ipAddress: "192.168.1.1",
+  sessionData: {
+    preferences: { theme: "dark", language: "en" },
+    lastActivity: new Date(),
+    deviceInfo: { type: "desktop", os: "Windows" }
+  }
+});
 ```
 
----
+## 🔄 Streaming
 
-## Events
+For large datasets, use streams to process data efficiently:
 
-The 3 main classes `S3db`, `Resource` and `S3Client` are extensions of Javascript's `EventEmitter`.
+### Readable Stream
 
-| S3Database | S3Client      | S3Resource | S3Resource Readable Stream |
-| ---------- | ------------- | ---------- | -------------------------- |
-| error      | error         | error      | error                      |
-| connected  | request       | insert     | id                         |
-|            | response      | get        | data                       |
-|            | response      | update     |                            |
-|            | getObject     | delete     |                            |
-|            | putObject     | count      |                            |
-|            | headObject    | insertMany |                            |
-|            | deleteObject  | deleteAll  |                            |
-|            | deleteObjects | listIds    |                            |
-|            | listObjects   | getMany    |                            |
-|            | count         | getAll     |                            |
-|            | getAllKeys    |            |                            |
+```javascript
+const readableStream = await users.readable();
 
-### S3Database
+readableStream.on("id", (id) => {
+  console.log("Processing user ID:", id);
+});
 
-#### error
+readableStream.on("data", (user) => {
+  console.log("User:", user.name);
+  // Process each user
+});
 
-```javascript
-s3db.on("error", (error) => console.error(error));
+readableStream.on("end", () => {
+  console.log("Finished processing all users");
+});
+
+readableStream.on("error", (error) => {
+  console.error("Stream error:", error);
+});
 ```
 
-#### connected
+### Writable Stream
 
 ```javascript
-s3db.on("connected", () => {});
-```
+const writableStream = await users.writable();
 
-### S3Client
+// Write data to stream
+writableStream.write({
+  name: "User 1",
+  email: "user1@example.com"
+});
 
-Using this reference for the events:
+writableStream.write({
+  name: "User 2", 
+  email: "user2@example.com"
+});
 
-```javascript
-const client = s3db.client;
+// End stream
+writableStream.end();
 ```
 
-#### error
+### Stream to CSV
 
 ```javascript
-client.on("error", (error) => console.error(error));
-```
+import fs from "fs";
+import { createObjectCsvWriter } from "csv-writer";
+
+const csvWriter = createObjectCsvWriter({
+  path: "users.csv",
+  header: [
+    { id: "id", title: "ID" },
+    { id: "name", title: "Name" },
+    { id: "email", title: "Email" }
+  ]
+});
 
-#### request
+const readableStream = await users.readable();
+const records = [];
 
-Emitted when a request is generated to AWS.
+readableStream.on("data", (user) => {
+  records.push(user);
+});
 
-```javascript
-client.on("request", (action, params) => {});
+readableStream.on("end", async () => {
+  await csvWriter.writeRecords(records);
+  console.log("CSV file created successfully");
+});
 ```
 
-#### response
+## 🔐 Security & Encryption
 
-Emitted when a response is received from AWS.
+### Field-Level Encryption
+
+Use the `"secret"` type for sensitive data:
 
 ```javascript
-client.on("response", (action, params, response) => {});
-```
+const users = await s3db.createResource({
+  name: "users",
+  attributes: {
+    username: "string",
+    email: "email",
+    password: "secret",        // Encrypted
+    apiKey: "secret",          // Encrypted
+    creditCard: "secret"       // Encrypted
+  }
+});
 
-#### getObject
+// Data is automatically encrypted/decrypted
+const user = await users.insert({
+  username: "john_doe",
+  email: "john@example.com",
+  password: "my_secure_password",  // Stored encrypted
+  apiKey: "sk_live_123456789",     // Stored encrypted
+  creditCard: "4111111111111111"   // Stored encrypted
+});
 
-```javascript
-client.on("getObject", (options, response) => {});
+// Retrieved data is automatically decrypted
+const retrieved = await users.get(user.id);
+console.log(retrieved.password); // "my_secure_password" (decrypted)
 ```
 
-#### putObject
+### Custom Encryption Key
 
 ```javascript
-client.on("putObject", (options, response) => {});
+import fs from "fs";
+
+const s3db = new S3db({
+  uri: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp",
+  passphrase: fs.readFileSync("./private-key.pem") // Custom encryption key
+});
 ```
 
-#### headObject
+## 💰 Cost Analysis
 
-```javascript
-client.on("headObject", (options, response) => {});
-```
+### Understanding S3 Costs
 
-#### deleteObject
+- **PUT Requests**: $0.000005 per 1,000 requests
+- **GET Requests**: $0.0000004 per 1,000 requests  
+- **Data Transfer**: $0.09 per GB
+- **Storage**: $0.023 per GB (but s3db.js uses 0-byte files)
 
-```javascript
-client.on("deleteObject", (options, response) => {});
-```
+### Cost Examples
 
-#### deleteObjects
+#### Small Application (1,000 users)
 
 ```javascript
-client.on("deleteObjects", (options, response) => {});
-```
+// Setup cost (one-time)
+const setupCost = 0.005; // 1,000 PUT requests
 
-#### listObjects
+// Monthly read cost
+const monthlyReadCost = 0.0004; // 1,000 GET requests
 
-```javascript
-client.on("listObjects", (options, response) => {});
+console.log(`Setup: $${setupCost}`);
+console.log(`Monthly reads: $${monthlyReadCost}`);
 ```
 
-#### count
+#### Large Application (1,000,000 users)
 
 ```javascript
-client.on("count", (options, response) => {});
-```
+// Setup cost (one-time)
+const setupCost = 5.00; // 1,000,000 PUT requests
 
-#### getAllKeys
+// Monthly read cost
+const monthlyReadCost = 0.40; // 1,000,000 GET requests
 
-```javascript
-client.on("getAllKeys", (options, response) => {});
+console.log(`Setup: $${setupCost}`);
+console.log(`Monthly reads: $${monthlyReadCost}`);
 ```
 
-### S3Resource
-
-Using this reference for the events:
+### Cost Tracking Plugin
 
 ```javascript
-const resource = s3db.resource("leads");
-```
+import { CostsPlugin } from "s3db.js";
 
-#### error
+const s3db = new S3db({
+  uri: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp",
+  plugins: [CostsPlugin]
+});
 
-```javascript
-resource.on("error", (err) => console.error(err));
+// After operations
+console.log("Total cost:", s3db.client.costs.total.toFixed(4), "USD");
+console.log("Requests made:", s3db.client.costs.requests.total);
 ```
 
-#### insert
+## 🎛️ Advanced Features
 
-```javascript
-resource.on("insert", (data) => {});
-```
+### AutoEncrypt / AutoDecrypt
 
-#### get
+Fields with the type `secret` are automatically encrypted and decrypted using the resource's passphrase. This ensures sensitive data is protected at rest.
 
-```javascript
-resource.on("get", (data) => {});
-```
+```js
+const users = await s3db.createResource({
+  name: "users",
+  attributes: {
+    username: "string",
+    password: "secret" // Will be encrypted
+  }
+});
 
-#### update
+const user = await users.insert({
+  username: "john_doe",
+  password: "my_secret_password"
+});
 
-```javascript
-resource.on("update", (attrs, data) => {});
+// The password is stored encrypted in S3, but automatically decrypted when retrieved
+const retrieved = await users.get(user.id);
+console.log(retrieved.password); // "my_secret_password"
 ```
 
-#### delete
+### Resource Events
 
-```javascript
-resource.on("delete", (id) => {});
+All resources emit events for key operations. You can listen to these events for logging, analytics, or custom workflows.
+
+```js
+users.on("insert", (data) => console.log("User inserted:", data.id));
+users.on("get", (data) => console.log("User retrieved:", data.id));
+users.on("update", (attrs, data) => console.log("User updated:", data.id));
+users.on("delete", (id) => console.log("User deleted:", id));
 ```
 
-#### count
+### Resource Schema Export/Import
 
-```javascript
-resource.on("count", (count) => {});
-```
+You can export and import resource schemas for backup, migration, or versioning purposes.
 
-#### insertMany
+```js
+// Export schema
+const schemaData = users.schema.export();
 
-```javascript
-resource.on("insertMany", (count) => {});
+// Import schema
+const importedSchema = Schema.import(schemaData);
 ```
 
-#### getMany
+## Partitions
 
-```javascript
-resource.on("getMany", (count) => {});
+`s3db.js` supports **partitions** to organize and query your data efficiently. Partitions allow you to group documents by one or more fields, making it easy to filter, archive, or manage large datasets.
+
+### Defining partitions
+
+You can define partitions when creating a resource using the `options.partitions` property:
+
+```js
+const users = await s3db.createResource({
+  name: "users",
+  attributes: {
+    name: "string",
+    email: "email",
+    region: "string",
+    ageGroup: "string"
+  },
+  options: {
+    partitions: {
+      byRegion: {
+        fields: { region: "string" }
+      },
+      byAgeGroup: {
+        fields: { ageGroup: "string" }
+      }
+    }
+  }
+});
 ```
 
-#### getAll
+### Querying by partition
 
-```javascript
-resource.on("getAll", (count) => {});
+```js
+// Find all users in the 'south' region
+const usersSouth = await users.query({ region: "south" });
+
+// Find all users in the 'adult' age group
+const adults = await users.query({ ageGroup: "adult" });
 ```
 
-#### deleteAll
+### Example: Time-based partition
 
-```javascript
-resource.on("deleteAll", (count) => {});
+```js
+const logs = await s3db.createResource({
+  name: "logs",
+  attributes: {
+    message: "string",
+    level: "string",
+    createdAt: "date"
+  },
+  options: {
+    partitions: {
+      byDate: {
+        fields: { createdAt: "date|maxlength:10" }
+      }
+    }
+  }
+});
+
+// Query logs for a specific day
+const logsToday = await logs.query({ createdAt: "2024-06-27" });
 ```
 
-#### listIds
+## Hooks
+
+`s3db.js` provides a powerful hooks system to let you run custom logic before and after key operations on your resources. Hooks can be used for validation, transformation, logging, or any custom workflow.
+
+### Supported hooks
+- `preInsert` / `afterInsert`
+- `preUpdate` / `afterUpdate`
+- `preDelete` / `afterDelete`
+
+### Registering hooks
+You can register hooks when creating a resource or dynamically:
+
+```js
+const users = await s3db.createResource({
+  name: "users",
+  attributes: { name: "string", email: "email" },
+  options: {
+    hooks: {
+      preInsert: [async (data) => {
+        if (!data.email.includes("@")) throw new Error("Invalid email");
+        return data;
+      }],
+      afterInsert: [async (data) => {
+        console.log("User inserted:", data.id);
+      }]
+    }
+  }
+});
 
-```javascript
-resource.on("listIds", (count) => {});
+// Or dynamically:
+users.addHook('preInsert', async (data) => {
+  // Custom logic
+  return data;
+});
 ```
 
----
+### Hook execution order
+- Internal hooks run first, user hooks run last (in the order they were added).
+- Hooks can be async and can modify the data (for `pre*` hooks).
+- If a hook throws, the operation is aborted.
 
 ## Plugins
 
-Anatomy of a plugin:
+`s3db.js` supports plugins to extend or customize its behavior. Plugins can hook into lifecycle events, add new methods, or integrate with external systems.
 
-```javascript
+### Example: Custom plugin
+
+```js
 const MyPlugin = {
-  setup(s3db: S3db) {},
-  start() {},
+  setup(s3db) {
+    console.log("Plugin setup");
+  },
+  start() {
+    console.log("Plugin started");
+  },
+  onUserCreated(user) {
+    console.log("New user created:", user.id);
+  }
 };
+
+const s3db = new S3db({
+  uri: "s3://...",
+  plugins: [MyPlugin]
+});
 ```
 
-We have an example of a _costs simulator plugin_ [here!](https://github.com/forattini-dev/s3db.js/blob/main/src/plugins/costs.plugin.js)
+## 🚨 Limitations & Best Practices
 
----
+### Limitations
 
-## Cost simulation
+1. **Document Size**: Maximum ~2KB per document (metadata only) - **💡 Use behaviors to handle larger documents**
+2. **No Complex Queries**: No SQL-like WHERE clauses or joins
+3. **No Indexes**: No automatic indexing for fast lookups
+4. **Sequential IDs**: Best performance with sequential IDs (00001, 00002, etc.)
+5. **No Transactions**: No ACID transactions across multiple operations
+6. **S3 Pagination**: S3 lists objects in pages of 1000 items maximum, and these operations are not parallelizable, which can make listing large datasets slow
 
-S3's pricing deep dive:
+**💡 Overcoming the 2KB Limit**: Use resource behaviors to handle documents that exceed the 2KB metadata limit:
+- **`body-overflow`**: Stores excess data in S3 object body (preserves all data)
+- **`data-truncate`**: Intelligently truncates data to fit within limits
+- **`enforce-limits`**: Strict validation to prevent oversized documents
+- **`user-management`**: Default behavior with warnings and monitoring
 
-- Data volume [1 GB x 0.023 USD]: it relates to the total volume of storage used and requests volume but, in this implementation, we just upload `0 bytes` files.
-- GET Requests [1,000 GET requests in a month x 0.0000004 USD per request = 0.0004 USD]: every read requests
-- PUT Requests [1,000 PUT requests for S3 Standard Storage x 0.000005 USD per request = 0.005 USD]: every write request
-- Data transfer [Internet: 1 GB x 0.09 USD per GB = 0.09 USD]:
+### ✅ Recent Improvements
 
-Check by yourself the pricing page details at https://aws.amazon.com/s3/pricing/ and https://calculator.aws/#/addService/S3.
+**🔧 Enhanced Data Serialization (v3.3.2+)**
 
-### Big example
+s3db.js now handles complex data structures robustly:
 
-Lets try to simulate a big project where you have a database with a few tables:
+- **Empty Arrays**: `[]` correctly serialized and preserved
+- **Null Arrays**: `null` values maintained without corruption  
+- **Special Characters**: Arrays with pipe `|` characters properly escaped
+- **Empty Objects**: `{}` correctly mapped and stored
+- **Null Objects**: `null` object values preserved during serialization
+- **Nested Structures**: Complex nested objects with mixed empty/null values supported
 
-- pageviews: 100,000,000 lines of 100 bytes each
-- leads: 1,000,000 lines of 200 bytes each
+### Best Practices
 
-```javascript
-const Fakerator = require("fakerator");
-const fake = Fakerator("pt-BR");
+#### 1. Design for Document Storage
 
-const pageview = {
-  ip: this.faker.internet.ip(),
-  domain: this.faker.internet.url(),
-  path: this.faker.internet.url(),
-  query: `?q=${this.faker.lorem.word()}`,
+```javascript
+// ✅ Good: Nested structure is fine
+const user = {
+  id: "user-123",
+  name: "John Doe",
+  email: "john@example.com",
+  profile: {
+    bio: "Software developer",
+    avatar: "https://example.com/avatar.jpg",
+    preferences: {
+      theme: "dark",
+      notifications: true
+    }
+  }
 };
 
-const lead = {
-  name: fake.names.name(),
-  mobile: fake.phone.number(),
-  email: fake.internet.email(),
-  country: "Brazil",
-  city: fake.address.city(),
-  state: fake.address.countryCode(),
-  address: fake.address.street(),
+// ❌ Avoid: Large arrays in documents
+const user = {
+  id: "user-123",
+  name: "John Doe",
+  // This could exceed metadata limits
+  purchaseHistory: [
+    { id: "order-1", date: "2023-01-01", total: 99.99 },
+    { id: "order-2", date: "2023-01-15", total: 149.99 },
+    // ... many more items
+  ]
 };
 ```
 
-If you write the whole database of:
+#### 2. Use Sequential IDs
 
-- pageviews:
-  - 100,000,000 PUT requests for S3 Standard Storage x 0.000005 USD per request = 500.00 USD (S3 Standard PUT requests cost)
-- leads:
-  - 1,000,000 PUT requests for S3 Standard Storage x 0.000005 USD per request = 5.00 USD (S3 Standard PUT requests cost)
+```javascript
+// ✅ Good: Sequential IDs for better performance
+const users = ["00001", "00002", "00003", "00004"];
 
-It will cost 505.00 USD, once.
+// ⚠️ Acceptable: Random IDs (but ensure sufficient uniqueness)
+const users = ["abc123", "def456", "ghi789", "jkl012"];
 
-If you want to read the whole database:
+// ❌ Avoid: Random IDs with low combinations (risk of collisions)
+const users = ["a1", "b2", "c3", "d4"]; // Only 26*10 = 260 combinations
+```
 
-- pageviews:
-  - 100,000,000 GET requests in a month x 0.0000004 USD per request = 40.00 USD (S3 Standard GET requests cost)
-  - (100,000,000 × 100 bytes)÷(1024×1000×1000) ≅ 10 Gb
-    Internet: 10 GB x 0.09 USD per GB = 0.90 USD
-- leads:
-  - 1,000,000 GET requests in a month x 0.0000004 USD per request = 0.40 USD (S3 Standard GET requests cost)
-  - (1,000,000 × 200 bytes)÷(1024×1000×1000) ≅ 0.19 Gb
-    Internet: 1 GB x 0.09 USD per GB = 0.09 USD
+#### 3. Optimize for Read Patterns
 
-It will cost 41.39 USD, once.
+```javascript
+// ✅ Good: Store frequently accessed data together
+const order = {
+  id: "order-123",
+  customerId: "customer-456",
+  customerName: "John Doe",  // Denormalized for quick access
+  items: ["product-1", "product-2"],
+  total: 99.99
+};
 
-### Small example
+// ❌ Avoid: Requiring multiple lookups
+const order = {
+  id: "order-123",
+  customerId: "customer-456",  // Requires separate lookup
+  items: ["product-1", "product-2"]
+};
+```
 
-Lets save some JWT tokens using the [RFC:7519](https://www.rfc-editor.org/rfc/rfc7519.html).
+#### 4. Use Streaming for Large Datasets
 
 ```javascript
-await s3db.createResource({
-  name: "tokens",
-  attributes: {
-    iss: 'url|max:256',
-    sub: 'string',
-    aud: 'string',
-    exp: 'number',
-    email: 'email',
-    name: 'string',
-    scope: 'string',
-    email_verified: 'boolean',
-  })
-
-function generateToken () {
-  const token = createTokenLib(...)
-
-  await resource.insert({
-    id: token.jti || md5(token)
-    ...token,
-  })
-
-  return token
-}
+// ✅ Good: Use streams for large operations
+const readableStream = await users.readable();
+readableStream.on("data", (user) => {
+  // Process each user individually
+});
+
+// ❌ Avoid: Loading all data at once
+const allUsers = await users.getAll(); // May timeout with large datasets
+```
 
-function validateToken (token) {
-  const id = token.jti || md5(token)
+#### 5. Implement Proper Error Handling
 
-  if (!validateTokenSignature(token, ...)) {
-    await resource.deleteById(id)
-    throw new Error('invalid-token')
+```javascript
+// Method 1: Try-catch with get()
+try {
+  const user = await users.get("non-existent-id");
+} catch (error) {
+  if (error.message.includes("does not exist")) {
+    console.log("User not found");
+  } else {
+    console.error("Unexpected error:", error);
   }
+}
 
-  return resource.getById(id)
+// Method 2: Check existence first (⚠️ Additional request cost)
+const userId = "user-123";
+if (await users.exists(userId)) {
+  const user = await users.get(userId);
+  console.log("User found:", user.name);
+} else {
+  console.log("User not found");
 }
 ```
 
-## Roadmap
+**⚠️ Cost Warning**: Using `exists()` creates an additional S3 request. For high-volume operations, prefer the try-catch approach to minimize costs.
+
+#### 6. Choose the Right Behavior Strategy
+
+```javascript
+// ✅ For development and testing - allows flexibility
+const devUsers = await s3db.createResource({
+  name: "users",
+  attributes: { name: "string", email: "email" },
+  options: { behavior: "user-management" }
+});
+
+// ✅ For production with strict data control
+const prodUsers = await s3db.createResource({
+  name: "users",
+  attributes: { name: "string", email: "email" },
+  options: { behavior: "enforce-limits" }
+});
+
+// ✅ For preserving all data with larger documents
+const blogPosts = await s3db.createResource({
+  name: "posts",
+  attributes: { title: "string", content: "string", author: "string" },
+  options: { behavior: "body-overflow" }
+});
+
+// ✅ For structured data where truncation is acceptable
+const productDescriptions = await s3db.createResource({
+  name: "products",
+  attributes: { name: "string", description: "string", price: "number" },
+  options: { behavior: "data-truncate" }
+});
+```
+
+**Behavior Selection Guide:**
+- **`user-management`**: Development, testing, or when you want full control
+- **`enforce-limits`**: Production systems requiring strict data validation
+- **`body-overflow`**: When data integrity is critical and you need to preserve all information
+- **`data-truncate`**: When you can afford to lose some data but want to maintain structure
 
-Tasks board can be found at [this link](https://github.com/orgs/forattini-dev/projects/5/views/1)!
+### Performance Tips
 
-Feel free to interact and PRs are welcome! :)
+1. **Enable Caching**: Use `cache: true` for frequently accessed data
+2. **Adjust Parallelism**: Increase `parallelism`