s3db.js 6.1.0 → 7.0.0

package/README.md

@@ -93,6 +93,8 @@
 
 ## 📋 Table of Contents
 
+- [🚀 What is s3db.js?](#-what-is-s3dbjs)
+- [✨ Key Features](#-key-features)
 - [🚀 Quick Start](#-quick-start)
 - [💾 Installation](#-installation)
 - [🎯 Core Concepts](#-core-concepts)
@@ -100,22 +102,14 @@
 - [🔄 Resource Versioning System](#-resource-versioning-system)
 - [🆔 Custom ID Generation](#-custom-id-generation)
 - [🔌 Plugin System](#-plugin-system)
-- [🎛️ Advanced Behaviors](#️-advanced-behaviors)
+- [🔄 Replicator System](#-replicator-system)
+- [🎛️ Resource Behaviors](#️-resource-behaviors)
 - [🔄 Advanced Streaming API](#-advanced-streaming-api)
 - [📁 Binary Content Management](#-binary-content-management)
 - [🗂️ Advanced Partitioning](#️-advanced-partitioning)
 - [🎣 Advanced Hooks System](#-advanced-hooks-system)
+- [🧩 Resource Middlewares](#-resource-middlewares)
 - [📖 API Reference](#-api-reference)
-- [🎨 Examples](#-examples)
-- [🔐 Security](#-security)
-- [⚙️ Advanced Configuration Options](#️-advanced-configuration-options)
-- [📡 Events and Emitters](#-events-and-emitters)
-- [🔧 Troubleshooting](#-troubleshooting)
-- [💰 Cost Analysis](#-cost-analysis)
-- [🚨 Best Practices](#-best-practices)
-- [🧪 Testing](#-testing)
-- [🤝 Contributing](#-contributing)
-- [📄 License](#-license)
 
 ---
 
@@ -206,23 +200,22 @@ yarn add s3db.js
 
 Some features require additional dependencies to be installed manually:
 
-#### Replication Dependencies
+#### replicator Dependencies
 
-If you plan to use the replication system with external services, install the corresponding dependencies:
+If you plan to use the replicator system with external services, install the corresponding dependencies:
 
 ```bash
-# For SQS replication (AWS SQS queues)
+# For SQS replicator (AWS SQS queues)
 npm install @aws-sdk/client-sqs
 
-# For BigQuery replication (Google BigQuery)
+# For BigQuery replicator (Google BigQuery)
 npm install @google-cloud/bigquery
 
-# For PostgreSQL replication (PostgreSQL databases)
+# For PostgreSQL replicator (PostgreSQL databases)
 npm install pg
 ```
 
 **Why manual installation?** These are marked as `peerDependencies` to keep the main package lightweight. Only install what you need!
-```
 
 ### Environment Setup
 
@@ -310,7 +303,7 @@ const users = await s3db.createResource({
     password: "secret"
   },
   timestamps: true,
-        behavior: "user-managed",
+  behavior: "user-managed",
   partitions: {
     byRegion: { fields: { region: "string" } }
   }
@@ -358,7 +351,7 @@ const products = await s3db.createResource({
   name: "products",
   attributes: { name: "string", price: "number" },
   hooks: {
-    preInsert: [async (data) => {
+    beforeInsert: [async (data) => {
       data.sku = `${data.category.toUpperCase()}-${Date.now()}`;
       return data;
     }],
@@ -384,21 +377,6 @@ readableStream.on("end", () => console.log("✅ Export completed"));
 const writableStream = await users.writable();
 importData.forEach(userData => writableStream.write(userData));
 writableStream.end();
-```
-    value: "string"
-  },
-  behavior: "enforce-limits" // Ensures data stays within 2KB
-});
-
-// Smart truncation - preserves structure, truncates content
-const summaries = await s3db.createResource({
-  name: "summaries",
-  attributes: {
-    title: "string",
-    description: "string"
-  },
-  behavior: "data-truncate" // Truncates to fit within limits
-});
 ```
 
 ### 🔄 Resource Versioning System
@@ -485,9 +463,25 @@ const timestampUsers = await s3db.createResource({
 });
 ```
 
+#### 📏 **Intelligent Data Compression**
+
+s3db.js automatically compresses numeric data using **Base62 encoding** to maximize your S3 metadata space (2KB limit):
+
+| Data Type | Original | Compressed | Space Saved |
+|-----------|----------|------------|-------------|
+| `10000` | `10000` (5 digits) | `2Bi` (3 digits) | **40%** |
+| `123456789` | `123456789` (9 digits) | `8m0Kx` (5 digits) | **44%** |
+| Large arrays | `[1,2,3,999999]` (13 chars) | `1,2,3,hBxM` (9 chars) | **31%** |
+
+**Performance Benefits:**
+- ⚡ **5x faster** encoding for large numbers vs Base36
+- 🗜️ **41% compression** for typical numeric data  
+- 🚀 **Space efficient** - more data fits in S3 metadata
+- 🔄 **Automatic** - no configuration required
+
 ### 🔌 Plugin System
 
-Extend functionality with powerful plugins. s3db.js supports multiple plugins working together seamlessly:
+Extend s3db.js with powerful plugins for caching, monitoring, replication, search, and more:
 
 ```javascript
 import { 
@@ -495,457 +489,49 @@ import {
   CostsPlugin, 
   FullTextPlugin, 
   MetricsPlugin, 
-  ReplicationPlugin, 
+  ReplicatorPlugin, 
   AuditPlugin 
 } from 's3db.js';
 
 const s3db = new S3db({
   connectionString: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp",
   plugins: [
-    new CachePlugin({ enabled: true }), // CachePlugin needs instantiation
-    CostsPlugin, // CostsPlugin is a static object
-    new FullTextPlugin({ fields: ['name', 'description'] }),
-    new MetricsPlugin({ enabled: true }),
-    new ReplicationPlugin({ 
-      enabled: true, 
-      replicators: [
-        {
-          driver: 's3db',
-          resources: ['users', 'products'],
-          config: {
-            connectionString: "s3://BACKUP_KEY:BACKUP_SECRET@BACKUP_BUCKET/backup"
-          }
-        }
-      ]
+    new CachePlugin(),                        // 💾 Intelligent caching
+    CostsPlugin,                              // 💰 Cost tracking
+    new FullTextPlugin({ fields: ['name'] }), // 🔍 Full-text search
+    new MetricsPlugin(),                      // 📊 Performance monitoring
+    new ReplicatorPlugin({                    // 🔄 Data replication
+      replicators: [{
+        driver: 's3db',
+        resources: ['users'],
+        config: { connectionString: "s3://backup-bucket/backup" }
+      }]
     }),
-    new AuditPlugin({ enabled: true })
+    new AuditPlugin()                         // 📝 Audit logging
   ]
 });
 
-// All plugins work together seamlessly
-await users.insert({ name: "John", email: "john@example.com" });
-// - Cache: Caches the operation
-// - Costs: Tracks S3 costs
-// - FullText: Indexes the data for search
-// - Metrics: Records performance metrics
-// - Replication: Syncs to configured replicators
-// - Audit: Logs the operation
-```
-
-### 🔄 Replicator System
-
-The Replication Plugin now supports a flexible driver-based system for replicating data to different targets. Each replicator driver handles a specific type of target system.
-
-#### Available Replicators
-
-**S3DB Replicator** - Replicates data to another s3db instance:
-```javascript
-{
-  driver: 's3db',
-  resources: ['users', 'products'], // <-- root level
-  config: {
-    connectionString: "s3://BACKUP_KEY:BACKUP_SECRET@BACKUP_BUCKET/backup",
-    region: 'us-west-2'
-  }
-}
-```
-
-**SQS Replicator** - Sends data to AWS SQS queues:
-```javascript
-{
-  driver: 'sqs',
-  resources: ['orders'],
-  config: {
-    queueUrl: 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue',
-    region: 'us-east-1',
-    messageGroupId: 's3db-replication', // For FIFO queues
-    deduplicationId: true // Enable deduplication
-  }
-}
-```
-
-**BigQuery Replicator** - Sends data to Google BigQuery:
-```javascript
-{
-  driver: 'bigquery',
-  resources: ['users', 'orders'],
-  config: {
-    projectId: 'my-project',
-    datasetId: 'analytics',
-    tableId: 's3db_replication',
-    location: 'US',
-    credentials: {
-      // Your Google Cloud service account credentials
-      client_email: 'service-account@project.iam.gserviceaccount.com',
-      private_key: '-----BEGIN PRIVATE KEY-----\n...'
-    }
-  }
-}
-```
-
-**PostgreSQL Replicator** - Sends data to PostgreSQL databases:
-```javascript
-{
-  driver: 'postgres',
-  resources: ['users'],
-  config: {
-    connectionString: 'postgresql://user:pass@localhost:5432/analytics',
-    // OR individual parameters:
-    // host: 'localhost',
-    // port: 5432,
-    // database: 'analytics',
-    // user: 'user',
-    // password: 'pass',
-    tableName: 's3db_replication',
-    ssl: false
-  }
-}
-```
-
-#### Replicator Features
-
-- **Resource Filtering**: Each replicator can be configured to handle specific resources only
-- **Event Emission**: All replicators emit events for monitoring and debugging
-- **Connection Testing**: Test connections to replicators before use
-- **Batch Operations**: Support for batch replication operations
-- **Error Handling**: Comprehensive error handling and retry logic
-- **Status Monitoring**: Get detailed status and statistics for each replicator
-
-#### Dependencies
-
-The replicators use optional peer dependencies. Install only what you need:
-
-```bash
-# For SQS replicator
-npm install @aws-sdk/client-sqs
-# or
-yarn add @aws-sdk/client-sqs
-# or
-pnpm add @aws-sdk/client-sqs
-
-# For BigQuery replicator
-npm install @google-cloud/bigquery
-# or
-yarn add @google-cloud/bigquery
-# or
-pnpm add @google-cloud/bigquery
-
-# For PostgreSQL replicator
-npm install pg
-# or
-yarn add pg
-# or
-pnpm add pg
-```
-
-**⚠️ Important:** These dependencies are marked as `peerDependencies` in the package.json, which means they are not automatically installed with s3db.js. You must install them manually if you plan to use the corresponding replicators. If you don't install the required dependency, the replicator will throw an error when trying to initialize.
-
-**Example error without dependency:**
-```
-Error: Cannot find module '@aws-sdk/client-sqs'
-```
-
-**Solution:** Install the missing dependency as shown above.
-
-#### Example Usage
-
-See `examples/e34-replicators.js` for a complete example using all four replicator types.
-
-**Prerequisites:** Make sure to install the required dependencies before running the example:
-
-```bash
-# Install all replication dependencies for the full example
-npm install @aws-sdk/client-sqs @google-cloud/bigquery pg
-```
-
-#### 🔄 Cache Plugin
-
-#### 🔄 Cache Plugin
-Intelligent caching to reduce API calls and improve performance:
-
-```javascript
-const s3db = new S3db({
-  connectionString: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp",
-  plugins: [new CachePlugin({
-    enabled: true,
-    ttl: 300000, // 5 minutes cache
-    maxSize: 1000, // Max 1000 items in cache
-    driverType: 'memory' // 'memory' or 's3'
-  })]
-});
-
-// Automatic caching for reads
-await users.count(); // Cached for 5 minutes
-await users.list();  // Cached for 5 minutes
-await users.insert({...}); // Automatically clears cache
-```
-
-#### 💰 Costs Plugin
-Track and monitor AWS S3 costs in real-time:
-
-```javascript
-const s3db = new S3db({
-  connectionString: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp",
-  plugins: [CostsPlugin]
-});
+await s3db.connect();
 
-// Track costs automatically
+// All plugins work together seamlessly
 await users.insert({ name: "John", email: "john@example.com" });
-await users.list();
-
-// Get cost information
-console.log(s3db.client.costs); 
-// { total: 0.000009, requests: { total: 3, get: 1, put: 1, list: 1 } }
+// ✅ Data cached, costs tracked, indexed for search, metrics recorded, replicated, and audited
 ```
 
-#### 🔍 Full-Text Search Plugin
-Powerful text search with automatic indexing:
+#### Available Plugins
 
-```javascript
-const s3db = new S3db({
-  connectionString: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp",
-  plugins: [new FullTextPlugin({
-    enabled: true,
-    fields: ['name', 'description', 'content'], // Fields to index
-    minWordLength: 3,
-    maxResults: 50,
-    language: 'en-US'
-  })]
-});
+- **💾 Cache Plugin** - Intelligent caching (memory/S3) for performance
+- **💰 Costs Plugin** - Real-time AWS S3 cost tracking  
+- **🔍 FullText Plugin** - Advanced search with automatic indexing
+- **📊 Metrics Plugin** - Performance monitoring and analytics
+- **🔄 Replicator Plugin** - Multi-target replication (S3DB, SQS, BigQuery, PostgreSQL)
+- **📝 Audit Plugin** - Comprehensive audit logging for compliance
+- **📬 Queue Consumer Plugin** - Message consumption from SQS/RabbitMQ
 
-// Create resource with searchable fields
-const products = await s3db.createResource({
-  name: "products",
-  attributes: {
-    name: "string|required",
-    description: "string",
-    content: "string"
-  }
-});
-
-// Insert data (automatically indexed)
-await products.insert({
-  name: "JavaScript Book",
-  description: "Learn JavaScript programming",
-  content: "Comprehensive guide to modern JavaScript"
-});
-
-// Search across all indexed fields
-const results = await s3db.plugins.fulltext.searchRecords('products', 'javascript');
-console.log(results); // Returns products with search scores
-
-// Example of search results:
-// [
-//   {
-//     id: "prod-123",
-//     name: "JavaScript Book",
-//     description: "Learn JavaScript programming",
-//     content: "Comprehensive guide to modern JavaScript",
-//     _searchScore: 0.85,
-//     _matchedFields: ["name", "description", "content"],
-//     _matchedWords: ["javascript"]
-//   },
-//   {
-//     id: "prod-456", 
-//     name: "Web Development Guide",
-//     description: "Includes JavaScript, HTML, and CSS",
-//     content: "Complete web development with JavaScript",
-//     _searchScore: 0.72,
-//     _matchedFields: ["description", "content"],
-//     _matchedWords: ["javascript"]
-//   }
-// ]
-```
+**📖 For detailed documentation, configuration options, and advanced examples, see:**
+**[📋 Complete Plugin Documentation](https://github.com/forattini-dev/s3db.js/blob/main/PLUGINS.md)**
 
-#### 📊 Metrics Plugin
-Monitor performance and usage metrics:
-
-```javascript
-const s3db = new S3db({
-  connectionString: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp",
-  plugins: [new MetricsPlugin({
-    enabled: true,
-    collectPerformance: true,
-    collectErrors: true,
-    collectUsage: true,
-    flushInterval: 60000 // Flush every minute
-  })]
-});
-
-// Metrics are collected automatically
-await users.insert({ name: "John" });
-await users.list();
-
-// Get metrics
-const metrics = await s3db.plugins.metrics.getMetrics();
-console.log(metrics); // Performance and usage data
-
-// Example of metrics object:
-// {
-//   performance: {
-//     averageResponseTime: 245, // milliseconds
-//     totalRequests: 1250,
-//     requestsPerSecond: 12.5,
-//     slowestOperations: [
-//       { operation: "list", resource: "users", avgTime: 450, count: 50 },
-//       { operation: "get", resource: "products", avgTime: 320, count: 200 }
-//     ]
-//   },
-//   usage: {
-//     resources: {
-//       users: { inserts: 150, updates: 75, deletes: 10, reads: 800 },
-//       products: { inserts: 300, updates: 120, deletes: 25, reads: 1200 }
-//     },
-//     totalOperations: 2680,
-//     mostActiveResource: "products",
-//     peakUsageHour: "14:00"
-//   },
-//   errors: {
-//     total: 15,
-//     byType: {
-//       "ValidationError": 8,
-//       "NotFoundError": 5,
-//       "PermissionError": 2
-//     },
-//     byResource: {
-//       users: 10,
-//       products: 5
-//     }
-//   },
-//   cache: {
-//     hitRate: 0.78, // 78% cache hit rate
-//     totalHits: 980,
-//     totalMisses: 270,
-//     averageCacheTime: 120 // milliseconds
-//   }
-// }
-```
-
-#### 🔄 Replication Plugin
-Replicate data to other buckets or regions:
-
-```javascript
-const s3db = new S3db({
-  connectionString: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp",
-  plugins: [new ReplicationPlugin({
-    enabled: true,
-    replicators: [
-      {
-        driver: 's3db',
-        resources: ['users', 'products'],
-        config: {
-          connectionString: "s3://BACKUP_KEY:BACKUP_SECRET@BACKUP_BUCKET/backup"
-        }
-      },
-      {
-        driver: 'sqs',
-        resources: ['orders', 'users', 'products'],
-        config: {
-          // Resource-specific queues
-          queues: {
-            users: 'https://sqs.us-east-1.amazonaws.com/123456789012/users-events.fifo',
-            orders: 'https://sqs.us-east-1.amazonaws.com/123456789012/orders-events.fifo',
-            products: 'https://sqs.us-east-1.amazonaws.com/123456789012/products-events.fifo'
-          },
-          // Fallback queue for unspecified resources
-          defaultQueueUrl: 'https://sqs.us-east-1.amazonaws.com/123456789012/default-events.fifo',
-          messageGroupId: 's3db-replication', // For FIFO queues
-          deduplicationId: true // Enable deduplication
-        }
-      },
-      {
-        driver: 'bigquery',
-        resources: ['users', 'orders'],
-        config: {
-          projectId: 'my-project',
-          datasetId: 'analytics',
-          tableId: 's3db_replication'
-        }
-      },
-      {
-        driver: 'postgres',
-        resources: ['users'],
-        config: {
-          connectionString: 'postgresql://user:pass@localhost:5432/analytics',
-          tableName: 's3db_replication'
-        }
-      }
-    ],
-    syncInterval: 300000 // Sync every 5 minutes
-  })]
-});
-
-// Data is automatically replicated to all configured targets
-await users.insert({ name: "John" }); // Synced to all replicators
-
-**SQS Message Structure:**
-
-The SQS replicator sends standardized messages with the following structure:
-
-```javascript
-// INSERT operation
-{
-  resource: "users",
-  action: "insert",
-  data: { _v: 0, id: "user-001", name: "John", email: "john@example.com" },
-  timestamp: "2024-01-01T10:00:00.000Z",
-  source: "s3db-replication"
-}
-
-// UPDATE operation (includes before/after data)
-{
-  resource: "users",
-  action: "update",
-  before: { _v: 0, id: "user-001", name: "John", age: 30 },
-  data: { _v: 1, id: "user-001", name: "John", age: 31 },
-  timestamp: "2024-01-01T10:05:00.000Z",
-  source: "s3db-replication"
-}
-
-// DELETE operation
-{
-  resource: "users",
-  action: "delete",
-  data: { _v: 1, id: "user-001", name: "John", age: 31 },
-  timestamp: "2024-01-01T10:10:00.000Z",
-  source: "s3db-replication"
-}
-```
-
-**Queue Routing:**
-- Each resource can have its own dedicated queue
-- Unspecified resources use the default queue
-- FIFO queues supported with deduplication
-- Messages are automatically routed to the appropriate queue
-```
-
-#### 📝 Audit Plugin
-Log all operations for compliance and traceability:
-
-```javascript
-const s3db = new S3db({
-  connectionString: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp",
-  plugins: [new AuditPlugin({
-    enabled: true,
-    trackOperations: ['insert', 'update', 'delete', 'get'],
-    includeData: false, // Don't log sensitive data
-    retentionDays: 90
-  })]
-});
-
-// All operations are logged
-await users.insert({ name: "John" });
-await users.update(userId, { age: 31 });
-
-// Get audit logs
-const logs = await s3db.plugins.audit.getAuditLogs({
-  resourceName: 'users',
-  operation: 'insert'
-});
-console.log(logs); // Audit trail
-```
-
-### 🎛️ Advanced Behaviors
+### 🎛️ Resource Behaviors
 
 Choose the right behavior strategy for your use case:
 
@@ -955,7 +541,7 @@ Choose the right behavior strategy for your use case:
 |------------------|-------------|-----------|----------------|-------------------------|
 | `user-managed`   | None        | Possible  | Warns          | Dev/Test/Advanced users |
 | `enforce-limits` | Strict      | No        | Throws         | Production              |
-| `data-truncate`  | Truncates   | Yes       | Warns          | Content Mgmt            |
+| `truncate-data`  | Truncates   | Yes       | Warns          | Content Mgmt            |
 | `body-overflow`  | Truncates/Splits | Yes   | Warns          | Large objects           |
 | `body-only`      | Unlimited   | No        | No             | Large JSON/Logs         |
 
@@ -1006,10 +592,10 @@ await users.insert({
 **Best Practices & Warnings:**
 - Exceeding S3 metadata limits will cause silent data loss or errors at the storage layer.
 - Use this behavior only if you have custom logic to handle warnings and enforce limits.
-- For production, prefer `enforce-limits` or `data-truncate` to avoid data loss.
+- For production, prefer `enforce-limits` or `truncate-data` to avoid data loss.
 
 **Migration Tips:**
-- To migrate to a stricter behavior, change the resource's behavior to `enforce-limits` or `data-truncate`.
+- To migrate to a stricter behavior, change the resource's behavior to `enforce-limits` or `truncate-data`.
 - Review emitted warnings to identify resources at risk of exceeding S3 limits.
 
 #### Enforce Limits Behavior
@@ -1040,7 +626,7 @@ const summaries = await s3db.createResource({
     description: "string",
     content: "string"
   },
-  behavior: "data-truncate"
+  behavior: "truncate-data"
 });
 
 // Automatically truncates to fit within 2KB
@@ -1348,6 +934,7 @@ const usUsers = await users.list({
 
 // Note: The system automatically manages partition references internally
 // Users should use standard list() method with partition parameters
+```
 
 #### Automatic Timestamp Partitions
 
@@ -1405,14 +992,14 @@ const users = await s3db.createResource({
   name: "users",
   attributes: { name: "string", email: "string" },
   hooks: {
-    preInsert: [
+    beforeInsert: [
       async (data) => {
-        console.log('1. Pre-insert hook 1');
+        console.log('1. Before-insert hook 1');
         data.timestamp = new Date().toISOString();
         return data;
       },
       async (data) => {
-        console.log('2. Pre-insert hook 2');
+        console.log('2. Before-insert hook 2');
         data.processed = true;
         return data;
       }
@@ -1430,7 +1017,7 @@ const users = await s3db.createResource({
   }
 });
 
-// Execution order: preInsert hooks → insert → afterInsert hooks
+// Execution order: beforeInsert hooks → insert → afterInsert hooks
 ```
 
 #### Version-Specific Hooks
@@ -1442,7 +1029,7 @@ const users = await s3db.createResource({
   attributes: { name: "string", email: "string" },
   versioningEnabled: true,
   hooks: {
-    preInsert: [
+    beforeInsert: [
       async (data) => {
         // Access resource context
         console.log('Current version:', this.version);
@@ -1464,19 +1051,19 @@ users.on('versionUpdated', ({ oldVersion, newVersion }) => {
 const users = await s3db.createResource({
   name: "users",
   attributes: { name: "string", email: "string" },
-  hooks: {
-    preInsert: [
-      async (data) => {
-        try {
-          // Validate external service
-          await validateEmail(data.email);
-          return data;
-        } catch (error) {
-          // Transform error or add context
-          throw new Error(`Email validation failed: ${error.message}`);
+      hooks: {
+      beforeInsert: [
+        async (data) => {
+          try {
+            // Validate external service
+            await validateEmail(data.email);
+            return data;
+          } catch (error) {
+            // Transform error or add context
+            throw new Error(`Email validation failed: ${error.message}`);
+          }
         }
-      }
-    ],
+      ],
     afterInsert: [
       async (data) => {
         try {
@@ -1498,7 +1085,7 @@ const users = await s3db.createResource({
   name: "users",
   attributes: { name: "string", email: "string" },
   hooks: {
-    preInsert: [
+    beforeInsert: [
       async function(data) {
         // 'this' is bound to the resource instance
         console.log('Resource name:', this.name);
@@ -1517,6 +1104,303 @@ const users = await s3db.createResource({
 });
 ```
 
+### 🧩 Resource Middlewares
+
+The Resource class supports a powerful middleware system, similar to Express/Koa, allowing you to intercept, modify, or extend the behavior of core methods like `insert`, `get`, `update`, `delete`, `list`, and more.
+
+**Supported methods for middleware:**
+- `get`
+- `list`
+- `listIds`
+- `getAll`
+- `count`
+- `page`
+- `insert`
+- `update`
+- `delete`
+- `deleteMany`
+- `exists`
+- `getMany`
+
+#### Middleware Signature
+```js
+async function middleware(ctx, next) {
+  // ctx.resource: Resource instance
+  // ctx.args: arguments array (for the method)
+  // ctx.method: method name (e.g., 'insert')
+  // next(): calls the next middleware or the original method
+}
+```
+
+#### Example: Logging Middleware for Insert
+```js
+const users = await s3db.createResource({
+  name: "users",
+  attributes: { name: "string", email: "string" }
+});
+
+users.useMiddleware('insert', async (ctx, next) => {
+  console.log('Before insert:', ctx.args[0]);
+  // You can modify ctx.args if needed
+  ctx.args[0].name = ctx.args[0].name.toUpperCase();
+  const result = await next();
+  console.log('After insert:', result);
+  return result;
+});
+
+await users.insert({ name: "john", email: "john@example.com" });
+// Output:
+// Before insert: { name: 'john', email: 'john@example.com' }
+// After insert: { id: '...', name: 'JOHN', email: 'john@example.com', ... }
+```
+
+#### Example: Validation or Metrics Middleware
+```js
+users.useMiddleware('update', async (ctx, next) => {
+  if (!ctx.args[1].email) throw new Error('Email is required for update!');
+  const start = Date.now();
+  const result = await next();
+  const duration = Date.now() - start;
+  console.log(`Update took ${duration}ms`);
+  return result;
+});
+```
+
+#### 🔒 Complete Example: Authentication & Audit Middleware
+
+Here's a practical example showing how to implement authentication and audit logging with middleware:
+
+```js
+import { S3db } from 's3db.js';
+
+// Create database and resources
+const database = new S3db({ connectionString: 's3://my-bucket/my-app' });
+await database.connect();
+
+const orders = await database.createResource({
+  name: 'orders',
+  attributes: {
+    id: 'string|required',
+    customerId: 'string|required', 
+    amount: 'number|required',
+    status: 'string|required'
+  }
+});
+
+// Authentication middleware - runs on all operations
+['insert', 'update', 'delete', 'get'].forEach(method => {
+  orders.useMiddleware(method, async (ctx, next) => {
+    // Extract user from context (e.g., from JWT token)
+    const user = ctx.user || ctx.args.find(arg => arg?.userId);
+    
+    if (!user || !user.userId) {
+      throw new Error(`Authentication required for ${method} operation`);
+    }
+    
+    // Add user info to context for other middlewares
+    ctx.authenticatedUser = user;
+    
+    return await next();
+  });
+});
+
+// Audit logging middleware - tracks all changes
+['insert', 'update', 'delete'].forEach(method => {
+  orders.useMiddleware(method, async (ctx, next) => {
+    const startTime = Date.now();
+    const user = ctx.authenticatedUser;
+    
+    try {
+      const result = await next();
+      
+      // Log successful operation
+      console.log(`[AUDIT] ${method.toUpperCase()}`, {
+        resource: 'orders',
+        userId: user.userId,
+        method,
+        args: ctx.args,
+        duration: Date.now() - startTime,
+        timestamp: new Date().toISOString(),
+        success: true
+      });
+      
+      return result;
+    } catch (error) {
+      // Log failed operation
+      console.log(`[AUDIT] ${method.toUpperCase()} FAILED`, {
+        resource: 'orders',
+        userId: user.userId,
+        method,
+        error: error.message,
+        duration: Date.now() - startTime,
+        timestamp: new Date().toISOString(),
+        success: false
+      });
+      
+      throw error;
+    }
+  });
+});
+
+// Permission middleware for sensitive operations
+orders.useMiddleware('delete', async (ctx, next) => {
+  const user = ctx.authenticatedUser;
+  
+  if (user.role !== 'admin') {
+    throw new Error('Only admins can delete orders');
+  }
+  
+  return await next();
+});
+
+// Usage examples
+try {
+  // This will require authentication and log the operation
+  const order = await orders.insert(
+    { 
+      id: 'order-123',
+      customerId: 'cust-456', 
+      amount: 99.99, 
+      status: 'pending' 
+    },
+    { user: { userId: 'user-789', role: 'customer' } }
+  );
+  
+  // This will fail - only admins can delete
+  await orders.delete('order-123', { 
+    user: { userId: 'user-789', role: 'customer' } 
+  });
+  
+} catch (error) {
+  console.error('Operation failed:', error.message);
+}
+
+/*
+Expected output:
+[AUDIT] INSERT {
+  resource: 'orders',
+  userId: 'user-789', 
+  method: 'insert',
+  args: [{ id: 'order-123', customerId: 'cust-456', amount: 99.99, status: 'pending' }],
+  duration: 245,
+  timestamp: '2024-01-15T10:30:45.123Z',
+  success: true
+}
+
+Operation failed: Only admins can delete orders
+[AUDIT] DELETE FAILED {
+  resource: 'orders',
+  userId: 'user-789',
+  method: 'delete', 
+  error: 'Only admins can delete orders',
+  duration: 12,
+  timestamp: '2024-01-15T10:30:45.456Z',
+  success: false
+}
+*/
+```
+
+**Key Benefits of This Approach:**
+- 🔐 **Centralized Authentication**: One middleware handles auth for all operations
+- 📊 **Comprehensive Auditing**: All operations are logged with timing and user info  
+- 🛡️ **Granular Permissions**: Different rules for different operations
+- ⚡ **Performance Tracking**: Built-in timing for operation monitoring
+- 🔧 **Easy to Maintain**: Add/remove middlewares without changing business logic
+
+- **Chaining:** You can add multiple middlewares for the same method; they run in registration order.
+- **Control:** You can short-circuit the chain by not calling `next()`, or modify arguments/results as needed.
+
+This system is ideal for cross-cutting concerns like logging, access control, custom validation, metrics, or request shaping.
+
+---
+
+### 🧩 Hooks vs Middlewares: Differences, Usage, and Coexistence
+
+s3db.js supports **both hooks and middlewares** for resources. They are complementary tools for customizing and extending resource behavior.
+
+#### **What are Hooks?**
+- Hooks are functions that run **before or after** specific operations (e.g., `beforeInsert`, `afterUpdate`).
+- They are ideal for **side effects**: logging, notifications, analytics, validation, etc.
+- Hooks **cannot block or replace** the original operation—they can only observe or modify the data passed to them.
+- Hooks are registered with `addHook(hookName, fn)` or via the `hooks` config.
+
+> **📝 Note:** Don't confuse hooks with **events**. Hooks are lifecycle functions (`beforeInsert`, `afterUpdate`, etc.) while events are actual EventEmitter events (`exceedsLimit`, `truncate`, `overflow`) that you listen to with `.on(eventName, handler)`.
+
+**Example:**
+```js
+users.addHook('afterInsert', async (data) => {
+  await sendWelcomeEmail(data.email);
+  return data;
+});
+```
+
+#### **What are Middlewares?**
+- Middlewares are functions that **wrap** the entire method call (like Express/Koa middlewares).
+- They can **intercept, modify, block, or replace** the operation.
+- Middlewares can transform arguments, short-circuit the call, or modify the result.
+- Middlewares are registered with `useMiddleware(method, fn)`.
+
+**Example:**
+```js
+users.useMiddleware('insert', async (ctx, next) => {
+  if (!ctx.args[0].email) throw new Error('Email required');
+  ctx.args[0].name = ctx.args[0].name.toUpperCase();
+  const result = await next();
+  return result;
+});
+```
+
+#### **Key Differences**
+| Feature         | Hooks                        | Middlewares                  |
+|----------------|------------------------------|------------------------------|
+| Placement      | Before/after operation       | Wraps the entire method      |
+| Control        | Cannot block/replace op      | Can block/replace op         |
+| Use case       | Side effects, logging, etc.  | Access control, transform    |
+| Registration   | `addHook(hookName, fn)`      | `useMiddleware(method, fn)`  |
+| Data access    | Receives data only           | Full context (args, method)  |
+| Chaining       | Runs in order, always passes | Runs in order, can short-circuit |
+
+#### **How They Work Together**
+Hooks and middlewares can be used **together** on the same resource and method. The order of execution is:
+
+1. **Middlewares** (before the operation)
+2. **Hooks** (`beforeX`)
+3. **Original operation**
+4. **Hooks** (`afterX`)
+5. **Middlewares** (after the operation, as the call stack unwinds)
+
+**Example: Using Both**
+```js
+// Middleware: transforms input and checks permissions
+users.useMiddleware('insert', async (ctx, next) => {
+  if (!userHasPermission(ctx.args[0])) throw new Error('Unauthorized');
+  ctx.args[0].name = ctx.args[0].name.toUpperCase();
+  const result = await next();
+  return result;
+});
+
+// Hook: sends notification after insert
+users.addHook('afterInsert', async (data) => {
+  await sendWelcomeEmail(data.email);
+  return data;
+});
+
+await users.insert({ name: 'john', email: 'john@example.com' });
+// Output:
+// Middleware runs (transforms/checks)
+// Hook runs (sends email)
+```
+
+#### **When to Use Each**
+- Use **hooks** for: logging, analytics, notifications, validation, side effects.
+- Use **middlewares** for: access control, input/output transformation, caching, rate limiting, blocking or replacing operations.
+- Use **both** for advanced scenarios: e.g., middleware for access control + hook for analytics.
+
+#### **Best Practices**
+- Hooks are lightweight and ideal for observing or reacting to events.
+- Middlewares are powerful and ideal for controlling or transforming operations.
+- You can safely combine both for maximum flexibility.
+
 ---
 
 ## 📖 API Reference
@@ -1603,4 +1487,5 @@ console.log(`Total users: ${allUsers.length}`);
 
 | Method | Description | Example |
 |--------|-------------|---------|
-| `
+| `readable(options?)` | Create readable stream | `await users.readable()` |
+| `writable(options?)` | Create writable stream | `await users.writable()` |