odac 1.4.7 → 1.4.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +17 -0
- package/client/odac.js +1 -1
- package/docs/ai/README.md +1 -1
- package/docs/ai/skills/SKILL.md +1 -1
- package/docs/ai/skills/backend/database.md +103 -12
- package/docs/ai/skills/backend/ipc.md +71 -12
- package/docs/backend/08-database/05-write-behind-cache.md +230 -0
- package/docs/backend/13-utilities/02-ipc.md +117 -0
- package/package.json +1 -1
- package/src/Database/WriteBuffer.js +605 -0
- package/src/Database.js +32 -1
- package/src/Ipc.js +343 -81
- package/src/Odac.js +2 -1
- package/src/Storage.js +4 -2
- package/test/Database/WriteBuffer/_recoverFromCheckpoint.test.js +207 -0
- package/test/Database/WriteBuffer/buffer.test.js +143 -0
- package/test/Database/WriteBuffer/flush.test.js +192 -0
- package/test/Database/WriteBuffer/get.test.js +72 -0
- package/test/Database/WriteBuffer/increment.test.js +118 -0
- package/test/Database/WriteBuffer/update.test.js +178 -0
- package/test/Ipc/hset.test.js +59 -0
- package/test/Ipc/incrBy.test.js +65 -0
- package/test/Ipc/lock.test.js +62 -0
- package/test/Ipc/rpush.test.js +68 -0
- package/test/Ipc/sadd.test.js +68 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,20 @@
|
|
|
1
|
+
### ✨ What's New
|
|
2
|
+
|
|
3
|
+
- **database:** add Write-Behind Cache with counter, update, and batch insert buffering
|
|
4
|
+
- **ipc,database:** extend Ipc with atomic ops and delegate WriteBuffer state to Ipc layer
|
|
5
|
+
|
|
6
|
+
### 🛠️ Fixes & Improvements
|
|
7
|
+
|
|
8
|
+
- atomic queue drain, transaction-safe flush, hgetall clone & unhandled rejection
|
|
9
|
+
- **odac:** ensure proper token query parameter handling in get method
|
|
10
|
+
- **storage:** ensure synchronous operations for put and remove methods
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
Powered by [⚡ ODAC](https://odac.run)
|
|
17
|
+
|
|
1
18
|
### ⚙️ Engine Tuning
|
|
2
19
|
|
|
3
20
|
- **test:** remove unused fsPromises and standardize async I/O in View tests
|
package/client/odac.js
CHANGED
package/docs/ai/README.md
CHANGED
|
@@ -24,7 +24,7 @@ Detailed instructions are organized into Backend and Frontend categories:
|
|
|
24
24
|
- `backend/config.md`: Configuration management and environment variables.
|
|
25
25
|
- `backend/controllers.md`: Request handling and Class-Based Controllers.
|
|
26
26
|
- `backend/cron.md`: Scheduled background tasks and automation.
|
|
27
|
-
- `backend/database.md`: High-performance query builder usage.
|
|
27
|
+
- `backend/database.md`: High-performance query builder usage, Write-Behind Cache (buffered counters, last-write-wins updates, batch inserts).
|
|
28
28
|
- `backend/forms.md`: Form processing and validation logic.
|
|
29
29
|
- `backend/ipc.md`: Inter-Process Communication and state sharing.
|
|
30
30
|
- `backend/mail.md`: Transactional email sending.
|
package/docs/ai/skills/SKILL.md
CHANGED
|
@@ -18,7 +18,7 @@ Read the specific rule files based on whether you are working on the Backend or
|
|
|
18
18
|
- [backend/config.md](backend/config.md) - Configuration management and environment variables
|
|
19
19
|
- [backend/controllers.md](backend/controllers.md) - Request handling and Class-Based Controllers
|
|
20
20
|
- [backend/cron.md](backend/cron.md) - Scheduled background tasks and automation
|
|
21
|
-
- [backend/database.md](backend/database.md) - Query Builder and DB best practices
|
|
21
|
+
- [backend/database.md](backend/database.md) - Query Builder, Write-Behind Cache, and DB best practices
|
|
22
22
|
- [backend/forms.md](backend/forms.md) - Form processing and Validation logic
|
|
23
23
|
- [backend/ipc.md](backend/ipc.md) - Inter-Process Communication and state sharing
|
|
24
24
|
- [backend/mail.md](backend/mail.md) - Transactional email sending
|
|
@@ -1,30 +1,32 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: backend-database-skill
|
|
3
|
-
description: High-performance ODAC database querying patterns
|
|
3
|
+
description: High-performance ODAC database querying patterns including the Write-Behind Cache for counters, last-write-wins updates, and buffered batch inserts.
|
|
4
4
|
metadata:
|
|
5
|
-
tags: backend, database, query-builder, sql,
|
|
5
|
+
tags: backend, database, query-builder, sql, write-behind-cache, performance, security
|
|
6
6
|
---
|
|
7
7
|
|
|
8
8
|
# Backend Database Skill
|
|
9
9
|
|
|
10
|
-
High-performance database operations using the ODAC Query Builder.
|
|
10
|
+
High-performance database operations using the ODAC Query Builder and Write-Behind Cache.
|
|
11
11
|
|
|
12
12
|
## Principles
|
|
13
13
|
1. **Directness**: Avoid ORM overhead. Use fluent Query Builder.
|
|
14
14
|
2. **Safety**: Always use parameterized queries (built-in).
|
|
15
15
|
3. **Efficiency**: Index foreign keys. No `SELECT *`.
|
|
16
|
+
4. **Write Coalescing**: Use `buffer` for high-frequency writes to avoid DB saturation.
|
|
16
17
|
|
|
17
|
-
## Patterns
|
|
18
|
+
## Query Builder Patterns
|
|
18
19
|
```javascript
|
|
19
|
-
const user = await Odac.DB.
|
|
20
|
+
const user = await Odac.DB.users
|
|
20
21
|
.select('id', 'name', 'email')
|
|
21
22
|
.where('status', 'active')
|
|
22
|
-
.first()
|
|
23
|
+
.first()
|
|
23
24
|
|
|
24
|
-
await Odac.DB.
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
25
|
+
await Odac.DB.posts.insert({title: 'Hello', user_id: 1})
|
|
26
|
+
|
|
27
|
+
await Odac.DB.users.where('id', 1).update({last_login: new Date()})
|
|
28
|
+
|
|
29
|
+
const count = await Odac.DB.users.where('active', true).count() // → Number
|
|
28
30
|
```
|
|
29
31
|
|
|
30
32
|
## ID Strategy (NanoID)
|
|
@@ -43,7 +45,96 @@ module.exports = {
|
|
|
43
45
|
}
|
|
44
46
|
|
|
45
47
|
// ID is generated automatically
|
|
46
|
-
await Odac.DB.
|
|
48
|
+
await Odac.DB.posts.insert({title: 'Hello'})
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
## Write-Behind Cache (`buffer`)
|
|
52
|
+
|
|
53
|
+
The `buffer` API routes all writes through `Odac.Ipc`. The active IPC driver determines the scaling model — no code changes required when switching drivers.
|
|
54
|
+
|
|
55
|
+
- **`memory` driver** (default): Primary process holds all state; workers communicate via cluster IPC.
|
|
56
|
+
- **`redis` driver**: State lives in Redis — works transparently across multiple servers.
|
|
57
|
+
|
|
58
|
+
### When to suggest `buffer`
|
|
59
|
+
|
|
60
|
+
Propose the Write-Behind Cache when the user's write operation meets **all three** of these criteria:
|
|
61
|
+
1. **High frequency** — the same column or row is written on many requests (counters, timestamps, last-seen values).
|
|
62
|
+
2. **Eventual consistency is acceptable** — a few seconds' delay before the value reaches the DB is fine.
|
|
63
|
+
3. **Not safety-critical** — losing the buffered value in a crash has no financial, legal, or user-facing consequence.
|
|
64
|
+
|
|
65
|
+
**Typical candidates:**
|
|
66
|
+
- Page / post / download view counters (`increment`)
|
|
67
|
+
- Like, upvote, share counts (`increment`)
|
|
68
|
+
- User last-active timestamp, last IP (`update`)
|
|
69
|
+
- Analytics events, audit logs, activity streams (`insert`)
|
|
70
|
+
- Score accumulators, play counts, rating tallies (`increment`)
|
|
71
|
+
|
|
72
|
+
**Do NOT suggest `buffer` for:**
|
|
73
|
+
- Payments, charges, refunds, or any monetary operation
|
|
74
|
+
- Order creation, inventory decrements
|
|
75
|
+
- Anything the same request needs to read back immediately
|
|
76
|
+
- Inserts that return a generated ID the caller uses
|
|
77
|
+
|
|
78
|
+
When in doubt, use a direct DB call.
|
|
79
|
+
|
|
80
|
+
**Three operation types:**
|
|
81
|
+
|
|
82
|
+
### 1. Counter Increment (Write Coalescing)
|
|
83
|
+
Accumulates deltas — multiple increments merge into one `UPDATE col = col + delta`.
|
|
84
|
+
`get()` returns `base + pending delta` (always current, no DB read needed).
|
|
85
|
+
```javascript
|
|
86
|
+
// Increment — returns current total (DB base + buffered delta)
|
|
87
|
+
const views = await Odac.DB.posts.buffer.where(postId).increment('views')
|
|
88
|
+
const likes = await Odac.DB.posts.buffer.where(postId).increment('likes', 5)
|
|
89
|
+
|
|
90
|
+
// Read buffered counter (no extra DB round-trip)
|
|
91
|
+
const current = await Odac.DB.posts.buffer.where(postId).get('views')
|
|
92
|
+
|
|
93
|
+
// Composite key
|
|
94
|
+
await Odac.DB.post_stats.buffer.where({post_id: 1, date: '2026-04-01'}).increment('views')
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
### 2. Last-Write-Wins Update (Field Coalescing)
|
|
98
|
+
Multiple updates to the same row merge column maps — 50 requests = 1 `UPDATE` at flush.
|
|
99
|
+
```javascript
|
|
100
|
+
// Columns are merged per row: first update + second update = single UPDATE at flush
|
|
101
|
+
await Odac.DB.users.buffer.where(userId).update({active_date: new Date()})
|
|
102
|
+
await Odac.DB.users.buffer.where(userId).update({last_ip: req.ip})
|
|
103
|
+
// → UPDATE users SET active_date = ?, last_ip = ? WHERE id = ? (one query)
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
### 3. Batch Insert (Queue)
|
|
107
|
+
Rows accumulate in memory; flushed in chunks of 1000. Auto-flushes when `maxQueueSize` is reached.
|
|
108
|
+
```javascript
|
|
109
|
+
await Odac.DB.activity_log.buffer.insert({user_id: userId, action: 'page_view', meta: url})
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
### Manual Flush
|
|
113
|
+
```javascript
|
|
114
|
+
await Odac.DB.posts.buffer.flush() // flush this table only
|
|
115
|
+
await Odac.DB.buffer.flush() // flush everything
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
## Write-Behind Cache — Key Rules
|
|
119
|
+
1. **Ipc-Backed**: All buffer state goes through `Odac.Ipc`. Memory driver = Primary holds state; Redis driver = Redis holds state.
|
|
120
|
+
2. **Horizontally Scalable**: With Redis driver, multiple servers share the same buffer state. Distributed lock (`Ipc.lock`) prevents duplicate flushes.
|
|
121
|
+
3. **Crash-Safe (memory)**: LMDB checkpoint written every 30s. On restart, pending data is recovered and flushed before serving traffic.
|
|
122
|
+
4. **Crash-Safe (redis)**: Redis persistence provides durability. LMDB checkpoints are skipped.
|
|
123
|
+
5. **get() is authoritative**: Always returns `DB base + buffered delta`. Never stale.
|
|
124
|
+
6. **Flush on shutdown**: `Database.close()` triggers a final flush automatically — no data loss on graceful shutdown.
|
|
125
|
+
7. **Error resilience**: If a flush fails, data is retained in Ipc for the next cycle. Never lost silently.
|
|
126
|
+
8. **NEVER use buffer for safety-critical writes**: Payment records, order confirmations, balance changes, inventory decrements — anything where data loss has real-world consequences MUST use direct DB transactions. The buffer does not guarantee delivery before a crash.
|
|
127
|
+
|
|
128
|
+
## Configuration (`odac.json`)
|
|
129
|
+
```json
|
|
130
|
+
{
|
|
131
|
+
"buffer": {
|
|
132
|
+
"flushInterval": 5000,
|
|
133
|
+
"checkpointInterval": 30000,
|
|
134
|
+
"maxQueueSize": 10000,
|
|
135
|
+
"primaryKey": "id"
|
|
136
|
+
}
|
|
137
|
+
}
|
|
47
138
|
```
|
|
48
139
|
|
|
49
140
|
## Migration Awareness
|
|
@@ -53,4 +144,4 @@ await Odac.DB.table('posts').insert({title: 'Hello'});
|
|
|
53
144
|
4. **Indexes**: Keep index definitions in schema so add/drop is managed automatically.
|
|
54
145
|
5. **Data Changes**: Use `migration/*.js` only for one-time data transformation.
|
|
55
146
|
|
|
56
|
-
See: [migrations.md](./migrations.md)
|
|
147
|
+
See: [migrations.md](./migrations.md) | [write-behind-cache user docs](../../../backend/08-database/05-write-behind-cache.md)
|
|
@@ -24,25 +24,81 @@ IPC abstracts the underlying driver, allowing seamless transition between `memor
|
|
|
24
24
|
### 1. Key-Value Storage
|
|
25
25
|
```javascript
|
|
26
26
|
// Set value with optional TTL (seconds)
|
|
27
|
-
await Odac.Ipc.set('maintenance_mode', true)
|
|
28
|
-
await Odac.Ipc.set('cache_key', {
|
|
27
|
+
await Odac.Ipc.set('maintenance_mode', true)
|
|
28
|
+
await Odac.Ipc.set('cache_key', {data: 123}, 60)
|
|
29
29
|
|
|
30
30
|
// Get value
|
|
31
|
-
const status = await Odac.Ipc.get('maintenance_mode')
|
|
31
|
+
const status = await Odac.Ipc.get('maintenance_mode')
|
|
32
32
|
|
|
33
33
|
// Delete value
|
|
34
|
-
await Odac.Ipc.del('maintenance_mode')
|
|
34
|
+
await Odac.Ipc.del('maintenance_mode')
|
|
35
35
|
```
|
|
36
36
|
|
|
37
|
-
### 2.
|
|
37
|
+
### 2. Atomic Counters
|
|
38
38
|
```javascript
|
|
39
|
-
//
|
|
39
|
+
// Increment / decrement a numeric key (returns new value)
|
|
40
|
+
await Odac.Ipc.incrBy('page:views', 1) // → 1
|
|
41
|
+
await Odac.Ipc.incrBy('page:views', 5) // → 6
|
|
42
|
+
await Odac.Ipc.decrBy('page:views', 2) // → 4
|
|
43
|
+
|
|
44
|
+
// Read current value
|
|
45
|
+
const views = await Odac.Ipc.get('page:views') // → 4
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
### 3. Hash Maps
|
|
49
|
+
```javascript
|
|
50
|
+
// Set / merge fields (last-write-wins per field)
|
|
51
|
+
await Odac.Ipc.hset('user:1', {active_date: new Date(), last_ip: '1.2.3.4'})
|
|
52
|
+
await Odac.Ipc.hset('user:1', {score: 42})
|
|
53
|
+
|
|
54
|
+
// Get all fields
|
|
55
|
+
const fields = await Odac.Ipc.hgetall('user:1')
|
|
56
|
+
// → {active_date: ..., last_ip: '1.2.3.4', score: 42}
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
### 4. Lists
|
|
60
|
+
```javascript
|
|
61
|
+
// Append items to the right (returns new length)
|
|
62
|
+
await Odac.Ipc.rpush('events', {type: 'click'}, {type: 'view'}) // → 2
|
|
63
|
+
|
|
64
|
+
// Read range (inclusive, -1 = last item)
|
|
65
|
+
const items = await Odac.Ipc.lrange('events', 0, -1)
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
### 5. Sets
|
|
69
|
+
```javascript
|
|
70
|
+
// Add members
|
|
71
|
+
await Odac.Ipc.sadd('online_users', 'user:1', 'user:2')
|
|
72
|
+
|
|
73
|
+
// Get all members
|
|
74
|
+
const users = await Odac.Ipc.smembers('online_users') // → ['user:1', 'user:2']
|
|
75
|
+
|
|
76
|
+
// Remove members (returns removed count)
|
|
77
|
+
await Odac.Ipc.srem('online_users', 'user:1')
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
### 6. Distributed Locks
|
|
81
|
+
```javascript
|
|
82
|
+
// Acquire lock with TTL in seconds (returns true if acquired, false if already held)
|
|
83
|
+
const acquired = await Odac.Ipc.lock('flush:lock', 10)
|
|
84
|
+
if (!acquired) return // another process is holding the lock
|
|
85
|
+
|
|
86
|
+
try {
|
|
87
|
+
// ... critical section ...
|
|
88
|
+
} finally {
|
|
89
|
+
await Odac.Ipc.unlock('flush:lock')
|
|
90
|
+
}
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
### 7. Pub/Sub Messaging
|
|
94
|
+
```javascript
|
|
95
|
+
// Subscribe to a channel
|
|
40
96
|
await Odac.Ipc.subscribe('notifications', (msg) => {
|
|
41
|
-
console.log('Received:', msg)
|
|
42
|
-
})
|
|
97
|
+
console.log('Received:', msg)
|
|
98
|
+
})
|
|
43
99
|
|
|
44
100
|
// Publish a message from any worker
|
|
45
|
-
await Odac.Ipc.publish('notifications', {
|
|
101
|
+
await Odac.Ipc.publish('notifications', {type: 'alert', text: 'System update'})
|
|
46
102
|
```
|
|
47
103
|
|
|
48
104
|
## Configuration
|
|
@@ -57,6 +113,9 @@ In `odac.json` or a config provider:
|
|
|
57
113
|
```
|
|
58
114
|
|
|
59
115
|
## Best Practices
|
|
60
|
-
-
|
|
61
|
-
-
|
|
62
|
-
-
|
|
116
|
+
- **Async First**: All IPC operations are asynchronous.
|
|
117
|
+
- **TTL Usage**: Always set a TTL for temporary cache data to prevent memory bloat.
|
|
118
|
+
- **Scaling**: Switch to `redis` driver when deploying across multiple containers or servers. No application code changes required.
|
|
119
|
+
- **Locks**: Always release locks in a `finally` block to prevent deadlocks.
|
|
120
|
+
- **Sets for indexes**: Use `sadd/smembers/srem` to track active keys — avoids expensive SCAN operations.
|
|
121
|
+
- **Atomic counters over get+set**: Use `incrBy`/`decrBy` instead of `get` → compute → `set` to prevent race conditions.
|
|
@@ -0,0 +1,230 @@
|
|
|
1
|
+
# Write-Behind Cache
|
|
2
|
+
|
|
3
|
+
At high traffic, individual database writes for common operations — like incrementing a page view counter or stamping a user's last-active date — quickly saturate your connection pool. One million page views = one million `UPDATE` queries.
|
|
4
|
+
|
|
5
|
+
ODAC's **Write-Behind Cache** solves this by buffering writes in memory and flushing them to the database in efficient batches. The only change to your code is adding `.buffer` to the chain.
|
|
6
|
+
|
|
7
|
+
```javascript
|
|
8
|
+
// Without buffer — 1 DB write per request
|
|
9
|
+
await Odac.DB.posts.where(postId).update({views: Odac.DB.raw('views + 1')})
|
|
10
|
+
|
|
11
|
+
// With buffer — 1 DB write per flush interval, for all requests combined
|
|
12
|
+
await Odac.DB.posts.buffer.where(postId).increment('views')
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
## How It Works
|
|
18
|
+
|
|
19
|
+
**Architecture: Ipc-Backed, Driver-Agnostic**
|
|
20
|
+
|
|
21
|
+
All buffered state is held in `Odac.Ipc`. The active IPC driver determines the scaling model:
|
|
22
|
+
|
|
23
|
+
| Driver | Scope | When to use |
|
|
24
|
+
|---|---|---|
|
|
25
|
+
| `memory` (default) | Single machine — cluster workers share state via Primary process | Single-server deployments |
|
|
26
|
+
| `redis` | Multi-machine — all servers share state in Redis | Horizontal scaling behind a load balancer |
|
|
27
|
+
|
|
28
|
+
```
|
|
29
|
+
// Memory driver (default)
|
|
30
|
+
Worker 1 ─┐
|
|
31
|
+
Worker 2 ─┼──→ Primary (Ipc memory store) ──→ DB (batch flush every 5s)
|
|
32
|
+
Worker N ─┘
|
|
33
|
+
|
|
34
|
+
// Redis driver
|
|
35
|
+
Server A ─┐
|
|
36
|
+
Server B ─┼──→ Redis (Ipc state) ──→ DB (flush — distributed lock prevents duplicate writes)
|
|
37
|
+
Server C ─┘
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
A **distributed lock** (`Ipc.lock`) guarantees that only one process or server flushes at a time, even across multiple machines.
|
|
41
|
+
|
|
42
|
+
**Crash Safety via LMDB Checkpoint** *(memory driver only)*
|
|
43
|
+
|
|
44
|
+
Every 30 seconds, pending buffer data is written to the local LMDB store. On a crash and restart, ODAC recovers this checkpoint and flushes it to the database before accepting any traffic. When using the Redis driver, Redis itself provides durability — LMDB checkpoints are skipped.
|
|
45
|
+
|
|
46
|
+
---
|
|
47
|
+
|
|
48
|
+
## Three Operations
|
|
49
|
+
|
|
50
|
+
### 1. Counter Increment
|
|
51
|
+
|
|
52
|
+
Accumulates numeric deltas. Multiple increments to the same column merge into a single `UPDATE col = col + delta` at flush time.
|
|
53
|
+
|
|
54
|
+
```javascript
|
|
55
|
+
// Increment by 1 (default)
|
|
56
|
+
await Odac.DB.posts.buffer.where(postId).increment('views')
|
|
57
|
+
|
|
58
|
+
// Increment by a custom amount
|
|
59
|
+
await Odac.DB.posts.buffer.where(postId).increment('likes', 5)
|
|
60
|
+
await Odac.DB.downloads.buffer.where(fileId).increment('count', 3)
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
**Read the current value** — returns `DB base + pending delta`, always accurate:
|
|
64
|
+
|
|
65
|
+
```javascript
|
|
66
|
+
const currentViews = await Odac.DB.posts.buffer.where(postId).get('views')
|
|
67
|
+
// → 4527 (e.g., 4500 in DB + 27 buffered, not yet flushed)
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
**Composite primary key:**
|
|
71
|
+
|
|
72
|
+
```javascript
|
|
73
|
+
await Odac.DB.post_stats.buffer
|
|
74
|
+
.where({post_id: 123, date: '2026-04-01'})
|
|
75
|
+
.increment('views')
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
---
|
|
79
|
+
|
|
80
|
+
### 2. Last-Write-Wins Update
|
|
81
|
+
|
|
82
|
+
Buffers column SET operations for a row. If the same row is updated multiple times before a flush, the values are merged — the latest value for each column wins. The entire pending set for a row is written in a single `UPDATE` at flush.
|
|
83
|
+
|
|
84
|
+
```javascript
|
|
85
|
+
// 50 requests update the same user → 1 UPDATE at flush
|
|
86
|
+
await Odac.DB.users.buffer.where(userId).update({active_date: new Date()})
|
|
87
|
+
await Odac.DB.users.buffer.where(userId).update({last_ip: req.ip})
|
|
88
|
+
// → UPDATE users SET active_date = ?, last_ip = ? WHERE id = ? (one query for all 50 requests)
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
**Composite primary key:**
|
|
92
|
+
|
|
93
|
+
```javascript
|
|
94
|
+
await Odac.DB.user_prefs.buffer
|
|
95
|
+
.where({user_id: 1, pref_key: 'theme'})
|
|
96
|
+
.update({pref_value: 'dark'})
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
**Combine with increment** — both flush in the same cycle:
|
|
100
|
+
|
|
101
|
+
```javascript
|
|
102
|
+
await Odac.DB.users.buffer.where(userId).increment('login_count')
|
|
103
|
+
await Odac.DB.users.buffer.where(userId).update({active_date: new Date(), last_ip: req.ip})
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
---
|
|
107
|
+
|
|
108
|
+
### 3. Batch Insert
|
|
109
|
+
|
|
110
|
+
Queues rows in memory and inserts them in chunks of 1,000 at flush time. Ideal for audit logs, analytics events, and activity streams where individual inserts are wasteful.
|
|
111
|
+
|
|
112
|
+
```javascript
|
|
113
|
+
await Odac.DB.activity_log.buffer.insert({
|
|
114
|
+
user_id: userId,
|
|
115
|
+
action: 'page_view',
|
|
116
|
+
meta: req.url,
|
|
117
|
+
created_at: Date.now()
|
|
118
|
+
})
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
The queue auto-flushes immediately if it exceeds `maxQueueSize` (default: 10,000 rows).
|
|
122
|
+
|
|
123
|
+
---
|
|
124
|
+
|
|
125
|
+
## Manual Flush
|
|
126
|
+
|
|
127
|
+
Force an immediate flush for a specific table or for all buffered tables:
|
|
128
|
+
|
|
129
|
+
```javascript
|
|
130
|
+
// Flush a single table
|
|
131
|
+
await Odac.DB.posts.buffer.flush()
|
|
132
|
+
|
|
133
|
+
// Flush all buffered tables across all connections
|
|
134
|
+
await Odac.DB.buffer.flush()
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
> Graceful shutdown (`SIGTERM`/`SIGINT`) triggers a final flush automatically before the DB connections are closed. You do not need to call `flush()` in your shutdown handlers.
|
|
138
|
+
|
|
139
|
+
---
|
|
140
|
+
|
|
141
|
+
## Configuration
|
|
142
|
+
|
|
143
|
+
Add a `buffer` section to your `odac.json`:
|
|
144
|
+
|
|
145
|
+
```json
|
|
146
|
+
{
|
|
147
|
+
"buffer": {
|
|
148
|
+
"flushInterval": 5000,
|
|
149
|
+
"checkpointInterval": 30000,
|
|
150
|
+
"maxQueueSize": 10000,
|
|
151
|
+
"primaryKey": "id"
|
|
152
|
+
}
|
|
153
|
+
}
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
| Option | Default | Description |
|
|
157
|
+
|---|---|---|
|
|
158
|
+
| `flushInterval` | `5000` | How often (ms) to flush pending data to the database |
|
|
159
|
+
| `checkpointInterval` | `30000` | How often (ms) to write a crash-recovery checkpoint to LMDB *(memory driver only)* |
|
|
160
|
+
| `maxQueueSize` | `10000` | Auto-flush the insert queue when it reaches this many rows |
|
|
161
|
+
| `primaryKey` | `"id"` | Default primary key column name for scalar `where()` values |
|
|
162
|
+
|
|
163
|
+
---
|
|
164
|
+
|
|
165
|
+
## Horizontal Scaling
|
|
166
|
+
|
|
167
|
+
To share buffer state across multiple servers, switch the `ipc` driver to `redis`:
|
|
168
|
+
|
|
169
|
+
```json
|
|
170
|
+
{
|
|
171
|
+
"ipc": {
|
|
172
|
+
"driver": "redis",
|
|
173
|
+
"redis": "default"
|
|
174
|
+
}
|
|
175
|
+
}
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
With the Redis driver active:
|
|
179
|
+
- All `increment`, `update`, and `insert` operations go to Redis atomically.
|
|
180
|
+
- Any server can trigger a `flush()` — the distributed lock ensures no server writes twice.
|
|
181
|
+
- LMDB checkpoints are skipped (Redis persistence provides the durability guarantee).
|
|
182
|
+
|
|
183
|
+
No code changes are required in your application. The `.buffer` API is identical regardless of driver.
|
|
184
|
+
|
|
185
|
+
---
|
|
186
|
+
|
|
187
|
+
## Named Database Connections
|
|
188
|
+
|
|
189
|
+
The buffer respects your multi-connection configuration. Access it via the named connection, then the table:
|
|
190
|
+
|
|
191
|
+
```javascript
|
|
192
|
+
// Default connection
|
|
193
|
+
await Odac.DB.posts.buffer.where(postId).increment('views')
|
|
194
|
+
|
|
195
|
+
// Named connection: 'analytics'
|
|
196
|
+
await Odac.DB.analytics.events.buffer.insert({type: 'click', target: '#cta'})
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
---
|
|
200
|
+
|
|
201
|
+
## Guarantees
|
|
202
|
+
|
|
203
|
+
| Scenario | Behaviour |
|
|
204
|
+
|---|---|
|
|
205
|
+
| Worker crash | No data loss — all state is in the Primary process (memory) or Redis |
|
|
206
|
+
| Primary crash | Pending data recovered from LMDB checkpoint on next startup *(memory driver)* |
|
|
207
|
+
| Server crash (Redis) | Pending data is durable in Redis — recovered on next flush cycle |
|
|
208
|
+
| DB flush error | Data is retained in Ipc and retried on the next flush cycle |
|
|
209
|
+
| Graceful shutdown | Automatic final flush before connections close |
|
|
210
|
+
| `get()` after `increment()` | Returns base + buffered delta — always accurate, no extra DB read |
|
|
211
|
+
| Concurrent workers | Primary serializes all writes (memory) or Redis atomic ops prevent races |
|
|
212
|
+
| Multiple servers | Distributed lock guarantees exactly one flush at a time |
|
|
213
|
+
|
|
214
|
+
---
|
|
215
|
+
|
|
216
|
+
## When to Use (and Not Use)
|
|
217
|
+
|
|
218
|
+
**Use Write-Behind Cache for:**
|
|
219
|
+
- Page/post view counters
|
|
220
|
+
- Download counters, like/upvote counts
|
|
221
|
+
- User last-active timestamps, last IP
|
|
222
|
+
- Activity logs, analytics events, audit trails
|
|
223
|
+
- Any write that is not immediately safety-critical and occurs on every request
|
|
224
|
+
|
|
225
|
+
**Do not use for:**
|
|
226
|
+
- Operations where the write must be visible to the *same* request that triggered it
|
|
227
|
+
- Inserts that return generated IDs you need immediately (use direct `insert()`)
|
|
228
|
+
|
|
229
|
+
> [!WARNING]
|
|
230
|
+
> **Never use Write-Behind Cache for financial or safety-critical operations** — payment records, order confirmations, balance changes, inventory decrements, or any write where data loss or a delayed flush would have real-world consequences. The buffer does not guarantee that data reaches the database before a crash. Use direct database transactions for anything that matters immediately.
|
|
@@ -71,3 +71,120 @@ await Odac.Ipc.publish('chat:global', { user: 'Emre', text: 'Hello World' });
|
|
|
71
71
|
|
|
72
72
|
> [!TIP]
|
|
73
73
|
> When using `memory` driver, the subscription listener is registered in the current worker. When a message is published, it goes to the Main process and is then broadcasted to all subscribed workers.
|
|
74
|
+
|
|
75
|
+
### Atomic Counters
|
|
76
|
+
|
|
77
|
+
Use `incrBy` / `decrBy` to atomically increment or decrement a numeric key. These are safe to call from multiple workers simultaneously — no read-then-write race conditions.
|
|
78
|
+
|
|
79
|
+
```javascript
|
|
80
|
+
// Increment by 1 — returns new value
|
|
81
|
+
await Odac.Ipc.incrBy('page:views', 1) // → 1
|
|
82
|
+
await Odac.Ipc.incrBy('page:views', 5) // → 6
|
|
83
|
+
|
|
84
|
+
// Decrement
|
|
85
|
+
await Odac.Ipc.decrBy('page:views', 2) // → 4
|
|
86
|
+
|
|
87
|
+
// Read the result
|
|
88
|
+
const views = await Odac.Ipc.get('page:views') // → 4
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
> [!NOTE]
|
|
92
|
+
> Keys that don't exist yet are initialised to `0` before the operation.
|
|
93
|
+
|
|
94
|
+
---
|
|
95
|
+
|
|
96
|
+
### Hash Maps
|
|
97
|
+
|
|
98
|
+
Store and retrieve structured per-key data. Fields are merged on every `hset` call — existing fields not mentioned in the call are preserved.
|
|
99
|
+
|
|
100
|
+
```javascript
|
|
101
|
+
// Set fields (merged, not overwritten)
|
|
102
|
+
await Odac.Ipc.hset('user:42', {active_date: new Date(), last_ip: '1.2.3.4'})
|
|
103
|
+
await Odac.Ipc.hset('user:42', {score: 100})
|
|
104
|
+
|
|
105
|
+
// Retrieve all fields
|
|
106
|
+
const data = await Odac.Ipc.hgetall('user:42')
|
|
107
|
+
// → {active_date: ..., last_ip: '1.2.3.4', score: 100}
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
---
|
|
111
|
+
|
|
112
|
+
### Lists
|
|
113
|
+
|
|
114
|
+
Append items to a shared list and read them back in order. Useful for queues and event streams.
|
|
115
|
+
|
|
116
|
+
```javascript
|
|
117
|
+
// Append items to the right — returns new list length
|
|
118
|
+
await Odac.Ipc.rpush('jobs', {type: 'email', to: 'a@b.com'})
|
|
119
|
+
await Odac.Ipc.rpush('jobs', {type: 'sms'}, {type: 'push'}) // → 3
|
|
120
|
+
|
|
121
|
+
// Read a range (0-indexed, -1 = last item)
|
|
122
|
+
const pending = await Odac.Ipc.lrange('jobs', 0, -1)
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
---
|
|
126
|
+
|
|
127
|
+
### Sets
|
|
128
|
+
|
|
129
|
+
Maintain a collection of unique string members.
|
|
130
|
+
|
|
131
|
+
```javascript
|
|
132
|
+
// Add members
|
|
133
|
+
await Odac.Ipc.sadd('online', 'user:1', 'user:2', 'user:3')
|
|
134
|
+
|
|
135
|
+
// List all members
|
|
136
|
+
const online = await Odac.Ipc.smembers('online') // → ['user:1', 'user:2', 'user:3']
|
|
137
|
+
|
|
138
|
+
// Remove members — returns number of members actually removed
|
|
139
|
+
await Odac.Ipc.srem('online', 'user:2')
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
---
|
|
143
|
+
|
|
144
|
+
### Distributed Locks
|
|
145
|
+
|
|
146
|
+
Acquire a mutex across all workers and servers before entering a critical section. The TTL prevents deadlocks if a process crashes while holding the lock.
|
|
147
|
+
|
|
148
|
+
```javascript
|
|
149
|
+
// Attempt to acquire the lock (TTL in seconds)
|
|
150
|
+
const acquired = await Odac.Ipc.lock('report:generate', 30)
|
|
151
|
+
|
|
152
|
+
if (!acquired) {
|
|
153
|
+
// Another process is already running this — skip
|
|
154
|
+
return
|
|
155
|
+
}
|
|
156
|
+
|
|
157
|
+
try {
|
|
158
|
+
// Critical section — only one process runs this at a time
|
|
159
|
+
await generateReport()
|
|
160
|
+
} finally {
|
|
161
|
+
// Always release, even on error
|
|
162
|
+
await Odac.Ipc.unlock('report:generate')
|
|
163
|
+
}
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
> [!TIP]
|
|
167
|
+
> With the `redis` driver, locks work across multiple servers — making them true distributed locks.
|
|
168
|
+
|
|
169
|
+
---
|
|
170
|
+
|
|
171
|
+
## Method Reference
|
|
172
|
+
|
|
173
|
+
| Method | Description |
|
|
174
|
+
|---|---|
|
|
175
|
+
| `set(key, value, ttl?)` | Store a value, with optional TTL in seconds |
|
|
176
|
+
| `get(key)` | Retrieve a value |
|
|
177
|
+
| `del(key)` | Delete a key |
|
|
178
|
+
| `incrBy(key, delta)` | Atomically increment a numeric key |
|
|
179
|
+
| `decrBy(key, delta)` | Atomically decrement a numeric key |
|
|
180
|
+
| `hset(key, fields)` | Merge fields into a hash map |
|
|
181
|
+
| `hgetall(key)` | Retrieve all fields of a hash map |
|
|
182
|
+
| `rpush(key, ...items)` | Append items to a list |
|
|
183
|
+
| `lrange(key, start, stop)` | Read a range of list items |
|
|
184
|
+
| `sadd(key, ...members)` | Add members to a set |
|
|
185
|
+
| `smembers(key)` | Get all members of a set |
|
|
186
|
+
| `srem(key, ...members)` | Remove members from a set |
|
|
187
|
+
| `lock(key, ttl)` | Acquire a mutex lock |
|
|
188
|
+
| `unlock(key)` | Release a mutex lock |
|
|
189
|
+
| `subscribe(channel, handler)` | Subscribe to a Pub/Sub channel |
|
|
190
|
+
| `publish(channel, message)` | Publish a message to a channel |
|