@reddb-io/cli 1.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +661 -0
- package/README.md +635 -0
- package/drivers/js/cli-postinstall.js +109 -0
- package/drivers/js/package.json +50 -0
- package/drivers/js/src/binary.js +66 -0
- package/drivers/js/src/cache.js +137 -0
- package/drivers/js/src/cli.js +25 -0
- package/drivers/js/src/config.js +66 -0
- package/drivers/js/src/http.js +200 -0
- package/drivers/js/src/index.js +432 -0
- package/drivers/js/src/internal/asset-fetcher/asset-name.js +37 -0
- package/drivers/js/src/internal/asset-fetcher/checksum.js +23 -0
- package/drivers/js/src/internal/asset-fetcher/download.js +89 -0
- package/drivers/js/src/internal/asset-fetcher/index.js +52 -0
- package/drivers/js/src/internal/bin-resolver/index.js +57 -0
- package/drivers/js/src/internal/version-compare/index.js +163 -0
- package/drivers/js/src/kv.js +70 -0
- package/drivers/js/src/protocol.js +157 -0
- package/drivers/js/src/redwire.js +723 -0
- package/drivers/js/src/spawn.js +177 -0
- package/drivers/js/src/url.js +271 -0
- package/drivers/js/src/vault.js +58 -0
- package/package.json +39 -0
package/README.md
ADDED
|
@@ -0,0 +1,635 @@
|
|
|
1
|
+
<p align="center">
|
|
2
|
+
<h1 align="center">RedDB</h1>
|
|
3
|
+
<p align="center"><strong>The AI-first multi-model database.</strong></p>
|
|
4
|
+
<p align="center">Tables. Documents. Graphs. Vectors. KV. One engine. Ask it anything.</p>
|
|
5
|
+
</p>
|
|
6
|
+
|
|
7
|
+
<p align="center">
|
|
8
|
+
<a href="https://github.com/reddb-io/reddb/releases"><img src="https://img.shields.io/github/v/release/reddb-io/reddb?style=flat-square" alt="Release"></a>
|
|
9
|
+
<a href="https://www.gnu.org/licenses/agpl-3.0"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue?style=flat-square" alt="License"></a>
|
|
10
|
+
<a href="https://www.npmjs.com/package/@reddb-io/cli"><img src="https://img.shields.io/npm/v/@reddb-io/cli?style=flat-square&label=npm" alt="npm"></a>
|
|
11
|
+
</p>
|
|
12
|
+
|
|
13
|
+
> [!IMPORTANT]
|
|
14
|
+
> In RedDB, a `collection` is the named logical container for data. Tables, documents, key-value, graphs,
|
|
15
|
+
> vectors, time-series, and queues are the user-facing models or semantics you use on top of collections.
|
|
16
|
+
> A collection is not a separate hierarchy layer that contains multiple tables and documents beneath it.
|
|
17
|
+
> Instead, `users` can be a collection used as a table, `events` can be a collection used for documents,
|
|
18
|
+
> and `config` can be a collection used as KV. Some models can also coexist in the same collection.
|
|
19
|
+
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
## The Killer Feature: `ASK`
|
|
23
|
+
|
|
24
|
+
```sql
|
|
25
|
+
ASK 'who owns passport AB1234567 and what services do they use?'
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
One command. RedDB searches across tables, graphs, vectors, documents, and key-value stores -- builds context -- calls an LLM -- returns a natural-language answer. No pipelines. No glue code. No other database does this.
|
|
29
|
+
|
|
30
|
+
---
|
|
31
|
+
|
|
32
|
+
## 7 Data Models, 1 Engine
|
|
33
|
+
|
|
34
|
+
Stop running Postgres + Neo4j + Pinecone + Redis + Mongo + InfluxDB + RabbitMQ. RedDB unifies them.
|
|
35
|
+
|
|
36
|
+
The key mental model is simple: the model is how you work with the data, and the collection is where
|
|
37
|
+
that data lives.
|
|
38
|
+
|
|
39
|
+
```sql
|
|
40
|
+
-- Relational rows
|
|
41
|
+
INSERT INTO users (name, email) VALUES ('Alice', 'alice@co.com')
|
|
42
|
+
|
|
43
|
+
-- JSON documents
|
|
44
|
+
INSERT INTO logs DOCUMENT (body) VALUES ({"level":"info","msg":"login"})
|
|
45
|
+
|
|
46
|
+
-- Graph edges
|
|
47
|
+
INSERT INTO network EDGE (label, from, to) VALUES ('CONNECTS', 1, 2)
|
|
48
|
+
|
|
49
|
+
-- Vector similarity search
|
|
50
|
+
SEARCH SIMILAR TEXT 'anomaly detected' COLLECTION events
|
|
51
|
+
|
|
52
|
+
-- Key-value
|
|
53
|
+
PUT config.theme = 'dark'
|
|
54
|
+
|
|
55
|
+
-- Time-series metrics (with retention & downsampling)
|
|
56
|
+
CREATE TIMESERIES cpu_metrics RETENTION 90 d
|
|
57
|
+
INSERT INTO cpu_metrics (metric, value, tags) VALUES ('cpu.idle', 95.2, {"host":"srv1"})
|
|
58
|
+
|
|
59
|
+
-- Hypertables + partition TTL + continuous aggregates (logs / events / telemetry)
|
|
60
|
+
CREATE HYPERTABLE access_log (ts BIGINT, service TEXT, status INT, latency_ms INT)
|
|
61
|
+
CHUNK_INTERVAL '1 day' WITH (ttl = '90d');
|
|
62
|
+
|
|
63
|
+
-- Append-only tables (audit, ledger, immutable events)
|
|
64
|
+
CREATE TABLE audit_log (id BIGINT, action TEXT) APPEND ONLY
|
|
65
|
+
|
|
66
|
+
-- Message queues (FIFO, priority, consumer groups)
|
|
67
|
+
CREATE QUEUE tasks MAX_SIZE 10000
|
|
68
|
+
QUEUE PUSH tasks {"job":"process","id":123}
|
|
69
|
+
QUEUE POP tasks
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Same file. Same engine. Same query language.
|
|
73
|
+
|
|
74
|
+
Want to use RedDB as your log store? Start with the
|
|
75
|
+
[Logs Quickstart](./docs/guides/logs-quickstart.md) or the full
|
|
76
|
+
[Using RedDB for Logs](./docs/guides/using-reddb-for-logs.md) guide.
|
|
77
|
+
|
|
78
|
+
---
|
|
79
|
+
|
|
80
|
+
## AI-Native From Day One
|
|
81
|
+
|
|
82
|
+
```sql
|
|
83
|
+
-- Semantic search without managing vectors yourself
|
|
84
|
+
SEARCH SIMILAR TEXT 'suspicious login' COLLECTION logs USING groq
|
|
85
|
+
|
|
86
|
+
-- Auto-embed on insert -- vectors are created for you
|
|
87
|
+
INSERT INTO articles (title, body) VALUES ('AI Safety', 'Alignment research...')
|
|
88
|
+
WITH AUTO EMBED (body) USING openai
|
|
89
|
+
|
|
90
|
+
-- Context search: find everything related to an entity across all models
|
|
91
|
+
SEARCH CONTEXT '192.168.1.1' FIELD ip DEPTH 2
|
|
92
|
+
|
|
93
|
+
-- Ask questions in plain English
|
|
94
|
+
ASK 'what vulnerabilities affect host 10.0.0.1?' USING anthropic
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
RedDB retrieves context from every data model, feeds it to the LLM, and gives you a grounded answer. RAG built into the database layer.
|
|
98
|
+
|
|
99
|
+
---
|
|
100
|
+
|
|
101
|
+
## 11 AI Providers
|
|
102
|
+
|
|
103
|
+
Swap providers with a keyword. No code changes.
|
|
104
|
+
|
|
105
|
+
| Provider | Keyword | API Key | ASK / Prompt | Embeddings |
|
|
106
|
+
|:------------|:---------------|:-------:|:------------:|:----------:|
|
|
107
|
+
| OpenAI | `openai` | Yes | ✅ | ✅ |
|
|
108
|
+
| Anthropic | `anthropic` | Yes | ✅ | rejected |
|
|
109
|
+
| Groq | `groq` | Yes | ✅ | ✅ |
|
|
110
|
+
| OpenRouter | `openrouter` | Yes | ✅ | ✅ |
|
|
111
|
+
| Together | `together` | Yes | ✅ | ✅ |
|
|
112
|
+
| Venice | `venice` | Yes | ✅ | ✅ |
|
|
113
|
+
| DeepSeek | `deepseek` | Yes | ✅ | ✅ |
|
|
114
|
+
| HuggingFace | `huggingface` | Yes | ✅ | ✅ |
|
|
115
|
+
| Ollama | `ollama` | No | ✅ | ✅ |
|
|
116
|
+
| Local | `local` | No | feature-gated | feature-gated |
|
|
117
|
+
| Custom URL | `https://...` | depends | ✅ | ✅ |
|
|
118
|
+
|
|
119
|
+
Most providers speak the OpenAI-compatible `POST /embeddings` shape;
|
|
120
|
+
HuggingFace has its own (`POST /pipeline/feature-extraction/{model}`)
|
|
121
|
+
and RedDB ships a dedicated client for it. Anthropic does not have an
|
|
122
|
+
embeddings API — RedDB rejects embedding calls against it explicitly
|
|
123
|
+
rather than silently re-routing to a different provider. `local`
|
|
124
|
+
requires the `local-models` feature flag at engine build time.
|
|
125
|
+
|
|
126
|
+
See [`docs/guides/ai-providers.md`](./docs/guides/ai-providers.md)
|
|
127
|
+
for the routing matrix, the wire shape per provider, and the
|
|
128
|
+
Anthropic-embeddings policy in detail.
|
|
129
|
+
|
|
130
|
+
```sql
|
|
131
|
+
ASK 'summarize alerts' USING groq MODEL 'llama-3.3-70b-versatile'
|
|
132
|
+
ASK 'summarize alerts' USING ollama MODEL 'llama3'
|
|
133
|
+
ASK 'summarize alerts' USING anthropic
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
Set a default provider so you can drop `USING` from every query:
|
|
137
|
+
|
|
138
|
+
```bash
|
|
139
|
+
# Set default provider -- no more USING on every query
|
|
140
|
+
curl -X POST http://127.0.0.1:8080/ai/credentials \
|
|
141
|
+
-d '{"provider":"groq","api_key":"gsk_xxx","default":true}'
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
```sql
|
|
145
|
+
-- Now ASK uses groq by default
|
|
146
|
+
ASK 'what happened?'
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
```bash
|
|
150
|
+
# Export/import all config as JSON
|
|
151
|
+
curl http://127.0.0.1:8080/config
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
---
|
|
155
|
+
|
|
156
|
+
## Probabilistic Data Structures
|
|
157
|
+
|
|
158
|
+
Built-in approximate data structures for real-time analytics at scale.
|
|
159
|
+
|
|
160
|
+
```sql
|
|
161
|
+
-- HyperLogLog: count unique visitors (~0.8% error, ~16KB memory)
|
|
162
|
+
CREATE HLL visitors
|
|
163
|
+
HLL ADD visitors 'user1' 'user2' 'user3'
|
|
164
|
+
HLL COUNT visitors
|
|
165
|
+
|
|
166
|
+
-- Count-Min Sketch: frequency estimation
|
|
167
|
+
CREATE SKETCH click_counter WIDTH 2000 DEPTH 7
|
|
168
|
+
SKETCH ADD click_counter 'button_a' 5
|
|
169
|
+
SKETCH COUNT click_counter 'button_a'
|
|
170
|
+
|
|
171
|
+
-- Cuckoo Filter: membership testing with deletion (unlike Bloom filters)
|
|
172
|
+
CREATE FILTER active_sessions CAPACITY 500000
|
|
173
|
+
FILTER ADD active_sessions 'session_abc'
|
|
174
|
+
FILTER CHECK active_sessions 'session_abc'
|
|
175
|
+
FILTER DELETE active_sessions 'session_abc'
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
---
|
|
179
|
+
|
|
180
|
+
## Advanced Indexes
|
|
181
|
+
|
|
182
|
+
Beyond B-tree. Create the right index for your workload.
|
|
183
|
+
|
|
184
|
+
```sql
|
|
185
|
+
-- Hash index: O(1) exact-match lookups
|
|
186
|
+
CREATE INDEX idx_email ON users (email) USING HASH
|
|
187
|
+
|
|
188
|
+
-- Bitmap index: fast analytical queries on low-cardinality columns
|
|
189
|
+
CREATE INDEX idx_status ON orders (status) USING BITMAP
|
|
190
|
+
|
|
191
|
+
-- R-Tree: spatial queries on geo data
|
|
192
|
+
CREATE INDEX idx_loc ON sites (location) USING RTREE
|
|
193
|
+
SEARCH SPATIAL RADIUS 48.8566 2.3522 10.0 COLLECTION sites COLUMN location LIMIT 50
|
|
194
|
+
SEARCH SPATIAL NEAREST 48.8566 2.3522 K 5 COLLECTION sites COLUMN location
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
---
|
|
198
|
+
|
|
199
|
+
## SQL Extensions
|
|
200
|
+
|
|
201
|
+
RedDB extends SQL with `WITH` clauses for operational semantics:
|
|
202
|
+
|
|
203
|
+
```sql
|
|
204
|
+
-- TTL: auto-expire records
|
|
205
|
+
INSERT INTO sessions (token) VALUES ('abc') WITH TTL 1 h
|
|
206
|
+
|
|
207
|
+
-- Context indexes for cross-model search
|
|
208
|
+
CREATE TABLE customers (passport TEXT) WITH CONTEXT INDEX ON (passport)
|
|
209
|
+
|
|
210
|
+
-- Graph expansion inline with SELECT
|
|
211
|
+
SELECT * FROM users WITH EXPAND GRAPH DEPTH 2
|
|
212
|
+
|
|
213
|
+
-- Metadata on write
|
|
214
|
+
INSERT INTO logs (msg) VALUES ('deploy') WITH METADATA (source = 'ci')
|
|
215
|
+
|
|
216
|
+
-- Absolute expiration
|
|
217
|
+
INSERT INTO events (name) VALUES ('launch') WITH EXPIRES AT 1735689600000
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
---
|
|
221
|
+
|
|
222
|
+
## 6 Query Languages
|
|
223
|
+
|
|
224
|
+
Write in whatever you think in. The engine auto-detects the language.
|
|
225
|
+
|
|
226
|
+
| Language | Example |
|
|
227
|
+
|:---------|:--------|
|
|
228
|
+
| **SQL** | `SELECT * FROM hosts WHERE os = 'linux'` |
|
|
229
|
+
| **Cypher** | `MATCH (a:User)-[:FOLLOWS]->(b) RETURN b.name` |
|
|
230
|
+
| **Gremlin** | `g.V().hasLabel('person').out('FOLLOWS').values('name')` |
|
|
231
|
+
| **SPARQL** | `SELECT ?name WHERE { ?p :name ?name }` |
|
|
232
|
+
| **Natural Language** | `show me all critical hosts` |
|
|
233
|
+
| **ASK (RAG)** | `ASK 'what changed in the last 24 hours?'` |
|
|
234
|
+
|
|
235
|
+
All six hit the same engine, same data, same indexes.
|
|
236
|
+
|
|
237
|
+
---
|
|
238
|
+
|
|
239
|
+
## Native Migrations — No External Tools
|
|
240
|
+
|
|
241
|
+
Stop reaching for Flyway, Liquibase, Drizzle Migrate, or Sequelize migrations. RedDB handles schema and data migrations as first-class SQL commands.
|
|
242
|
+
|
|
243
|
+
```sql
|
|
244
|
+
-- Register a migration
|
|
245
|
+
CREATE MIGRATION add_users_table AS
|
|
246
|
+
CREATE TABLE users (id BIGINT, email TEXT, created_at TIMESTAMP);
|
|
247
|
+
|
|
248
|
+
-- Register a dependent migration (RedDB also auto-infers deps from SQL body)
|
|
249
|
+
CREATE MIGRATION add_users_index DEPENDS ON add_users_table AS
|
|
250
|
+
CREATE INDEX idx_email ON users (email);
|
|
251
|
+
|
|
252
|
+
-- Apply everything in dependency order
|
|
253
|
+
APPLY MIGRATION *
|
|
254
|
+
|
|
255
|
+
-- Large data backfill in safe 5,000-row batches — resumes on crash
|
|
256
|
+
CREATE MIGRATION backfill_display_names BATCH 5000 ROWS AS
|
|
257
|
+
UPDATE users SET display_name = email WHERE display_name IS NULL;
|
|
258
|
+
|
|
259
|
+
-- Undo an applied migration (VCS revert under the hood)
|
|
260
|
+
ROLLBACK MIGRATION add_users_index
|
|
261
|
+
|
|
262
|
+
-- Inspect what a migration will do
|
|
263
|
+
EXPLAIN MIGRATION backfill_display_names
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
Every applied migration creates a **VCS commit** (RedDB's "Git for Data"). Rollback reverts that commit automatically — no rollback scripts to maintain. Dependency ordering is a DAG; RedDB detects cycles at `CREATE` time and auto-infers edges from your SQL body so you rarely need explicit `DEPENDS ON`.
|
|
267
|
+
|
|
268
|
+
→ [Native Migrations docs](./docs/migrations/overview.md)
|
|
269
|
+
|
|
270
|
+
---
|
|
271
|
+
|
|
272
|
+
## 48 Built-in Types
|
|
273
|
+
|
|
274
|
+
Not just `TEXT` and `INTEGER`. RedDB understands your domain.
|
|
275
|
+
|
|
276
|
+
**Network:** `IpAddr`, `Ipv4`, `Ipv6`, `MacAddr`, `Cidr`, `Subnet`, `Port`
|
|
277
|
+
**Geo:** `Latitude`, `Longitude`, `GeoPoint`
|
|
278
|
+
**Locale:** `Country2`, `Country3`, `Lang2`, `Lang5`, `Currency`
|
|
279
|
+
**Identity:** `Uuid`, `Email`, `Url`, `Phone`, `Semver`
|
|
280
|
+
**Visual:** `Color`, `ColorAlpha`
|
|
281
|
+
**Cross-model refs:** `NodeRef`, `EdgeRef`, `VectorRef`, `RowRef`, `KeyRef`, `DocRef`, `TableRef`, `PageRef`
|
|
282
|
+
**Primitives:** `Integer`, `UnsignedInteger`, `Float`, `Decimal`, `BigInt`, `Text`, `Blob`, `Boolean`, `Json`, `Array`, `Enum`
|
|
283
|
+
**Temporal:** `Timestamp`, `TimestampMs`, `Date`, `Time`, `Duration`
|
|
284
|
+
|
|
285
|
+
Validation on write. No parsing in your app.
|
|
286
|
+
|
|
287
|
+
---
|
|
288
|
+
|
|
289
|
+
## Backup & Recovery
|
|
290
|
+
|
|
291
|
+
Built-in backup scheduler, WAL archiving, Change Data Capture (CDC), and Point-in-Time Recovery framework:
|
|
292
|
+
|
|
293
|
+
```bash
|
|
294
|
+
# Poll real-time changes
|
|
295
|
+
curl 'localhost:8080/changes?since_lsn=0'
|
|
296
|
+
|
|
297
|
+
# Trigger manual backup
|
|
298
|
+
curl -X POST localhost:8080/backup/trigger
|
|
299
|
+
|
|
300
|
+
# Check backup status
|
|
301
|
+
curl localhost:8080/backup/status
|
|
302
|
+
```
|
|
303
|
+
|
|
304
|
+
Remote backends: S3, R2, DigitalOcean Spaces, GCS, Turso, Cloudflare D1, local filesystem.
|
|
305
|
+
|
|
306
|
+
For concrete RTO/RPO numbers per failure mode (process crash, disk loss,
|
|
307
|
+
PITR rollback, replica promotion), see
|
|
308
|
+
[`docs/operations/rto-rpo.md`](./docs/operations/rto-rpo.md).
|
|
309
|
+
|
|
310
|
+
---
|
|
311
|
+
|
|
312
|
+
## KV REST API
|
|
313
|
+
|
|
314
|
+
Every collection doubles as a key-value store with dedicated REST endpoints:
|
|
315
|
+
|
|
316
|
+
```bash
|
|
317
|
+
# Write a key
|
|
318
|
+
curl -X PUT http://127.0.0.1:8080/collections/settings/kvs/theme \
|
|
319
|
+
-H 'content-type: application/json' -d '{"value": "dark"}'
|
|
320
|
+
|
|
321
|
+
# Read a key
|
|
322
|
+
curl http://127.0.0.1:8080/collections/settings/kvs/theme
|
|
323
|
+
|
|
324
|
+
# Delete a key
|
|
325
|
+
curl -X DELETE http://127.0.0.1:8080/collections/settings/kvs/theme
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
Config keys work the same way -- read, write, or delete any `red_config` setting at runtime:
|
|
329
|
+
|
|
330
|
+
```bash
|
|
331
|
+
# Set a config key
|
|
332
|
+
curl -X PUT http://127.0.0.1:8080/config/red.ai.default.provider \
|
|
333
|
+
-d '{"value": "groq"}'
|
|
334
|
+
|
|
335
|
+
# Read a config key
|
|
336
|
+
curl http://127.0.0.1:8080/config/red.ai.default.provider
|
|
337
|
+
|
|
338
|
+
# Or manage config from SQL
|
|
339
|
+
SET CONFIG red.ai.default.provider = 'groq'
|
|
340
|
+
SHOW CONFIG red.ai
|
|
341
|
+
```
|
|
342
|
+
|
|
343
|
+
---
|
|
344
|
+
|
|
345
|
+
## 3 Deployment Modes
|
|
346
|
+
|
|
347
|
+
| Mode | Think of it as... | Access via |
|
|
348
|
+
|:-----|:-------------------|:-----------|
|
|
349
|
+
| **Embedded** | SQLite | Rust API -- `RedDB::open("data.rdb")` |
|
|
350
|
+
| **Server** | Postgres | HTTP + gRPC -- dual-stack |
|
|
351
|
+
| **Agent** | MCP tool | `red mcp` -- AI agent integration |
|
|
352
|
+
|
|
353
|
+
Same storage format across all three. Start embedded, scale to server, expose to agents -- no migration.
|
|
354
|
+
|
|
355
|
+
---
|
|
356
|
+
|
|
357
|
+
## Performance
|
|
358
|
+
|
|
359
|
+
> **Where RedDB wins.** Two scenarios in the canonical `duel-official`
|
|
360
|
+
> benchmark show measurable wins over Postgres and Mongo today:
|
|
361
|
+
>
|
|
362
|
+
> - `typed_insert` — RedDB ≈ **16× faster** than PostgreSQL on typed
|
|
363
|
+
> single-row inserts.
|
|
364
|
+
> - `disk_usage` — RedDB ≈ **1.5× faster** than MongoDB on the
|
|
365
|
+
> compact-write path.
|
|
366
|
+
>
|
|
367
|
+
> See [`docs/perf/wins.md`](docs/perf/wins.md) for the cited sessions
|
|
368
|
+
> and reproducible commands. The honest counterpart — gaps where RedDB
|
|
369
|
+
> is still behind — lives at
|
|
370
|
+
> [`docs/perf/when-not-reddb.md`](docs/perf/when-not-reddb.md).
|
|
371
|
+
|
|
372
|
+
RedDB uses multiple optimization techniques for fast queries at scale:
|
|
373
|
+
|
|
374
|
+
- **Result Cache** -- identical SELECT queries return in <1ms; auto-invalidated on INSERT/UPDATE/DELETE (30s TTL, max 1000 entries)
|
|
375
|
+
- **Hot Entity Cache** -- `get_any(id)` lookups served from an LRU cache (10K entries), O(1) instead of scanning all collections
|
|
376
|
+
- **Binary Bulk Insert** -- gRPC `BulkInsertBinary` with zero JSON overhead, protobuf native types -- 241K ops/sec
|
|
377
|
+
- **Concurrent HTTP** -- thread-per-connection model; each request handled in its own OS thread
|
|
378
|
+
- **Parallel Segment Scanning** -- sealed segments scanned in parallel via `std::thread::scope`; auto-detects single-core and skips parallelism
|
|
379
|
+
- **Hash Join** -- O(n+m) joins instead of O(n*m), auto-selected for large datasets
|
|
380
|
+
- **Lazy Graph Materialization** -- only loads reachable nodes instead of full graph
|
|
381
|
+
- **Pre-filtered Vector Search** -- metadata filters applied before HNSW indexing
|
|
382
|
+
- **Index-Assisted Scans** -- bloom filter + hash index hints for WHERE clauses
|
|
383
|
+
- **Column Projection Pushdown** -- only materializes SELECT columns
|
|
384
|
+
- **Query Plan Caching** -- LRU cache with 1h TTL for repeated queries
|
|
385
|
+
- **Batch Entity Lookup** -- multi-entity fetches resolved in a single pass
|
|
386
|
+
- **Background Maintenance Thread** -- backup scheduling, retention, and checkpoint run off the hot path
|
|
387
|
+
|
|
388
|
+
---
|
|
389
|
+
|
|
390
|
+
## Durability & Corruption Defense
|
|
391
|
+
|
|
392
|
+
RedDB uses 7 layers of protection to keep your data safe:
|
|
393
|
+
|
|
394
|
+
| Layer | What it does |
|
|
395
|
+
|:------|:-------------|
|
|
396
|
+
| **File Lock** | Exclusive `flock` prevents two processes from writing the same `.rdb` file |
|
|
397
|
+
| **Double-Write Buffer** | Pages written to `.rdb-dwb` first; survives torn writes on power loss |
|
|
398
|
+
| **Header Shadow** | Copy of page 0 in `.rdb-hdr`; auto-recovers if header corrupts |
|
|
399
|
+
| **Metadata Shadow** | Copy of page 1 in `.rdb-meta`; auto-recovers collection registry |
|
|
400
|
+
| **fsync Discipline** | All critical writes followed by `sync_all()` (not just flush) |
|
|
401
|
+
| **Two-Phase Checkpoint** | Crash-safe WAL→DB transfer with `checkpoint_in_progress` flag |
|
|
402
|
+
| **Binary Store CRC32** | V3 files have CRC32 footer + atomic write-to-temp-then-rename |
|
|
403
|
+
|
|
404
|
+
Every page has a CRC32 checksum (verified on read). Every WAL record has a CRC32 checksum. The binary store format (V3) includes a full-file CRC32 footer.
|
|
405
|
+
|
|
406
|
+
---
|
|
407
|
+
|
|
408
|
+
## Eventual Consistency
|
|
409
|
+
|
|
410
|
+
RedDB supports per-field eventual consistency via an append-only transaction log with periodic consolidation. Inspired by CRDT principles (commutative, associative reducers), it enables high-throughput write patterns while guaranteeing convergence.
|
|
411
|
+
|
|
412
|
+
```bash
|
|
413
|
+
# Track clicks with async consolidation (returns instantly)
|
|
414
|
+
curl -X POST localhost:8080/ec/urls/clicks/add -d '{"id": 1, "value": 1}'
|
|
415
|
+
|
|
416
|
+
# Check consolidated + pending value
|
|
417
|
+
curl localhost:8080/ec/urls/clicks/status?id=1
|
|
418
|
+
```
|
|
419
|
+
|
|
420
|
+
| Feature | Description |
|
|
421
|
+
|:--------|:------------|
|
|
422
|
+
| **6 reducers** | Sum, Max, Min, Count, Average, Last (last-write-wins) |
|
|
423
|
+
| **Sync mode** | Consolidates immediately (strong consistency) |
|
|
424
|
+
| **Async mode** | Background worker consolidates periodically (high throughput) |
|
|
425
|
+
| **Transaction log** | Immutable append-only audit trail per field |
|
|
426
|
+
| **SET checkpoint** | Resets base value, discards prior operations |
|
|
427
|
+
| **All modes** | Works in server, embedded (Rust API), and serverless |
|
|
428
|
+
|
|
429
|
+
See the [Eventual Consistency Guide](https://reddb-io.github.io/reddb/#/guides/eventual-consistency) for the theory (CAP theorem, CRDTs, convergence) and full API reference.
|
|
430
|
+
|
|
431
|
+
---
|
|
432
|
+
|
|
433
|
+
## Geographic Operations
|
|
434
|
+
|
|
435
|
+
Built-in geo functions with no external dependencies. Supports both spherical (Haversine) and ellipsoidal (Vincenty/WGS-84) models.
|
|
436
|
+
|
|
437
|
+
```sql
|
|
438
|
+
-- Distance from each store to a point (in km)
|
|
439
|
+
SELECT name, GEO_DISTANCE(location, POINT(-23.55, -46.63)) AS dist
|
|
440
|
+
FROM stores ORDER BY dist
|
|
441
|
+
|
|
442
|
+
-- Vincenty for sub-millimeter accuracy
|
|
443
|
+
SELECT name, GEO_DISTANCE_VINCENTY(location, POINT(40.71, -74.00)) AS dist
|
|
444
|
+
FROM airports
|
|
445
|
+
```
|
|
446
|
+
|
|
447
|
+
```bash
|
|
448
|
+
# HTTP API
|
|
449
|
+
curl -X POST localhost:8080/geo/distance -d '{
|
|
450
|
+
"from": {"lat": -23.55, "lon": -46.63},
|
|
451
|
+
"to": {"lat": -22.91, "lon": -43.17}
|
|
452
|
+
}'
|
|
453
|
+
```
|
|
454
|
+
|
|
455
|
+
| Function | What it computes |
|
|
456
|
+
|:---------|:-----------------|
|
|
457
|
+
| `GEO_DISTANCE` | Haversine distance (km) |
|
|
458
|
+
| `GEO_DISTANCE_VINCENTY` | WGS-84 geodesic distance (km) |
|
|
459
|
+
| `GEO_BEARING` | Compass direction (degrees) |
|
|
460
|
+
| `GEO_MIDPOINT` | Great-circle midpoint |
|
|
461
|
+
|
|
462
|
+
Also available: destination point, bounding box, polygon area, spatial search (RADIUS, BBOX, NEAREST). See the [Geo Operations Guide](https://reddb-io.github.io/reddb/#/guides/geo-operations).
|
|
463
|
+
|
|
464
|
+
---
|
|
465
|
+
|
|
466
|
+
## Vector Clustering
|
|
467
|
+
|
|
468
|
+
Standalone K-Means and DBSCAN clustering on vector collections, with SIMD-accelerated distance computation and automatic parallelization.
|
|
469
|
+
|
|
470
|
+
```bash
|
|
471
|
+
# K-Means: group products into 5 clusters
|
|
472
|
+
curl -X POST localhost:8080/vectors/cluster -d '{
|
|
473
|
+
"collection": "products", "algorithm": "kmeans", "k": 5
|
|
474
|
+
}'
|
|
475
|
+
|
|
476
|
+
# DBSCAN: discover clusters automatically (no K needed)
|
|
477
|
+
curl -X POST localhost:8080/vectors/cluster -d '{
|
|
478
|
+
"collection": "products", "algorithm": "dbscan", "eps": 0.5, "min_points": 3
|
|
479
|
+
}'
|
|
480
|
+
```
|
|
481
|
+
|
|
482
|
+
K-Means uses parallel assignment (multi-threaded for datasets > 1K vectors). DBSCAN labels unreachable points as noise (-1), useful for outlier detection. See the [Vector Clustering Guide](https://reddb-io.github.io/reddb/#/guides/vector-clustering).
|
|
483
|
+
|
|
484
|
+
---
|
|
485
|
+
|
|
486
|
+
## Native Drivers
|
|
487
|
+
|
|
488
|
+
One connection-string API, four languages. Every driver accepts the same
|
|
489
|
+
`connect(uri)` contract so application code ports across runtimes with zero
|
|
490
|
+
ceremony.
|
|
491
|
+
|
|
492
|
+
| Language | Package | Install | Backends |
|
|
493
|
+
|-------------------|------------------|--------------------------------|-------------------------------------|
|
|
494
|
+
| Rust | `reddb-client` | `cargo add reddb-client` | embedded ✅ · gRPC ✅ · HTTP ✅ |
|
|
495
|
+
| Node / Bun / Deno | `@reddb-io/sdk` (npm) | `pnpm add @reddb-io/sdk` | stdio subprocess ✅ |
|
|
496
|
+
| Python | `reddb` (PyPI) | `pip install reddb` *(soon)* | embedded ✅ · gRPC ✅ · wire ✅ |
|
|
497
|
+
|
|
498
|
+
All drivers accept the same URIs:
|
|
499
|
+
|
|
500
|
+
```
|
|
501
|
+
memory:// ephemeral in-memory
|
|
502
|
+
file:///absolute/path embedded engine on disk
|
|
503
|
+
grpc://host:port remote server (planned — tracked in PLAN_DRIVERS.md)
|
|
504
|
+
```
|
|
505
|
+
|
|
506
|
+
Example — the same app in three languages:
|
|
507
|
+
|
|
508
|
+
```rust
|
|
509
|
+
// Rust
|
|
510
|
+
let db = reddb_client::Reddb::connect("memory://").await?;
|
|
511
|
+
db.insert("users", &JsonValue::object([("name", JsonValue::string("Alice"))])).await?;
|
|
512
|
+
let rows = db.query("SELECT * FROM users").await?;
|
|
513
|
+
```
|
|
514
|
+
|
|
515
|
+
```js
|
|
516
|
+
// Node, Bun, Deno
|
|
517
|
+
import { connect } from '@reddb-io/sdk'
|
|
518
|
+
const db = await connect('memory://')
|
|
519
|
+
await db.insert('users', { name: 'Alice' })
|
|
520
|
+
const rows = await db.query('SELECT * FROM users')
|
|
521
|
+
```
|
|
522
|
+
|
|
523
|
+
```python
|
|
524
|
+
# Python
|
|
525
|
+
import reddb
|
|
526
|
+
with reddb.connect("memory://") as db:
|
|
527
|
+
db.insert("users", {"name": "Alice"})
|
|
528
|
+
print(db.query("SELECT * FROM users"))
|
|
529
|
+
```
|
|
530
|
+
|
|
531
|
+
Driver docs live in `crates/reddb-client/README.md`, `drivers/js/README.md`, and
|
|
532
|
+
`drivers/python/README.md`. The full protocol spec and roadmap are in
|
|
533
|
+
[`PLAN_DRIVERS.md`](./PLAN_DRIVERS.md).
|
|
534
|
+
|
|
535
|
+
For JavaScript and TypeScript, RedDB ships three packages under the
|
|
536
|
+
`@reddb-io/` scope. Pick the one that matches your scenario — see the
|
|
537
|
+
[JavaScript / TypeScript driver guide](./docs/guides/javascript-typescript-driver.md#package-matrix)
|
|
538
|
+
for the full matrix and [ADR 0007](./docs/adr/0007-npm-package-matrix.md)
|
|
539
|
+
for the rationale.
|
|
540
|
+
|
|
541
|
+
```bash
|
|
542
|
+
# App code in Node, Bun, or Deno — full SDK with embedded, gRPC, and HTTP transports
|
|
543
|
+
pnpm add @reddb-io/sdk
|
|
544
|
+
|
|
545
|
+
# Thin remote-only client for serverless, edge, CI, or sidecar runtimes (~5 MB)
|
|
546
|
+
pnpm add @reddb-io/client
|
|
547
|
+
|
|
548
|
+
# CLI launcher — installs the `red` binary on PATH
|
|
549
|
+
pnpm add -g @reddb-io/cli
|
|
550
|
+
```
|
|
551
|
+
|
|
552
|
+
Application code with the SDK:
|
|
553
|
+
|
|
554
|
+
```ts
|
|
555
|
+
import { connect } from '@reddb-io/sdk'
|
|
556
|
+
|
|
557
|
+
const db = await connect('memory://')
|
|
558
|
+
const result = await db.query('SELECT * FROM users')
|
|
559
|
+
await db.close()
|
|
560
|
+
```
|
|
561
|
+
|
|
562
|
+
Launch the server from npm without a separate install step:
|
|
563
|
+
|
|
564
|
+
```bash
|
|
565
|
+
npx @reddb-io/cli@latest version
|
|
566
|
+
npx @reddb-io/cli@latest server --wire-bind 127.0.0.1:5050 --http-bind 127.0.0.1:8080 --path ./data.rdb
|
|
567
|
+
```
|
|
568
|
+
|
|
569
|
+
---
|
|
570
|
+
|
|
571
|
+
## Quick Start
|
|
572
|
+
|
|
573
|
+
```bash
|
|
574
|
+
# Install
|
|
575
|
+
curl -fsSL https://raw.githubusercontent.com/reddb-io/reddb/main/install.sh | bash
|
|
576
|
+
|
|
577
|
+
# Start the server (wire: 5050, gRPC: 5055, HTTP: 8080)
|
|
578
|
+
red server --wire-bind 127.0.0.1:5050 --grpc-bind 127.0.0.1:5055 --http-bind 127.0.0.1:8080 --path ./data.rdb
|
|
579
|
+
|
|
580
|
+
# Insert data
|
|
581
|
+
curl -X POST http://127.0.0.1:8080/query \
|
|
582
|
+
-H 'content-type: application/json' \
|
|
583
|
+
-d '{"query":"INSERT INTO hosts (ip, os) VALUES ('\''10.0.0.1'\'', '\''linux'\'')"}'
|
|
584
|
+
|
|
585
|
+
# Query it
|
|
586
|
+
curl -X POST http://127.0.0.1:8080/query \
|
|
587
|
+
-H 'content-type: application/json' \
|
|
588
|
+
-d '{"query":"SELECT * FROM hosts"}'
|
|
589
|
+
```
|
|
590
|
+
|
|
591
|
+
Or via npm CLI launcher:
|
|
592
|
+
|
|
593
|
+
```bash
|
|
594
|
+
npx @reddb-io/cli@latest server --wire-bind 127.0.0.1:5050 --http-bind 127.0.0.1:8080
|
|
595
|
+
```
|
|
596
|
+
|
|
597
|
+
Or via Docker:
|
|
598
|
+
|
|
599
|
+
```bash
|
|
600
|
+
docker run --rm -p 5050:5050 -p 5055:5055 -p 8080:8080 ghcr.io/reddb-io/reddb:latest
|
|
601
|
+
```
|
|
602
|
+
|
|
603
|
+
Or, if you only need the thin remote-only client (~7 MB image):
|
|
604
|
+
|
|
605
|
+
```bash
|
|
606
|
+
docker run --rm ghcr.io/reddb-io/reddb-client:latest red://reddb.example.com:5050 -c "SELECT 1"
|
|
607
|
+
```
|
|
608
|
+
|
|
609
|
+
For production-secure Docker (vault + secrets) and Kubernetes, see
|
|
610
|
+
[`docs/getting-started/docker.md`](./docs/getting-started/docker.md)
|
|
611
|
+
and [`docs/security/vault.md`](./docs/security/vault.md).
|
|
612
|
+
|
|
613
|
+
---
|
|
614
|
+
|
|
615
|
+
## Workspace layout
|
|
616
|
+
|
|
617
|
+
RedDB ships as a Cargo workspace. The `reddb` crate is the
|
|
618
|
+
umbrella that hosts the `red` binary; the engine, thin client,
|
|
619
|
+
gRPC stubs, and wire vocabulary live in sibling crates. See the
|
|
620
|
+
[workspace migration guide](./docs/migration/workspace-split.md)
|
|
621
|
+
for what moved where and which crate to pick when depending on
|
|
622
|
+
RedDB from another Rust project.
|
|
623
|
+
|
|
624
|
+
## Links
|
|
625
|
+
|
|
626
|
+
- [Documentation](https://reddb-io.github.io/reddb)
|
|
627
|
+
- [GitHub](https://github.com/reddb-io/reddb)
|
|
628
|
+
- [npm driver package](https://www.npmjs.com/package/reddb)
|
|
629
|
+
- [npm CLI launcher](https://www.npmjs.com/package/@reddb-io/cli)
|
|
630
|
+
- [Releases](https://github.com/reddb-io/reddb/releases)
|
|
631
|
+
- [Workspace migration guide](./docs/migration/workspace-split.md)
|
|
632
|
+
|
|
633
|
+
---
|
|
634
|
+
|
|
635
|
+
**AGPL-3.0 License** -- Built by [RedDB.io](https://github.com/reddb-io)
|