@rip-lang/db 0.10.0 → 1.0.1

package/INTERNALS.md

@@ -0,0 +1,324 @@
+# DuckDB Internals
+
+This document covers the internal architecture, binary protocol, and implementation
+details for rip-db's native DuckDB integration. For usage, see README.md.
+
+## Architecture
+
+```
+┌────────────────────────────────────────────────────────────────────────────┐
+│                              Browser                                       │
+│  ┌──────────────────────────────────────────────────────────────────────┐  │
+│  │  DuckDB UI (React App)                                               │  │
+│  │  - Loaded from http://localhost:4213/ (proxied from ui.duckdb.org)   │  │
+│  │  - Makes API calls to relative URLs (/ddb/run, /ddb/tokenize, etc.)  │  │
+│  │  - Uses EventSource for /localEvents (catalog updates)               │  │
+│  └──────────────────────────────────────────────────────────────────────┘  │
+│                                    │                                       │
+│                                    │ Same-origin requests                  │
+│                                    ▼                                       │
+└────────────────────────────────────────────────────────────────────────────┘
+                                     │
+                                     ▼
+┌────────────────────────────────────────────────────────────────────────────┐
+│                           rip-db Server (:4213)                            │
+│                                                                            │
+│  ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────────────┐ │
+│  │ Static Proxy    │    │ Binary API      │    │ SSE Events              │ │
+│  │ GET /*          │    │ POST /ddb/run   │    │ GET /localEvents        │ │
+│  │ → ui.duckdb.org │    │ POST /ddb/token │    │ → catalog updates       │ │
+│  └─────────────────┘    │ POST /ddb/intr  │    └─────────────────────────┘ │
+│                         └─────────────────┘                                │
+│                                    │                                       │
+│                                    ▼                                       │
+│  ┌──────────────────────────────────────────────────────────────────────┐  │
+│  │                    High-Performance Zig Bindings                     │  │
+│  │  - Single FFI call per query                                         │  │
+│  │  - Zero-copy for numeric columns (direct memory access)              │  │
+│  │  - Binary serialization done in Zig (no JS overhead)                 │  │
+│  │  - Pre-allocated output buffer (no allocations per query)            │  │
+│  └──────────────────────────────────────────────────────────────────────┘  │
+│                                    │                                       │
+│                                    ▼                                       │
+│                         ┌─────────────────┐                                │
+│                         │     DuckDB      │                                │
+│                         │    (native)     │                                │
+│                         └─────────────────┘                                │
+└────────────────────────────────────────────────────────────────────────────┘
+```
+
+### Why Zig?
+
+The naive approach (per-value FFI calls) has severe problems:
+
+```
+┌─────────┐    FFI call     ┌──────────┐   per-value    ┌──────────┐
+│ DuckDB  │ ──────────────▶ │   Zig    │ ─────────────▶ │ Bun/JS   │
+│ Result  │ ◀────────────── │ Wrapper  │ ◀───────────── │ Extracts │
+└─────────┘   ptr to data   └──────────┘   100k calls   └──────────┘
+                                                              │
+                                                              ▼
+                                          ┌──────────────────────────┐
+                                          │  JavaScript Objects      │
+                                          │  (100k allocations)      │
+                                          └──────────────────────────┘
+```
+
+**Problems:**
+- ~100,000 FFI calls for 10k rows × 10 columns
+- String pointers become invalid after result freed (segfaults!)
+- V8 heap pressure from intermediate objects
+
+**Our solution:** Do everything in Zig with a single FFI call:
+
+```
+┌─────────┐   single call   ┌──────────────────────────────┐
+│ DuckDB  │ ──────────────▶ │         Zig                  │
+│ Result  │                 │  ┌─────────────────────────┐ │
+└─────────┘                 │  │ Binary Serializer       │ │
+     │                      │  │ (direct memory access)  │ │
+     │ column data ptrs     │  └───────────┬─────────────┘ │
+     └──────────────────────┼──────────────┘               │
+                            │              ▼               │
+                            │  ┌─────────────────────────┐ │
+                            │  │ Output Buffer           │ │
+                            │  │ (pre-allocated)         │ │
+                            │  └─────────────────────────┘ │
+                            └──────────────────────────────┘
+                                           │
+                                           ▼
+                            ┌──────────────────────────────┐
+                            │        HTTP Response         │
+                            └──────────────────────────────┘
+```
+
+| Metric | Naive (JS) | Zig | Improvement |
+|--------|------------|-----|-------------|
+| FFI calls (10k×10) | ~100,000 | 1 | 100,000× |
+| Allocations | O(rows×cols) | O(1) | ∞ |
+| Memory copies | 3-4 per value | 1 total | 3-4× |
+| String handling | Unsafe (crash) | Safe | ✓ |
+
+---
+
+## Binary Protocol
+
+The DuckDB UI uses a custom binary format for query results. All numbers are little-endian.
+
+### Primitives
+
+#### varint (Variable-length Integer)
+```
+while (byte & 0x80):
+    result |= (byte & 0x7F) << shift
+    shift += 7
+```
+
+#### Field ID
+```
+id: uint16 (little-endian)
+end marker: 0xFFFF
+```
+
+#### string
+```
+length: varint
+data:   UTF-8 bytes
+```
+
+#### list<T>
+```
+count: varint
+items: T[]
+```
+
+#### nullable<T>
+```
+present: uint8 (0 = null, non-zero = present)
+value:   T (only if present)
+```
+
+### Response Types
+
+#### SuccessResult
+```
+field_100: boolean (true)
+field_101: ColumnNamesAndTypes
+field_102: list<DataChunk>
+0xFFFF
+```
+
+#### ErrorResult
+```
+field_100: boolean (false)
+field_101: string (error message)
+0xFFFF
+```
+
+#### TokenizeResult
+```
+field_100: list<varint> (offsets)
+field_101: list<varint> (token types)
+0xFFFF
+```
+
+### ColumnNamesAndTypes
+```
+field_100: list<string> (column names)
+field_101: list<Type> (column types)
+0xFFFF
+```
+
+### Type
+```
+field_100: uint8 (LogicalTypeId)
+field_101: nullable<TypeInfo>
+0xFFFF
+```
+
+### DataChunk
+```
+field_100: varint (row count)
+field_101: list<Vector>
+0xFFFF
+```
+
+### Vector
+
+```
+field_100: uint8 (allValid: 0 = all valid, 1 = has bitmap)
+field_101: data (validity bitmap - only if allValid != 0)
+field_102: data (values)
+0xFFFF
+```
+
+**Validity bitmap:** LSB first, 1 = valid, 0 = NULL, size = ceil(rows/8) bytes.
+
+### LogicalTypeId
+
+| ID | Type | Bytes |
+|----|------|-------|
+| 10 | BOOLEAN | 1 |
+| 11 | TINYINT | 1 |
+| 12 | SMALLINT | 2 |
+| 13 | INTEGER | 4 |
+| 14 | BIGINT | 8 |
+| 15 | DATE | 4 (days since epoch) |
+| 16 | TIME | 8 (microseconds) |
+| 19 | TIMESTAMP | 8 (microseconds since epoch) |
+| 22 | FLOAT | 4 |
+| 23 | DOUBLE | 8 |
+| 25 | VARCHAR | variable (list<string>) |
+| 26 | BLOB | variable |
+| 50 | HUGEINT | 16 |
+| 54 | UUID | 16 |
+| 100 | STRUCT | nested |
+| 101 | LIST | nested |
+| 102 | MAP | nested |
+
+---
+
+## HTTP Endpoints
+
+### Required for DuckDB UI
+
+| Endpoint | Method | Request | Response |
+|----------|--------|---------|----------|
+| `/ddb/run` | POST | SQL text | Binary result |
+| `/ddb/interrupt` | POST | Empty | Empty result |
+| `/ddb/tokenize` | POST | SQL text | Binary tokens |
+| `/info` | GET | - | Headers only |
+| `/version` | GET | - | JSON `{"origin":"host",...}` |
+| `/config` | GET | - | Proxied + version headers |
+| `/localToken` | GET | - | Empty (no MotherDuck) |
+| `/localEvents` | GET | - | SSE stream |
+
+### Request Headers
+
+| Header | Encoding | Purpose |
+|--------|----------|---------|
+| `Origin` | - | Security check (must match server URL) |
+| `X-DuckDB-UI-Result-Row-Limit` | Plain | Max rows to return |
+| `X-DuckDB-UI-Database-Name` | Base64 | Target database |
+| `X-DuckDB-UI-Schema-Name` | Base64 | Target schema |
+| `X-DuckDB-UI-Parameter-Count` | Plain | Prepared statement params |
+| `X-DuckDB-UI-Parameter-Value-{n}` | Base64 | Param values |
+
+### Response Headers (Required)
+
+```
+X-DuckDB-Version: 1.4.1
+X-DuckDB-Platform: rip-db
+X-DuckDB-UI-Extension-Version: 139-944c08a214
+```
+
+The UI checks `X-DuckDB-UI-Extension-Version` to decide between HTTP and WASM modes.
+
+### Security
+
+The official DuckDB server checks Origin header:
+```cpp
+if (origin != local_url) {
+    res.status = 401;  // UNAUTHORIZED
+}
+```
+
+Our solution: Proxy UI assets from `ui.duckdb.org` so all requests are same-origin.
+
+---
+
+## Building
+
+```bash
+cd packages/db
+
+# Build (outputs to lib/{os}-{arch}/duckdb.node)
+# ReleaseFast is the default, so no -Doptimize flag needed
+zig build --prefix .
+
+# Run tests
+zig build test
+
+# Debug build
+zig build -Doptimize=Debug --prefix .
+```
+
+Requires DuckDB headers and library. Set `DUCKDB_DIR` if not at `/opt/homebrew`:
+```bash
+DUCKDB_DIR=/path/to/duckdb zig build
+```
+
+---
+
+## Debugging
+
+### Test endpoints
+
+```bash
+# Start server
+rip db.rip :memory: --port 4213
+
+# Test binary endpoint
+curl -X POST http://localhost:4213/ddb/run \
+  -H "Origin: http://localhost:4213" \
+  -d "SELECT 42" | xxd | head -5
+
+# Should see: 6400 01 = field 100, value true (success)
+```
+
+### Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| 401 on /ddb/run | Wrong Origin | Load UI from localhost, not ui.duckdb.org |
+| UI shows "WASM" | Missing version header | Add `X-DuckDB-UI-Extension-Version` |
+| Syntax highlighting broken | /ddb/tokenize error | Check server logs |
+| Segfault | Memory lifetime | Use Zig bindings, not JS FFI per-value |
+
+---
+
+## References
+
+- [DuckDB UI GitHub](https://github.com/duckdb/duckdb-ui)
+- [DuckDB UI Client](https://github.com/duckdb/duckdb-ui/tree/main/ts/pkgs/duckdb-ui-client)
+- [BinaryDeserializer.ts](https://github.com/duckdb/duckdb-ui/blob/main/ts/pkgs/duckdb-ui-client/src/serialization/classes/BinaryDeserializer.ts)