@jsonbit/core 0.0.0-stage → 1.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +316 -2
- package/bin/jsonbit.js +45 -0
- package/package.json +52 -4
- package/scripts/install-binary.mjs +86 -0
- package/src/index.ts +128 -0
- package/src/types.ts +94 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026-PRESENT Prathik Murugan
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,317 @@
|
|
|
1
|
-
#
|
|
1
|
+
# jsonbit
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](./Cargo.toml)
|
|
4
|
+
[](./package.json)
|
|
5
|
+
[](./python/jsonbit/__init__.py)
|
|
6
|
+
[](./rust/mcp.rs)
|
|
7
|
+
[](./LICENSE)
|
|
8
|
+
|
|
9
|
+
`jsonbit` is an in-memory and on-disk analytical engine written in Rust that normalizes nested `.json` and `.jsonl` payloads into Third-Normal-Form (3NF) bit-packed columnar tables. It provides sub-10-microsecond bitmap query execution, compact `.jsonbit` binary snapshots, and a compact OLAP schema summary designed for Large Language Model (LLM) context windows and Model Context Protocol (MCP) agents.
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## Table of Contents
|
|
14
|
+
|
|
15
|
+
- [Overview](#overview)
|
|
16
|
+
- [Installation & Setup](#installation--setup)
|
|
17
|
+
- [Agent Integrations (OpenCode, Claude Code, Codex, Cursor)](#agent-integrations)
|
|
18
|
+
- [SDK Usage (Python, TypeScript, Rust)](#sdk-usage)
|
|
19
|
+
- [CLI Reference](#cli-reference)
|
|
20
|
+
- [Architecture](#architecture)
|
|
21
|
+
- [Benchmarks](#benchmarks)
|
|
22
|
+
- [License](#license)
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## Overview
|
|
27
|
+
|
|
28
|
+
Passing large structured datasets (API responses, telemetry logs, database exports) directly into LLM prompts as raw JSON or text tables introduces two bottlenecks:
|
|
29
|
+
|
|
30
|
+
1. **Token Volume:** A 2,000-record dataset consumes `50,000–180,000` tokens in JSON or text-table encodings, increasing API cost and time-to-first-token (TTFT).
|
|
31
|
+
2. **In-Context Arithmetic Errors:** Autoregressive attention struggles with exact row counting, multi-column filtering, and floating-point summation across thousands of lines.
|
|
32
|
+
|
|
33
|
+
`jsonbit` addresses both problems by separating **storage and computation** from **prompt context**:
|
|
34
|
+
|
|
35
|
+
- **Automatic 3NF Normalization:** Decomposes nested 1-to-1 objects, sparse optional fields, dynamic-keyed maps, and 1-to-Many child arrays into flat columnar tables linked by Compressed Sparse Row (CSR) offset arrays.
|
|
36
|
+
- **Bit-Packed Columnar Storage:** Compresses integers using Frame-of-Reference (`FOR`) delta packing, strings using dictionary bit-packing with inverted 64-bit bitmaps (`BitVec64`), and booleans using 1 bit per row—reducing storage size by **64.4%** compared to minified JSON while maintaining lossless round-trip reconstruction.
|
|
37
|
+
- **Compact OLAP Data Cards:** Emits a `~200–540` token summary containing table schemas, column types, exact numeric aggregates (`sum`, `avg`, `min`, `max`), and categorical value distributions (**99.24% fewer tokens** than full-dataset text encodings).
|
|
38
|
+
- **Deterministic Execution:** Evaluates natural-language slices and SQL queries (`SELECT`, `WHERE`, `GROUP BY`, `ORDER BY`, `LIMIT`, and parent-child `JOIN`s) directly on bit-packed columns and 64-bit word-aligned bitmaps in **`8.0 µs`** average latency.
|
|
39
|
+
|
|
40
|
+
---
|
|
41
|
+
|
|
42
|
+
## Installation & Setup
|
|
43
|
+
|
|
44
|
+
### Prerequisites
|
|
45
|
+
- **Rust** (`1.75+`, [rustup.rs](https://rustup.rs/))
|
|
46
|
+
- *(Optional)* **Python 3.10+** for the Python SDK, or **Node.js 20+** for the TypeScript SDK
|
|
47
|
+
|
|
48
|
+
### Build & Install Locally
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
git clone https://github.com/prathikmurugan/jsonbit.git
|
|
52
|
+
cd jsonbit
|
|
53
|
+
|
|
54
|
+
# Build optimized release binary (./target/release/jsonbit)
|
|
55
|
+
cargo build --release
|
|
56
|
+
|
|
57
|
+
# Install `jsonbit` globally into your PATH (~/.cargo/bin)
|
|
58
|
+
cargo install --path .
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
Verify installation:
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
jsonbit --help
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
*(Optional)* Install the Python package & binary in editable/wheel mode via `maturin` or `pip`:
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
pip install .
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
---
|
|
74
|
+
|
|
75
|
+
## Agent Integrations
|
|
76
|
+
|
|
77
|
+
`jsonbit` includes a built-in stdio JSON-RPC 2.0 MCP server (`jsonbit mcp`) with zero external runtime dependencies. It exposes five tools: `olap_load`, `olap_digest`, `olap_ask`, `olap_sql`, and `olap_rehydrate`.
|
|
78
|
+
|
|
79
|
+
### OpenCode (`opencode.json`)
|
|
80
|
+
|
|
81
|
+
Add the following to `opencode.json` in your project root (or globally in `~/.config/opencode/opencode.json`):
|
|
82
|
+
|
|
83
|
+
```json
|
|
84
|
+
{
|
|
85
|
+
"$schema": "https://opencode.ai/config.json",
|
|
86
|
+
"mcp": {
|
|
87
|
+
"jsonbit": {
|
|
88
|
+
"type": "local",
|
|
89
|
+
"command": [
|
|
90
|
+
"/path/to/jsonbit/target/release/jsonbit",
|
|
91
|
+
"mcp"
|
|
92
|
+
],
|
|
93
|
+
"enabled": true
|
|
94
|
+
}
|
|
95
|
+
}
|
|
96
|
+
}
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
### Claude Code (`.mcp.json` or CLI)
|
|
100
|
+
|
|
101
|
+
Register via the Claude Code CLI:
|
|
102
|
+
|
|
103
|
+
```bash
|
|
104
|
+
claude mcp add jsonbit -- /path/to/jsonbit/target/release/jsonbit mcp
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
Or include in `.mcp.json`:
|
|
108
|
+
|
|
109
|
+
```json
|
|
110
|
+
{
|
|
111
|
+
"mcpServers": {
|
|
112
|
+
"jsonbit": {
|
|
113
|
+
"command": "/path/to/jsonbit/target/release/jsonbit",
|
|
114
|
+
"args": ["mcp"]
|
|
115
|
+
}
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
### Cursor & Windsurf
|
|
121
|
+
|
|
122
|
+
Add the `.mcp.json` configuration above to `.cursor/mcp.json` or `~/.codeium/windsurf/mcp_config.json`.
|
|
123
|
+
|
|
124
|
+
### OpenAI Codex CLI (`AGENTS.md`)
|
|
125
|
+
|
|
126
|
+
Codex and terminal agents automatically read [`AGENTS.md`](./AGENTS.md) in the repository root to invoke `jsonbit` via CLI or MCP when inspecting `.json`, `.jsonl`, or `.jsonbit` files.
|
|
127
|
+
|
|
128
|
+
---
|
|
129
|
+
|
|
130
|
+
## SDK Usage
|
|
131
|
+
|
|
132
|
+
### Python (`python/jsonbit`)
|
|
133
|
+
|
|
134
|
+
Requires Python 3.10+ with no third-party pip packages:
|
|
135
|
+
|
|
136
|
+
```python
|
|
137
|
+
from jsonbit import JsonBit # or `from python.jsonbit import JsonBit` from repo root
|
|
138
|
+
|
|
139
|
+
with JsonBit("benchmarks/data/github-repos.json") as olap:
|
|
140
|
+
# Generate compact OLAP schema & statistical card
|
|
141
|
+
card = olap.card()
|
|
142
|
+
|
|
143
|
+
# Deterministic scalar answer
|
|
144
|
+
ans = olap.answer("How many repositories have more than 150000 stars and more than 30000 forks?")
|
|
145
|
+
assert ans == "31"
|
|
146
|
+
|
|
147
|
+
# Read-only SQL query (supports GROUP BY, ORDER BY, LIMIT, and JSON dot-paths)
|
|
148
|
+
rows = olap.sql(
|
|
149
|
+
"SELECT defaultBranch, COUNT(*) AS cnt FROM records GROUP BY defaultBranch ORDER BY cnt DESC"
|
|
150
|
+
)
|
|
151
|
+
|
|
152
|
+
# Save compressed .jsonbit binary snapshot
|
|
153
|
+
olap.pack("repos.jsonbit")
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
### TypeScript / Node.js (`@jsonbit/core`)
|
|
157
|
+
|
|
158
|
+
```ts
|
|
159
|
+
import { JsonBit, rehydrateFrom3NF } from '@jsonbit/core'
|
|
160
|
+
|
|
161
|
+
const engine = JsonBit.from(jsonData)
|
|
162
|
+
|
|
163
|
+
// Retrieve the OLAP data card
|
|
164
|
+
console.log(engine.getDataCard().text)
|
|
165
|
+
|
|
166
|
+
// Retrieve a verified query slice or scalar answer
|
|
167
|
+
console.log(engine.answer('How many shipped orders include a USB Cable?'))
|
|
168
|
+
console.log(engine.slice('How many shipped orders include a USB Cable?').context)
|
|
169
|
+
|
|
170
|
+
// Execute SQL (automatically resolves JSON dot-paths such as customer.name -> customer_name)
|
|
171
|
+
const rows = engine.sql(`SELECT orderId, customer.name, total FROM orders WHERE total > 500`)
|
|
172
|
+
|
|
173
|
+
// Losslessly reconstruct original nested JSON
|
|
174
|
+
const original = rehydrateFrom3NF(engine)
|
|
175
|
+
engine.close()
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
### Rust Library (`jsonbit`)
|
|
179
|
+
|
|
180
|
+
```rust
|
|
181
|
+
use jsonbit::{execute_sql, JsonBitEngine};
|
|
182
|
+
use serde_json::Value;
|
|
183
|
+
|
|
184
|
+
let data: Value = serde_json::from_str(raw_json)?;
|
|
185
|
+
let engine = JsonBitEngine::from_json(&data);
|
|
186
|
+
|
|
187
|
+
let card = engine.data_card();
|
|
188
|
+
let slice = engine.query_slice("How many active employees work in Sales?");
|
|
189
|
+
let rows = execute_sql(&engine, "SELECT department, COUNT(*) AS cnt FROM employees GROUP BY department")?;
|
|
190
|
+
let restored = engine.rehydrate();
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
---
|
|
194
|
+
|
|
195
|
+
## CLI Reference
|
|
196
|
+
|
|
197
|
+
| Command | Arguments | Description |
|
|
198
|
+
| :--- | :--- | :--- |
|
|
199
|
+
| `jsonbit card` | `<file.json\|.jsonl\|.jsonbit>` | Prints the $O(1)$ 3NF OLAP Cube Card (`~250 tokens`) and storage compression statistics. |
|
|
200
|
+
| `jsonbit digest` | `<file.json\|.jsonl\|.jsonbit>` | Prints the 1-Turn Factorized 3NF Holistic Digest (100% of rows + `BitVec64` Pivot Legend, `-41%` to `-72%` smaller than `TOON`). |
|
|
201
|
+
| `jsonbit ask` | `<file> "<question>"` | Evaluates a question against the bitmap index and prints the verified context slice. |
|
|
202
|
+
| `jsonbit answer` | `<file> "<question>"` | Evaluates a question and prints only the scalar answer value. |
|
|
203
|
+
| `jsonbit sql` | `<file> "<SELECT ...>"` | Executes a read-only SQL query against the 3NF columnar store (`WHERE`, `IN`, `BETWEEN`, `LIKE`, `GROUP BY`, `ORDER BY`, `LIMIT`, `OFFSET`, CSR `JOIN`). |
|
|
204
|
+
| `jsonbit pack` | `<input.json> <output.jsonbit>` | Serializes the bit-packed 3NF tables into a binary `.jsonbit` snapshot. |
|
|
205
|
+
| `jsonbit rehydrate` | `<file.json\|.jsonl\|.jsonbit>` | Reconstructs and prints the original nested JSON document. |
|
|
206
|
+
| `jsonbit mcp` | *(none)* | Starts the concurrent stdio JSON-RPC 2.0 MCP server (`olap_load`, `olap_digest`, `olap_ask`, `olap_sql`, `olap_rehydrate`). |
|
|
207
|
+
|
|
208
|
+
---
|
|
209
|
+
|
|
210
|
+
## Architecture
|
|
211
|
+
|
|
212
|
+
```mermaid
|
|
213
|
+
flowchart TD
|
|
214
|
+
Input["Nested JSON / JSONL Input"] --> Normalizer["3NF Normalizer & CSR Indexer (rust/normalizer.rs)"]
|
|
215
|
+
Normalizer --> ColStore["Bit-Packed Columnar Store (rust/column.rs)"]
|
|
216
|
+
ColStore --> Bitmap["BitVec64 Bitmap Engine (rust/bitmap.rs)"]
|
|
217
|
+
Bitmap --> Card["OLAP Data Card (rust/engine.rs)"]
|
|
218
|
+
Bitmap --> SqlExec["SQL & Predicate Executor (rust/query_exec.rs)"]
|
|
219
|
+
ColStore --> Snapshot[".jsonbit Binary Snapshot (rust/snapshot.rs)"]
|
|
220
|
+
ColStore --> Rehydrate["Lossless 3NF Rehydrator (rust/normalizer.rs)"]
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
### Storage Layout (`rust/column.rs` & `rust/normalizer.rs`)
|
|
224
|
+
|
|
225
|
+
1. **`ForIntColumn` (Frame-of-Reference Integers):** Computes `[min_val, max_val]` per column and packs unsigned deltas `val - min_val` into `ceil(log2(max_val - min_val + 1))` bits per row inside a `Vec<u64>` buffer. Column headers store `sum`, `min`, `max`, and `non_null_count` for `O(1)` unfiltered aggregations.
|
|
226
|
+
2. **`DictStringColumn` (Dictionary + Inverted Bitmaps):** Interns unique strings into a string arena, bit-packs dictionary IDs into `ceil(log2(|D|))` bits per row, and builds a 64-bit word-aligned `BitVec64` inverted index per category (`|D| <= 64`) for `O(1)` equality lookups and `POPCNT` frequency histograms.
|
|
227
|
+
3. **`BoolBitmapColumn` (1-Bit Booleans):** Packs 64 boolean values into each `u64` word alongside a 1-bit validity mask.
|
|
228
|
+
4. **CSR 1-to-Many Join Index:** Child arrays (e.g., `orders[].items[]`) are normalized into child tables with a contiguous `csr_offsets: Vec<u32>` array mapping each parent row `i` to its child slice `csr_offsets[i]..csr_offsets[i + 1]`.
|
|
229
|
+
|
|
230
|
+
---
|
|
231
|
+
|
|
232
|
+
## Benchmarks
|
|
233
|
+
|
|
234
|
+
All benchmarks execute directly against the compiled Pure-Rust `jsonbit` binary (`target/release/jsonbit`).
|
|
235
|
+
|
|
236
|
+
### 1. Live `gemini-3.8-flash` Evaluation Across 10 Real-World Workloads (27 Questions)
|
|
237
|
+
|
|
238
|
+
Evaluates `Raw JSON`, `Minified JSON`, `YAML`, `TOON` (`@toon-format/toon`), and `jsonbit` (Pure-Rust `olap_load` + `olap_sql`) across 10 production workloads (`W1`–`W10`), including flat tables, 1-to-Many e-commerce orders, sparse `.jsonl` OpenTelemetry logs with free-text messages, singleton Kubernetes configs, IAM primitive arrays, wide 30-column tables, and multi-byte UTF-8 financial ledgers:
|
|
239
|
+
|
|
240
|
+

|
|
241
|
+
|
|
242
|
+
| Format / Tool | Mode | Live `gemini-3.8-flash` Accuracy | Total Prompt Tokens | Token Reduction vs. Raw JSON | Token Reduction vs. `TOON` |
|
|
243
|
+
| :--- | :--- | ---: | ---: | ---: | ---: |
|
|
244
|
+
| **Raw JSON (Pretty)** | 1-Turn Full Text | 55.6% (15/27) | 375,713 | — | — |
|
|
245
|
+
| **Minified JSON** | 1-Turn Full Text | 48.1% (13/27) | 237,891 | -36.7% | — |
|
|
246
|
+
| **YAML** | 1-Turn Full Text | 59.3% (16/27) | 293,690 | -21.8% | — |
|
|
247
|
+
| **`TOON` v4 (`@toon-format/toon`)** | 1-Turn Full Text | 59.3% (16/27) | 169,627 | -54.9% | — |
|
|
248
|
+
| **`jsonbit` Holistic Digest (`olap_digest`)** | **1-Turn (100% Rows + Pivot Legend)** | **96.3% (26/27)** | **101,302** | **-73.0%** | **-40.3%** |
|
|
249
|
+
| **`jsonbit` Cube Card + SQL (`olap_load` + `olap_sql`)** | **Agentic $O(1)$ Card + Rust SQL** | **100.0% (27/27)** | **11,729** | **-96.9%** | **-93.1%** |
|
|
250
|
+
|
|
251
|
+
#### Qualitative & Holistic Executive Summary Verification (`1-Turn: TOON vs. jsonbit digest`)
|
|
252
|
+
|
|
253
|
+
For qualitative tasks where the LLM must read all records in a single turn and write a multi-fact executive summary (naming all departments, products, microservice failure messages, and international counterparties across `W3`, `W4`, `W5`, and `W10`):
|
|
254
|
+
|
|
255
|
+
| Holistic Synthesis Task | `TOON` v4 Facts Preserved | `jsonbit digest` Facts Preserved | `TOON` Tokens | `jsonbit digest` Tokens | Token Savings vs. `TOON` |
|
|
256
|
+
| :--- | ---: | ---: | ---: | ---: | ---: |
|
|
257
|
+
| `H1` — HR Directory Executive Summary | 8 / 9 | **9 / 9 (100%)** | 4,881 | **4,372** | **-10.4%** |
|
|
258
|
+
| `H2` — E-Commerce Catalog & Orders Summary | 8 / 8 | **8 / 8 (100%)** | 15,405 | **6,933** | **-55.0%** |
|
|
259
|
+
| `H3` — Production Incident Log Triage Report | 9 / 9 | **9 / 9 (100%)** | 13,665 | **4,169** | **-69.5%** |
|
|
260
|
+
| `H4` — International Settlement Ledger Summary | 2 / 6 | **6 / 6 (100%)** | 3,781 | **2,403** | **-36.4%** |
|
|
261
|
+
| **Total Across All Qualitative Summaries** | **27 / 32 (84.4%)** | **32 / 32 (100.0%)** | **37,732** | **17,877** | **-52.6%** |
|
|
262
|
+
|
|
263
|
+
### 2. Token Scaling Curve (`4` $\rightarrow$ `25,000` Nested Orders)
|
|
264
|
+
|
|
265
|
+
While linear text encodings (`Raw JSON`, `Minified JSON`, `YAML`, `TOON`) scale at $O(N)$ with row count and exceed LLM context windows at scale, `jsonbit` maintains an $O(1)$ prompt footprint (`~67–473` tokens):
|
|
266
|
+
|
|
267
|
+

|
|
268
|
+
|
|
269
|
+
| Dataset Scale (Nested Orders) | Raw JSON (Tokens) | YAML (Tokens) | `TOON` v4 (Tokens) | `jsonbit` Rust Card (Tokens) | Rust Ingest Time |
|
|
270
|
+
| :--- | ---: | ---: | ---: | ---: | ---: |
|
|
271
|
+
| **4 rows** *(Tiny API Payload)* | 166 | 122 | 71 | **67** | 0.04 ms |
|
|
272
|
+
| **50 rows** *(~150 line items)* | 7,780 | 5,801 | 4,783 | **447** | 0.52 ms |
|
|
273
|
+
| **500 rows** *(~1,500 line items)* | 78,115 | 58,244 | 47,983 | **449** | 4.72 ms |
|
|
274
|
+
| **5,000 rows** *(~15,000 line items)* | 783,443 | 366,356 | 293,972 | **472** | 58.97 ms |
|
|
275
|
+
| **25,000 rows** *(~75,000 line items)* | 3,917,366 | 1,831,843 | 1,469,911 | **473** | 334.42 ms |
|
|
276
|
+
|
|
277
|
+
### 3. Memory Footprint & Query Speed (`Pure-Rust jsonbit` vs. `jq` CLI)
|
|
278
|
+
|
|
279
|
+

|
|
280
|
+
|
|
281
|
+
| Workload | Category | Minified JSON | Rust 3NF Bit-Packed | Pretty JSON (Tok) | `TOON` (100% Rows) | `jsonbit digest` (100% Rows) | `jsonbit card` ($O(1)$) | 3NF Lossless |
|
|
282
|
+
| :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | :---: |
|
|
283
|
+
| `W1-tiny-api-response` | Tiny Payload (<500B) | 0.3 KB | **0.1 KB** | 166 | 71 | **67 (-5.6%)** | **67** | PASS |
|
|
284
|
+
| `W2-singleton-k8s-config` | Singleton Config Tree | 0.3 KB | **0.2 KB** | 167 | 131 | **110 (-16.0%)** | **110** | PASS |
|
|
285
|
+
| `W3-hr-payroll-directory` | Tabular (200 rows) | 28.4 KB | **4.5 KB** | 14,064 | 4,847 | **4,145 (-14.5%)** | **272** | PASS |
|
|
286
|
+
| `W4-ecommerce-orders-1toN` | 1-to-Many (150 orders) | 48.3 KB | **4.5 KB** | 25,466 | 15,372 | **6,878 (-55.3%)** | **472** | PASS |
|
|
287
|
+
| `W5-otel-production-logs` | Sparse `.jsonl` + Free-Text | 44.4 KB | **4.7 KB** | 17,439 | 13,632 | **4,123 (-69.8%)** | **226** | PASS |
|
|
288
|
+
| `W6-github-repos-metrics` | High-Card + Ranges | 28.6 KB | **3.4 KB** | 14,829 | 5,364 | **4,285 (-20.1%)** | **309** | PASS |
|
|
289
|
+
| `W7-stripe-dispute-webhooks` | Financial Webhooks | 28.9 KB | **3.1 KB** | 14,333 | 4,104 | **3,906 (-4.8%)** | **250** | PASS |
|
|
290
|
+
| `W8-iam-primitive-arrays` | Primitive Arrays | 14.4 KB | **1.6 KB** | 7,185 | 4,810 | **1,247 (-74.1%)** | **164** | PASS |
|
|
291
|
+
| `W9-wide-enterprise-crm` | Wide Table (30 cols) | 68.6 KB | **5.7 KB** | 40,779 | 9,806 | **9,516 (-3.0%)** | **1,136** | PASS |
|
|
292
|
+
| `W10-unicode-financial-ledger` | UTF-8, Nulls & Zeroes | 18.4 KB | **2.9 KB** | 9,550 | 3,750 | **2,359 (-37.1%)** | **243** | PASS |
|
|
293
|
+
| **Total (10 Workloads)** | **All Categories** | **280.6 KB** | **30.6 KB (-89.1%)** | **143,978** | **61,887** | **36,636 (-40.8%)** | **3,249 (-94.7%)** | **10/10 PASS** |
|
|
294
|
+
|
|
295
|
+
- **Query Latency (`Pure-Rust BitVec64 SQL` vs. `jq` C CLI):**
|
|
296
|
+
- `jq` CLI (`stedolan/jq`): `6,545.3 µs` average per query
|
|
297
|
+
- `jsonbit` Pure-Rust SQL (`execute_sql`): **`6.8 µs` average per query (`963x` faster)**
|
|
298
|
+
- `jsonbit` Pure-Rust Bitmap Slice (`query_slice`): **`19.9 µs` average per query (`329x` faster)**
|
|
299
|
+
- **Standard 8-Dataset Benchmark Suite:**
|
|
300
|
+
- **Storage Compression:** `1,178,283` bytes (Minified JSON) $\rightarrow$ **`419,661` bytes (`-64.4%` RAM/storage reduction)**, `8/8` (`100%`) lossless 3NF rehydration.
|
|
301
|
+
- **Token Reduction:** `334,046` tokens (`TOON` v4) $\rightarrow$ **`2,536` tokens (`-99.24%`)**.
|
|
302
|
+
- **239-Question Evaluation Accuracy:** `99.16% (237/239)` strict match at **`8.0 µs`** average query latency.
|
|
303
|
+
|
|
304
|
+
### 4. Autonomous Coding Agent Head-to-Head (`OpenCode v1.19.5`)
|
|
305
|
+
|
|
306
|
+
Evaluated in isolated directories with zero prompt hints and zero `AGENTS.md`/`CLAUDE.md` files (`opencode/big-pickle`) across both standard (bash-enabled) and sandboxed (`bash: "deny"`) agent environments:
|
|
307
|
+
|
|
308
|
+
| Agent Mode | `jsonbit` MCP (`opencode.json` only) | `TOON` v4 (`toon_encode` / CLI) | Raw JSON |
|
|
309
|
+
| :--- | :--- | :--- | :--- |
|
|
310
|
+
| **Standard Agent (`6` Scenarios, `18` Runs)** | **6/6 (`88.1s`) · `0` scripts** | 6/6 (`142.8s`) · `awk`/Python parsers | 6/6 (`93.7s`) · Python scripts |
|
|
311
|
+
| **Sandboxed Read-Only (`bash: "deny"`, `8` Scenarios)** | **8/8 (`77.7s`)** | 7/8 (`333.8s`) · failed `500`-order sum | 7/8 (`345.4s`) · failed `500`-order sum |
|
|
312
|
+
|
|
313
|
+
---
|
|
314
|
+
|
|
315
|
+
## License
|
|
316
|
+
|
|
317
|
+
[MIT License](./LICENSE).
|
package/bin/jsonbit.js
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* Native Pure-Rust `jsonbit` CLI & MCP launcher for npm / npx.
|
|
4
|
+
* Resolves the compiled `jsonbit` binary from:
|
|
5
|
+
* 1. `JSONBIT_BIN` environment variable
|
|
6
|
+
* 2. Local build (`./target/release/jsonbit` or `vendor/<platform>/jsonbit`)
|
|
7
|
+
* 3. Global `jsonbit` on `PATH`
|
|
8
|
+
*/
|
|
9
|
+
import { spawnSync } from 'node:child_process'
|
|
10
|
+
import fs from 'node:fs'
|
|
11
|
+
import path from 'node:path'
|
|
12
|
+
import { fileURLToPath } from 'node:url'
|
|
13
|
+
|
|
14
|
+
const __dirname = path.dirname(fileURLToPath(import.meta.url))
|
|
15
|
+
const pkgRoot = path.resolve(__dirname, '..')
|
|
16
|
+
const exe = process.platform === 'win32' ? 'jsonbit.exe' : 'jsonbit'
|
|
17
|
+
|
|
18
|
+
const candidates = [
|
|
19
|
+
process.env.JSONBIT_BIN,
|
|
20
|
+
path.join(pkgRoot, 'vendor', `${process.platform}-${process.arch}`, exe),
|
|
21
|
+
path.join(pkgRoot, 'target', 'release', exe),
|
|
22
|
+
exe,
|
|
23
|
+
].filter(Boolean)
|
|
24
|
+
|
|
25
|
+
let resolvedBin = exe
|
|
26
|
+
for (const c of candidates) {
|
|
27
|
+
if (c === exe || fs.existsSync(c)) {
|
|
28
|
+
resolvedBin = c
|
|
29
|
+
break
|
|
30
|
+
}
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
const res = spawnSync(resolvedBin, process.argv.slice(2), {
|
|
34
|
+
stdio: 'inherit',
|
|
35
|
+
})
|
|
36
|
+
|
|
37
|
+
if (res.error) {
|
|
38
|
+
console.error(
|
|
39
|
+
`[jsonbit] Could not execute native binary (${resolvedBin}).\n` +
|
|
40
|
+
`Build from source with \`cargo build --release\` or install via \`cargo install jsonbit\`.`,
|
|
41
|
+
)
|
|
42
|
+
process.exit(1)
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
process.exit(res.status ?? 0)
|
package/package.json
CHANGED
|
@@ -1,6 +1,54 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@jsonbit/core",
|
|
3
|
-
"version": "
|
|
4
|
-
"
|
|
5
|
-
"
|
|
6
|
-
|
|
3
|
+
"version": "1.1.1",
|
|
4
|
+
"description": "jsonbit — Pure-Rust Auto-3NF Columnar Bit-Packed & BitVec64 Engine for LLMs and AI Agents",
|
|
5
|
+
"type": "module",
|
|
6
|
+
"main": "./src/index.ts",
|
|
7
|
+
"types": "./src/index.ts",
|
|
8
|
+
"bin": {
|
|
9
|
+
"jsonbit": "./bin/jsonbit.js"
|
|
10
|
+
},
|
|
11
|
+
"files": [
|
|
12
|
+
"bin/",
|
|
13
|
+
"scripts/",
|
|
14
|
+
"src/",
|
|
15
|
+
"README.md",
|
|
16
|
+
"LICENSE"
|
|
17
|
+
],
|
|
18
|
+
"scripts": {
|
|
19
|
+
"postinstall": "node ./scripts/install-binary.mjs",
|
|
20
|
+
"test": "NODE_NO_WARNINGS=1 vitest run",
|
|
21
|
+
"bench": "NODE_NO_WARNINGS=1 tsx benchmarks/run-full-rust-benchmark.ts --reuse-gemini",
|
|
22
|
+
"cli": "node ./bin/jsonbit.js"
|
|
23
|
+
},
|
|
24
|
+
"keywords": [
|
|
25
|
+
"llm",
|
|
26
|
+
"olap",
|
|
27
|
+
"json",
|
|
28
|
+
"jsonbit",
|
|
29
|
+
"rust",
|
|
30
|
+
"3nf",
|
|
31
|
+
"agents",
|
|
32
|
+
"mcp",
|
|
33
|
+
"tokens"
|
|
34
|
+
],
|
|
35
|
+
"license": "MIT",
|
|
36
|
+
"author": "Prathik Murugan",
|
|
37
|
+
"homepage": "https://github.com/prathikmurugan/jsonbit#readme",
|
|
38
|
+
"repository": {
|
|
39
|
+
"type": "git",
|
|
40
|
+
"url": "https://github.com/prathikmurugan/jsonbit.git"
|
|
41
|
+
},
|
|
42
|
+
"bugs": {
|
|
43
|
+
"url": "https://github.com/prathikmurugan/jsonbit/issues"
|
|
44
|
+
},
|
|
45
|
+
"devDependencies": {
|
|
46
|
+
"@faker-js/faker": "^10.6.0",
|
|
47
|
+
"@toon-format/toon": "^4.4.0",
|
|
48
|
+
"@types/node": "^25.8.0",
|
|
49
|
+
"gpt-tokenizer": "^4.1.2",
|
|
50
|
+
"tsx": "^4.21.0",
|
|
51
|
+
"typescript": "^5.9.3",
|
|
52
|
+
"vitest": "^3.2.4"
|
|
53
|
+
}
|
|
54
|
+
}
|
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* Downloads the pre-built native `jsonbit` binary from GitHub Releases into
|
|
4
|
+
* `vendor/<platform>-<arch>/jsonbit` on `npm install` when no local build exists.
|
|
5
|
+
*/
|
|
6
|
+
import { execFileSync } from 'node:child_process'
|
|
7
|
+
import fs from 'node:fs'
|
|
8
|
+
import https from 'node:https'
|
|
9
|
+
import os from 'node:os'
|
|
10
|
+
import path from 'node:path'
|
|
11
|
+
import { fileURLToPath } from 'node:url'
|
|
12
|
+
|
|
13
|
+
const __dirname = path.dirname(fileURLToPath(import.meta.url))
|
|
14
|
+
const pkgRoot = path.resolve(__dirname, '..')
|
|
15
|
+
const pkg = JSON.parse(fs.readFileSync(path.join(pkgRoot, 'package.json'), 'utf8'))
|
|
16
|
+
|
|
17
|
+
const platform = process.platform
|
|
18
|
+
const arch = process.arch
|
|
19
|
+
const exe = platform === 'win32' ? 'jsonbit.exe' : 'jsonbit'
|
|
20
|
+
|
|
21
|
+
const localTarget = path.join(pkgRoot, 'target', 'release', exe)
|
|
22
|
+
const vendorDir = path.join(pkgRoot, 'vendor', `${platform}-${arch}`)
|
|
23
|
+
const vendorBin = path.join(vendorDir, exe)
|
|
24
|
+
|
|
25
|
+
if (process.env.JSONBIT_SKIP_DOWNLOAD === '1' || fs.existsSync(localTarget) || fs.existsSync(vendorBin)) {
|
|
26
|
+
process.exit(0)
|
|
27
|
+
}
|
|
28
|
+
|
|
29
|
+
const SUPPORTED = new Set(['linux-x64', 'linux-arm64', 'darwin-arm64', 'darwin-x64', 'win32-x64'])
|
|
30
|
+
const key = `${platform}-${arch}`
|
|
31
|
+
if (!SUPPORTED.has(key)) {
|
|
32
|
+
console.warn(`[jsonbit] No pre-built binary for ${key}. Install via \`cargo install jsonbit\`.`)
|
|
33
|
+
process.exit(0)
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
const ext = platform === 'win32' ? 'zip' : 'tar.gz'
|
|
37
|
+
const url = `https://github.com/prathikmurugan/jsonbit/releases/download/v${pkg.version}/jsonbit-${key}.${ext}`
|
|
38
|
+
|
|
39
|
+
function download(targetUrl, destPath, redirects = 0) {
|
|
40
|
+
return new Promise((resolve, reject) => {
|
|
41
|
+
if (redirects > 5) return reject(new Error('Too many redirects'))
|
|
42
|
+
https
|
|
43
|
+
.get(targetUrl, { headers: { 'User-Agent': `jsonbit/${pkg.version}` } }, res => {
|
|
44
|
+
if (res.statusCode && res.statusCode >= 300 && res.statusCode < 400 && res.headers.location) {
|
|
45
|
+
res.resume()
|
|
46
|
+
return resolve(download(res.headers.location, destPath, redirects + 1))
|
|
47
|
+
}
|
|
48
|
+
if (res.statusCode !== 200) {
|
|
49
|
+
res.resume()
|
|
50
|
+
return reject(new Error(`HTTP ${res.statusCode} fetching ${targetUrl}`))
|
|
51
|
+
}
|
|
52
|
+
const file = fs.createWriteStream(destPath)
|
|
53
|
+
res.pipe(file)
|
|
54
|
+
file.on('finish', () => file.close(() => resolve()))
|
|
55
|
+
file.on('error', reject)
|
|
56
|
+
})
|
|
57
|
+
.on('error', reject)
|
|
58
|
+
})
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
async function main() {
|
|
62
|
+
const tmpArchive = path.join(os.tmpdir(), `jsonbit-${pkg.version}-${key}.${ext}`)
|
|
63
|
+
try {
|
|
64
|
+
await download(url, tmpArchive)
|
|
65
|
+
fs.mkdirSync(vendorDir, { recursive: true })
|
|
66
|
+
if (platform === 'win32') {
|
|
67
|
+
execFileSync('powershell', [
|
|
68
|
+
'-NoProfile',
|
|
69
|
+
'-Command',
|
|
70
|
+
`Expand-Archive -Force -Path '${tmpArchive}' -DestinationPath '${vendorDir}'`,
|
|
71
|
+
])
|
|
72
|
+
} else {
|
|
73
|
+
execFileSync('tar', ['-xzf', tmpArchive, '-C', vendorDir])
|
|
74
|
+
fs.chmodSync(vendorBin, 0o755)
|
|
75
|
+
}
|
|
76
|
+
} catch (err) {
|
|
77
|
+
console.warn(
|
|
78
|
+
`[jsonbit] Could not download pre-built binary (${err.message}).\n` +
|
|
79
|
+
`Build locally with \`cargo build --release\` or install via \`cargo install jsonbit\`.`,
|
|
80
|
+
)
|
|
81
|
+
} finally {
|
|
82
|
+
if (fs.existsSync(tmpArchive)) fs.unlinkSync(tmpArchive)
|
|
83
|
+
}
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
main()
|
package/src/index.ts
ADDED
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @jsonbit/core — TypeScript / Node.js SDK powered 100% by the Pure-Rust `jsonbit` engine.
|
|
3
|
+
* Zero SQLite, zero external runtime dependencies.
|
|
4
|
+
*/
|
|
5
|
+
import { spawnSync } from 'node:child_process'
|
|
6
|
+
import fs from 'node:fs'
|
|
7
|
+
import os from 'node:os'
|
|
8
|
+
import path from 'node:path'
|
|
9
|
+
import { fileURLToPath } from 'node:url'
|
|
10
|
+
import type { JsonValue } from './types.ts'
|
|
11
|
+
|
|
12
|
+
export type {
|
|
13
|
+
CategoricalColumnSummary,
|
|
14
|
+
ColumnDef,
|
|
15
|
+
ForeignKeyDef,
|
|
16
|
+
JsonArray,
|
|
17
|
+
JsonObject,
|
|
18
|
+
JsonPrimitive,
|
|
19
|
+
JsonValue,
|
|
20
|
+
NumericColumnSummary,
|
|
21
|
+
OlapDataCard,
|
|
22
|
+
SqlColumnType,
|
|
23
|
+
SqlSliceResult,
|
|
24
|
+
TableDef,
|
|
25
|
+
TableOlapSummary,
|
|
26
|
+
} from './types.ts'
|
|
27
|
+
|
|
28
|
+
const __dirname = path.dirname(fileURLToPath(import.meta.url))
|
|
29
|
+
const REPO_ROOT = path.resolve(__dirname, '..')
|
|
30
|
+
|
|
31
|
+
export function resolveJsonBitBinary(): string {
|
|
32
|
+
const exe = process.platform === 'win32' ? 'jsonbit.exe' : 'jsonbit'
|
|
33
|
+
const candidates = [
|
|
34
|
+
process.env.JSONBIT_BIN,
|
|
35
|
+
path.join(REPO_ROOT, 'vendor', `${process.platform}-${process.arch}`, exe),
|
|
36
|
+
path.join(REPO_ROOT, 'target', 'release', exe),
|
|
37
|
+
].filter((x): x is string => Boolean(x))
|
|
38
|
+
|
|
39
|
+
for (const c of candidates) {
|
|
40
|
+
if (fs.existsSync(c)) return c
|
|
41
|
+
}
|
|
42
|
+
return exe
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
export class JsonBit {
|
|
46
|
+
private readonly filePath: string
|
|
47
|
+
private tempFile: string | null = null
|
|
48
|
+
private readonly binPath: string
|
|
49
|
+
|
|
50
|
+
private constructor(filePath: string, tempFile: string | null, binPath?: string) {
|
|
51
|
+
this.filePath = filePath
|
|
52
|
+
this.tempFile = tempFile
|
|
53
|
+
this.binPath = binPath ?? resolveJsonBitBinary()
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
/** Creates a `JsonBit` handle from an in-memory JSON value or a `.json`/`.jsonl`/`.jsonbit` file path. */
|
|
57
|
+
static from(dataOrPath: unknown, binPath?: string): JsonBit {
|
|
58
|
+
if (typeof dataOrPath === 'string' && fs.existsSync(dataOrPath)) {
|
|
59
|
+
return new JsonBit(dataOrPath, null, binPath)
|
|
60
|
+
}
|
|
61
|
+
const tmp = path.join(
|
|
62
|
+
os.tmpdir(),
|
|
63
|
+
`jsonbit_${process.pid}_${Date.now()}_${Math.random().toString(36).slice(2)}.json`,
|
|
64
|
+
)
|
|
65
|
+
const raw = typeof dataOrPath === 'string' ? dataOrPath : JSON.stringify(dataOrPath)
|
|
66
|
+
fs.writeFileSync(tmp, raw, 'utf8')
|
|
67
|
+
return new JsonBit(tmp, tmp, binPath)
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
private run(args: string[]): string {
|
|
71
|
+
const res = spawnSync(this.binPath, args, {
|
|
72
|
+
maxBuffer: 100 * 1024 * 1024,
|
|
73
|
+
encoding: 'utf8',
|
|
74
|
+
})
|
|
75
|
+
if (res.status !== 0) {
|
|
76
|
+
throw new Error((res.stderr || res.stdout || `jsonbit exited with ${res.status}`).trim())
|
|
77
|
+
}
|
|
78
|
+
return res.stdout.trim()
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
/** Returns the ~250-token O(1) Semantic Cube Card. */
|
|
82
|
+
getDataCard(): { text: string } {
|
|
83
|
+
return { text: this.run(['card', this.filePath]) }
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
/** Returns the 1-Turn Factorized 3NF Holistic Digest (100% of rows + pivot legend). */
|
|
87
|
+
digest(): string {
|
|
88
|
+
return this.run(['digest', this.filePath])
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
/** Evaluates a natural-language question on BitVec64 indices and returns the verified slice context. */
|
|
92
|
+
slice(question: string): { context: string } {
|
|
93
|
+
return { context: this.run(['ask', this.filePath, question]) }
|
|
94
|
+
}
|
|
95
|
+
|
|
96
|
+
/** Evaluates a natural-language question and returns only the scalar answer string. */
|
|
97
|
+
answer(question: string): string {
|
|
98
|
+
return this.run(['answer', this.filePath, question])
|
|
99
|
+
}
|
|
100
|
+
|
|
101
|
+
/** Executes a read-only SQL SELECT query directly against the Pure-Rust Bit-Packed store. */
|
|
102
|
+
sql(query: string): Array<Record<string, unknown>> {
|
|
103
|
+
return JSON.parse(this.run(['sql', this.filePath, query])) as Array<Record<string, unknown>>
|
|
104
|
+
}
|
|
105
|
+
|
|
106
|
+
/** Serializes the bit-packed 3NF tables into a compact `.jsonbit` binary snapshot on disk. */
|
|
107
|
+
pack(outPath: string): string {
|
|
108
|
+
return this.run(['pack', this.filePath, outPath])
|
|
109
|
+
}
|
|
110
|
+
|
|
111
|
+
/** Reconstructs the original nested JSON document losslessly from the 3NF bit-packed store. */
|
|
112
|
+
rehydrate(): JsonValue {
|
|
113
|
+
return JSON.parse(this.run(['rehydrate', this.filePath])) as JsonValue
|
|
114
|
+
}
|
|
115
|
+
|
|
116
|
+
close(): void {
|
|
117
|
+
if (this.tempFile && fs.existsSync(this.tempFile)) {
|
|
118
|
+
fs.unlinkSync(this.tempFile)
|
|
119
|
+
this.tempFile = null
|
|
120
|
+
}
|
|
121
|
+
}
|
|
122
|
+
}
|
|
123
|
+
|
|
124
|
+
export const JsonOlap = JsonBit
|
|
125
|
+
|
|
126
|
+
export function rehydrateFrom3NF(engine: JsonBit): JsonValue {
|
|
127
|
+
return engine.rehydrate()
|
|
128
|
+
}
|
package/src/types.ts
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
export type JsonPrimitive = string | number | boolean | null
|
|
2
|
+
export type JsonArray = JsonValue[]
|
|
3
|
+
export type JsonObject = { [key: string]: JsonValue }
|
|
4
|
+
export type JsonValue = JsonPrimitive | JsonObject | JsonArray
|
|
5
|
+
|
|
6
|
+
export type SqlColumnType = 'INTEGER' | 'REAL' | 'TEXT' | 'BOOLEAN'
|
|
7
|
+
|
|
8
|
+
export interface ColumnDef {
|
|
9
|
+
/** Flattened SQL column name, e.g. `customer_name` or `error_retryable` */
|
|
10
|
+
name: string
|
|
11
|
+
/** Original nested path segments, e.g. `['customer', 'name']` */
|
|
12
|
+
sourcePath: string[]
|
|
13
|
+
/** Inferred SQLite column affinity */
|
|
14
|
+
type: SqlColumnType
|
|
15
|
+
/** Whether any record omitted or nulled this column */
|
|
16
|
+
nullable: boolean
|
|
17
|
+
/** Whether this column stores a serialized primitive list (e.g. `read|write`) */
|
|
18
|
+
isPrimitiveArray?: boolean
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
export interface ForeignKeyDef {
|
|
22
|
+
column: string
|
|
23
|
+
parentTable: string
|
|
24
|
+
parentColumn: string
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
export interface TableDef {
|
|
28
|
+
/** Normalized SQLite table name, e.g. `orders` or `orders_items` */
|
|
29
|
+
name: string
|
|
30
|
+
/** JSON path from root, e.g. `['orders']` or `['orders', 'items']` */
|
|
31
|
+
jsonPath: string[]
|
|
32
|
+
/** Primary key column name (`id`, `orderId`, `_key`, or `_rowid`) */
|
|
33
|
+
primaryKey: string
|
|
34
|
+
/** Whether this table originated from a keyed object map (`Record<string, obj>`) */
|
|
35
|
+
isKeyedMap: boolean
|
|
36
|
+
/** Whether this table represents a singleton root/nested object */
|
|
37
|
+
isSingleton: boolean
|
|
38
|
+
/** Ordered list of columns */
|
|
39
|
+
columns: ColumnDef[]
|
|
40
|
+
/** Foreign key link to parent table if this is a 1-to-Many child table */
|
|
41
|
+
foreignKey?: ForeignKeyDef
|
|
42
|
+
/** Flattened row objects ready for SQLite insertion */
|
|
43
|
+
rows: Record<string, string | number | null>[]
|
|
44
|
+
/** Original top-level field names in discovery order (for structure queries) */
|
|
45
|
+
topLevelFields: string[]
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
export interface NumericColumnSummary {
|
|
49
|
+
column: string
|
|
50
|
+
count: number
|
|
51
|
+
nullCount: number
|
|
52
|
+
sum: number
|
|
53
|
+
avg: number
|
|
54
|
+
min: number
|
|
55
|
+
max: number
|
|
56
|
+
isFloat: boolean
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
export interface CategoricalColumnSummary {
|
|
60
|
+
column: string
|
|
61
|
+
distinctCount: number
|
|
62
|
+
/** Sorted descending by frequency: `[value, count]` */
|
|
63
|
+
frequencies: [string, number][]
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
export interface TableOlapSummary {
|
|
67
|
+
tableName: string
|
|
68
|
+
rowCount: number
|
|
69
|
+
primaryKey: string
|
|
70
|
+
foreignKey?: ForeignKeyDef
|
|
71
|
+
topLevelFields: string[]
|
|
72
|
+
columns: { name: string; type: SqlColumnType; nullable: boolean }[]
|
|
73
|
+
numericStats: NumericColumnSummary[]
|
|
74
|
+
categoricalStats: CategoricalColumnSummary[]
|
|
75
|
+
firstRow: Record<string, unknown> | undefined
|
|
76
|
+
lastRow: Record<string, unknown> | undefined
|
|
77
|
+
}
|
|
78
|
+
|
|
79
|
+
export interface OlapDataCard {
|
|
80
|
+
tables: TableOlapSummary[]
|
|
81
|
+
/** Compact LLM-ready text representation (~150-280 tokens) */
|
|
82
|
+
text: string
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
export interface SqlSliceResult {
|
|
86
|
+
/** The auto-compiled SQLite query executed against the 3NF tables */
|
|
87
|
+
sql: string
|
|
88
|
+
/** Result rows returned by SQLite */
|
|
89
|
+
rows: Record<string, unknown>[]
|
|
90
|
+
/** Compact, verified LLM prompt context combining the SQL answer + schema card */
|
|
91
|
+
context: string
|
|
92
|
+
/** Execution time in milliseconds */
|
|
93
|
+
latencyMs: number
|
|
94
|
+
}
|