@jsonbit/core 0.0.0-stage → 1.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026-PRESENT Prathik Murugan
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,317 @@
1
- # Temporary Holding Version
1
+ # jsonbit
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ [![Rust](https://img.shields.io/badge/rust-1.75%2B-orange.svg)](./Cargo.toml)
4
+ [![TypeScript](https://img.shields.io/badge/typescript-5.9-blue.svg)](./package.json)
5
+ [![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](./python/jsonbit/__init__.py)
6
+ [![MCP](https://img.shields.io/badge/MCP-2024--11--05-00ADD8.svg)](./rust/mcp.rs)
7
+ [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](./LICENSE)
8
+
9
+ `jsonbit` is an in-memory and on-disk analytical engine written in Rust that normalizes nested `.json` and `.jsonl` payloads into Third-Normal-Form (3NF) bit-packed columnar tables. It provides sub-10-microsecond bitmap query execution, compact `.jsonbit` binary snapshots, and a compact OLAP schema summary designed for Large Language Model (LLM) context windows and Model Context Protocol (MCP) agents.
10
+
11
+ ---
12
+
13
+ ## Table of Contents
14
+
15
+ - [Overview](#overview)
16
+ - [Installation & Setup](#installation--setup)
17
+ - [Agent Integrations (OpenCode, Claude Code, Codex, Cursor)](#agent-integrations)
18
+ - [SDK Usage (Python, TypeScript, Rust)](#sdk-usage)
19
+ - [CLI Reference](#cli-reference)
20
+ - [Architecture](#architecture)
21
+ - [Benchmarks](#benchmarks)
22
+ - [License](#license)
23
+
24
+ ---
25
+
26
+ ## Overview
27
+
28
+ Passing large structured datasets (API responses, telemetry logs, database exports) directly into LLM prompts as raw JSON or text tables introduces two bottlenecks:
29
+
30
+ 1. **Token Volume:** A 2,000-record dataset consumes `50,000–180,000` tokens in JSON or text-table encodings, increasing API cost and time-to-first-token (TTFT).
31
+ 2. **In-Context Arithmetic Errors:** Autoregressive attention struggles with exact row counting, multi-column filtering, and floating-point summation across thousands of lines.
32
+
33
+ `jsonbit` addresses both problems by separating **storage and computation** from **prompt context**:
34
+
35
+ - **Automatic 3NF Normalization:** Decomposes nested 1-to-1 objects, sparse optional fields, dynamic-keyed maps, and 1-to-Many child arrays into flat columnar tables linked by Compressed Sparse Row (CSR) offset arrays.
36
+ - **Bit-Packed Columnar Storage:** Compresses integers using Frame-of-Reference (`FOR`) delta packing, strings using dictionary bit-packing with inverted 64-bit bitmaps (`BitVec64`), and booleans using 1 bit per row—reducing storage size by **64.4%** compared to minified JSON while maintaining lossless round-trip reconstruction.
37
+ - **Compact OLAP Data Cards:** Emits a `~200–540` token summary containing table schemas, column types, exact numeric aggregates (`sum`, `avg`, `min`, `max`), and categorical value distributions (**99.24% fewer tokens** than full-dataset text encodings).
38
+ - **Deterministic Execution:** Evaluates natural-language slices and SQL queries (`SELECT`, `WHERE`, `GROUP BY`, `ORDER BY`, `LIMIT`, and parent-child `JOIN`s) directly on bit-packed columns and 64-bit word-aligned bitmaps in **`8.0 µs`** average latency.
39
+
40
+ ---
41
+
42
+ ## Installation & Setup
43
+
44
+ ### Prerequisites
45
+ - **Rust** (`1.75+`, [rustup.rs](https://rustup.rs/))
46
+ - *(Optional)* **Python 3.10+** for the Python SDK, or **Node.js 20+** for the TypeScript SDK
47
+
48
+ ### Build & Install Locally
49
+
50
+ ```bash
51
+ git clone https://github.com/prathikmurugan/jsonbit.git
52
+ cd jsonbit
53
+
54
+ # Build optimized release binary (./target/release/jsonbit)
55
+ cargo build --release
56
+
57
+ # Install `jsonbit` globally into your PATH (~/.cargo/bin)
58
+ cargo install --path .
59
+ ```
60
+
61
+ Verify installation:
62
+
63
+ ```bash
64
+ jsonbit --help
65
+ ```
66
+
67
+ *(Optional)* Install the Python package & binary in editable/wheel mode via `maturin` or `pip`:
68
+
69
+ ```bash
70
+ pip install .
71
+ ```
72
+
73
+ ---
74
+
75
+ ## Agent Integrations
76
+
77
+ `jsonbit` includes a built-in stdio JSON-RPC 2.0 MCP server (`jsonbit mcp`) with zero external runtime dependencies. It exposes five tools: `olap_load`, `olap_digest`, `olap_ask`, `olap_sql`, and `olap_rehydrate`.
78
+
79
+ ### OpenCode (`opencode.json`)
80
+
81
+ Add the following to `opencode.json` in your project root (or globally in `~/.config/opencode/opencode.json`):
82
+
83
+ ```json
84
+ {
85
+ "$schema": "https://opencode.ai/config.json",
86
+ "mcp": {
87
+ "jsonbit": {
88
+ "type": "local",
89
+ "command": [
90
+ "/path/to/jsonbit/target/release/jsonbit",
91
+ "mcp"
92
+ ],
93
+ "enabled": true
94
+ }
95
+ }
96
+ }
97
+ ```
98
+
99
+ ### Claude Code (`.mcp.json` or CLI)
100
+
101
+ Register via the Claude Code CLI:
102
+
103
+ ```bash
104
+ claude mcp add jsonbit -- /path/to/jsonbit/target/release/jsonbit mcp
105
+ ```
106
+
107
+ Or include in `.mcp.json`:
108
+
109
+ ```json
110
+ {
111
+ "mcpServers": {
112
+ "jsonbit": {
113
+ "command": "/path/to/jsonbit/target/release/jsonbit",
114
+ "args": ["mcp"]
115
+ }
116
+ }
117
+ }
118
+ ```
119
+
120
+ ### Cursor & Windsurf
121
+
122
+ Add the `.mcp.json` configuration above to `.cursor/mcp.json` or `~/.codeium/windsurf/mcp_config.json`.
123
+
124
+ ### OpenAI Codex CLI (`AGENTS.md`)
125
+
126
+ Codex and terminal agents automatically read [`AGENTS.md`](./AGENTS.md) in the repository root to invoke `jsonbit` via CLI or MCP when inspecting `.json`, `.jsonl`, or `.jsonbit` files.
127
+
128
+ ---
129
+
130
+ ## SDK Usage
131
+
132
+ ### Python (`python/jsonbit`)
133
+
134
+ Requires Python 3.10+ with no third-party pip packages:
135
+
136
+ ```python
137
+ from jsonbit import JsonBit # or `from python.jsonbit import JsonBit` from repo root
138
+
139
+ with JsonBit("benchmarks/data/github-repos.json") as olap:
140
+ # Generate compact OLAP schema & statistical card
141
+ card = olap.card()
142
+
143
+ # Deterministic scalar answer
144
+ ans = olap.answer("How many repositories have more than 150000 stars and more than 30000 forks?")
145
+ assert ans == "31"
146
+
147
+ # Read-only SQL query (supports GROUP BY, ORDER BY, LIMIT, and JSON dot-paths)
148
+ rows = olap.sql(
149
+ "SELECT defaultBranch, COUNT(*) AS cnt FROM records GROUP BY defaultBranch ORDER BY cnt DESC"
150
+ )
151
+
152
+ # Save compressed .jsonbit binary snapshot
153
+ olap.pack("repos.jsonbit")
154
+ ```
155
+
156
+ ### TypeScript / Node.js (`@jsonbit/core`)
157
+
158
+ ```ts
159
+ import { JsonBit, rehydrateFrom3NF } from '@jsonbit/core'
160
+
161
+ const engine = JsonBit.from(jsonData)
162
+
163
+ // Retrieve the OLAP data card
164
+ console.log(engine.getDataCard().text)
165
+
166
+ // Retrieve a verified query slice or scalar answer
167
+ console.log(engine.answer('How many shipped orders include a USB Cable?'))
168
+ console.log(engine.slice('How many shipped orders include a USB Cable?').context)
169
+
170
+ // Execute SQL (automatically resolves JSON dot-paths such as customer.name -> customer_name)
171
+ const rows = engine.sql(`SELECT orderId, customer.name, total FROM orders WHERE total > 500`)
172
+
173
+ // Losslessly reconstruct original nested JSON
174
+ const original = rehydrateFrom3NF(engine)
175
+ engine.close()
176
+ ```
177
+
178
+ ### Rust Library (`jsonbit`)
179
+
180
+ ```rust
181
+ use jsonbit::{execute_sql, JsonBitEngine};
182
+ use serde_json::Value;
183
+
184
+ let data: Value = serde_json::from_str(raw_json)?;
185
+ let engine = JsonBitEngine::from_json(&data);
186
+
187
+ let card = engine.data_card();
188
+ let slice = engine.query_slice("How many active employees work in Sales?");
189
+ let rows = execute_sql(&engine, "SELECT department, COUNT(*) AS cnt FROM employees GROUP BY department")?;
190
+ let restored = engine.rehydrate();
191
+ ```
192
+
193
+ ---
194
+
195
+ ## CLI Reference
196
+
197
+ | Command | Arguments | Description |
198
+ | :--- | :--- | :--- |
199
+ | `jsonbit card` | `<file.json\|.jsonl\|.jsonbit>` | Prints the $O(1)$ 3NF OLAP Cube Card (`~250 tokens`) and storage compression statistics. |
200
+ | `jsonbit digest` | `<file.json\|.jsonl\|.jsonbit>` | Prints the 1-Turn Factorized 3NF Holistic Digest (100% of rows + `BitVec64` Pivot Legend, `-41%` to `-72%` smaller than `TOON`). |
201
+ | `jsonbit ask` | `<file> "<question>"` | Evaluates a question against the bitmap index and prints the verified context slice. |
202
+ | `jsonbit answer` | `<file> "<question>"` | Evaluates a question and prints only the scalar answer value. |
203
+ | `jsonbit sql` | `<file> "<SELECT ...>"` | Executes a read-only SQL query against the 3NF columnar store (`WHERE`, `IN`, `BETWEEN`, `LIKE`, `GROUP BY`, `ORDER BY`, `LIMIT`, `OFFSET`, CSR `JOIN`). |
204
+ | `jsonbit pack` | `<input.json> <output.jsonbit>` | Serializes the bit-packed 3NF tables into a binary `.jsonbit` snapshot. |
205
+ | `jsonbit rehydrate` | `<file.json\|.jsonl\|.jsonbit>` | Reconstructs and prints the original nested JSON document. |
206
+ | `jsonbit mcp` | *(none)* | Starts the concurrent stdio JSON-RPC 2.0 MCP server (`olap_load`, `olap_digest`, `olap_ask`, `olap_sql`, `olap_rehydrate`). |
207
+
208
+ ---
209
+
210
+ ## Architecture
211
+
212
+ ```mermaid
213
+ flowchart TD
214
+ Input["Nested JSON / JSONL Input"] --> Normalizer["3NF Normalizer & CSR Indexer (rust/normalizer.rs)"]
215
+ Normalizer --> ColStore["Bit-Packed Columnar Store (rust/column.rs)"]
216
+ ColStore --> Bitmap["BitVec64 Bitmap Engine (rust/bitmap.rs)"]
217
+ Bitmap --> Card["OLAP Data Card (rust/engine.rs)"]
218
+ Bitmap --> SqlExec["SQL & Predicate Executor (rust/query_exec.rs)"]
219
+ ColStore --> Snapshot[".jsonbit Binary Snapshot (rust/snapshot.rs)"]
220
+ ColStore --> Rehydrate["Lossless 3NF Rehydrator (rust/normalizer.rs)"]
221
+ ```
222
+
223
+ ### Storage Layout (`rust/column.rs` & `rust/normalizer.rs`)
224
+
225
+ 1. **`ForIntColumn` (Frame-of-Reference Integers):** Computes `[min_val, max_val]` per column and packs unsigned deltas `val - min_val` into `ceil(log2(max_val - min_val + 1))` bits per row inside a `Vec<u64>` buffer. Column headers store `sum`, `min`, `max`, and `non_null_count` for `O(1)` unfiltered aggregations.
226
+ 2. **`DictStringColumn` (Dictionary + Inverted Bitmaps):** Interns unique strings into a string arena, bit-packs dictionary IDs into `ceil(log2(|D|))` bits per row, and builds a 64-bit word-aligned `BitVec64` inverted index per category (`|D| <= 64`) for `O(1)` equality lookups and `POPCNT` frequency histograms.
227
+ 3. **`BoolBitmapColumn` (1-Bit Booleans):** Packs 64 boolean values into each `u64` word alongside a 1-bit validity mask.
228
+ 4. **CSR 1-to-Many Join Index:** Child arrays (e.g., `orders[].items[]`) are normalized into child tables with a contiguous `csr_offsets: Vec<u32>` array mapping each parent row `i` to its child slice `csr_offsets[i]..csr_offsets[i + 1]`.
229
+
230
+ ---
231
+
232
+ ## Benchmarks
233
+
234
+ All benchmarks execute directly against the compiled Pure-Rust `jsonbit` binary (`target/release/jsonbit`).
235
+
236
+ ### 1. Live `gemini-3.8-flash` Evaluation Across 10 Real-World Workloads (27 Questions)
237
+
238
+ Evaluates `Raw JSON`, `Minified JSON`, `YAML`, `TOON` (`@toon-format/toon`), and `jsonbit` (Pure-Rust `olap_load` + `olap_sql`) across 10 production workloads (`W1`–`W10`), including flat tables, 1-to-Many e-commerce orders, sparse `.jsonl` OpenTelemetry logs with free-text messages, singleton Kubernetes configs, IAM primitive arrays, wide 30-column tables, and multi-byte UTF-8 financial ledgers:
239
+
240
+ ![Live Gemini 3.8 Flash Accuracy and Token Efficiency](./benchmarks/charts/gemini-accuracy-and-tokens.svg)
241
+
242
+ | Format / Tool | Mode | Live `gemini-3.8-flash` Accuracy | Total Prompt Tokens | Token Reduction vs. Raw JSON | Token Reduction vs. `TOON` |
243
+ | :--- | :--- | ---: | ---: | ---: | ---: |
244
+ | **Raw JSON (Pretty)** | 1-Turn Full Text | 55.6% (15/27) | 375,713 | — | — |
245
+ | **Minified JSON** | 1-Turn Full Text | 48.1% (13/27) | 237,891 | -36.7% | — |
246
+ | **YAML** | 1-Turn Full Text | 59.3% (16/27) | 293,690 | -21.8% | — |
247
+ | **`TOON` v4 (`@toon-format/toon`)** | 1-Turn Full Text | 59.3% (16/27) | 169,627 | -54.9% | — |
248
+ | **`jsonbit` Holistic Digest (`olap_digest`)** | **1-Turn (100% Rows + Pivot Legend)** | **96.3% (26/27)** | **101,302** | **-73.0%** | **-40.3%** |
249
+ | **`jsonbit` Cube Card + SQL (`olap_load` + `olap_sql`)** | **Agentic $O(1)$ Card + Rust SQL** | **100.0% (27/27)** | **11,729** | **-96.9%** | **-93.1%** |
250
+
251
+ #### Qualitative & Holistic Executive Summary Verification (`1-Turn: TOON vs. jsonbit digest`)
252
+
253
+ For qualitative tasks where the LLM must read all records in a single turn and write a multi-fact executive summary (naming all departments, products, microservice failure messages, and international counterparties across `W3`, `W4`, `W5`, and `W10`):
254
+
255
+ | Holistic Synthesis Task | `TOON` v4 Facts Preserved | `jsonbit digest` Facts Preserved | `TOON` Tokens | `jsonbit digest` Tokens | Token Savings vs. `TOON` |
256
+ | :--- | ---: | ---: | ---: | ---: | ---: |
257
+ | `H1` — HR Directory Executive Summary | 8 / 9 | **9 / 9 (100%)** | 4,881 | **4,372** | **-10.4%** |
258
+ | `H2` — E-Commerce Catalog & Orders Summary | 8 / 8 | **8 / 8 (100%)** | 15,405 | **6,933** | **-55.0%** |
259
+ | `H3` — Production Incident Log Triage Report | 9 / 9 | **9 / 9 (100%)** | 13,665 | **4,169** | **-69.5%** |
260
+ | `H4` — International Settlement Ledger Summary | 2 / 6 | **6 / 6 (100%)** | 3,781 | **2,403** | **-36.4%** |
261
+ | **Total Across All Qualitative Summaries** | **27 / 32 (84.4%)** | **32 / 32 (100.0%)** | **37,732** | **17,877** | **-52.6%** |
262
+
263
+ ### 2. Token Scaling Curve (`4` $\rightarrow$ `25,000` Nested Orders)
264
+
265
+ While linear text encodings (`Raw JSON`, `Minified JSON`, `YAML`, `TOON`) scale at $O(N)$ with row count and exceed LLM context windows at scale, `jsonbit` maintains an $O(1)$ prompt footprint (`~67–473` tokens):
266
+
267
+ ![LLM Context Tokens vs Dataset Scale](./benchmarks/charts/token-scaling.svg)
268
+
269
+ | Dataset Scale (Nested Orders) | Raw JSON (Tokens) | YAML (Tokens) | `TOON` v4 (Tokens) | `jsonbit` Rust Card (Tokens) | Rust Ingest Time |
270
+ | :--- | ---: | ---: | ---: | ---: | ---: |
271
+ | **4 rows** *(Tiny API Payload)* | 166 | 122 | 71 | **67** | 0.04 ms |
272
+ | **50 rows** *(~150 line items)* | 7,780 | 5,801 | 4,783 | **447** | 0.52 ms |
273
+ | **500 rows** *(~1,500 line items)* | 78,115 | 58,244 | 47,983 | **449** | 4.72 ms |
274
+ | **5,000 rows** *(~15,000 line items)* | 783,443 | 366,356 | 293,972 | **472** | 58.97 ms |
275
+ | **25,000 rows** *(~75,000 line items)* | 3,917,366 | 1,831,843 | 1,469,911 | **473** | 334.42 ms |
276
+
277
+ ### 3. Memory Footprint & Query Speed (`Pure-Rust jsonbit` vs. `jq` CLI)
278
+
279
+ ![Pure-Rust Memory Footprint and Query Speed](./benchmarks/charts/storage-and-query-speed.svg)
280
+
281
+ | Workload | Category | Minified JSON | Rust 3NF Bit-Packed | Pretty JSON (Tok) | `TOON` (100% Rows) | `jsonbit digest` (100% Rows) | `jsonbit card` ($O(1)$) | 3NF Lossless |
282
+ | :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | :---: |
283
+ | `W1-tiny-api-response` | Tiny Payload (<500B) | 0.3 KB | **0.1 KB** | 166 | 71 | **67 (-5.6%)** | **67** | PASS |
284
+ | `W2-singleton-k8s-config` | Singleton Config Tree | 0.3 KB | **0.2 KB** | 167 | 131 | **110 (-16.0%)** | **110** | PASS |
285
+ | `W3-hr-payroll-directory` | Tabular (200 rows) | 28.4 KB | **4.5 KB** | 14,064 | 4,847 | **4,145 (-14.5%)** | **272** | PASS |
286
+ | `W4-ecommerce-orders-1toN` | 1-to-Many (150 orders) | 48.3 KB | **4.5 KB** | 25,466 | 15,372 | **6,878 (-55.3%)** | **472** | PASS |
287
+ | `W5-otel-production-logs` | Sparse `.jsonl` + Free-Text | 44.4 KB | **4.7 KB** | 17,439 | 13,632 | **4,123 (-69.8%)** | **226** | PASS |
288
+ | `W6-github-repos-metrics` | High-Card + Ranges | 28.6 KB | **3.4 KB** | 14,829 | 5,364 | **4,285 (-20.1%)** | **309** | PASS |
289
+ | `W7-stripe-dispute-webhooks` | Financial Webhooks | 28.9 KB | **3.1 KB** | 14,333 | 4,104 | **3,906 (-4.8%)** | **250** | PASS |
290
+ | `W8-iam-primitive-arrays` | Primitive Arrays | 14.4 KB | **1.6 KB** | 7,185 | 4,810 | **1,247 (-74.1%)** | **164** | PASS |
291
+ | `W9-wide-enterprise-crm` | Wide Table (30 cols) | 68.6 KB | **5.7 KB** | 40,779 | 9,806 | **9,516 (-3.0%)** | **1,136** | PASS |
292
+ | `W10-unicode-financial-ledger` | UTF-8, Nulls & Zeroes | 18.4 KB | **2.9 KB** | 9,550 | 3,750 | **2,359 (-37.1%)** | **243** | PASS |
293
+ | **Total (10 Workloads)** | **All Categories** | **280.6 KB** | **30.6 KB (-89.1%)** | **143,978** | **61,887** | **36,636 (-40.8%)** | **3,249 (-94.7%)** | **10/10 PASS** |
294
+
295
+ - **Query Latency (`Pure-Rust BitVec64 SQL` vs. `jq` C CLI):**
296
+ - `jq` CLI (`stedolan/jq`): `6,545.3 µs` average per query
297
+ - `jsonbit` Pure-Rust SQL (`execute_sql`): **`6.8 µs` average per query (`963x` faster)**
298
+ - `jsonbit` Pure-Rust Bitmap Slice (`query_slice`): **`19.9 µs` average per query (`329x` faster)**
299
+ - **Standard 8-Dataset Benchmark Suite:**
300
+ - **Storage Compression:** `1,178,283` bytes (Minified JSON) $\rightarrow$ **`419,661` bytes (`-64.4%` RAM/storage reduction)**, `8/8` (`100%`) lossless 3NF rehydration.
301
+ - **Token Reduction:** `334,046` tokens (`TOON` v4) $\rightarrow$ **`2,536` tokens (`-99.24%`)**.
302
+ - **239-Question Evaluation Accuracy:** `99.16% (237/239)` strict match at **`8.0 µs`** average query latency.
303
+
304
+ ### 4. Autonomous Coding Agent Head-to-Head (`OpenCode v1.19.5`)
305
+
306
+ Evaluated in isolated directories with zero prompt hints and zero `AGENTS.md`/`CLAUDE.md` files (`opencode/big-pickle`) across both standard (bash-enabled) and sandboxed (`bash: "deny"`) agent environments:
307
+
308
+ | Agent Mode | `jsonbit` MCP (`opencode.json` only) | `TOON` v4 (`toon_encode` / CLI) | Raw JSON |
309
+ | :--- | :--- | :--- | :--- |
310
+ | **Standard Agent (`6` Scenarios, `18` Runs)** | **6/6 (`88.1s`) · `0` scripts** | 6/6 (`142.8s`) · `awk`/Python parsers | 6/6 (`93.7s`) · Python scripts |
311
+ | **Sandboxed Read-Only (`bash: "deny"`, `8` Scenarios)** | **8/8 (`77.7s`)** | 7/8 (`333.8s`) · failed `500`-order sum | 7/8 (`345.4s`) · failed `500`-order sum |
312
+
313
+ ---
314
+
315
+ ## License
316
+
317
+ [MIT License](./LICENSE).
package/bin/jsonbit.js ADDED
@@ -0,0 +1,45 @@
1
+ #!/usr/bin/env node
2
+ /**
3
+ * Native Pure-Rust `jsonbit` CLI & MCP launcher for npm / npx.
4
+ * Resolves the compiled `jsonbit` binary from:
5
+ * 1. `JSONBIT_BIN` environment variable
6
+ * 2. Local build (`./target/release/jsonbit` or `vendor/<platform>/jsonbit`)
7
+ * 3. Global `jsonbit` on `PATH`
8
+ */
9
+ import { spawnSync } from 'node:child_process'
10
+ import fs from 'node:fs'
11
+ import path from 'node:path'
12
+ import { fileURLToPath } from 'node:url'
13
+
14
+ const __dirname = path.dirname(fileURLToPath(import.meta.url))
15
+ const pkgRoot = path.resolve(__dirname, '..')
16
+ const exe = process.platform === 'win32' ? 'jsonbit.exe' : 'jsonbit'
17
+
18
+ const candidates = [
19
+ process.env.JSONBIT_BIN,
20
+ path.join(pkgRoot, 'vendor', `${process.platform}-${process.arch}`, exe),
21
+ path.join(pkgRoot, 'target', 'release', exe),
22
+ exe,
23
+ ].filter(Boolean)
24
+
25
+ let resolvedBin = exe
26
+ for (const c of candidates) {
27
+ if (c === exe || fs.existsSync(c)) {
28
+ resolvedBin = c
29
+ break
30
+ }
31
+ }
32
+
33
+ const res = spawnSync(resolvedBin, process.argv.slice(2), {
34
+ stdio: 'inherit',
35
+ })
36
+
37
+ if (res.error) {
38
+ console.error(
39
+ `[jsonbit] Could not execute native binary (${resolvedBin}).\n` +
40
+ `Build from source with \`cargo build --release\` or install via \`cargo install jsonbit\`.`,
41
+ )
42
+ process.exit(1)
43
+ }
44
+
45
+ process.exit(res.status ?? 0)
package/package.json CHANGED
@@ -1,6 +1,54 @@
1
1
  {
2
2
  "name": "@jsonbit/core",
3
- "version": "0.0.0-stage",
4
- "stub": true,
5
- "description": "Temporary package placeholder for staged publishing"
6
- }
3
+ "version": "1.1.1",
4
+ "description": "jsonbit — Pure-Rust Auto-3NF Columnar Bit-Packed & BitVec64 Engine for LLMs and AI Agents",
5
+ "type": "module",
6
+ "main": "./src/index.ts",
7
+ "types": "./src/index.ts",
8
+ "bin": {
9
+ "jsonbit": "./bin/jsonbit.js"
10
+ },
11
+ "files": [
12
+ "bin/",
13
+ "scripts/",
14
+ "src/",
15
+ "README.md",
16
+ "LICENSE"
17
+ ],
18
+ "scripts": {
19
+ "postinstall": "node ./scripts/install-binary.mjs",
20
+ "test": "NODE_NO_WARNINGS=1 vitest run",
21
+ "bench": "NODE_NO_WARNINGS=1 tsx benchmarks/run-full-rust-benchmark.ts --reuse-gemini",
22
+ "cli": "node ./bin/jsonbit.js"
23
+ },
24
+ "keywords": [
25
+ "llm",
26
+ "olap",
27
+ "json",
28
+ "jsonbit",
29
+ "rust",
30
+ "3nf",
31
+ "agents",
32
+ "mcp",
33
+ "tokens"
34
+ ],
35
+ "license": "MIT",
36
+ "author": "Prathik Murugan",
37
+ "homepage": "https://github.com/prathikmurugan/jsonbit#readme",
38
+ "repository": {
39
+ "type": "git",
40
+ "url": "https://github.com/prathikmurugan/jsonbit.git"
41
+ },
42
+ "bugs": {
43
+ "url": "https://github.com/prathikmurugan/jsonbit/issues"
44
+ },
45
+ "devDependencies": {
46
+ "@faker-js/faker": "^10.6.0",
47
+ "@toon-format/toon": "^4.4.0",
48
+ "@types/node": "^25.8.0",
49
+ "gpt-tokenizer": "^4.1.2",
50
+ "tsx": "^4.21.0",
51
+ "typescript": "^5.9.3",
52
+ "vitest": "^3.2.4"
53
+ }
54
+ }
@@ -0,0 +1,86 @@
1
+ #!/usr/bin/env node
2
+ /**
3
+ * Downloads the pre-built native `jsonbit` binary from GitHub Releases into
4
+ * `vendor/<platform>-<arch>/jsonbit` on `npm install` when no local build exists.
5
+ */
6
+ import { execFileSync } from 'node:child_process'
7
+ import fs from 'node:fs'
8
+ import https from 'node:https'
9
+ import os from 'node:os'
10
+ import path from 'node:path'
11
+ import { fileURLToPath } from 'node:url'
12
+
13
+ const __dirname = path.dirname(fileURLToPath(import.meta.url))
14
+ const pkgRoot = path.resolve(__dirname, '..')
15
+ const pkg = JSON.parse(fs.readFileSync(path.join(pkgRoot, 'package.json'), 'utf8'))
16
+
17
+ const platform = process.platform
18
+ const arch = process.arch
19
+ const exe = platform === 'win32' ? 'jsonbit.exe' : 'jsonbit'
20
+
21
+ const localTarget = path.join(pkgRoot, 'target', 'release', exe)
22
+ const vendorDir = path.join(pkgRoot, 'vendor', `${platform}-${arch}`)
23
+ const vendorBin = path.join(vendorDir, exe)
24
+
25
+ if (process.env.JSONBIT_SKIP_DOWNLOAD === '1' || fs.existsSync(localTarget) || fs.existsSync(vendorBin)) {
26
+ process.exit(0)
27
+ }
28
+
29
+ const SUPPORTED = new Set(['linux-x64', 'linux-arm64', 'darwin-arm64', 'darwin-x64', 'win32-x64'])
30
+ const key = `${platform}-${arch}`
31
+ if (!SUPPORTED.has(key)) {
32
+ console.warn(`[jsonbit] No pre-built binary for ${key}. Install via \`cargo install jsonbit\`.`)
33
+ process.exit(0)
34
+ }
35
+
36
+ const ext = platform === 'win32' ? 'zip' : 'tar.gz'
37
+ const url = `https://github.com/prathikmurugan/jsonbit/releases/download/v${pkg.version}/jsonbit-${key}.${ext}`
38
+
39
+ function download(targetUrl, destPath, redirects = 0) {
40
+ return new Promise((resolve, reject) => {
41
+ if (redirects > 5) return reject(new Error('Too many redirects'))
42
+ https
43
+ .get(targetUrl, { headers: { 'User-Agent': `jsonbit/${pkg.version}` } }, res => {
44
+ if (res.statusCode && res.statusCode >= 300 && res.statusCode < 400 && res.headers.location) {
45
+ res.resume()
46
+ return resolve(download(res.headers.location, destPath, redirects + 1))
47
+ }
48
+ if (res.statusCode !== 200) {
49
+ res.resume()
50
+ return reject(new Error(`HTTP ${res.statusCode} fetching ${targetUrl}`))
51
+ }
52
+ const file = fs.createWriteStream(destPath)
53
+ res.pipe(file)
54
+ file.on('finish', () => file.close(() => resolve()))
55
+ file.on('error', reject)
56
+ })
57
+ .on('error', reject)
58
+ })
59
+ }
60
+
61
+ async function main() {
62
+ const tmpArchive = path.join(os.tmpdir(), `jsonbit-${pkg.version}-${key}.${ext}`)
63
+ try {
64
+ await download(url, tmpArchive)
65
+ fs.mkdirSync(vendorDir, { recursive: true })
66
+ if (platform === 'win32') {
67
+ execFileSync('powershell', [
68
+ '-NoProfile',
69
+ '-Command',
70
+ `Expand-Archive -Force -Path '${tmpArchive}' -DestinationPath '${vendorDir}'`,
71
+ ])
72
+ } else {
73
+ execFileSync('tar', ['-xzf', tmpArchive, '-C', vendorDir])
74
+ fs.chmodSync(vendorBin, 0o755)
75
+ }
76
+ } catch (err) {
77
+ console.warn(
78
+ `[jsonbit] Could not download pre-built binary (${err.message}).\n` +
79
+ `Build locally with \`cargo build --release\` or install via \`cargo install jsonbit\`.`,
80
+ )
81
+ } finally {
82
+ if (fs.existsSync(tmpArchive)) fs.unlinkSync(tmpArchive)
83
+ }
84
+ }
85
+
86
+ main()
package/src/index.ts ADDED
@@ -0,0 +1,128 @@
1
+ /**
2
+ * @jsonbit/core — TypeScript / Node.js SDK powered 100% by the Pure-Rust `jsonbit` engine.
3
+ * Zero SQLite, zero external runtime dependencies.
4
+ */
5
+ import { spawnSync } from 'node:child_process'
6
+ import fs from 'node:fs'
7
+ import os from 'node:os'
8
+ import path from 'node:path'
9
+ import { fileURLToPath } from 'node:url'
10
+ import type { JsonValue } from './types.ts'
11
+
12
+ export type {
13
+ CategoricalColumnSummary,
14
+ ColumnDef,
15
+ ForeignKeyDef,
16
+ JsonArray,
17
+ JsonObject,
18
+ JsonPrimitive,
19
+ JsonValue,
20
+ NumericColumnSummary,
21
+ OlapDataCard,
22
+ SqlColumnType,
23
+ SqlSliceResult,
24
+ TableDef,
25
+ TableOlapSummary,
26
+ } from './types.ts'
27
+
28
+ const __dirname = path.dirname(fileURLToPath(import.meta.url))
29
+ const REPO_ROOT = path.resolve(__dirname, '..')
30
+
31
+ export function resolveJsonBitBinary(): string {
32
+ const exe = process.platform === 'win32' ? 'jsonbit.exe' : 'jsonbit'
33
+ const candidates = [
34
+ process.env.JSONBIT_BIN,
35
+ path.join(REPO_ROOT, 'vendor', `${process.platform}-${process.arch}`, exe),
36
+ path.join(REPO_ROOT, 'target', 'release', exe),
37
+ ].filter((x): x is string => Boolean(x))
38
+
39
+ for (const c of candidates) {
40
+ if (fs.existsSync(c)) return c
41
+ }
42
+ return exe
43
+ }
44
+
45
+ export class JsonBit {
46
+ private readonly filePath: string
47
+ private tempFile: string | null = null
48
+ private readonly binPath: string
49
+
50
+ private constructor(filePath: string, tempFile: string | null, binPath?: string) {
51
+ this.filePath = filePath
52
+ this.tempFile = tempFile
53
+ this.binPath = binPath ?? resolveJsonBitBinary()
54
+ }
55
+
56
+ /** Creates a `JsonBit` handle from an in-memory JSON value or a `.json`/`.jsonl`/`.jsonbit` file path. */
57
+ static from(dataOrPath: unknown, binPath?: string): JsonBit {
58
+ if (typeof dataOrPath === 'string' && fs.existsSync(dataOrPath)) {
59
+ return new JsonBit(dataOrPath, null, binPath)
60
+ }
61
+ const tmp = path.join(
62
+ os.tmpdir(),
63
+ `jsonbit_${process.pid}_${Date.now()}_${Math.random().toString(36).slice(2)}.json`,
64
+ )
65
+ const raw = typeof dataOrPath === 'string' ? dataOrPath : JSON.stringify(dataOrPath)
66
+ fs.writeFileSync(tmp, raw, 'utf8')
67
+ return new JsonBit(tmp, tmp, binPath)
68
+ }
69
+
70
+ private run(args: string[]): string {
71
+ const res = spawnSync(this.binPath, args, {
72
+ maxBuffer: 100 * 1024 * 1024,
73
+ encoding: 'utf8',
74
+ })
75
+ if (res.status !== 0) {
76
+ throw new Error((res.stderr || res.stdout || `jsonbit exited with ${res.status}`).trim())
77
+ }
78
+ return res.stdout.trim()
79
+ }
80
+
81
+ /** Returns the ~250-token O(1) Semantic Cube Card. */
82
+ getDataCard(): { text: string } {
83
+ return { text: this.run(['card', this.filePath]) }
84
+ }
85
+
86
+ /** Returns the 1-Turn Factorized 3NF Holistic Digest (100% of rows + pivot legend). */
87
+ digest(): string {
88
+ return this.run(['digest', this.filePath])
89
+ }
90
+
91
+ /** Evaluates a natural-language question on BitVec64 indices and returns the verified slice context. */
92
+ slice(question: string): { context: string } {
93
+ return { context: this.run(['ask', this.filePath, question]) }
94
+ }
95
+
96
+ /** Evaluates a natural-language question and returns only the scalar answer string. */
97
+ answer(question: string): string {
98
+ return this.run(['answer', this.filePath, question])
99
+ }
100
+
101
+ /** Executes a read-only SQL SELECT query directly against the Pure-Rust Bit-Packed store. */
102
+ sql(query: string): Array<Record<string, unknown>> {
103
+ return JSON.parse(this.run(['sql', this.filePath, query])) as Array<Record<string, unknown>>
104
+ }
105
+
106
+ /** Serializes the bit-packed 3NF tables into a compact `.jsonbit` binary snapshot on disk. */
107
+ pack(outPath: string): string {
108
+ return this.run(['pack', this.filePath, outPath])
109
+ }
110
+
111
+ /** Reconstructs the original nested JSON document losslessly from the 3NF bit-packed store. */
112
+ rehydrate(): JsonValue {
113
+ return JSON.parse(this.run(['rehydrate', this.filePath])) as JsonValue
114
+ }
115
+
116
+ close(): void {
117
+ if (this.tempFile && fs.existsSync(this.tempFile)) {
118
+ fs.unlinkSync(this.tempFile)
119
+ this.tempFile = null
120
+ }
121
+ }
122
+ }
123
+
124
+ export const JsonOlap = JsonBit
125
+
126
+ export function rehydrateFrom3NF(engine: JsonBit): JsonValue {
127
+ return engine.rehydrate()
128
+ }
package/src/types.ts ADDED
@@ -0,0 +1,94 @@
1
+ export type JsonPrimitive = string | number | boolean | null
2
+ export type JsonArray = JsonValue[]
3
+ export type JsonObject = { [key: string]: JsonValue }
4
+ export type JsonValue = JsonPrimitive | JsonObject | JsonArray
5
+
6
+ export type SqlColumnType = 'INTEGER' | 'REAL' | 'TEXT' | 'BOOLEAN'
7
+
8
+ export interface ColumnDef {
9
+ /** Flattened SQL column name, e.g. `customer_name` or `error_retryable` */
10
+ name: string
11
+ /** Original nested path segments, e.g. `['customer', 'name']` */
12
+ sourcePath: string[]
13
+ /** Inferred SQLite column affinity */
14
+ type: SqlColumnType
15
+ /** Whether any record omitted or nulled this column */
16
+ nullable: boolean
17
+ /** Whether this column stores a serialized primitive list (e.g. `read|write`) */
18
+ isPrimitiveArray?: boolean
19
+ }
20
+
21
+ export interface ForeignKeyDef {
22
+ column: string
23
+ parentTable: string
24
+ parentColumn: string
25
+ }
26
+
27
+ export interface TableDef {
28
+ /** Normalized SQLite table name, e.g. `orders` or `orders_items` */
29
+ name: string
30
+ /** JSON path from root, e.g. `['orders']` or `['orders', 'items']` */
31
+ jsonPath: string[]
32
+ /** Primary key column name (`id`, `orderId`, `_key`, or `_rowid`) */
33
+ primaryKey: string
34
+ /** Whether this table originated from a keyed object map (`Record<string, obj>`) */
35
+ isKeyedMap: boolean
36
+ /** Whether this table represents a singleton root/nested object */
37
+ isSingleton: boolean
38
+ /** Ordered list of columns */
39
+ columns: ColumnDef[]
40
+ /** Foreign key link to parent table if this is a 1-to-Many child table */
41
+ foreignKey?: ForeignKeyDef
42
+ /** Flattened row objects ready for SQLite insertion */
43
+ rows: Record<string, string | number | null>[]
44
+ /** Original top-level field names in discovery order (for structure queries) */
45
+ topLevelFields: string[]
46
+ }
47
+
48
+ export interface NumericColumnSummary {
49
+ column: string
50
+ count: number
51
+ nullCount: number
52
+ sum: number
53
+ avg: number
54
+ min: number
55
+ max: number
56
+ isFloat: boolean
57
+ }
58
+
59
+ export interface CategoricalColumnSummary {
60
+ column: string
61
+ distinctCount: number
62
+ /** Sorted descending by frequency: `[value, count]` */
63
+ frequencies: [string, number][]
64
+ }
65
+
66
+ export interface TableOlapSummary {
67
+ tableName: string
68
+ rowCount: number
69
+ primaryKey: string
70
+ foreignKey?: ForeignKeyDef
71
+ topLevelFields: string[]
72
+ columns: { name: string; type: SqlColumnType; nullable: boolean }[]
73
+ numericStats: NumericColumnSummary[]
74
+ categoricalStats: CategoricalColumnSummary[]
75
+ firstRow: Record<string, unknown> | undefined
76
+ lastRow: Record<string, unknown> | undefined
77
+ }
78
+
79
+ export interface OlapDataCard {
80
+ tables: TableOlapSummary[]
81
+ /** Compact LLM-ready text representation (~150-280 tokens) */
82
+ text: string
83
+ }
84
+
85
+ export interface SqlSliceResult {
86
+ /** The auto-compiled SQLite query executed against the 3NF tables */
87
+ sql: string
88
+ /** Result rows returned by SQLite */
89
+ rows: Record<string, unknown>[]
90
+ /** Compact, verified LLM prompt context combining the SQL answer + schema card */
91
+ context: string
92
+ /** Execution time in milliseconds */
93
+ latencyMs: number
94
+ }