vectorjson 0.1.0 → 0.2.1

package/README.md

@@ -1,6 +1,9 @@
 # VectorJSON
 
 [![CI](https://github.com/teamchong/vectorjson/actions/workflows/ci.yml/badge.svg)](https://github.com/teamchong/vectorjson/actions/workflows/ci.yml)
+[![npm](https://img.shields.io/npm/v/vectorjson)](https://www.npmjs.com/package/vectorjson)
+[![gzip size](https://img.shields.io/badge/gzip-~47kB-blue)](https://www.npmjs.com/package/vectorjson)
+[![license](https://img.shields.io/npm/l/vectorjson)](https://github.com/teamchong/vectorjson/blob/main/LICENSE)
 
 O(n) streaming JSON parser for LLM tool calls, built on WASM SIMD. Agents act faster with field-level streaming, detect wrong outputs early to abort and save tokens, and offload parsing to Workers with transferable ArrayBuffers.
 
@@ -27,24 +30,26 @@ for await (const chunk of stream) {
 }
 ```
 
-A 50KB tool call streamed in ~12-char chunks means ~4,000 full re-parses — O(n²). At 100KB, Vercel AI SDK spends 4.1 seconds just parsing. Anthropic SDK spends 9.3 seconds.
+A 50KB tool call streamed in ~12-char chunks means ~4,000 full re-parses — O(n²). At 100KB, Vercel AI SDK spends 6.1 seconds just parsing. Anthropic SDK spends 13.4 seconds.
 
 ## Quick Start
 
-Drop-in replacement for your SDK's partial JSON parser:
+Zero-config — just import and use. No `init()`, no WASM setup:
 
 ```js
-import { init } from "vectorjson";
-const vj = await init();
+import { parse, createParser, createEventParser } from "vectorjson";
 
-// Before (JS parser — what your SDK does today):
-for await (const chunk of stream) {
-  buffer += chunk;
-  result = parsePartialJson(buffer); // re-parses entire buffer every time
-}
+// One-shot parse
+const result = parse('{"tool":"file_edit","path":"app.ts"}');
+result.value.tool;  // "file_edit" — lazy Proxy over WASM tape
+```
+
+**Streaming** — O(n) incremental parsing, feed chunks, get a live object:
+
+```js
+import { createParser } from "vectorjson";
 
-// After (VectorJSON — O(n) live document builder):
-const parser = vj.createParser();
+const parser = createParser();
 for await (const chunk of stream) {
   parser.feed(chunk);
   result = parser.getValue();        // O(1) — returns live object
@@ -57,7 +62,7 @@ parser.destroy();
 **Or skip intermediate access entirely** — if you only need the final value:
 
 ```js
-const parser = vj.createParser();
+const parser = createParser();
 for await (const chunk of stream) {
   const s = parser.feed(chunk);      // O(1) — appends bytes to WASM buffer
   if (s === "complete") break;
@@ -69,7 +74,9 @@ parser.destroy();
 **Event-driven** — react to fields as they arrive, O(n) total, no re-parsing:
 
 ```js
-const parser = vj.createEventParser();
+import { createEventParser } from "vectorjson";
+
+const parser = createEventParser();
 
 parser.on('tool', (e) => showToolUI(e.value));             // fires immediately
 parser.onDelta('code', (e) => editor.append(e.value));     // streams char-by-char
@@ -85,7 +92,7 @@ parser.destroy();
 
 ```js
 const abort = new AbortController();
-const parser = vj.createEventParser();
+const parser = createEventParser();
 
 parser.on('name', (e) => {
   if (e.value !== 'str_replace_editor') {
@@ -111,7 +118,7 @@ const buf = parser.getRawBuffer();
 postMessage(buf, [buf]); // O(1) transfer — moves pointer, no copy
 
 // On Main thread:
-const result = vj.parse(new Uint8Array(buf)); // lazy Proxy
+const result = parse(new Uint8Array(buf)); // lazy Proxy
 result.value.name; // only materializes what you touch
 ```
 
@@ -125,18 +132,22 @@ Apple-to-apple: both sides produce a materialized partial object on every chunk.
 
 | Payload | Product | Original | + VectorJSON | Speedup |
 |---------|---------|----------|-------------|---------|
-| 1 KB | Vercel AI SDK | 4.2 ms | 162 µs | **26×** |
-| | Anthropic SDK | 1.6 ms | 162 µs | **10×** |
-| | TanStack AI | 1.8 ms | 162 µs | **11×** |
-| | OpenClaw | 2.0 ms | 162 µs | **12×** |
-| 10 KB | Vercel AI SDK | 49 ms | 470 µs | **104×** |
-| | Anthropic SDK | 93 ms | 470 µs | **198×** |
-| | TanStack AI | 96 ms | 470 µs | **204×** |
-| | OpenClaw | 113 ms | 470 µs | **240×** |
-| 100 KB | Vercel AI SDK | 4.1 s | 4.6 ms | **892×** |
-| | Anthropic SDK | 9.3 s | 4.6 ms | **2016×** |
-| | TanStack AI | 7.5 s | 4.6 ms | **1644×** |
-| | OpenClaw | 8.1 s | 4.6 ms | **1757×** |
+| 1 KB | Vercel AI SDK | 3.9 ms | 283 µs | **14×** |
+| | Anthropic SDK | 3.3 ms | 283 µs | **12×** |
+| | TanStack AI | 3.2 ms | 283 µs | **11×** |
+| | OpenClaw | 3.8 ms | 283 µs | **14×** |
+| 5 KB | Vercel AI SDK | 23.1 ms | 739 µs | **31×** |
+| | Anthropic SDK | 34.7 ms | 739 µs | **47×** |
+| | TanStack AI | — | 739 µs | — |
+| | OpenClaw | — | 739 µs | — |
+| 50 KB | Vercel AI SDK | 1.80 s | 2.7 ms | **664×** |
+| | Anthropic SDK | 3.39 s | 2.7 ms | **1255×** |
+| | TanStack AI | 2.34 s | 2.7 ms | **864×** |
+| | OpenClaw | 2.73 s | 2.7 ms | **1011×** |
+| 100 KB | Vercel AI SDK | 6.1 s | 6.6 ms | **920×** |
+| | Anthropic SDK | 13.4 s | 6.6 ms | **2028×** |
+| | TanStack AI | 7.0 s | 6.6 ms | **1065×** |
+| | OpenClaw | 8.0 s | 6.6 ms | **1222×** |
 
 Stock parsers re-parse the full buffer on every chunk — O(n²). VectorJSON maintains a **live JS object** that grows incrementally on each `feed()`, so `getValue()` is O(1). Total work: O(n).
 
@@ -148,10 +159,10 @@ The real cost isn't just CPU time — it's blocking the agent's main thread. Sim
 
 | Payload | Stock total | VectorJSON total | Main thread freed |
 |---------|-----------|-----------------|-------------------|
-| 10 KB | 24 ms | 1 ms | 23 ms sooner |
-| 100 KB | 1.5 s | 3 ms | **1.5 seconds sooner** |
-| 500 KB | 39 s | 29 ms | **39 seconds sooner** |
-| 1 MB | 2 min 41 s | 44 ms | **161 seconds sooner** |
+| 1 KB | 4.0 ms | 1.7 ms | 2.3 ms sooner |
+| 10 KB | 36.7 ms | 1.9 ms | 35 ms sooner |
+| 50 KB | 665 ms | 3.8 ms | **661 ms sooner** |
+| 100 KB | 2.42 s | 10.2 ms | **2.4 seconds sooner** |
 
 Both approaches detect the tool name (`.name`) at the same chunk — the LLM hasn't streamed more yet. But while VectorJSON finishes processing all chunks in milliseconds, the stock parser blocks the main thread for the entire duration. The agent can't render UI, stream code to the editor, or start running tools until parsing is done.
 
@@ -188,19 +199,24 @@ The event parser (`createEventParser`) adds path-matching on top: it diffs the t
 
 ```bash
 npm install vectorjson
+# or
+pnpm add vectorjson
+# or
+bun add vectorjson
+# or
+yarn add vectorjson
 ```
 
 ## Usage
 
-### Drop-in: Replace your SDK's partial JSON parser
+### Streaming parse
 
-Every AI SDK has a `parsePartialJson` function that re-parses the full buffer on every chunk. Replace it with VectorJSON's streaming parser:
+Feed chunks as they arrive from any source — raw fetch, WebSocket, SSE, or your own transport:
 
 ```js
-import { init } from "vectorjson";
-const vj = await init();
+import { createParser } from "vectorjson";
 
-const parser = vj.createParser();
+const parser = createParser();
 for await (const chunk of stream) {
   const s = parser.feed(chunk);
   if (s === "complete" || s === "end_early") break;
@@ -209,7 +225,9 @@ const result = parser.getValue(); // lazy Proxy — materializes on access
 parser.destroy();
 ```
 
-Or use the Vercel AI SDK-compatible signature as a 1-line swap:
+### Vercel AI SDK-compatible signature
+
+If you have code that calls `parsePartialJson`, VectorJSON provides a compatible function:
 
 ```js
 // Before
@@ -217,17 +235,20 @@ import { parsePartialJson } from "ai";
 const { value, state } = parsePartialJson(buffer);
 
 // After
-import { init } from "vectorjson";
-const vj = await init();
-const { value, state } = vj.parsePartialJson(buffer);
+import { parsePartialJson } from "vectorjson";
+const { value, state } = parsePartialJson(buffer);
 ```
 
+> **Note:** AI SDKs (Vercel, Anthropic, TanStack) parse JSON internally inside `streamObject()`, `MessageStream`, etc. — you don't get access to the raw chunks. To use VectorJSON today, work with the raw LLM stream directly (raw fetch, WebSocket, SSE).
+
 ### Event-driven: React to fields as they stream in
 
 When an LLM streams a tool call, you usually care about specific fields at specific times. `createEventParser` lets you subscribe to paths and get notified the moment a value completes or a string grows:
 
 ```js
-const parser = vj.createEventParser();
+import { createEventParser } from "vectorjson";
+
+const parser = createEventParser();
 
 // Get the tool name the moment it's complete
 parser.on('tool_calls[*].name', (e) => {
@@ -254,7 +275,9 @@ parser.destroy();
 Some LLM APIs stream multiple JSON values separated by newlines. VectorJSON auto-resets between values:
 
 ```js
-const parser = vj.createEventParser({
+import { createEventParser } from "vectorjson";
+
+const parser = createEventParser({
   multiRoot: true,
   onRoot(event) {
     console.log(`Root #${event.index}:`, event.value);
@@ -272,7 +295,9 @@ parser.destroy();
 Some models emit thinking text before JSON, or wrap JSON in code fences. VectorJSON finds the JSON automatically:
 
 ```js
-const parser = vj.createEventParser();
+import { createEventParser } from "vectorjson";
+
+const parser = createEventParser();
 parser.on('answer', (e) => console.log(e.value));
 parser.onText((text) => thinkingPanel.append(text)); // opt-in
 
@@ -291,9 +316,11 @@ Validate and auto-infer types with Zod, Valibot, ArkType, or any lib with `.safe
 
 ```ts
 import { z } from 'zod';
+import { createParser } from "vectorjson";
+
 const User = z.object({ name: z.string(), age: z.number() });
 
-const parser = vj.createParser(User);       // T inferred from schema
+const parser = createParser(User);           // T inferred from schema
 for await (const chunk of stream) {
   parser.feed(chunk);
   const partial = parser.getValue();         // { name: "Ali" } mid-stream — always available
@@ -307,7 +334,9 @@ parser.destroy();
 **Partial JSON** — returns `DeepPartial<T>` because incomplete JSON has missing fields:
 
 ```ts
-const { value, state } = vj.parsePartialJson('{"name":"Al', User);
+import { parsePartialJson } from "vectorjson";
+
+const { value, state } = parsePartialJson('{"name":"Al', User);
 // value: { name: "Al" }     — partial object, typed as DeepPartial<{ name: string; age: number }>
 // state: "repaired-parse"
 // TypeScript type: { name?: string; age?: number } | undefined
@@ -326,12 +355,41 @@ parser.on('tool_calls[*]', ToolCall, (event) => {
 
 Schema-agnostic: any object with `{ safeParse(v) → { success: boolean; data?: T } }` works.
 
+### Deep compare — compare JSON without materializing
+
+Compare two parsed values directly in WASM memory. Returns a boolean — no JS objects allocated, no Proxy traps fired. Useful for diffing LLM outputs, caching, or deduplication:
+
+```js
+import { parse, deepCompare } from "vectorjson";
+
+const a = parse('{"name":"Alice","age":30}').value;
+const b = parse('{"age":30,"name":"Alice"}').value;
+
+deepCompare(a, b);                          // true — key order ignored by default
+deepCompare(a, b, { ignoreKeyOrder: false }); // false — keys must be in same order
+```
+
+By default, `deepCompare` ignores key order — `{"a":1,"b":2}` equals `{"b":2,"a":1}`, just like `fast-deep-equal`. Set `{ ignoreKeyOrder: false }` for strict key order comparison, which is ~2× faster when you know both values come from the same source.
+
+```
+bun --expose-gc bench/deep-compare.mjs
+
+  Equal objects (560 KB):
+  JS deepEqual (recursive)         848 ops/s    heap Δ  2.4 MB
+  VJ ignore key order (default)  1.63K ops/s    heap Δ  0.1 MB    2× faster
+  VJ strict key order            3.41K ops/s    heap Δ  0.1 MB    4× faster
+```
+
+Works with any combination: two VJ proxies (fast WASM path), plain JS objects, or mixed (falls back to `JSON.stringify` comparison).
+
 ### Lazy access — only materialize what you touch
 
-`vj.parse()` returns a lazy Proxy backed by the WASM tape. Fields are only materialized into JS objects when you access them. On a 2 MB payload, reading one field is 2× faster than `JSON.parse` because the other 99% is never allocated:
+`parse()` returns a lazy Proxy backed by the WASM tape. Fields are only materialized into JS objects when you access them. On a 2 MB payload, reading one field is 2× faster than `JSON.parse` because the other 99% is never allocated:
 
 ```js
-const result = vj.parse(huge2MBToolCall);
+import { parse } from "vectorjson";
+
+const result = parse(huge2MBToolCall);
 result.value.tool;   // "file_edit" — reads from WASM tape, 2.3ms
 result.value.path;   // "app.ts"
 // result.value.code (the 50KB field) is never materialized in JS memory
@@ -351,18 +409,28 @@ bun --expose-gc bench/partial-access.mjs
 For non-streaming use cases:
 
 ```js
-const result = vj.parse('{"users": [{"name": "Alice"}]}');
+import { parse } from "vectorjson";
+
+const result = parse('{"users": [{"name": "Alice"}]}');
 result.status;       // "complete" | "complete_early" | "incomplete" | "invalid"
 result.value.users;  // lazy Proxy — materializes on access
 ```
 
 ## API Reference
 
+### Direct exports (recommended)
+
+All functions are available as direct imports — no `init()` needed:
+
+```js
+import { parse, parsePartialJson, deepCompare, createParser, createEventParser, materialize } from "vectorjson";
+```
+
 ### `init(options?): Promise<VectorJSON>`
 
-Loads WASM once, returns cached singleton. `{ engineWasm?: string | URL | BufferSource }` for custom WASM location.
+Returns cached singleton. Useful for passing custom WASM via `{ engineWasm?: string | URL | BufferSource }`. Called automatically on import.
 
-### `vj.parse(input: string | Uint8Array): ParseResult`
+### `parse(input: string | Uint8Array): ParseResult`
 
 ```ts
 interface ParseResult {
@@ -380,7 +448,7 @@ interface ParseResult {
 - **`incomplete`** — truncated JSON; value is autocompleted, `isComplete()` tells you what's real
 - **`invalid`** — broken JSON
 
-### `vj.createParser(schema?): StreamingParser<T>`
+### `createParser(schema?): StreamingParser<T>`
 
 Each `feed()` processes only new bytes — O(n) total. Pass an optional schema to auto-validate and infer the return type.
 
@@ -400,16 +468,18 @@ While incomplete, `getValue()` returns the **live document** — a mutable JS ob
 
 ```ts
 import { z } from 'zod';
+import { createParser } from "vectorjson";
+
 const User = z.object({ name: z.string(), age: z.number() });
 
-const parser = vj.createParser(User);
+const parser = createParser(User);
 parser.feed('{"name":"Alice","age":30}');
 const val = parser.getValue(); // { name: string; age: number } | undefined ✅
 ```
 
 Works with Zod, Valibot, ArkType — any library with `{ safeParse(v) → { success, data? } }`.
 
-### `vj.parsePartialJson(input, schema?): PartialJsonResult<DeepPartial<T>>`
+### `parsePartialJson(input, schema?): PartialJsonResult<DeepPartial<T>>`
 
 Compatible with Vercel AI SDK's `parsePartialJson` signature. Returns a plain JS object (not a Proxy). Pass an optional schema for type-safe validation.
 
@@ -427,7 +497,7 @@ type DeepPartial<T> = T extends object
   : T;
 ```
 
-### `vj.createEventParser(options?): EventParser`
+### `createEventParser(options?): EventParser`
 
 Event-driven streaming parser. Events fire synchronously during `feed()`.
 
@@ -485,7 +555,23 @@ interface RootEvent {
 }
 ```
 
-### `vj.materialize(value): unknown`
+### `deepCompare(a, b, options?): boolean`
+
+Compare two values for deep equality without materializing JS objects. When both values are VJ proxies, comparison runs entirely in WASM memory — zero allocations, zero Proxy traps.
+
+```ts
+deepCompare(
+  a: unknown,
+  b: unknown,
+  options?: { ignoreKeyOrder?: boolean }  // default: true
+): boolean
+```
+
+- **`ignoreKeyOrder: true`** (default) — `{"a":1,"b":2}` equals `{"b":2,"a":1}`. Same semantics as `fast-deep-equal`.
+- **`ignoreKeyOrder: false`** — keys must appear in the same order. ~2× faster for same-source comparisons.
+- Falls back to `JSON.stringify` comparison when either value is a plain JS object.
+
+### `materialize(value): unknown`
 
 Convert a lazy Proxy into a plain JS object tree. No-op on plain values.
 
@@ -493,26 +579,41 @@ Convert a lazy Proxy into a plain JS object tree. No-op on plain values.
 
 | Runtime | Status | Notes |
 |---------|--------|-------|
-| Node.js 20+ | ✅ | WASM loaded from disk automatically |
-| Bun | ✅ | WASM loaded from disk automatically |
-| Browsers | ✅ | Pass `engineWasm` as `ArrayBuffer` or `URL` to `init()` |
-| Deno | ✅ | Pass `engineWasm` as `URL` to `init()` |
-| Cloudflare Workers | ✅ | Import WASM as module, pass as `ArrayBuffer` to `init()` |
+| Node.js 20+ | ✅ | WASM embedded in bundle — zero config |
+| Bun | ✅ | WASM embedded in bundle — zero config |
+| Browsers | ✅ | WASM embedded in bundle — zero config |
+| Deno | ✅ | WASM embedded in bundle — zero config |
+| Cloudflare Workers | ✅ | WASM embedded in bundle — zero config |
+
+WASM is embedded as base64 in the JS bundle and auto-initialized via top-level `await`. No setup required — just `import { parse } from "vectorjson"`.
 
-For environments without filesystem access, provide the WASM binary explicitly:
+For advanced use cases, you can still provide a custom WASM binary via `init()`:
 
 ```js
 import { init } from "vectorjson";
+const vj = await init({ engineWasm: customWasmBytes });
+```
+
+Bundle size: ~148 KB JS with embedded WASM (~47 KB gzipped). No runtime dependencies.
+
+## Runnable Examples
+
+The `examples/` directory has working demos you can run immediately:
+
+```bash
+# Anthropic tool call — streams fields as they arrive, early abort demo
+bun examples/anthropic-tool-call.ts --mock
+bun examples/anthropic-tool-call.ts --mock --wrong-tool   # early abort
 
-// Option 1: URL (browsers, Deno)
-const vj = await init({ engineWasm: new URL('./engine.wasm', import.meta.url) });
+# OpenAI function call — streams function arguments via EventParser
+bun examples/openai-function-call.ts --mock
 
-// Option 2: ArrayBuffer (Workers, custom loaders)
-const wasmBytes = await fetch('/engine.wasm').then(r => r.arrayBuffer());
-const vj = await init({ engineWasm: wasmBytes });
+# With a real API key:
+ANTHROPIC_API_KEY=sk-ant-... bun examples/anthropic-tool-call.ts
+OPENAI_API_KEY=sk-...       bun examples/openai-function-call.ts
 ```
 
-Bundle size: ~92 KB WASM + ~20 KB JS (~37 KB gzipped total). No runtime dependencies.
+See also `examples/ai-usage.ts` for additional patterns (MCP stdio, Vercel AI SDK `streamObject`, NDJSON embeddings).
 
 ## Building from Source
 
@@ -528,7 +629,7 @@ sudo apt-get install -y binaryen
 
 ```bash
 bun run build        # Zig → WASM → wasm-opt → TypeScript
-bun run test         # 557 tests including 100MB stress payloads
+bun run test         # 724+ tests including 100MB stress payloads
 bun run test:worker  # Worker transferable tests (Playwright + Chromium)
 ```
 
@@ -538,9 +639,10 @@ To reproduce benchmarks:
 bun --expose-gc bench/parse-stream.mjs           # one-shot + streaming parse
 cd bench/ai-parsers && bun install && bun --expose-gc bench.mjs  # AI SDK comparison
 bun run bench:worker                             # Worker transfer vs structured clone benchmark
+node --expose-gc bench/deep-compare.mjs          # deep compare: VJ vs JS deepEqual
 ```
 
-Benchmark numbers in this README were measured on an Apple M-series Mac. Results vary by machine but relative speedups are consistent.
+Benchmark numbers in this README were measured on GitHub Actions (Ubuntu, x86_64). Results vary by machine but relative speedups are consistent.
 
 ## License