boom-format 0.9.0

package/README.md

@@ -0,0 +1,1079 @@
+# BOOM
+
+**Binary Object Optimised Markup**
+
+A compact binary serialisation format with an optional human-readable debug mode. Smaller than JSON, faster to parse, yet still debuggable.
+
+[![npm version](https://img.shields.io/npm/v/boom-format.svg)](https://www.npmjs.com/package/boom-format)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+**[Documentation](https://boomformat.dev)** | **[Getting Started](https://boomformat.dev/guide/getting-started.html)** | **[Playground](https://boomformat.dev/converter.html)** | **[Benchmarks](https://boomformat.dev/benchmarks.html)**
+
+## Features
+
+- 🗜️ **40-70% smaller** than JSON
+- 🧠 **50-85% fewer LLM tokens** - best-in-class for AI applications
+- ⚡ **1.8-19x faster** than JSON (encode & decode)
+- 📖 **299-entry dictionary** - single-byte encoding for common keys
+- 🔍 **Human-readable debug format** - `.boom.txt`
+- 🔄 **Self-describing** - no schema required
+- 📦 **String interning** - automatic deduplication
+- 🌍 **13 implementations** - JS, Go, Rust, PHP, Java, .NET, Swift, C++, Python
+- 📝 **TypeScript-first** - full type safety
+- 🏷️ **Script type support** - `<script type="text/boom">`
+
+## Installation
+
+```bash
+npm install boom-format
+```
+
+## Quick Start
+
+```typescript
+import { encode, decode, stringify, parseText } from 'boom-format';
+
+// Binary format
+const data = { 
+  name: 'Tom', 
+  skills: ['TypeScript', 'Node.js'],
+  nested: { deep: { value: 42 } }
+};
+
+const binary = encode(data);      // Uint8Array
+const restored = decode(binary);  // Back to object
+
+// Text format (for debugging)
+const text = stringify(data);     // Human-readable
+const parsed = parseText(text);   // Back to object
+```
+
+## Command Line Interface
+
+BOOM includes a CLI for converting files between JSON and BOOM formats:
+
+```bash
+# Install globally
+npm install -g boom-format
+
+# Or use with npx
+npx boom --help
+```
+
+### Basic Usage
+
+```bash
+# Encode JSON to BOOM binary
+boom encode data.json                    # Output: data.boom
+
+# Encode JSON to BOOM text (human-readable)
+boom encode data.json --text             # Output: data.boom.txt
+
+# Decode BOOM to JSON
+boom decode data.boom                    # Output: data.json
+boom decode data.boom.txt                # Decode text format
+
+# Show file info
+boom info data.boom
+```
+
+### CLI Options
+
+```
+Usage:
+  boom encode <input.json> [output]        Encode JSON to BOOM format
+  boom decode <input.boom> [output.json]   Decode BOOM to JSON
+  boom info <input.boom>                   Show info about a BOOM file
+  boom <input.json>                        Shorthand for encode
+
+Output Formats:
+  --binary, -b   Output BOOM binary format (default)
+  --text, -t     Output BOOM text format (.boom.txt)
+  --json, -j     Output JSON format (for decode)
+
+Options:
+  --help, -h     Show this help message
+  --version, -v  Show version
+  --pretty, -p   Pretty-print output (JSON and text)
+  --dict, -d     Use shared dictionary (default: true)
+  --no-dict      Disable shared dictionary
+```
+
+### Stdin/Stdout Support
+
+```bash
+# Pipe from stdin
+cat data.json | boom encode - > data.boom
+cat data.boom | boom decode - > data.json
+
+# Convert and pipe to another command
+boom encode data.json | curl -X POST -H "Content-Type: application/x-boom" --data-binary @- https://api.example.com
+```
+
+### Sample Files
+
+The package includes sample files in `samples/` to help you understand the format:
+
+```bash
+# View sample files
+cat node_modules/boom-format/samples/users.boom.txt
+
+# Test round-trip conversion
+boom decode node_modules/boom-format/samples/users.boom --pretty
+```
+
+## Size Comparison
+
+```typescript
+import { compare } from 'boom-format';
+
+const data = {
+  users: [
+    { id: 1, name: 'Alice', active: true },
+    { id: 2, name: 'Bob', active: false },
+    { id: 3, name: 'Charlie', active: true }
+  ],
+  count: 3,
+  page: 1
+};
+
+console.log(compare(data));
+// {
+//   jsonSize: 156,
+//   jsonMinSize: 128,
+//   boomBinarySize: 52,
+//   boomTextSize: 98,
+//   savings: { vsJson: '66.7%', vsJsonMin: '59.4%' }
+// }
+```
+
+## API Reference
+
+### Binary Format
+
+#### `encode(value, options?): Uint8Array`
+
+Encode a value to BOOM binary format.
+
+```typescript
+const binary = encode({ hello: 'world' });
+```
+
+#### `decode(buffer, options?): BoomValue`
+
+Decode BOOM binary format to a value.
+
+```typescript
+const value = decode(binary);
+```
+
+### Text Format
+
+#### `stringify(value, options?): string`
+
+Convert a value to BOOM text format.
+
+```typescript
+const text = stringify({ name: 'Tom', age: 42 });
+// {
+//   name: "Tom"
+//   age: 42
+// }
+```
+
+#### `parseText(input, options?): BoomValue`
+
+Parse BOOM text format.
+
+```typescript
+const value = parseText(`{
+  name: "Tom"
+  age: 42
+  tags: ["web" "ai"]
+}`);
+```
+
+### JSON Interop
+
+#### `fromJSON(json, options?): Uint8Array`
+
+Convert JSON string to BOOM binary.
+
+#### `toJSON(buffer, options?): string`
+
+Convert BOOM binary to JSON string.
+
+### Options
+
+```typescript
+interface BoomOptions {
+  maxDepth?: number;        // Max nesting (default: 64)
+  maxStringLength?: number; // Max string bytes (default: 16MB)
+  maxArrayLength?: number;  // Max array items (default: 1M)
+  enableInterning?: boolean; // String dedup (default: true)
+}
+
+interface BoomTextOptions extends BoomOptions {
+  compact?: boolean;        // Single-line output
+  indentString?: string;    // Indent chars (default: "  ")
+}
+```
+
+## Framework Integration
+
+### React
+
+```tsx
+import { encode, decode } from 'boom-format';
+import { useState, useEffect } from 'react';
+
+// Custom hook for BOOM API calls
+function useBoom<T>(url: string) {
+  const [data, setData] = useState<T | null>(null);
+  const [loading, setLoading] = useState(true);
+
+  useEffect(() => {
+    fetch(url, { headers: { Accept: 'application/x-boom' } })
+      .then(res => res.arrayBuffer())
+      .then(buf => setData(decode(new Uint8Array(buf)) as T))
+      .finally(() => setLoading(false));
+  }, [url]);
+
+  return { data, loading };
+}
+
+// Usage in component
+function UserList() {
+  const { data: users, loading } = useBoom<User[]>('/api/users');
+
+  if (loading) return <div>Loading...</div>;
+  return <ul>{users?.map(u => <li key={u.id}>{u.name}</li>)}</ul>;
+}
+```
+
+### Next.js
+
+```typescript
+// app/api/users/route.ts (App Router)
+import { encode, decode } from 'boom-format';
+
+export async function GET() {
+  const users = await db.users.findMany();
+  return new Response(encode(users), {
+    headers: { 'Content-Type': 'application/x-boom' }
+  });
+}
+
+export async function POST(req: Request) {
+  const body = decode(new Uint8Array(await req.arrayBuffer()));
+  const user = await db.users.create({ data: body });
+  return new Response(encode(user), {
+    headers: { 'Content-Type': 'application/x-boom' }
+  });
+}
+
+// Client component with Server Actions
+'use client';
+import { boomFetch } from 'boom-format';
+
+export default function Dashboard() {
+  const [users, setUsers] = useState([]);
+
+  useEffect(() => {
+    boomFetch('/api/users').then(setUsers);
+  }, []);
+
+  return <UserTable users={users} />;
+}
+```
+
+### Vue 3
+
+```vue
+<script setup lang="ts">
+import { ref, onMounted } from 'vue';
+import { decode } from 'boom-format';
+
+const users = ref<User[]>([]);
+const loading = ref(true);
+
+onMounted(async () => {
+  const res = await fetch('/api/users', {
+    headers: { Accept: 'application/x-boom' }
+  });
+  users.value = decode(new Uint8Array(await res.arrayBuffer()));
+  loading.value = false;
+});
+</script>
+
+<template>
+  <div v-if="loading">Loading...</div>
+  <ul v-else>
+    <li v-for="user in users" :key="user.id">{{ user.name }}</li>
+  </ul>
+</template>
+```
+
+### Nuxt 3
+
+```typescript
+// server/api/users.ts
+import { encode, decode } from 'boom-format';
+
+export default defineEventHandler(async (event) => {
+  const accept = getHeader(event, 'accept');
+  const users = await $fetch('/external-api/users');
+
+  if (accept === 'application/x-boom') {
+    setHeader(event, 'Content-Type', 'application/x-boom');
+    return encode(users);
+  }
+  return users;
+});
+
+// composables/useBoom.ts
+export function useBoom<T>(url: string) {
+  return useFetch<T>(url, {
+    headers: { Accept: 'application/x-boom' },
+    transform: (data) => decode(new Uint8Array(data))
+  });
+}
+```
+
+### Redux / Redux Toolkit
+
+```typescript
+import { encode, decode } from 'boom-format';
+import { createSlice, createAsyncThunk } from '@reduxjs/toolkit';
+
+// Async thunk with BOOM encoding
+export const fetchUsers = createAsyncThunk('users/fetch', async () => {
+  const res = await fetch('/api/users', {
+    headers: { Accept: 'application/x-boom' }
+  });
+  return decode(new Uint8Array(await res.arrayBuffer()));
+});
+
+// Persist Redux state with BOOM (40-70% smaller than JSON)
+const persistBoom = {
+  save: (state: RootState) => {
+    localStorage.setItem('app-state',
+      btoa(String.fromCharCode(...encode(state)))
+    );
+  },
+  load: (): RootState | undefined => {
+    const saved = localStorage.getItem('app-state');
+    if (!saved) return undefined;
+    const bytes = Uint8Array.from(atob(saved), c => c.charCodeAt(0));
+    return decode(bytes) as RootState;
+  }
+};
+```
+
+## Mobile Development
+
+BOOM is ideal for mobile apps - smaller payloads mean faster loads on cellular networks and better battery life.
+
+### React Native
+
+```typescript
+import { encode, decode } from 'boom-format';
+
+// Custom fetch wrapper for React Native
+async function boomRequest<T>(url: string, options?: RequestInit): Promise<T> {
+  const res = await fetch(url, {
+    ...options,
+    headers: {
+      ...options?.headers,
+      Accept: 'application/x-boom',
+      'Content-Type': options?.body ? 'application/x-boom' : undefined,
+    },
+    body: options?.body ? encode(options.body) : undefined,
+  });
+
+  const buffer = await res.arrayBuffer();
+  return decode(new Uint8Array(buffer)) as T;
+}
+
+// Usage
+const users = await boomRequest<User[]>('https://api.example.com/users');
+
+// Offline storage with AsyncStorage (60% smaller than JSON)
+import AsyncStorage from '@react-native-async-storage/async-storage';
+
+const saveOffline = async (key: string, data: any) => {
+  const encoded = encode(data);
+  await AsyncStorage.setItem(key, btoa(String.fromCharCode(...encoded)));
+};
+
+const loadOffline = async <T>(key: string): Promise<T | null> => {
+  const saved = await AsyncStorage.getItem(key);
+  if (!saved) return null;
+  const bytes = Uint8Array.from(atob(saved), c => c.charCodeAt(0));
+  return decode(bytes) as T;
+};
+```
+
+### iOS (Swift)
+
+```swift
+import Boom
+
+// Network request with BOOM
+func fetchUsers() async throws -> [User] {
+    var request = URLRequest(url: URL(string: "https://api.example.com/users")!)
+    request.setValue("application/x-boom", forHTTPHeaderField: "Accept")
+
+    let (data, _) = try await URLSession.shared.data(for: request)
+    return try Boom.decode([User].self, from: data)
+}
+
+// Encode for POST
+func createUser(_ user: User) async throws -> User {
+    var request = URLRequest(url: URL(string: "https://api.example.com/users")!)
+    request.httpMethod = "POST"
+    request.setValue("application/x-boom", forHTTPHeaderField: "Content-Type")
+    request.httpBody = try Boom.encode(user)
+
+    let (data, _) = try await URLSession.shared.data(for: request)
+    return try Boom.decode(User.self, from: data)
+}
+
+// Cache to disk (uses 60% less storage than JSON)
+let encoded = try Boom.encode(users)
+try encoded.write(to: cacheURL)
+```
+
+### Android (Java/Kotlin)
+
+```kotlin
+import io.boomformat.Boom
+
+// With Retrofit + custom converter
+val retrofit = Retrofit.Builder()
+    .baseUrl("https://api.example.com/")
+    .addConverterFactory(BoomConverterFactory.create())
+    .build()
+
+interface ApiService {
+    @GET("users")
+    @Headers("Accept: application/x-boom")
+    suspend fun getUsers(): List<User>
+
+    @POST("users")
+    @Headers("Content-Type: application/x-boom")
+    suspend fun createUser(@Body user: User): User
+}
+
+// Direct usage
+val users = Boom.decode<List<User>>(response.bytes())
+val encoded = Boom.encode(newUser)
+
+// Room database with BOOM serialization (smaller DB files)
+@TypeConverter
+fun fromBoom(data: ByteArray): UserData = Boom.decode(data)
+
+@TypeConverter
+fun toBoom(userData: UserData): ByteArray = Boom.encode(userData)
+```
+
+### Why BOOM for Mobile?
+
+| Benefit | Impact |
+|---------|--------|
+| **40-70% smaller payloads** | Faster downloads on 3G/4G/5G |
+| **2-5x faster parsing** | Smoother UI, less jank |
+| **Less data transfer** | Lower cellular bills, data caps |
+| **Reduced battery drain** | Less radio usage, less CPU |
+| **Offline storage** | 60% smaller cache files |
+| **Native SDKs** | Swift, Java, C++ - no bridging overhead |
+
+## Browser Usage
+
+### Script Type
+
+BOOM supports inline data via `<script type="text/boom">`:
+
+```html
+<script type="text/boom" id="config">
+{
+  apiUrl: "https://api.example.com"
+  features: {
+    darkMode: true
+    analytics: false
+  }
+  limits: [100 200 500]
+}
+</script>
+
+<script type="module">
+  import { processBoomScripts, getBoomData } from 'boom-format';
+  
+  // Process all boom scripts
+  processBoomScripts();
+  
+  // Get specific data
+  const config = getBoomData('config');
+  console.log(config.apiUrl);
+</script>
+```
+
+### Auto-initialisation
+
+```typescript
+import { autoInit } from 'boom-format';
+
+// Automatically process boom scripts when DOM is ready
+autoInit();
+```
+
+### CDN Usage
+
+```html
+<script type="module">
+  import * as boom from 'https://esm.sh/boom-format';
+  
+  const data = boom.parseText('{name: "Tom"}');
+</script>
+```
+
+## Text Format Syntax
+
+```boom
+# Comments start with #
+
+# Strings (quoted)
+name: "Tom Taylor"
+
+# Numbers (bare)
+age: 42
+price: 19.99
+bignum: 9007199254740993n  # BigInt
+
+# Booleans & null
+active: true
+deleted: false
+metadata: null
+
+# Arrays (space or newline separated)
+tags: ["web" "ai" "performance"]
+matrix: [
+  [1 2 3]
+  [4 5 6]
+]
+
+# Nested objects
+user: {
+  name: "Tom"
+  address: {
+    city: "Liverpool"
+    country: "UK"
+  }
+}
+
+# Arrays of objects
+users: [
+  { id: 1 name: "Alice" }
+  { id: 2 name: "Bob" }
+]
+
+# Binary data (base64)
+avatar: <binary:SGVsbG8gV29ybGQ=>
+```
+
+## Binary Format Specification
+
+See [SPECIFICATION.md](./SPECIFICATION.md) for the full binary format specification.
+
+### Header
+
+| Offset | Size | Description |
+|--------|------|-------------|
+| 0 | 4 | Magic: `BOOM` (0x42 0x4F 0x4F 0x4D) |
+| 4 | 1 | Version: 0x01 |
+| 5 | 1 | Flags (reserved) |
+| 6 | ... | Root value |
+
+### Type Markers
+
+| Byte | Type |
+|------|------|
+| 0x00 | null |
+| 0x01 | false |
+| 0x02 | true |
+| 0x10-0x19 | integers (various sizes) |
+| 0x20-0x21 | floats |
+| 0x30 | string |
+| 0x31 | binary |
+| 0x40 | array |
+| 0x50 | object |
+| 0x60 | string reference (local interning) |
+| 0x61 | dictionary reference (shared dictionary) |
+
+## Built-in Dictionary
+
+BOOM includes a shared dictionary of **299 common strings** optimized for API responses, LLM streaming, and analytics data. Strings in the dictionary are encoded as 1-2 byte references instead of full UTF-8. The first 128 entries (indices 0-127) use single-byte varint encoding for maximum efficiency.
+
+```typescript
+import { encode, decode, BOOM_DICTIONARY_V1, analyzeDictionary } from 'boom-format';
+
+// Dictionary is used automatically
+const data = {
+  id: 1,
+  status: 'active',      // "id" and "status" are in dictionary
+  role: 'assistant',     // "role" and "assistant" are in dictionary
+  content: 'Hello!'      // "content" is in dictionary
+};
+
+const encoded = encode(data);  // Keys encoded as 1-byte refs
+
+// Analyze dictionary coverage
+const analysis = analyzeDictionary(data);
+console.log(analysis.stats.dictionaryHitRate);  // e.g., 0.85 (85% hit rate)
+console.log(analysis.missingStrings);           // Strings not in dictionary
+```
+
+### Dictionary Contents (First 128 = 1-byte encoding)
+
+The first 128 entries get optimal 1-byte encoding:
+
+- **Core keys**: id, type, name, status, data, value, content, message, error
+- **AI/LLM**: role, model, choices, delta, finish_reason, usage, prompt_tokens, completion_tokens
+- **API patterns**: request, response, params, headers, body, method, url, path
+- **Timestamps**: created_at, updated_at, timestamp, expires_at
+- **Status values**: active, pending, completed, failed, enabled, disabled
+- **HTTP**: GET, POST, PUT, DELETE, Content-Type, Authorization
+
+### Disable Dictionary
+
+```typescript
+const encoded = encode(data, { useBuiltInDictionary: false });
+```
+
+### Strict Binary Mode (Production Security)
+
+Reject non-BOOM input to prevent JSON injection attacks:
+
+```typescript
+const decoded = decode(buffer, { strictBinaryMode: true });
+// Throws if buffer doesn't start with BOOM magic bytes
+```
+
+## TypeScript Support
+
+Full type definitions included:
+
+```typescript
+import type { BoomValue, BoomObject, BoomArray, BoomOptions } from 'boom-format';
+
+const data: BoomObject = {
+  name: 'Tom',
+  scores: [100, 95, 88] as BoomArray,
+};
+```
+
+## Error Handling
+
+```typescript
+import { encode, decode, BoomError } from 'boom-format';
+
+try {
+  const value = decode(corruptedData);
+} catch (error) {
+  if (error instanceof BoomError) {
+    console.error('Invalid BOOM data:', error.message);
+  }
+}
+```
+
+## Benchmarks
+
+Run benchmarks:
+
+```bash
+npm run benchmark         # Size/token benchmarks
+npm run benchmark:compare # Full format comparison with speed tests
+```
+
+### Speed Benchmarks (Node.js v22)
+
+BOOM is optimised for real-world data with repeated strings (API responses, logs, exports):
+
+| Data Type | JSON Size | BOOM Size | Savings | Encode Speed | Decode Speed |
+|-----------|-----------|-----------|---------|--------------|--------------|
+| **Small** | 37 bytes | 35 bytes | **5.4%** | **2x faster** | **2x faster** |
+| **Medium** | 9,727 bytes | 5,389 bytes | **44.6%** | **1.3x faster** | **1.1x faster** |
+| **Tabular (500 rows)** | 52,226 bytes | 18,093 bytes | **65.4%** | **2x faster** | **2.5x faster** |
+| **Large (float matrix)** | 192,966 bytes | 80,291 bytes | **58.4%** | **8x faster** | **19x faster** |
+
+For typical API data, BOOM is **1.3-2x faster** than JSON for encoding and **1.1-2.5x faster** for decoding.
+Float matrices benefit most from native binary encoding: **8-19x faster**.
+
+### Cross-Language Performance
+
+BOOM is available in 13 implementations with consistent performance:
+
+| Language | Encode Speed | Decode Speed | Size Savings |
+|----------|-------------|--------------|--------------|
+| **JavaScript** | **1.3-2x faster** | **1.1-2.5x faster** | 40-65% |
+| **Go** | **2.0-2.6x faster** | **3.6-4.1x faster** | 40-70% |
+| **Rust** | **2x+ faster** | **2x+ faster** | 40-70% |
+| **C++** | **1.3-3.2x faster** | **1.5-1.6x faster** | 29-69% |
+| **Java** | ~1x (Jackson optimized) | **2.8-5.3x faster** | 36-73% |
+| **.NET** | **1.3-2.7x faster** | **3.4-4.9x faster** | 36-72% |
+| **PHP** (C ext) | **1.2x faster** | **1.4x faster** | 40-70% |
+| **Swift** | **2.2x faster** | **3.5x faster** | 40-70% |
+
+**Key findings:**
+- BOOM decode is **1.5-5x faster** than JSON across all languages
+- BOOM encode is **1.2-3x faster** in most languages
+- Float/binary data sees **8-19x** speedup
+- Size savings of **36-73%** depending on data structure
+- Tabular data with repeated strings shows the best improvements
+
+### Cross-Platform Compatibility (Production-Safe)
+
+BOOM is **100% compatible** across all backend providers and mobile/frontend consumers. Every provider-consumer combination is automatically tested on every commit.
+
+| Backend Provider | iOS (Swift) | Android (Java) | Web (JS) | Native (C++) |
+|-----------------|-------------|----------------|----------|--------------|
+| **Go** | ✅ | ✅ | ✅ | ✅ |
+| **PHP** | ✅ | ✅ | ✅ | ✅ |
+| **Rust** | ✅ | ✅ | ✅ | ✅ |
+| **Java** | ✅ | ✅ | ✅ | ✅ |
+| **.NET** | ✅ | ✅ | ✅ | ✅ |
+| **Node.js** | ✅ | ✅ | ✅ | ✅ |
+
+**Why this matters:**
+- **Safe for production** - Any backend can serve any mobile app
+- **CI-verified** - Automated tests on every commit via GitHub Actions
+- **No surprises** - Binary format is byte-for-byte consistent
+- **Mix and match** - Use different providers for different services
+
+See [Interoperability Tests](https://boomformat.dev/interoperability.html) for live CI results.
+
+#### Go Performance
+
+```
+go test -bench=. -benchmem
+
+Encode Small:   BOOM 775ns   vs JSON 1,619ns  = 2.1x faster
+Encode Medium:  BOOM 131µs   vs JSON 312µs    = 2.4x faster
+Decode Small:   BOOM 787ns   vs JSON 3,210ns  = 4.1x faster
+Decode Medium:  BOOM 111µs   vs JSON 352µs    = 3.2x faster
+Size savings:   48.3%
+```
+
+#### Java Performance (vs Jackson)
+
+```
+mvn exec:java
+
+Decode Small:   BOOM 1,083K ops/s  vs Jackson 204K ops/s  = 5.3x faster
+Decode Medium:  BOOM 20.4K ops/s   vs Jackson 7.3K ops/s  = 2.8x faster
+Decode Tabular: BOOM 5.1K ops/s    vs Jackson 1.2K ops/s  = 4.4x faster
+Size savings:   36% (medium), 73% (tabular)
+```
+
+#### .NET Performance (vs System.Text.Json)
+
+```
+dotnet run -c Release
+
+Decode Small:   BOOM 411ns   vs JSON 1,572ns   = 3.8x faster
+Decode Medium:  BOOM 72µs    vs JSON 349µs     = 4.9x faster
+Decode Tabular: BOOM 578µs   vs JSON 1,973µs   = 3.4x faster
+Encode Small:   BOOM 447ns   vs JSON 588ns     = 1.3x faster
+Encode Medium:  BOOM 48µs    vs JSON 101µs     = 2.1x faster
+Encode Tabular: BOOM 255µs   vs JSON 700µs     = 2.7x faster
+Size savings:   36% (medium), 72% (tabular)
+```
+
+#### PHP Performance (Native C Extension)
+
+```
+php benchmark.php
+
+Decode:         BOOM Native 1.92x faster than json_decode
+Size savings:   49%
+```
+
+#### Rust Performance
+
+Rust's `serde_json` is highly optimized. BOOM trades speed for size:
+
+```
+cargo bench
+
+Size savings:   66.3% on tabular data
+Note: serde_json is faster for pure speed, but BOOM reduces network transfer significantly
+```
+
+### Payload Size Thresholds (Optional)
+
+BOOM includes an **optional** threshold system for automatic format selection based on payload size. This is a tuning mechanism for users who want to optimize for their specific use case.
+
+> **Note:** This feature is entirely optional. BOOM aims to be faster than JSON for all sizes, but the threshold system allows fine-tuning for edge cases where native JSON implementations may have slightly less overhead for very small payloads.
+
+#### Default Thresholds (Benchmark-Based)
+
+| Payload Size | Default Format | Reason |
+|--------------|----------------|--------|
+| < 1KB | JSON | Native JSON may have less overhead |
+| 1KB - 100KB | BOOM Binary | Good balance of speed and size |
+| > 100KB | BOOM Binary (forced) | Significant size/speed benefits |
+
+#### Configuration
+
+```typescript
+import { setThresholdConfig, selectFormat, estimateSize } from 'boom-format';
+
+// Configure thresholds (optional - defaults work well for most cases)
+setThresholdConfig({
+  minSize: 1024,      // Below: may use JSON (default: 1KB)
+  preferSize: 10240,  // Above: prefer BOOM (default: 10KB)
+  forceSize: 102400,  // Above: always BOOM (default: 100KB)
+  autoDetect: true,   // Enable auto format selection
+});
+
+// Auto-select format based on estimated size
+const data = { users: [...] };
+const { mode, reason } = selectFormat(estimateSize(data));
+// mode: 'boom-binary' or 'json'
+// reason: 'size-below-threshold', 'size-above-force', etc.
+
+// Or use with boomFetch for automatic selection
+const result = await boomFetch('/api/data', {
+  body: largePayload,
+  autoFormat: true,  // Automatically selects format based on size
+});
+```
+
+#### HTTP Header Negotiation
+
+Servers can announce their threshold policy, and clients can hint expected payload sizes:
+
+```
+# Server response headers
+X-Boom-Min-Size: 1024
+X-Boom-Prefer-Size: 10240
+X-Boom-Force-Size: 102400
+
+# Client request header
+X-Boom-Expected-Size: large   # or 'small', or byte count like '50000'
+```
+
+```typescript
+import { createThresholdHeaders, parseThresholdHeaders } from 'boom-format';
+
+// Server: Create headers to announce policy
+app.use((req, res, next) => {
+  const headers = createThresholdHeaders();
+  Object.entries(headers).forEach(([k, v]) => res.setHeader(k, v));
+  next();
+});
+
+// Client: Parse server's threshold policy
+const serverConfig = parseThresholdHeaders(response.headers);
+```
+
+#### When to Use Thresholds
+
+- **Most users:** Don't need to configure thresholds - defaults work well
+- **API developers:** May want to tune based on typical payload sizes
+- **Performance-critical apps:** Can fine-tune based on their specific data patterns
+- **Hybrid environments:** Useful when some clients don't support BOOM
+
+### BOOM vs JSON + HTTP Compression
+
+Modern HTTP/2 and HTTP/3 use compression (gzip, brotli, deflate). BOOM still wins:
+
+| Data Type | JSON | JSON+gzip | JSON+brotli | BOOM | BOOM+brotli |
+|-----------|------|-----------|-------------|------|-------------|
+| **Small** (3 keys) | 39 B | 59 B | 43 B | **37 B** | 41 B |
+| **Medium** (100 rows) | 8,109 B | 1,064 B | 597 B | **4,623 B** | **624 B** |
+| **Tabular** (500 rows) | 75,176 B | 523 B | 137 B | **39,587 B** | 191 B |
+| **API Response** | 6,967 B | 719 B | 462 B | **3,071 B** | 554 B |
+
+**Key findings:**
+
+1. **BOOM uncompressed is 40-70% smaller than JSON** - before any HTTP compression
+2. **For tabular data, BOOM is smaller than JSON+gzip** - string interning eliminates redundancy at source
+3. **BOOM + HTTP compression works** - BOOM can be further compressed via gzip/brotli
+4. **No CPU overhead difference** - both formats get HTTP-compressed anyway
+
+**For HTTP/2 and HTTP/3:**
+```
+# Server returns BOOM binary
+Content-Type: application/x-boom
+Content-Encoding: br  # Brotli compression by HTTP layer
+
+# Result: Smallest possible payload + fast binary parsing
+```
+
+**Recommendation:**
+- API servers: Return BOOM binary, let HTTP layer handle compression
+- Mobile apps: Accept BOOM binary, benefit from smaller downloads + faster parsing
+- Web apps: Accept BOOM binary, faster parsing than JSON even after decompression
+
+Run the compression comparison:
+```bash
+node benchmark/compression-comparison.cjs
+```
+
+### Why BOOM Wins for LLM Use Cases
+
+| Metric | JSON | BOOM Binary | Winner |
+|--------|------|-------------|--------|
+| Decoding speed | Baseline | **240% faster** | **BOOM** |
+| Encoding speed | Baseline | **125%** | **BOOM** |
+| Payload size | 52KB | 18KB | **BOOM (65% smaller)** |
+| Token count | ~13,000 | ~5,700 | **BOOM (56% fewer)** |
+| Network transfer | Slower | **Faster** | **BOOM** |
+| LLM processing | Slower | **Faster** | **BOOM** |
+| API cost | $X | **$0.44X** | **BOOM (56% cheaper)** |
+
+For the typical LLM workflow:
+- Network latency (50-200ms) dominates
+- LLM inference time (500-5000ms) is the bottleneck
+- BOOM's size reduction speeds up both network AND inference
+- BOOM decode being 2.4x faster helps when processing responses
+
+## Format Comparison
+
+BOOM vs JSON, TRON, TOON, YAML, XML, CSV, and VSC - comprehensive benchmarks across different data structures.
+
+### Token Efficiency Rankings (LLM cost savings vs JSON)
+
+| Rank | Format | Avg Token Reduction | Data Support |
+|------|--------|---------------------|--------------|
+| **1** | **BOOM binary** | **68.6%** | **All data types** |
+| 2 | VSC | 64.5% | Flat tabular only |
+| 3 | CSV | 63.8% | Flat tabular only |
+| 4 | TRON | 55.2% | Uniform arrays only |
+| 5 | TOON | 50.8% | Uniform arrays only |
+| 6 | YAML | 20.6% | All data types |
+| 7 | BOOM text | 3.9% | All data types |
+| 8 | XML | -46.3% (more!) | All data types |
+
+**Key insight:** BOOM binary beats CSV/VSC on token efficiency while supporting ALL data structures, including nested objects.
+
+### Size Efficiency (bytes saved vs JSON)
+
+| Format | Simple Data | Employees (100) | Nested Orders | Deep Config | API Response |
+|--------|-------------|-----------------|---------------|-------------|--------------|
+| **BOOM binary** | **6%** | **63%** | **53%** | **24%** | **36%** |
+| TRON | 12% | 48% | 58% | -12% | 24% |
+| TOON | 16% | 51% | 64% | -18% | 27% |
+| CSV/VSC* | - | 57% | 65% | - | - |
+| YAML | 13% | -11% | -23% | -18% | -25% |
+| XML | -67% | -87% | -124% | -106% | -157% |
+
+*CSV/VSC only work for flat tabular data
+
+### Why BOOM Beats TRON & TOON
+
+[TRON](https://github.com/nicois/tron) and [TOON](https://toonformat.dev) claim fewer tokens than JSON. Our benchmarks show:
+- **BOOM binary: 80.6% fewer tokens than JSON**
+- **TRON: 55.2% fewer tokens than JSON**
+- **TOON: 50.8% fewer tokens than JSON**
+- **BOOM uses 31.2% fewer tokens than TRON**
+- **BOOM uses 37.8% fewer tokens than TOON**
+
+| Metric | BOOM Binary | TRON | TOON | Winner |
+|--------|-------------|------|------|--------|
+| Token savings vs JSON | 80.6% | 55.2% | 50.8% | **BOOM** |
+| Nested structures | Excellent | Poor | Poor | **BOOM** |
+| Mixed data types | Full support | Limited | Limited | **BOOM** |
+| Binary data | Native | None | None | **BOOM** |
+| Schema required | No | No | No | Tie |
+| Human readable | Via .boom.txt | Yes | Yes | TRON/TOON |
+
+**TRON & TOON limitations:**
+- Only efficient for uniform arrays of objects
+- Nested structures lose efficiency
+- No binary format option
+- No string interning
+
+**BOOM advantages:**
+- Efficient for ALL data structures
+- Binary format for maximum compression
+- String interning for repeated values
+- Native BigInt and binary data support
+- Self-describing (no schema needed)
+
+### Real-World Example
+
+```javascript
+// 100-employee directory
+const data = {
+  context: { task: 'Employee directory', department: 'Engineering' },
+  employees: Array.from({ length: 100 }, (_, i) => ({
+    id: i + 1,
+    name: `Employee ${i + 1}`,
+    email: `emp${i + 1}@company.com`,
+    department: ['Engineering', 'Sales', 'Marketing', 'HR'][i % 4],
+    salary: 50000 + (i * 500),
+    active: i % 5 !== 0
+  })),
+  total: 100
+};
+
+// Results:
+// JSON:        11,165 bytes | 5,993 tokens
+// TRON:         5,812 bytes | 2,687 tokens (48% smaller, 55% fewer tokens)
+// TOON:         5,499 bytes | 2,299 tokens (51% smaller, 62% fewer tokens)
+// BOOM binary:  4,134 bytes | 1,378 tokens (63% smaller, 77% fewer tokens)
+// CSV:          5,209 bytes | 2,261 tokens (flat array only)
+// VSC:          4,850 bytes | 2,224 tokens (flat array only)
+// XML:         20,837 bytes | 8,417 tokens (87% larger!)
+```
+
+**BOOM wins on both size AND tokens** - with tabular encoding, BOOM binary beats CSV/VSC for flat data too.
+
+### When to Use Each Format
+
+| Format | Use When |
+|--------|----------|
+| **BOOM binary** | LLM API calls, AI agents, maximum efficiency |
+| **BOOM text** | Debugging, config files, human editing |
+| JSON | Interoperability with existing systems |
+| TRON | Uniform tabular data, human-readable priority |
+| TOON | Uniform tabular data, human-readable priority |
+| YAML | Human-edited config files |
+| CSV/VSC | Flat tabular data, spreadsheet export |
+| XML | Never for LLMs (2x the tokens of JSON) |
+
+Run the full comparison benchmark:
+```bash
+node benchmark/format-comparison.js
+```
+
+## Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Run tests: `npm test`
+4. Submit a merge request
+
+## License
+
+MIT © Tom Taylor
+
+## Links
+
+- **Website**: [boomformat.dev](https://boomformat.dev)
+- **npm**: [npmjs.com/package/boom-format](https://www.npmjs.com/package/boom-format)
+- **Go**: [pkg.go.dev/github.com/t0mtaylor/boom-format/go](https://pkg.go.dev/github.com/t0mtaylor/boom-format/go)
+- **Rust**: [crates.io/crates/boom-format](https://crates.io/crates/boom-format)
+- **Java**: [Maven Central](https://central.sonatype.com/artifact/io.boom-format/boom)
+- **.NET**: [NuGet](https://www.nuget.org/packages/BoomFormat)
+- **PHP**: [Packagist](https://packagist.org/packages/boom-format/boom)
+- **Swift**: [Swift Package](https://github.com/t0mtaylor/boom-format)
+- **C++**: [Header-only](https://github.com/t0mtaylor/boom-format/tree/main/plugins/cpp)
+
+---
+
+Made with ❤️ by [The Tom Taylor Company](https://tomtaylor.co.uk) • [𝕏 @tom_taylor](https://x.com/tom_taylor)
+
+![](https://st.rs.thetomtaylor.co.uk/matomo.php?idsite=28&rec=1)