slimjson 1.0.2 → 1.0.3

package/README.md

@@ -1,14 +1,17 @@
 # slimjson
 
+中文 | [English](./README_EN.md)
+
 轻量级对象数组压缩工具 — 将重复 key 的 JSON 对象数组转换为 `{ keys, rows }` 紧凑格式，并支持序列化时省略 `null` 以进一步减小体积。
 
 ## 适用场景
 
-- 后端返回列表接口时，每个对象都携带相同的 key 名，大量冗余
-- 不同对象可能拥有不同的字段（后端按需 omit null 字段）
-- 需要在网络传输中极致压缩 JSON 文本体积
+- **API 列表接口**：后端返回列表接口时，每个对象都携带相同的 key 名，大量冗余
+- **异构字段**：不同对象可能拥有不同的字段（后端按需 omit null 字段）
+- **网络传输压缩**：需要在网络传输中极致压缩 JSON 文本体积
 - **大模型上下文压缩**：将大量结构化数据（如数据库查询结果、API 响应、知识库条目）压缩后送入 prompt，减少 token 消耗，降低调用成本
 - **大模型工具调用**：function calling / tool_use 返回的结果往往是结构化的对象数组，压缩后再回传给模型，可显著减少上下文窗口占用，让模型在有限 token 内处理更复杂的数据
+- **大模型识别友好**：压缩后的 `{ keys, rows }` 格式将 schema（字段定义）与数据分离，key 只出现一次，模型能更准确地理解数据结构、按字段名提取信息，比重复 key 的原始 JSON 更不容易混淆
 
 ## 安装
 
@@ -23,7 +26,7 @@ npm install slimjson
 将对象数组压缩为 `{ keys, rows }` 结构：
 
 ```js
-const { compress } = require('slimjson');
+import { compress } from 'slimjson';
 
 const users = [
   { name: 'Alice', age: 25, city: 'NYC' },
@@ -137,6 +140,13 @@ stringify(compress(orders));
 //            ^^^^^^^^^^^^^^^^^^^^^^^^^^^ specs 中缺失字段用空槽省略 null ^^^^^^^^^^^^^^^^^^^^^^^^
 ```
 
+#### 单对象示例
+```js
+compress({ name: 'Alice', age: 25 });
+// 等价于 compress([{ name: 'Alice', age: 25 }])
+// { keys: ['name', 'age'], rows: [['Alice', 25]] }
+```
+
 ### `decompress(compressed)`
 
 将 `{ keys, rows }` 还原为原始对象数组：
@@ -148,7 +158,7 @@ const restored = decompress(compressed);
 
 ### `stringify(compressed)`
 
-将 compress 结果序列化为文本，数组中 `null` 值被省略（保留逗号占位），安全的字符串省略引号：
+将 compress 结果序列化为紧凑文本。相比 `JSON.stringify`，应用了以下优化规则：
 
 ```js
 const data = [
@@ -158,23 +168,65 @@ const data = [
 
 const text = stringify(compress(data));
 // {keys:[name,age],rows:[[Alice,25],[Bob,30]]}
-```
 
-对比 JSON.stringify：
-```js
 JSON.stringify(compress(data));
 // {"keys":["name","age"],"rows":[["Alice",25],["Bob",30]]}
-//  ↑ 引号 ↑                    ↑ 引号 ↑
 ```
 
-数组 null 省略规则：
+#### 序列化规则一览
+
+| 值类型 | 序列化结果 | 说明 |
+|--------|-----------|------|
+| `null` / `undefined` | `null` | — |
+| 有限数字 | `25` | 直接输出，无引号 |
+| `NaN` / `Infinity` | `null` | 非有限数统一输出 null |
+| `true` / `false` | `true` / `false` | — |
+| 安全字符串 | `Alice` | 省略引号（见下方规则） |
+| 非安全字符串 | `"hello world"` | 保留 JSON 引号和转义 |
+| 嵌套对象 `{k: v}` | `{k:v}` | key 同样区分安全/非安全 |
+| 数组 | 见下方 null 省略规则 | — |
+
+#### 安全字符串（可省略引号的条件）
+
+满足以下**全部**条件的字符串可省略引号，否则保留 `JSON.stringify` 转义：
+
+1. 非空字符串
+2. 不是关键字字面量：`null`、`true`、`false`
+3. 不匹配数字模式：`/^-?(0|[1-9]\d*)(\.\d+)?([eE][+-]?\d+)?$/`（如 `"123"`、`"-3.14"`、`"1e10"` 均保留引号）
+4. 不以数字或减号 `-` 开头
+5. 不含空白、`[`、`]`、`{`、`}`、`,`、`:`、`"` 等字符
+
+| 字符串 | 结果 | 原因 |
+|--------|------|------|
+| `"Alice"` | `Alice` | 安全，省略引号 |
+| `"hello world"` | `"hello world"` | 含空格 |
+| `"123"` | `"123"` | 看起来像数字 |
+| `"-3.14"` | `"-3.14"` | 看起来像数字 |
+| `"null"` | `"null"` | 关键字 |
+| `""` | `""` | 空字符串 |
+| `"-abc"` | `"-abc"` | 以减号开头 |
+| `"a:b"` | `"a:b"` | 含冒号 |
+
+#### 对象 key 引号规则
+
+`keys` 中的嵌套对象 key 同样适用安全字符串判断：
+
+```js
+stringify({ keys: [{ profile: ['name', 'age'] }], rows: [...] });
+// {keys:[{profile:[name,age]}],rows:[...]}   ← profile 是安全 key，省略引号
+
+stringify({ keys: [{ "my-key": ['name'] }], rows: [...] });
+// {keys:[{"my-key":[name]}],rows:[...]}      ← my-key 含减号，保留引号
+```
+
+#### 数组 null 省略规则
+
+数组中的 `null` / `undefined` 被省略为逗号空槽，不占文字体积：
 
 | 原始数组                 | 序列化结果      | 说明 |
 |----------------------|------------|------|
 | `["a", null, null]`  | `[a,,]`    | 尾部两个空槽 |
 | `[null, 1, null]`    | `[,1,]`    | 前后空槽 |
-| `[null, "1", null]`  | `[,"1",]`  | 前后空槽 |
-| `[null, "1a", null]` | `[,"1a",]` | 前后空槽 |
 | `[]`                 | `[]`       | 空数组 |
 | `[null]`             | `[null]`   | **特殊**：`[,]` 代表 2 个 null，因此单 null 保留文字 |
 
@@ -192,7 +244,7 @@ const parsed = parse(text);
 ## 完整使用示例
 
 ```js
-const { compress, decompress, stringify, parse } = require('slimjson');
+import { compress, decompress, stringify, parse } from 'slimjson';
 
 const data = [
   { name: '张三', age: 28, profile: { avatar: 'a.jpg', bio: 'Hello' } },

package/README_EN.md

@@ -0,0 +1,309 @@
+# slimjson
+
+[中文](./README.md) | English
+
+A lightweight object array compression tool — converts JSON object arrays with repeated keys into a compact `{ keys, rows }` format, with support for omitting `null` values during serialization to further reduce size.
+
+## Use Cases
+
+- **API List Endpoints**: Backend list endpoints where every object carries the same key names, resulting in massive redundancy
+- **Heterogeneous Fields**: Objects with different fields (backend omits null fields on demand)
+- **Network Transfer Compression**: Minimizing JSON text size for network transmission
+- **LLM Context Compression**: Compress large structured data (e.g. database query results, API responses, knowledge base entries) before sending to prompts, reducing token consumption and API costs
+- **LLM Tool Calling**: function calling / tool_use results are often structured object arrays — compressing them before feeding back to the model significantly reduces context window usage, enabling the model to handle more complex data within limited tokens
+- **LLM-Friendly Format**: The compressed `{ keys, rows }` format separates schema (field definitions) from data, with each key appearing only once. Models can more accurately understand data structures and extract information by field name, with less confusion compared to raw JSON with repeated keys
+
+## Installation
+
+```bash
+npm install slimjson
+```
+
+## API
+
+### `compress(source)`
+
+Compresses an object array into a `{ keys, rows }` structure:
+
+```js
+import { compress } from 'slimjson';
+
+const users = [
+  { name: 'Alice', age: 25, city: 'NYC' },
+  { name: 'Bob',   age: 30, city: 'LA' },
+];
+
+const compressed = compress(users);
+// {
+//   keys: ['name', 'age', 'city'],
+//   rows: [
+//     ['Alice', 25, 'NYC'],
+//     ['Bob',   30, 'LA' ]
+//   ]
+// }
+```
+
+**Features:**
+- `keys` takes the union of all object keys, ordered by first appearance
+- Missing fields in an object → fill `null` at the corresponding row position
+- Nested objects are recursively processed: represented as `{ "fieldName": [childKeys] }` in `keys`
+- Object arrays (e.g. order items) are recursively compressed the same way
+- When a plain object is passed (not an array), it is treated as a single-element array
+
+```js
+compress({ name: 'Alice', age: 25 });
+// Equivalent to compress([{ name: 'Alice', age: 25 }])
+// { keys: ['name', 'age'], rows: [['Alice', 25]] }
+```
+
+#### Nested Object Example
+
+```js
+const data = [
+  { name: 'Alice', age: 28, profile: { avatar: 'a.jpg', bio: 'Hello' } },
+  { name: 'Bob',   age: 35, profile: { avatar: 'b.jpg' } },  // missing bio
+];
+
+compress(data);
+// {
+//   keys: ['name', 'age', { profile: ['avatar', 'bio'] }],
+//   rows: [
+//     ['Alice', 28, ['a.jpg', 'Hello']],
+//     ['Bob',   35, ['b.jpg', null    ]]
+//   ]
+// }
+
+stringify(compress(data));
+// {keys:[name,age,{profile:[avatar,bio]}],rows:[[Alice,28,[a.jpg,Hello]],[Bob,35,[b.jpg,]]]}
+//                                                                                         ^^ null omitted, comma retained
+```
+
+#### Object Array Example (Order Scenario)
+
+```js
+const orders = [
+  { orderId: 'A001', items: [{ name: 'Keyboard', price: 299 }, { name: 'Mouse', price: 99 }] },
+  { orderId: 'A002', items: [{ name: 'Monitor', price: 1999 }] },
+];
+
+compress(orders);
+// {
+//   keys: ['orderId', { items: ['name', 'price'] }],
+//   rows: [
+//     ['A001', [['Keyboard', 299], ['Mouse', 99]]],
+//     ['A002', [['Monitor', 1999]]]
+//   ]
+// }
+
+stringify(compress(orders));
+// {keys:[orderId,{items:[name,price]}],rows:[[A001,[[Keyboard,299],[Mouse,99]]],[A002,[[Monitor,1999]]]]}
+//                ^^^^^ nested object key, no quotes    ^^^^ safe string value, no quotes
+```
+
+#### Three-Level Nesting Example (Order → Item → Specs)
+
+```js
+const orders = [
+  {
+    orderId: 'A001',
+    customer: 'Alice',
+    items: [
+      { name: 'Keyboard', price: 299, specs: { color: 'Black', layout: '104-key' } },
+      { name: 'Mouse',    price: 99,  specs: { color: 'White', dpi: '4000' } },
+    ]
+  },
+  {
+    orderId: 'A002',
+    customer: 'Bob',
+    items: [
+      { name: 'Monitor', price: 1999, specs: { color: 'Silver', size: '27in' } },
+    ]
+  },
+];
+
+compress(orders);
+// {
+//   keys: [
+//     'orderId',
+//     'customer',
+//     { items: ['name', 'price', { specs: ['color', 'layout', 'dpi', 'size'] }] }
+//   ],
+//   rows: [
+//     ['A001', 'Alice', [
+//       ['Keyboard', 299, ['Black', '104-key', null, null]],
+//       ['Mouse',    99,  ['White', null, '4000', null]]
+//     ]],
+//     ['A002', 'Bob', [
+//       ['Monitor', 1999, ['Silver', null, null, '27in']]
+//     ]]
+//   ]
+// }
+// specs keys take the union: order 1 has layout, order 2 has size → both kept, missing fields filled with null
+
+stringify(compress(orders));
+// {keys:[orderId,customer,{items:[name,price,{specs:[color,layout,dpi,size]}]}],rows:[[
+//   A001,Alice,[[Keyboard,299,[Black,104-key,,]],[Mouse,99,[White,,4000,]]]],[A002,Bob,[[Monitor,1999,[Silver,,,27in]]]]]}
+//            ^^^^^^^^^^^^^^^^^^^^^^^^^^^ missing fields in specs omitted as empty slots ^^^^^^^^^^^^^^^^^^^^^^^^
+```
+
+### `decompress(compressed)`
+
+Restores `{ keys, rows }` back to the original object array:
+
+```js
+const restored = decompress(compressed);
+// deep-equal to the original array
+```
+
+### `stringify(compressed)`
+
+Serializes the compress result into compact text. Compared to `JSON.stringify`, the following optimization rules are applied:
+
+```js
+const data = [
+  { name: 'Alice', age: 25 },
+  { name: 'Bob',   age: 30 },
+];
+
+const text = stringify(compress(data));
+// {keys:[name,age],rows:[[Alice,25],[Bob,30]]}
+
+JSON.stringify(compress(data));
+// {"keys":["name","age"],"rows":[["Alice",25],["Bob",30]]}
+```
+
+#### Serialization Rules
+
+| Value Type | Serialized Result | Notes |
+|------------|------------------|-------|
+| `null` / `undefined` | `null` | — |
+| Finite number | `25` | Direct output, no quotes |
+| `NaN` / `Infinity` | `null` | Non-finite numbers unified to null |
+| `true` / `false` | `true` / `false` | — |
+| Safe string | `Alice` | Quotes omitted (see rules below) |
+| Unsafe string | `"hello world"` | JSON quotes and escaping retained |
+| Nested object `{k: v}` | `{k:v}` | Keys follow same safe/unsafe rules |
+| Array | See null omission rules below | — |
+
+#### Safe Strings (Conditions for Omitting Quotes)
+
+A string can omit quotes only when it satisfies **all** of the following conditions; otherwise `JSON.stringify` escaping is applied:
+
+1. Non-empty string
+2. Not a keyword literal: `null`, `true`, `false`
+3. Does not match number pattern: `/^-?(0|[1-9]\d*)(\.\d+)?([eE][+-]?\d+)?$/` (e.g. `"123"`, `"-3.14"`, `"1e10"` all retain quotes)
+4. Does not start with a digit or minus sign `-`
+5. Does not contain whitespace, `[`, `]`, `{`, `}`, `,`, `:`, `"` or similar characters
+
+| String | Result | Reason |
+|--------|--------|--------|
+| `"Alice"` | `Alice` | Safe, quotes omitted |
+| `"hello world"` | `"hello world"` | Contains space |
+| `"123"` | `"123"` | Looks like a number |
+| `"-3.14"` | `"-3.14"` | Looks like a number |
+| `"null"` | `"null"` | Keyword |
+| `""` | `""` | Empty string |
+| `"-abc"` | `"-abc"` | Starts with minus sign |
+| `"a:b"` | `"a:b"` | Contains colon |
+
+#### Object Key Quoting Rules
+
+Nested object keys in `keys` follow the same safe string check:
+
+```js
+stringify({ keys: [{ profile: ['name', 'age'] }], rows: [...] });
+// {keys:[{profile:[name,age]}],rows:[...]}   ← profile is safe, quotes omitted
+
+stringify({ keys: [{ "my-key": ['name'] }], rows: [...] });
+// {keys:[{"my-key":[name]}],rows:[...]}      ← my-key contains hyphen, quotes retained
+```
+
+#### Array Null Omission Rules
+
+`null` / `undefined` values in arrays are omitted as comma slots, taking no text space:
+
+| Original Array | Serialized Result | Notes |
+|---------------|------------------|-------|
+| `["a", null, null]` | `[a,,]` | Two trailing empty slots |
+| `[null, 1, null]` | `[,1,]` | Leading and trailing empty slots |
+| `[]` | `[]` | Empty array |
+| `[null]` | `[null]` | **Special**: `[,]` means 2 nulls, so single null retains literal |
+
+### `parse(text)`
+
+Parses text produced by `stringify`, restoring omitted `null` values:
+
+```js
+const parsed = parse(text);
+// deep-equal to compressed
+```
+
+Supports full JSON type parsing (strings, numbers, booleans, null, nested objects/arrays), compatible with escape characters and Unicode.
+
+## Complete Example
+
+```js
+import { compress, decompress, stringify, parse } from 'slimjson';
+
+const data = [
+  { name: 'Alice', age: 28, profile: { avatar: 'a.jpg', bio: 'Hello' } },
+  { name: 'Bob',   age: 35, profile: { avatar: 'b.jpg' } },              // missing bio
+];
+
+// Compress → Stringify → Parse → Decompress
+const compressed = compress(data);
+const text       = stringify(compressed);
+const parsed     = parse(text);
+const restored   = decompress(parsed);
+
+// restored is deep-equal to data
+```
+
+### Compression Ratio Calculation
+
+```js
+const originalSize   = Buffer.byteLength(JSON.stringify(data));
+const compressedSize = Buffer.byteLength(stringify(compress(data)));
+const ratio = ((originalSize - compressedSize) / originalSize * 100).toFixed(1);
+console.log(`Compression ratio: ${ratio}%`);
+```
+
+## Compression Results
+
+Based on actual data from `compress-test.js` benchmarks (18 test cases, average compression ratio **52.20%**, all roundtrip decompressions verified):
+
+| Data Type | Object Count | Original Size | Compressed | Ratio |
+|-----------|-------------|---------------|------------|-------|
+| Simple users | 1,000 | 147.85 KB | 87.36 KB | **40.91%** |
+| Simple users | 10,000 | 1.45 MB | 882.34 KB | **40.69%** |
+| Nested users (with profile.social) | 1,000 | 235.28 KB | 153.03 KB | **34.96%** |
+| Orders (1-5 items per order) | 500 | 166.34 KB | 72.10 KB | **56.66%** |
+| School data (6 grades x 4 classes x 30 students) | 24 | 215.47 KB | 88.39 KB | **58.98%** |
+| Sparse fields (500 records x 30 fields) | 500 | 144.68 KB | 45.39 KB | **68.62%** |
+| Sparse fields (2000 records x 50 fields) | 2,000 | 947.88 KB | 292.74 KB | **69.12%** |
+| Deep nesting (5-level org structure) | 5 | 634.65 KB | 288.75 KB | **54.50%** |
+
+**Conclusions:**
+1. Longer field names and more fields yield better compression
+2. Object arrays (order items, student lists) show significant compression (55–59%)
+3. Sparse fields achieve the highest compression — missing field nulls omitted as empty slots (67–69%)
+4. Deeper nested structures achieve better compression
+5. `stringify` quote omission further reduces text size
+
+## Development
+
+```bash
+# Run tests
+npm test
+
+# Run compression ratio benchmarks
+node compress-test.js
+```
+
+## GitHub
+
+[https://github.com/LastHeaven/slimjson](https://github.com/LastHeaven/slimjson)
+
+## License
+
+MIT

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "slimjson",
-  "version": "1.0.2",
+  "version": "1.0.3",
   "main": "compress.js",
   "exports": {
     ".": {