json-toon-parser 1.1.1 β 1.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +36 -0
- package/README.md +138 -42
- package/dist/cli.js +11 -1
- package/dist/cli.js.map +1 -1
- package/docs/API.md +185 -0
- package/package.json +90 -10
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
## [1.1.0] - 2025-11-22
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
- π **Web UI Server** - Beautiful browser-based interface for JSON to TOON conversion
|
|
9
|
+
- β‘ **`serve` Command** - Start a local web server with `json-toon serve`
|
|
10
|
+
- π **Real-time Conversion** - Instant token analysis and optimization in the browser
|
|
11
|
+
- πΎ **Download Feature** - Download converted TOON files directly from the UI
|
|
12
|
+
- π¨ **Sample Data** - Pre-loaded examples (Simple, Array, Nested, Complex)
|
|
13
|
+
- π **Copy/Paste** - Easy copy buttons for JSON and TOON output
|
|
14
|
+
- β¨ **JSON Formatter** - Auto-format messy JSON with one click
|
|
15
|
+
- π± **Responsive Design** - Works on desktop, tablet, and mobile devices
|
|
16
|
+
|
|
17
|
+
### Changed
|
|
18
|
+
- Updated CLI with new `serve` command
|
|
19
|
+
- Enhanced README with Web UI documentation
|
|
20
|
+
- Updated package description
|
|
21
|
+
- Added `public/` directory to published files
|
|
22
|
+
- Improved server output with beautiful ASCII art banner
|
|
23
|
+
|
|
24
|
+
### Technical
|
|
25
|
+
- Added `server.ts` for HTTP server functionality
|
|
26
|
+
- Created standalone `public/index.html` with embedded TOON library
|
|
27
|
+
- Updated TypeScript build configuration
|
|
28
|
+
- Added new npm scripts: `serve` and `dev`
|
|
29
|
+
|
|
30
|
+
## [1.0.1] - Previous Release
|
|
31
|
+
|
|
32
|
+
### Features
|
|
33
|
+
- CLI tool for JSON to TOON conversion
|
|
34
|
+
- Compare, convert, and update commands
|
|
35
|
+
- Token efficiency analysis
|
|
36
|
+
- Programmatic API support
|
package/README.md
CHANGED
|
@@ -1,18 +1,25 @@
|
|
|
1
1
|
# json-toon-parser
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**Version:** 1.1.3 β’ **License:** MIT β’ **Format:** TOON
|
|
4
|
+
|
|
5
|
+
High-efficiency conversion and analysis toolkit: JSON β TOON with CLI, API, and Web UI server for realβtime token savings. Ideal for reducing LLM prompt payload size and optimizing costs.
|
|
4
6
|
|
|
5
7
|
## Installation
|
|
8
|
+
Global (CLI + Web UI):
|
|
6
9
|
```bash
|
|
7
10
|
npm install -g json-toon-parser
|
|
8
11
|
```
|
|
12
|
+
Project (Programmatic API):
|
|
13
|
+
```bash
|
|
14
|
+
npm install json-toon-parser
|
|
15
|
+
```
|
|
9
16
|
|
|
10
|
-
## Quick Start
|
|
11
|
-
|
|
17
|
+
## Quick Start β Web UI
|
|
18
|
+
Start the local server:
|
|
12
19
|
```bash
|
|
13
20
|
json-toon serve
|
|
14
21
|
```
|
|
15
|
-
|
|
22
|
+
Browse to: `http://localhost:3000` β paste JSON β convert β copy/download TOON.
|
|
16
23
|
|
|
17
24
|
## CLI Usage
|
|
18
25
|
```bash
|
|
@@ -50,46 +57,114 @@ const toonString = jsonToToon({ hello: "world" });
|
|
|
50
57
|
|
|
51
58
|
## Features
|
|
52
59
|
|
|
53
|
-
- π
|
|
54
|
-
- β
|
|
55
|
-
- π
|
|
56
|
-
- πΎ
|
|
57
|
-
- π§ CLI
|
|
58
|
-
- π¦ Programmatic API
|
|
59
|
-
-
|
|
60
|
-
-
|
|
60
|
+
- π Web UI β Responsive, modern conversion dashboard
|
|
61
|
+
- β
Direct JSON β TOON conversion (string or file)
|
|
62
|
+
- π Token efficiency metrics (JSON vs TOON counts, savings %)
|
|
63
|
+
- πΎ Timestamped output file generation
|
|
64
|
+
- π§ CLI commands: `serve`, `compare`, `convert`, `update`
|
|
65
|
+
- π¦ Programmatic API with types & tree-shakeable exports
|
|
66
|
+
- β‘ Real-time stats + sample datasets in UI
|
|
67
|
+
- π§ͺ GPT-style token counting via `gpt-3-encoder`
|
|
68
|
+
|
|
69
|
+
## API (Summary)
|
|
70
|
+
# json-toon-parser
|
|
71
|
+
|
|
72
|
+
[](https://www.npmjs.com/package/json-toon-parser)
|
|
73
|
+
[](https://www.npmjs.com/package/json-toon-parser)
|
|
74
|
+
[](https://opensource.org/licenses/MIT)
|
|
75
|
+
|
|
76
|
+
================================================================================
|
|
77
|
+
|
|
78
|
+
π json-toon-parser β JSON β TOON converter, token optimizer, and developer toolkit
|
|
79
|
+
|
|
80
|
+
Convert verbose JSON into compact TOON, reduce LLM token usage, and measure savings.
|
|
81
|
+
|
|
82
|
+
Perfect for teams and engineers integrating large structured payloads into ChatGPT/OpenAI/Claude prompts or storage pipelines.
|
|
83
|
+
|
|
84
|
+
Official NPM package: https://www.npmjs.com/package/json-toon-parser
|
|
85
|
+
|
|
86
|
+
Documentation: `docs/API.md`
|
|
87
|
+
|
|
88
|
+
================================================================================
|
|
89
|
+
|
|
90
|
+
## Quick Install
|
|
91
|
+
|
|
92
|
+
Global (CLI + Web UI):
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
npm install -g json-toon-parser
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
Programmatic (project):
|
|
99
|
+
|
|
100
|
+
```bash
|
|
101
|
+
npm install json-toon-parser
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Verify:
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
json-toon --version
|
|
108
|
+
json-toon serve
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
================================================================================
|
|
112
|
+
|
|
113
|
+
## Why TOON? β The Value Proposition
|
|
114
|
+
|
|
115
|
+
TOON is a compact, human-friendly serialization optimized for token-efficient LLM prompts. It removes syntactic noise (quotes, braces, commas), reduces repeated keys, and uses concise array and nesting notation β which directly translates to lower API costs and faster transmission.
|
|
116
|
+
|
|
117
|
+
- Reduce per-request token counts by 40β70% (depends on structure)
|
|
118
|
+
- Keep payloads human-readable and reversible via the TOON library
|
|
119
|
+
- Ideal for prompts, telemetry, logs, and mobile/edge transfers
|
|
120
|
+
|
|
121
|
+
================================================================================
|
|
122
|
+
|
|
123
|
+
## Features
|
|
124
|
+
|
|
125
|
+
- CLI: `serve`, `compare`, `convert`, `update`
|
|
126
|
+
- Web UI: interactive editor, samples, copy & download
|
|
127
|
+
- Programmatic API (TypeScript types included)
|
|
128
|
+
- Token analysis using `gpt-3-encoder` for GPT-style counts
|
|
129
|
+
- Save TOON output with timestamped filenames
|
|
61
130
|
|
|
62
|
-
|
|
131
|
+
================================================================================
|
|
63
132
|
|
|
64
|
-
|
|
65
|
-
Compare JSON data with TOON format and calculate token efficiency.
|
|
133
|
+
## Usage Examples
|
|
66
134
|
|
|
67
|
-
|
|
68
|
-
- `jsonData`: Any JSON-serializable data
|
|
69
|
-
- `options`: Optional configuration
|
|
70
|
-
- `outputDir`: Output directory (default: "output")
|
|
71
|
-
- `saveToFile`: Save TOON file (default: false)
|
|
72
|
-
- `prettyJson`: Pretty print JSON (default: true)
|
|
73
|
-
- `fileName`: Custom filename
|
|
135
|
+
CLI
|
|
74
136
|
|
|
75
|
-
|
|
137
|
+
```bash
|
|
138
|
+
# Start web UI
|
|
139
|
+
json-toon serve
|
|
76
140
|
|
|
77
|
-
|
|
78
|
-
|
|
141
|
+
# Compare & save TOON
|
|
142
|
+
json-toon compare data/input.json --save
|
|
79
143
|
|
|
80
|
-
|
|
81
|
-
|
|
144
|
+
# Convert file
|
|
145
|
+
json-toon convert data/input.json output.toon
|
|
82
146
|
|
|
83
|
-
|
|
84
|
-
|
|
147
|
+
# Update JSON and produce comparison
|
|
148
|
+
json-toon update data/input.json
|
|
149
|
+
```
|
|
85
150
|
|
|
86
|
-
|
|
87
|
-
Update JSON file and generate TOON comparison.
|
|
151
|
+
Programmatic
|
|
88
152
|
|
|
89
|
-
|
|
90
|
-
|
|
153
|
+
```ts
|
|
154
|
+
import { compare, jsonToToon } from 'json-toon-parser';
|
|
155
|
+
|
|
156
|
+
const data = { users: [{ id: 1, name: 'Alice' }] };
|
|
157
|
+
const toon = jsonToToon(data);
|
|
158
|
+
const result = compare(data, { saveToFile: false });
|
|
159
|
+
console.log(result.toonOutput);
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
================================================================================
|
|
163
|
+
|
|
164
|
+
## Example Comparison
|
|
165
|
+
|
|
166
|
+
Use this example to demonstrate typical savings β include it in docs, blog posts, or README highlights:
|
|
91
167
|
|
|
92
|
-
## Example Output
|
|
93
168
|
```
|
|
94
169
|
============================================================
|
|
95
170
|
COMPARISON RESULTS
|
|
@@ -102,18 +177,39 @@ Output File: output/data_DateTime.toon
|
|
|
102
177
|
============================================================
|
|
103
178
|
```
|
|
104
179
|
|
|
105
|
-
|
|
180
|
+
================================================================================
|
|
181
|
+
|
|
182
|
+
## When TOON Is Most Effective
|
|
183
|
+
|
|
184
|
+
- Nested JSON with many structural characters
|
|
185
|
+
- Large arrays of similar objects
|
|
186
|
+
- Repetitive keys repeated across objects
|
|
187
|
+
- Prompts that include large contexts for LLMs
|
|
188
|
+
|
|
189
|
+
================================================================================
|
|
190
|
+
|
|
191
|
+
## Performance Notes
|
|
106
192
|
|
|
107
|
-
-
|
|
108
|
-
- πΎ **Data Compression**: Convert JSON to more efficient TOON format
|
|
109
|
-
- π **Batch Processing**: Process multiple JSON files
|
|
110
|
-
- π οΈ **CI/CD Integration**: Automate format conversion
|
|
111
|
-
- π **Cost Optimization**: Reduce API token usage for LLMs
|
|
193
|
+
Token counting is approximated with `gpt-3-encoder`. Results may vary slightly by tokenizers used by different LLM providers, but the relative savings are consistent.
|
|
112
194
|
|
|
113
|
-
|
|
195
|
+
================================================================================
|
|
114
196
|
|
|
115
|
-
|
|
197
|
+
## Links & Resources
|
|
198
|
+
|
|
199
|
+
- NPM: https://www.npmjs.com/package/json-toon-parser
|
|
200
|
+
- TOON format: https://github.com/toon-format/toon
|
|
201
|
+
- API docs: `docs/API.md`
|
|
202
|
+
|
|
203
|
+
================================================================================
|
|
204
|
+
|
|
205
|
+
## Contributing
|
|
206
|
+
|
|
207
|
+
See `CONTRIBUTING.md` to get started. We welcome PRs, issues, and ideas.
|
|
208
|
+
|
|
209
|
+
================================================================================
|
|
116
210
|
|
|
117
211
|
## License
|
|
118
212
|
|
|
119
213
|
MIT Β© Bhushan Chaudhari
|
|
214
|
+
|
|
215
|
+
================================================================================
|
package/dist/cli.js
CHANGED
|
@@ -7,10 +7,20 @@ import path from "node:path";
|
|
|
7
7
|
const __filename = fileURLToPath(import.meta.url);
|
|
8
8
|
const __dirname = path.dirname(__filename);
|
|
9
9
|
const program = new Command();
|
|
10
|
+
// Dynamically derive version from package.json to avoid manual sync issues.
|
|
11
|
+
import { createRequire } from "node:module";
|
|
12
|
+
const require = createRequire(import.meta.url);
|
|
13
|
+
let pkgVersion = "0.0.0";
|
|
14
|
+
try {
|
|
15
|
+
const pkg = require("../package.json");
|
|
16
|
+
pkgVersion = pkg.version || pkgVersion;
|
|
17
|
+
}
|
|
18
|
+
catch {
|
|
19
|
+
}
|
|
10
20
|
program
|
|
11
21
|
.name("json-toon")
|
|
12
22
|
.description("Compare and convert JSON to TOON format")
|
|
13
|
-
.version(
|
|
23
|
+
.version(pkgVersion);
|
|
14
24
|
program
|
|
15
25
|
.command("compare")
|
|
16
26
|
.description("Compare JSON file with TOON format")
|
package/dist/cli.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"cli.js","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AACA,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AACpC,OAAO,EAAE,WAAW,EAAE,WAAW,EAAE,YAAY,EAAE,cAAc,EAAE,MAAM,YAAY,CAAC;AACpF,OAAO,EAAE,KAAK,EAAE,MAAM,oBAAoB,CAAC;AAC3C,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AACzC,OAAO,IAAI,MAAM,WAAW,CAAC;AAE7B,MAAM,UAAU,GAAG,aAAa,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AAClD,MAAM,SAAS,GAAG,IAAI,CAAC,OAAO,CAAC,UAAU,CAAC,CAAC;AAE3C,MAAM,OAAO,GAAG,IAAI,OAAO,EAAE,CAAC;AAE9B,OAAO;KACJ,IAAI,CAAC,WAAW,CAAC;KACjB,WAAW,CAAC,yCAAyC,CAAC;KACtD,OAAO,CAAC,
|
|
1
|
+
{"version":3,"file":"cli.js","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AACA,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AACpC,OAAO,EAAE,WAAW,EAAE,WAAW,EAAE,YAAY,EAAE,cAAc,EAAE,MAAM,YAAY,CAAC;AACpF,OAAO,EAAE,KAAK,EAAE,MAAM,oBAAoB,CAAC;AAC3C,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AACzC,OAAO,IAAI,MAAM,WAAW,CAAC;AAE7B,MAAM,UAAU,GAAG,aAAa,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AAClD,MAAM,SAAS,GAAG,IAAI,CAAC,OAAO,CAAC,UAAU,CAAC,CAAC;AAE3C,MAAM,OAAO,GAAG,IAAI,OAAO,EAAE,CAAC;AAE9B,4EAA4E;AAC5E,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAC5C,MAAM,OAAO,GAAG,aAAa,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AAC/C,IAAI,UAAU,GAAG,OAAO,CAAC;AACzB,IAAI,CAAC;IACH,MAAM,GAAG,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAAC;IACvC,UAAU,GAAG,GAAG,CAAC,OAAO,IAAI,UAAU,CAAC;AACzC,CAAC;AAAC,MAAM,CAAC;AACT,CAAC;AAED,OAAO;KACJ,IAAI,CAAC,WAAW,CAAC;KACjB,WAAW,CAAC,yCAAyC,CAAC;KACtD,OAAO,CAAC,UAAU,CAAC,CAAC;AAEvB,OAAO;KACJ,OAAO,CAAC,SAAS,CAAC;KAClB,WAAW,CAAC,oCAAoC,CAAC;KACjD,QAAQ,CAAC,QAAQ,EAAE,sBAAsB,CAAC;KAC1C,MAAM,CAAC,oBAAoB,EAAE,kBAAkB,EAAE,QAAQ,CAAC;KAC1D,MAAM,CAAC,YAAY,EAAE,gBAAgB,EAAE,KAAK,CAAC;KAC7C,MAAM,CAAC,CAAC,IAAY,EAAE,OAA0C,EAAE,EAAE;IACnE,IAAI,CAAC;QACH,MAAM,MAAM,GAAG,WAAW,CAAC,IAAI,EAAE;YAC/B,SAAS,EAAE,OAAO,CAAC,MAAM;YACzB,UAAU,EAAE,OAAO,CAAC,IAAI;SACzB,CAAC,CAAC;QACH,OAAO,CAAC,GAAG,CAAC,YAAY,CAAC,MAAM,CAAC,CAAC,CAAC;IACpC,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,OAAO,CAAC,KAAK,CAAC,QAAQ,EAAE,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC;QACxE,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;AACH,CAAC,CAAC,CAAC;AAEL,OAAO;KACJ,OAAO,CAAC,SAAS,CAAC;KAClB,WAAW,CAAC,kCAAkC,CAAC;KAC/C,QAAQ,CAAC,SAAS,EAAE,iBAAiB,CAAC;KACtC,QAAQ,CAAC,UAAU,EAAE,kBAAkB,CAAC;KACxC,MAAM,CAAC,CAAC,KAAa,EAAE,MAAe,EAAE,EAAE;IACzC,IAAI,CAAC;QACH,MAAM,UAAU,GAAG,WAAW,CAAC,KAAK,EAAE,MAAM,CAAC,CAAC;QAC9C,OAAO,CAAC,GAAG,CAAC,2BAA2B,CAAC,CAAC;QACzC,OAAO,CAAC,GAAG,CAAC,cAAc,UAAU,EAAE,CAAC,CAAC;IAC1C,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,OAAO,CAAC,KAAK,CAAC,QAAQ,EAAE,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC;QACxE,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;AACH,CAAC,CAAC,CAAC;AAEL,OAAO;KACJ,OAAO,CAAC,QAAQ,CAAC;KACjB,WAAW,CAAC,+CAA+C,CAAC;KAC5D,QAAQ,CAAC,QAAQ,EAAE,qBAAqB,CAAC;KACzC,MAAM,CAAC,qBAAqB,EAAE,8CAA8C,CAAC;KAC7E,MAAM,CAAC,CAAC,IAAY,EAAE,OAA4B,EAAE,EAAE;IACrD,IAAI,CAAC;QACH,MAAM,MAAM,GAAG,cAAc,CAAC,IAAI,EAAE,OAAO,CAAC,MAAM,CAAC,CAAC;QACpD,OAAO,CAAC,GAAG,CAAC,sBAAsB,CAAC,CAAC;QACpC,OAAO,CAAC,GAAG,CAAC,YAAY,CAAC,MAAM,CAAC,CAAC,CAAC;QAClC,OAAO,CAAC,GAAG,CAAC,oBAAoB,MAAM,CAAC,eAAe,EAAE,CAAC,CAAC;IAC5D,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,OAAO,CAAC,KAAK,CAAC,QAAQ,EAAE,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC;QACxE,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;AACH,CAAC,CAAC,CAAC;AAEL,OAAO;KACJ,OAAO,CAAC,OAAO,CAAC;KAChB,WAAW,CAAC,iDAAiD,CAAC;KAC9D,MAAM,CAAC,mBAAmB,EAAE,aAAa,EAAE,MAAM,CAAC;KAClD,MAAM,CAAC,CAAC,OAAyB,EAAE,EAAE;IACpC,MAAM,UAAU,GAAG,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,WAAW,CAAC,CAAC;IACrD,MAAM,GAAG,GAAG,EAAE,GAAG,OAAO,CAAC,GAAG,EAAE,IAAI,EAAE,OAAO,CAAC,IAAI,EAAE,CAAC;IAEnD,MAAM,aAAa,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,UAAU,CAAC,EAAE;QAChD,GAAG;QACH,KAAK,EAAE,SAAS;KACjB,CAAC,CAAC;IAEH,aAAa,CAAC,EAAE,CAAC,OAAO,EAAE,CAAC,KAAK,EAAE,EAAE;QAClC,OAAO,CAAC,KAAK,CAAC,yBAAyB,EAAE,KAAK,CAAC,OAAO,CAAC,CAAC;QACxD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC,CAAC,CAAC;AACL,CAAC,CAAC,CAAC;AAEL,OAAO,CAAC,KAAK,EAAE,CAAC"}
|
package/docs/API.md
ADDED
|
@@ -0,0 +1,185 @@
|
|
|
1
|
+
# json-toon-parser API Documentation
|
|
2
|
+
|
|
3
|
+
Version: 1.1.2
|
|
4
|
+
|
|
5
|
+
The `json-toon-parser` package provides utilities to convert JSON data into the TOON format and analyze token efficiency. It exposes a CLI, programmatic API, and a Web UI server.
|
|
6
|
+
|
|
7
|
+
## Table of Contents
|
|
8
|
+
- Core Concepts
|
|
9
|
+
- Installation
|
|
10
|
+
- CLI Usage
|
|
11
|
+
- Web UI Server
|
|
12
|
+
- Programmatic API
|
|
13
|
+
- compare
|
|
14
|
+
- compareFile
|
|
15
|
+
- jsonToToon
|
|
16
|
+
- convertFile
|
|
17
|
+
- updateJsonFile
|
|
18
|
+
- formatResult
|
|
19
|
+
- Types
|
|
20
|
+
- Error Handling
|
|
21
|
+
- Performance Notes
|
|
22
|
+
- Examples
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
## Core Concepts
|
|
26
|
+
TOON is a compact serialization format aimed at reducing token counts when sending structured data to LLMs. This library allows:
|
|
27
|
+
- Measuring token savings between pretty JSON and TOON.
|
|
28
|
+
- Converting individual JSON objects or whole files.
|
|
29
|
+
- Serving a browser-based interface for interactive conversion.
|
|
30
|
+
|
|
31
|
+
## Installation
|
|
32
|
+
```bash
|
|
33
|
+
npm install -g json-toon-parser
|
|
34
|
+
# or locally
|
|
35
|
+
npm install json-toon-parser
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## CLI Usage
|
|
39
|
+
```bash
|
|
40
|
+
# Start Web UI on default port 3000
|
|
41
|
+
json-toon serve
|
|
42
|
+
# Custom port
|
|
43
|
+
json-toon serve --port 8080
|
|
44
|
+
|
|
45
|
+
# Compare a JSON file and save TOON output
|
|
46
|
+
json-toon compare data.json --save
|
|
47
|
+
|
|
48
|
+
# Convert JSON file directly
|
|
49
|
+
json-toon convert data.json output.toon
|
|
50
|
+
|
|
51
|
+
# Update JSON file and output comparison stats
|
|
52
|
+
json-toon update data.json
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
## Web UI Server
|
|
56
|
+
The `serve` command launches a lightweight HTTP server that serves `public/index.html` and provides a POST `/api/convert` endpoint.
|
|
57
|
+
- Interactive editor for JSON input and TOON output.
|
|
58
|
+
- Shows token metrics: JSON tokens, TOON tokens, savings, efficiency percentage.
|
|
59
|
+
- Provides sample datasets, formatting, copy, and download capabilities.
|
|
60
|
+
|
|
61
|
+
## Programmatic API
|
|
62
|
+
Import functions from the root export:
|
|
63
|
+
```ts
|
|
64
|
+
import { compare, compareFile, jsonToToon, convertFile, updateJsonFile, formatResult } from 'json-to-toon-parser';
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
### compare(jsonData, options?)
|
|
68
|
+
Compares raw JSON-serializable data to its TOON representation.
|
|
69
|
+
|
|
70
|
+
Parameters:
|
|
71
|
+
- `jsonData: unknown` β Any serializable data.
|
|
72
|
+
- `options?: CompareOptions`
|
|
73
|
+
- `outputDir?: string` (default `output`) β Directory for saved TOON file.
|
|
74
|
+
- `saveToFile?: boolean` (default `false`) β Whether to write TOON to disk.
|
|
75
|
+
- `prettyJson?: boolean` (default `true`) β Whether to pretty-print JSON when counting tokens.
|
|
76
|
+
- `fileName?: string` β Custom base file name.
|
|
77
|
+
|
|
78
|
+
Returns `ComparisonResult`:
|
|
79
|
+
- `jsonTokens: number`
|
|
80
|
+
- `toonTokens: number`
|
|
81
|
+
- `savings: number`
|
|
82
|
+
- `savingsPercentage: number`
|
|
83
|
+
- `toonOutput: string`
|
|
84
|
+
- `toonOutputPath?: string`
|
|
85
|
+
|
|
86
|
+
Usage:
|
|
87
|
+
```ts
|
|
88
|
+
const res = compare({ user: 'alice', roles: ['admin','editor'] }, { saveToFile: true });
|
|
89
|
+
console.log(res.savingsPercentage);
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
### compareFile(filePath, options?)
|
|
93
|
+
Loads JSON from a file, performs comparison, and always saves TOON output.
|
|
94
|
+
|
|
95
|
+
```ts
|
|
96
|
+
const res = compareFile('data/input.json', { outputDir: 'out' });
|
|
97
|
+
```
|
|
98
|
+
Throws if the file does not exist or contains invalid JSON.
|
|
99
|
+
|
|
100
|
+
### jsonToToon(jsonData)
|
|
101
|
+
Returns only the TOON string.
|
|
102
|
+
```ts
|
|
103
|
+
const toon = jsonToToon({ a: 1, b: 2 });
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
### convertFile(inputPath, outputPath?)
|
|
107
|
+
Converts a JSON file to TOON and writes the result. Generates a timestamped path when `outputPath` is omitted.
|
|
108
|
+
```ts
|
|
109
|
+
const outPath = convertFile('data/input.json');
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
### updateJsonFile(filePath, outputPath?)
|
|
113
|
+
Runs comparison on the existing JSON file and then rewrites (or copies) the JSON file (not mutated contentβkept formatted). Returns `ComparisonResult & { updatedJsonPath: string }`.
|
|
114
|
+
```ts
|
|
115
|
+
const updated = updateJsonFile('data/input.json');
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
### formatResult(result)
|
|
119
|
+
Formats a `ComparisonResult` into a readable multiline string for terminal output.
|
|
120
|
+
```ts
|
|
121
|
+
console.log(formatResult(res));
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
## Types
|
|
125
|
+
```ts
|
|
126
|
+
interface ComparisonResult {
|
|
127
|
+
jsonTokens: number;
|
|
128
|
+
toonTokens: number;
|
|
129
|
+
savings: number;
|
|
130
|
+
savingsPercentage: number;
|
|
131
|
+
toonOutput: string;
|
|
132
|
+
toonOutputPath?: string;
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
interface CompareOptions {
|
|
136
|
+
outputDir?: string;
|
|
137
|
+
saveToFile?: boolean;
|
|
138
|
+
prettyJson?: boolean;
|
|
139
|
+
fileName?: string;
|
|
140
|
+
}
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
## Error Handling
|
|
144
|
+
- File operations throw when paths are invalid.
|
|
145
|
+
- JSON parsing errors surface with native `SyntaxError` messages.
|
|
146
|
+
- Always wrap CLI or programmatic calls in try/catch for robustness.
|
|
147
|
+
|
|
148
|
+
## Performance Notes
|
|
149
|
+
- Token counting uses `gpt-3-encoder` for approximate GPT token metrics.
|
|
150
|
+
- Pretty-printing JSON increases token count slightly; disable with `{ prettyJson: false }` for raw comparison.
|
|
151
|
+
- TOON encoding efficiency improves with deeply nested or repetitive structures.
|
|
152
|
+
|
|
153
|
+
## Examples
|
|
154
|
+
```ts
|
|
155
|
+
import { compareFile } from 'json-to-toon-parser';
|
|
156
|
+
const result = compareFile('data/big.json', { outputDir: 'analysis' });
|
|
157
|
+
console.log(`Saved ${result.savings} tokens (${result.savingsPercentage.toFixed(2)}%)`);
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
```ts
|
|
161
|
+
import { jsonToToon } from 'json-to-toon-parser';
|
|
162
|
+
const payload = { items: Array.from({ length: 50 }, (_, i) => ({ id: i, flag: i % 2 === 0 })) };
|
|
163
|
+
const toon = jsonToToon(payload);
|
|
164
|
+
// Send toon to LLM or store compactly
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
## Decoding (If Needed)
|
|
168
|
+
If you need to restore JSON later, use the underlying TOON library directly:
|
|
169
|
+
```ts
|
|
170
|
+
import { decode } from '@toon-format/toon';
|
|
171
|
+
const original = decode(toonString);
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
## Versioning Policy
|
|
175
|
+
- Patch: Internal fixes / metadata.
|
|
176
|
+
- Minor: New features (e.g., Web UI server added in 1.1.x).
|
|
177
|
+
- Major: Breaking API or CLI changes.
|
|
178
|
+
|
|
179
|
+
## Contributing
|
|
180
|
+
1. Fork & branch (`feat/my-feature`).
|
|
181
|
+
2. Run `npm run build` before submitting PRs.
|
|
182
|
+
3. Add tests/examples where relevant.
|
|
183
|
+
|
|
184
|
+
## License
|
|
185
|
+
MIT Β© Bhushan Chaudhari
|
package/package.json
CHANGED
|
@@ -1,9 +1,18 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "json-toon-parser",
|
|
3
|
-
"version": "1.1.
|
|
4
|
-
"description": "Compare and convert JSON to TOON format with token efficiency analysis
|
|
3
|
+
"version": "1.1.3",
|
|
4
|
+
"description": "Compare and convert JSON to TOON format with token efficiency analysis β includes CLI, API, and Web UI server.",
|
|
5
5
|
"main": "./dist/index.js",
|
|
6
6
|
"types": "./dist/index.d.ts",
|
|
7
|
+
"exports": {
|
|
8
|
+
".": {
|
|
9
|
+
"types": "./dist/index.d.ts",
|
|
10
|
+
"import": "./dist/index.js"
|
|
11
|
+
},
|
|
12
|
+
"./server": {
|
|
13
|
+
"import": "./dist/server.js"
|
|
14
|
+
}
|
|
15
|
+
},
|
|
7
16
|
"bin": {
|
|
8
17
|
"json-toon": "./dist/cli.js"
|
|
9
18
|
},
|
|
@@ -20,28 +29,99 @@
|
|
|
20
29
|
"dist",
|
|
21
30
|
"public",
|
|
22
31
|
"README.md",
|
|
23
|
-
"LICENSE"
|
|
32
|
+
"LICENSE",
|
|
33
|
+
"CHANGELOG.md",
|
|
34
|
+
"docs"
|
|
24
35
|
],
|
|
36
|
+
"repository": {
|
|
37
|
+
"type": "git",
|
|
38
|
+
"url": "git+https://github.com/bhushan020/json-toon-parser.git"
|
|
39
|
+
},
|
|
40
|
+
"homepage": "https://github.com/bhushan020/json-toon-parser#readme",
|
|
25
41
|
"keywords": [
|
|
26
42
|
"json",
|
|
27
|
-
"toon",
|
|
28
43
|
"parser",
|
|
29
44
|
"converter",
|
|
30
45
|
"tokenization",
|
|
46
|
+
"toon",
|
|
47
|
+
"toon-format",
|
|
48
|
+
"toon-parser",
|
|
49
|
+
"toon-tokenizer",
|
|
50
|
+
"json-to-toon",
|
|
51
|
+
"json-toon-parser",
|
|
52
|
+
"JSON-to-TOON",
|
|
31
53
|
"gpt",
|
|
54
|
+
"llm",
|
|
55
|
+
"prompt-optimizer",
|
|
56
|
+
"token-optimization",
|
|
57
|
+
"token-efficiency",
|
|
32
58
|
"compression",
|
|
33
|
-
"
|
|
34
|
-
"
|
|
35
|
-
"
|
|
59
|
+
"data-compression",
|
|
60
|
+
"text-compression",
|
|
61
|
+
"minifier",
|
|
62
|
+
"custom-format",
|
|
63
|
+
"format-converter",
|
|
64
|
+
"webui",
|
|
65
|
+
"cli",
|
|
66
|
+
"developer-tools",
|
|
67
|
+
"nodejs",
|
|
68
|
+
"javascript",
|
|
69
|
+
"npm-package",
|
|
70
|
+
"open-source",
|
|
71
|
+
"json-optimizer",
|
|
72
|
+
"json-minifier",
|
|
73
|
+
"json-transform",
|
|
74
|
+
"json-formatter",
|
|
75
|
+
"json-analysis",
|
|
76
|
+
"json-compression",
|
|
77
|
+
"structured-data",
|
|
78
|
+
"structured-format",
|
|
79
|
+
"compact-format",
|
|
80
|
+
"compact-data",
|
|
81
|
+
"lightweight-serialization",
|
|
82
|
+
"serialization",
|
|
83
|
+
"deserialization",
|
|
84
|
+
"data-encoding",
|
|
85
|
+
"data-decoding",
|
|
86
|
+
"token-counter",
|
|
87
|
+
"token-metrics",
|
|
88
|
+
"token-saver",
|
|
89
|
+
"llm-tools",
|
|
90
|
+
"chatgpt-tools",
|
|
91
|
+
"openai-tools",
|
|
92
|
+
"ai-developer-tools",
|
|
93
|
+
"prompt-tools",
|
|
94
|
+
"prompt-engineering",
|
|
95
|
+
"payload-optimizer",
|
|
96
|
+
"text-optimizer",
|
|
97
|
+
"data-optimizer",
|
|
98
|
+
"json-cli",
|
|
99
|
+
"json-webui",
|
|
100
|
+
"llm-friendly",
|
|
101
|
+
"prompt-friendly",
|
|
102
|
+
"structured-compression",
|
|
103
|
+
"encoding-tools",
|
|
104
|
+
"decoder",
|
|
105
|
+
"encoder",
|
|
106
|
+
"tree-shakeable",
|
|
107
|
+
"typescript",
|
|
108
|
+
"ts-library",
|
|
109
|
+
"backend-tools",
|
|
110
|
+
"automation-tools",
|
|
111
|
+
"productivity-tools",
|
|
112
|
+
"developer-productivity",
|
|
113
|
+
"dev-cli",
|
|
114
|
+
"api-tools",
|
|
115
|
+
"file-conversion",
|
|
116
|
+
"file-transform",
|
|
117
|
+
"data-utilities"
|
|
36
118
|
],
|
|
37
119
|
"author": "Bhushan Chaudhari <bhushanschaudhari4@gmail.com>",
|
|
38
120
|
"license": "MIT",
|
|
39
121
|
"dependencies": {
|
|
40
122
|
"@toon-format/toon": "^1.0.0",
|
|
41
|
-
"1.0.1": "^1.0.0",
|
|
42
123
|
"commander": "^12.0.0",
|
|
43
|
-
"gpt-3-encoder": "^1.1.4"
|
|
44
|
-
"json-toon-parser": "^1.0.1"
|
|
124
|
+
"gpt-3-encoder": "^1.1.4"
|
|
45
125
|
},
|
|
46
126
|
"devDependencies": {
|
|
47
127
|
"@types/node": "^20.0.0",
|