@toon-format/spec 2.0.0 → 2.0.1

package/README.md

@@ -1,10 +1,10 @@
 # TOON Format Specification
 
 [![SPEC v2.0](https://img.shields.io/badge/spec-v2.0-lightgrey)](./SPEC.md)
-[![Tests](https://img.shields.io/badge/tests-340-green)](./tests/fixtures/)
+[![Tests](https://img.shields.io/badge/tests-342-green)](./tests/fixtures/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)
 
-This repository contains the official specification for **Token-Oriented Object Notation (TOON)**, a compact, human-readable serialization format designed for passing structured data to Large Language Models with significantly reduced token usage.
+This repository contains the official specification for **Token-Oriented Object Notation (TOON)**, a compact, human-readable encoding of the JSON data model for LLM prompts. It provides a lossless serialization of the same objects, arrays, and primitives as JSON, but in a syntax that minimizes tokens and makes structure easy for models to follow.
 
 ## 📋 Specification
 
@@ -18,38 +18,76 @@ The specification includes complete grammar (ABNF), encoding rules, validation r
 
 ## What is TOON?
 
-**Token-Oriented Object Notation** is a compact, human-readable serialization format designed for passing structured data to Large Language Models with significantly reduced token usage. It's intended for LLM input, not output.
+> [!IMPORTANT]
+> For a high-level overview of TOON, its features and benefits, design goals, and comparisons to other formats, see the [`toon-format/toon` repository](https://github.com/toon-format/toon).
 
-TOON's sweet spot is **uniform arrays of objects** – multiple fields per row, same structure across items. It borrows YAML's indentation-based structure for nested objects and CSV's tabular format for uniform data rows, then optimizes both for token efficiency in LLM contexts. For deeply nested or non-uniform data, JSON may be more efficient.
+## Serialization Example
 
-**Key Features:**
-
-- 💸 **Token-efficient:** typically 30–60% fewer tokens than JSON
-- 🤿 **LLM-friendly guardrails:** explicit lengths and fields enable validation
-- 🍱 **Minimal syntax:** removes redundant punctuation (braces, brackets, most quotes)
-- 📐 **Indentation-based structure:** like YAML, uses whitespace instead of braces
-- 🧺 **Tabular arrays:** declare keys once, stream data as rows
-
-## Quick Example
-
-**JSON:**
+<table>
+<tr>
+<th>JSON</th>
+<th>TOON</th>
+</tr>
+<tr>
+<td>
 
 ```json
 {
-  "users": [
-    { "id": 1, "name": "Alice", "role": "admin" },
-    { "id": 2, "name": "Bob", "role": "user" }
+  "context": {
+    "task": "Our favorite hikes together",
+    "location": "Boulder",
+    "season": "spring_2025"
+  },
+  "friends": ["ana", "luis", "sam"],
+  "hikes": [
+    {
+      "id": 1,
+      "name": "Blue Lake Trail",
+      "distanceKm": 7.5,
+      "elevationGain": 320,
+      "companion": "ana",
+      "wasSunny": true
+    },
+    {
+      "id": 2,
+      "name": "Ridge Overlook",
+      "distanceKm": 9.2,
+      "elevationGain": 540,
+      "companion": "luis",
+      "wasSunny": false
+    },
+    {
+      "id": 3,
+      "name": "Wildflower Loop",
+      "distanceKm": 5.1,
+      "elevationGain": 180,
+      "companion": "sam",
+      "wasSunny": true
+    }
   ]
 }
 ```
 
-**TOON:**
+</td>
+<td>
 
+```toon
+context:
+  task: Our favorite hikes together
+  location: Boulder
+  season: spring_2025
+
+friends[3]: ana,luis,sam
+
+hikes[3]{id,name,distanceKm,elevationGain,companion,wasSunny}:
+  1,Blue Lake Trail,7.5,320,ana,true
+  2,Ridge Overlook,9.2,540,luis,false
+  3,Wildflower Loop,5.1,180,sam,true
 ```
-users[2]{id,name,role}:
-  1,Alice,admin
-  2,Bob,user
-```
+
+</td>
+</tr>
+</table>
 
 ## Reference Implementation
 

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@toon-format/spec",
   "type": "module",
-  "version": "2.0.0",
+  "version": "2.0.1",
   "packageManager": "pnpm@10.19.0",
   "description": "Official specification for Token-Oriented Object Notation (TOON)",
   "author": "Johann Schopplich <hello@johannschopplich.com>",

package/tests/fixtures/decode/arrays-nested.json

@@ -69,10 +69,10 @@
     },
     {
       "name": "parses objects containing arrays (including empty arrays) in list format",
-      "input": "items[1]:\n  - name: test\n    data[0]:",
+      "input": "items[1]:\n  - name: Ada\n    data[0]:",
       "expected": {
         "items": [
-          { "name": "test", "data": [] }
+          { "name": "Ada", "data": [] }
         ]
       },
       "specSection": "9.4"
@@ -120,35 +120,41 @@
       "specSection": "9.2"
     },
     {
-      "name": "parses root arrays of primitives (inline)",
+      "name": "parses root-level primitive array inline",
       "input": "[5]: x,y,\"true\",true,10",
       "expected": ["x", "y", "true", true, 10],
       "specSection": "9.1"
     },
     {
-      "name": "parses root arrays of uniform objects in tabular format",
+      "name": "parses root-level array of uniform objects in tabular format",
       "input": "[2]{id}:\n  1\n  2",
       "expected": [{ "id": 1 }, { "id": 2 }],
       "specSection": "9.3"
     },
     {
-      "name": "parses root arrays of non-uniform objects in list format",
+      "name": "parses root-level array of non-uniform objects in list format",
       "input": "[2]:\n  - id: 1\n  - id: 2\n    name: Ada",
       "expected": [{ "id": 1 }, { "id": 2, "name": "Ada" }],
       "specSection": "9.4"
     },
     {
-      "name": "parses empty root arrays",
-      "input": "[0]:",
-      "expected": [],
-      "specSection": "9.1"
+      "name": "parses root-level array mixing primitive, object, and array of objects in list format",
+      "input": "[3]:\n  - summary\n  - id: 1\n    name: Ada\n  - [2]:\n    - id: 2\n    - status: draft",
+      "expected": ["summary", { "id": 1, "name": "Ada" }, [{ "id": 2 }, { "status": "draft" }]],
+      "specSection": "9.4"
     },
     {
-      "name": "parses root arrays of arrays",
+      "name": "parses root-level array of arrays",
       "input": "[2]:\n  - [2]: 1,2\n  - [0]:",
       "expected": [[1, 2], []],
       "specSection": "9.2"
     },
+    {
+      "name": "parses empty root-level array",
+      "input": "[0]:",
+      "expected": [],
+      "specSection": "9.1"
+    },
     {
       "name": "parses complex mixed object with arrays and nested objects",
       "input": "user:\n  id: 123\n  name: Ada\n  tags[2]: reading,gaming\n  active: true\n  prefs[0]:",
@@ -164,7 +170,7 @@
       "specSection": "8"
     },
     {
-      "name": "parses arrays mixing primitives, objects and strings (list format)",
+      "name": "parses arrays mixing primitives, objects, and strings in list format",
       "input": "items[3]:\n  - 1\n  - a: 1\n  - text",
       "expected": {
         "items": [1, { "a": 1 }, "text"]

package/tests/fixtures/decode/arrays-tabular.json

@@ -59,7 +59,7 @@
       "specSection": "9.3"
     },
     {
-      "name": "unquoted colon terminates tabular rows and starts key-value pair",
+      "name": "treats unquoted colon as terminator for tabular rows and start of key-value pair",
       "input": "items[2]{id,name}:\n  1,Alice\n  2,Bob\ncount: 2",
       "expected": {
         "items": [

package/tests/fixtures/decode/delimiters.json

@@ -66,7 +66,7 @@
       "specSection": "11"
     },
     {
-      "name": "nested arrays inside list items default to comma delimiter",
+      "name": "parses nested arrays inside list items with default comma delimiter",
       "input": "items[1\t]:\n  - tags[3]: a,b,c",
       "expected": {
         "items": [{ "tags": ["a", "b", "c"] }]
@@ -75,7 +75,7 @@
       "note": "Parent uses tab, nested defaults to comma"
     },
     {
-      "name": "nested arrays inside list items default to comma with pipe parent",
+      "name": "parses nested arrays inside list items with default comma delimiter when parent uses pipe",
       "input": "items[1|]:\n  - tags[3]: a,b,c",
       "expected": {
         "items": [{ "tags": ["a", "b", "c"] }]
@@ -83,25 +83,25 @@
       "specSection": "11"
     },
     {
-      "name": "parses root arrays with tab delimiter",
+      "name": "parses root-level array with tab delimiter",
       "input": "[3\t]: x\ty\tz",
       "expected": ["x", "y", "z"],
       "specSection": "11"
     },
     {
-      "name": "parses root arrays with pipe delimiter",
+      "name": "parses root-level array with pipe delimiter",
       "input": "[3|]: x|y|z",
       "expected": ["x", "y", "z"],
       "specSection": "11"
     },
     {
-      "name": "parses root arrays of objects with tab delimiter",
+      "name": "parses root-level array of objects with tab delimiter",
       "input": "[2\t]{id}:\n  1\n  2",
       "expected": [{ "id": 1 }, { "id": 2 }],
       "specSection": "11"
     },
     {
-      "name": "parses root arrays of objects with pipe delimiter",
+      "name": "parses root-level array of objects with pipe delimiter",
       "input": "[2|]{id}:\n  1\n  2",
       "expected": [{ "id": 1 }, { "id": 2 }],
       "specSection": "11"

package/tests/fixtures/decode/indentation-errors.json

@@ -4,7 +4,7 @@
   "description": "Strict mode indentation validation - non-multiple indentation, tab characters, custom indent sizes",
   "tests": [
     {
-      "name": "throws when object field has non-multiple indentation (3 spaces with indent=2)",
+      "name": "throws on object field with non-multiple indentation (3 spaces with indent=2)",
       "input": "a:\n   b: 1",
       "expected": null,
       "shouldError": true,
@@ -15,7 +15,7 @@
       "specSection": "14.3"
     },
     {
-      "name": "throws when list item has non-multiple indentation (3 spaces with indent=2)",
+      "name": "throws on list item with non-multiple indentation (3 spaces with indent=2)",
       "input": "items[2]:\n   - id: 1\n   - id: 2",
       "expected": null,
       "shouldError": true,
@@ -26,7 +26,7 @@
       "specSection": "14.3"
     },
     {
-      "name": "throws with custom indent size when non-multiple (3 spaces with indent=4)",
+      "name": "throws on non-multiple indentation with custom indent=4 (3 spaces)",
       "input": "a:\n   b: 1",
       "expected": null,
       "shouldError": true,
@@ -51,7 +51,7 @@
       "specSection": "12"
     },
     {
-      "name": "throws when tab character used in indentation",
+      "name": "throws on tab character used in indentation",
       "input": "a:\n\tb: 1",
       "expected": null,
       "shouldError": true,
@@ -61,7 +61,7 @@
       "specSection": "14.3"
     },
     {
-      "name": "throws when mixed tabs and spaces in indentation",
+      "name": "throws on mixed tabs and spaces in indentation",
       "input": "a:\n \tb: 1",
       "expected": null,
       "shouldError": true,
@@ -71,7 +71,7 @@
       "specSection": "14.3"
     },
     {
-      "name": "throws when tab at start of line",
+      "name": "throws on tab at start of line",
       "input": "\ta: 1",
       "expected": null,
       "shouldError": true,
@@ -144,7 +144,7 @@
       "specSection": "12"
     },
     {
-      "name": "empty lines do not trigger validation errors",
+      "name": "parses empty lines without validation errors",
       "input": "a: 1\n\nb: 2",
       "expected": {
         "a": 1,
@@ -156,7 +156,7 @@
       "specSection": "12"
     },
     {
-      "name": "root-level content (0 indentation) is always valid",
+      "name": "parses root-level content (0 indentation) as always valid",
       "input": "a: 1\nb: 2\nc: 3",
       "expected": {
         "a": 1,
@@ -169,7 +169,7 @@
       "specSection": "12"
     },
     {
-      "name": "lines with only spaces are not validated if empty",
+      "name": "parses lines with only spaces without validation if empty",
       "input": "a: 1\n   \nb: 2",
       "expected": {
         "a": 1,

package/tests/fixtures/decode/root-form.json

@@ -4,7 +4,7 @@
   "description": "Root form detection - empty document, single primitive, multiple primitives",
   "tests": [
     {
-      "name": "empty document decodes to empty object",
+      "name": "parses empty document as empty object",
       "input": "",
       "expected": {},
       "options": {

package/tests/fixtures/decode/validation-errors.json

@@ -18,14 +18,14 @@
       "specSection": "14.1"
     },
     {
-      "name": "throws when tabular row value count does not match header field count",
+      "name": "throws on tabular row value count mismatch with header field count",
       "input": "items[2]{id,name}:\n  1,Ada\n  2",
       "expected": null,
       "shouldError": true,
       "specSection": "14.1"
     },
     {
-      "name": "throws when tabular row count does not match header length",
+      "name": "throws on tabular row count mismatch with header length",
       "input": "[1]{id}:\n  1\n  2",
       "expected": null,
       "shouldError": true,

package/tests/fixtures/decode/whitespace.json

@@ -49,7 +49,7 @@
       "specSection": "12"
     },
     {
-      "name": "empty tokens decode to empty string",
+      "name": "parses empty tokens as empty string",
       "input": "items[3]: a,,c",
       "expected": {
         "items": ["a", "", "c"]

package/tests/fixtures/encode/arrays-nested.json

@@ -54,10 +54,10 @@
       "specSection": "9.4"
     },
     {
-      "name": "encodes empty root-level array",
-      "input": [],
-      "expected": "[0]:",
-      "specSection": "9.1"
+      "name": "encodes root-level array mixing primitive, object, and array of objects in list format",
+      "input": ["summary", { "id": 1, "name": "Ada" }, [{ "id": 2 }, { "status": "draft" }]],
+      "expected": "[3]:\n  - summary\n  - id: 1\n    name: Ada\n  - [2]:\n    - id: 2\n    - status: draft",
+      "specSection": "9.4"
     },
     {
       "name": "encodes root-level arrays of arrays",
@@ -65,6 +65,12 @@
       "expected": "[2]:\n  - [2]: 1,2\n  - [0]:",
       "specSection": "9.2"
     },
+    {
+      "name": "encodes empty root-level array",
+      "input": [],
+      "expected": "[0]:",
+      "specSection": "9.1"
+    },
     {
       "name": "encodes complex nested structure",
       "input": {

package/tests/fixtures/encode/arrays-objects.json

@@ -27,17 +27,17 @@
     {
       "name": "preserves field order in list items - array first",
       "input": {
-        "items": [{ "nums": [1, 2, 3], "name": "test" }]
+        "items": [{ "nums": [1, 2, 3], "name": "Ada" }]
       },
-      "expected": "items[1]:\n  - nums[3]: 1,2,3\n    name: test",
+      "expected": "items[1]:\n  - nums[3]: 1,2,3\n    name: Ada",
       "specSection": "10"
     },
     {
       "name": "preserves field order in list items - primitive first",
       "input": {
-        "items": [{ "name": "test", "nums": [1, 2, 3] }]
+        "items": [{ "name": "Ada", "nums": [1, 2, 3] }]
       },
-      "expected": "items[1]:\n  - name: test\n    nums[3]: 1,2,3",
+      "expected": "items[1]:\n  - name: Ada\n    nums[3]: 1,2,3",
       "specSection": "10"
     },
     {
@@ -90,10 +90,10 @@
       "name": "encodes objects with empty arrays in list format",
       "input": {
         "items": [
-          { "name": "test", "data": [] }
+          { "name": "Ada", "data": [] }
         ]
       },
-      "expected": "items[1]:\n  - name: test\n    data[0]:",
+      "expected": "items[1]:\n  - name: Ada\n    data[0]:",
       "specSection": "10"
     },
     {
@@ -124,7 +124,7 @@
       "specSection": "9.3"
     },
     {
-      "name": "uses list format when one object has nested column",
+      "name": "uses list format when one object has nested field",
       "input": {
         "items": [
           { "id": 1, "data": "string" },

package/tests/fixtures/encode/arrays-tabular.json

@@ -4,7 +4,7 @@
   "description": "Tabular array encoding - arrays of uniform objects with primitive values",
   "tests": [
     {
-      "name": "encodes arrays of similar objects in tabular format",
+      "name": "encodes arrays of uniform objects in tabular format",
       "input": {
         "items": [
           { "sku": "A1", "qty": 2, "price": 9.99 },

package/tests/fixtures/encode/delimiters.json

@@ -87,7 +87,7 @@
       "specSection": "11"
     },
     {
-      "name": "encodes root arrays with tab delimiter",
+      "name": "encodes root-level array with tab delimiter",
       "input": ["x", "y", "z"],
       "expected": "[3\t]: x\ty\tz",
       "options": {
@@ -96,7 +96,7 @@
       "specSection": "11"
     },
     {
-      "name": "encodes root arrays with pipe delimiter",
+      "name": "encodes root-level array with pipe delimiter",
       "input": ["x", "y", "z"],
       "expected": "[3|]: x|y|z",
       "options": {
@@ -105,7 +105,7 @@
       "specSection": "11"
     },
     {
-      "name": "encodes root arrays of objects with tab delimiter",
+      "name": "encodes root-level array of objects with tab delimiter",
       "input": [{ "id": 1 }, { "id": 2 }],
       "expected": "[2\t]{id}:\n  1\n  2",
       "options": {
@@ -114,7 +114,7 @@
       "specSection": "11"
     },
     {
-      "name": "encodes root arrays of objects with pipe delimiter",
+      "name": "encodes root-level array of objects with pipe delimiter",
       "input": [{ "id": 1 }, { "id": 2 }],
       "expected": "[2|]{id}:\n  1\n  2",
       "options": {