@toon-format/spec 1.3.3 → 1.5.0

package/VERSIONING.md

@@ -13,7 +13,7 @@ The TOON specification follows [Semantic Versioning](https://semver.org/) with a
 - **MAJOR version** - Incremented for breaking changes that are incompatible with previous versions
 - **MINOR version** - Incremented for backward-compatible additions, clarifications, or non-breaking changes
 
-**Example:** Moving from v1.3 to v1.4 means your implementation keeps working. Moving from v1.3 to v2.0 means you'll likely need to update your code.
+**Example:** Moving from v1.5 to v1.6 means your implementation keeps working. Moving from v1.5 to v2.0 means you'll likely need to update your code.
 
 ## What Constitutes a Breaking Change
 
@@ -122,19 +122,6 @@ Your implementation can support multiple spec versions. If you do:
 - Previous version implementations remain valid
 - We'll provide a migration guide
 
-## Version Numbering Examples
-
-### Current: v1.3
-
-**Next minor update (clarifications, additions):**
-- v1.4
-
-**Next major update (breaking changes):**
-- v2.0
-
-**After v2.0, next minor update:**
-- v2.1
-
 ## Version History
 
 See [CHANGELOG.md](./CHANGELOG.md) for detailed version history.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@toon-format/spec",
   "type": "module",
-  "version": "1.3.3",
+  "version": "1.5.0",
   "packageManager": "pnpm@10.19.0",
   "description": "Official specification for Token-Oriented Object Notation (TOON)",
   "author": "Johann Schopplich <hello@johannschopplich.com>",

package/tests/README.md

@@ -25,19 +25,23 @@ tests/
 │   │   ├── arrays-nested.json
 │   │   ├── arrays-objects.json
 │   │   ├── delimiters.json
-│   │   ├── normalization.json
 │   │   ├── whitespace.json
-│   │   └── options.json
+│   │   ├── options.json
+│   │   └── key-folding.json
 │   └── decode/             # Decoding tests (TOON → JSON)
 │       ├── primitives.json
+│       ├── numbers.json
 │       ├── objects.json
 │       ├── arrays-primitive.json
 │       ├── arrays-tabular.json
 │       ├── arrays-nested.json
 │       ├── delimiters.json
+│       ├── whitespace.json
+│       ├── root-form.json
 │       ├── validation-errors.json
 │       ├── indentation-errors.json
-│       └── blank-lines.json
+│       ├── blank-lines.json
+│       └── path-expansion.json
 └── README.md               # This file
 ```
 
@@ -47,7 +51,7 @@ All test fixtures follow a standard JSON structure defined in [`fixtures.schema.
 
 ```json
 {
-  "version": "1.3",
+  "version": "1.4",
   "category": "encode",
   "description": "Brief description of test category",
   "tests": [
@@ -88,25 +92,31 @@ All test fixtures follow a standard JSON structure defined in [`fixtures.schema.
 {
   "delimiter": ",",
   "indent": 2,
-  "lengthMarker": ""
+  "lengthMarker": "#",
+  "keyFolding": "safe",
+  "flattenDepth": 3
 }
 ```
 
-- `delimiter`: `","` (comma, default), `"\t"` (tab), or `"|"` (pipe)
+- `delimiter`: `","` (comma, default), `"\t"` (tab), or `"|"` (pipe). Affects encoder output; decoders parse the delimiter declared in array headers
 - `indent`: Number of spaces per indentation level (default: `2`)
-- `lengthMarker`: `"#"` to prefix array lengths, or `""` for no marker (default: `""`)
+- `lengthMarker`: Optional. Set to `"#"` to prefix array lengths (e.g., `[#3]`). Omit this property to disable length markers
+- `keyFolding`: `"off"` (default) or `"safe"`. Enables key folding to collapse single-key object chains into dotted-path notation (v1.5+)
+- `flattenDepth`: Integer. Maximum depth to fold key chains when `keyFolding` is `"safe"` (default: Infinity). Values less than 2 have no practical folding effect (v1.5+)
 
 #### Decoding Options
 
 ```json
 {
   "indent": 2,
-  "strict": true
+  "strict": true,
+  "expandPaths": "safe"
 }
 ```
 
-- `indent`: Expected number of spaces per level (default: `2`)
-- `strict`: Enable strict validation (default: `true`)
+- `indent`: Expected number of spaces per indentation level (default: `2`)
+- `strict`: Enable strict validation (default: `true`). When `expandPaths` is `"safe"`, strict mode controls conflict resolution: errors on conflicts when `true`, LWW when `false` (v1.5+)
+- `expandPaths`: `"off"` (default) or `"safe"`. Enables path expansion to split dotted keys into nested object structures (v1.5+)
 
 ### Error Tests
 
@@ -143,30 +153,34 @@ The fixture format is language-agnostic JSON, so you can load and iterate it usi
 
 | File | Description | Spec Sections |
 |------|-------------|---------------|
-| `primitives.json` | String, number, boolean, null encoding and escaping | §5 |
-| `objects.json` | Simple objects, nested objects, key encoding | §6 |
-| `arrays-primitive.json` | Inline primitive arrays, empty arrays | §7.1 |
-| `arrays-tabular.json` | Tabular format with header and rows | §7.2 |
-| `arrays-nested.json` | Arrays of arrays, mixed arrays | §7.3 |
-| `arrays-objects.json` | Objects as list items, complex nesting | §7 |
-| `delimiters.json` | Tab and pipe delimiter options | §8 |
-| `normalization.json` | BigInt, Date, undefined, NaN, Infinity handling | §5 |
-| `whitespace.json` | Formatting invariants and indentation | §4 |
+| `primitives.json` | String, number, boolean, null encoding and escaping | §7.1/§7.2, §2 |
+| `objects.json` | Simple objects, nested objects, key encoding | §8 (keys: §7.3/§7.1) |
+| `arrays-primitive.json` | Inline primitive arrays, empty arrays | §9.1 |
+| `arrays-tabular.json` | Tabular format with header and rows | §9.3 |
+| `arrays-nested.json` | Arrays of arrays, mixed arrays | §9.2/§9.4 |
+| `arrays-objects.json` | Objects as list items, complex nesting | §9, §10 |
+| `delimiters.json` | Tab and pipe delimiter options | §11 |
+| `whitespace.json` | Formatting invariants and indentation | §12 |
 | `options.json` | Length marker and delimiter option combinations | §3 |
+| `key-folding.json` | Key folding with safe mode, depth control, collision avoidance | §13.4 |
 
 ### Decoding Tests (`fixtures/decode/`)
 
 | File | Description | Spec Sections |
 |------|-------------|---------------|
-| `primitives.json` | Parsing primitives, unescaping, ambiguity | §5 |
-| `objects.json` | Parsing objects, keys, nesting | §6 |
-| `arrays-primitive.json` | Inline array parsing | §7.1 |
-| `arrays-tabular.json` | Tabular format parsing | §7.2 |
-| `arrays-nested.json` | Nested and mixed array parsing | §7.3 |
-| `delimiters.json` | Delimiter detection and parsing | §8 |
-| `validation-errors.json` | Syntax errors, length mismatches, malformed input | §9 |
-| `indentation-errors.json` | Strict mode indentation validation | §9 |
-| `blank-lines.json` | Blank line handling in arrays | §9 |
+| `primitives.json` | Parsing primitives, unescaping, ambiguity | §4, §7.1/§7.4 |
+| `numbers.json` | Number edge cases, exponent forms, leading zeros | §4 |
+| `objects.json` | Parsing objects, keys, nesting | §8 (keys: §7.3/§7.1) |
+| `arrays-primitive.json` | Inline array parsing | §9.1 |
+| `arrays-tabular.json` | Tabular format parsing | §9.3 |
+| `arrays-nested.json` | Nested and mixed array parsing | §9.2/§9.4 |
+| `delimiters.json` | Delimiter detection and parsing | §11 |
+| `whitespace.json` | Whitespace tolerance and token trimming | §12 |
+| `root-form.json` | Root form detection (empty, single primitive) | §5 |
+| `validation-errors.json` | Syntax errors, length mismatches, malformed input | §14 |
+| `indentation-errors.json` | Strict mode indentation validation | §14.3, §12 |
+| `blank-lines.json` | Blank line handling in arrays | §14.4, §12 |
+| `path-expansion.json` | Path expansion with safe mode, deep merge, strict-mode conflicts | §13.4, §14.5 |
 
 ## Validating Fixtures
 

package/tests/fixtures/decode/arrays-nested.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.3",
+  "version": "1.4",
   "category": "decode",
   "description": "Nested and mixed array decoding - list format, arrays of arrays, root arrays, mixed types",
   "tests": [
@@ -12,7 +12,7 @@
           { "id": 2, "name": "Second", "extra": true }
         ]
       },
-      "specSection": "7"
+      "specSection": "9.4"
     },
     {
       "name": "parses list arrays with empty items",
@@ -20,7 +20,7 @@
       "expected": {
         "items": ["first", "second", {}]
       },
-      "specSection": "7.3"
+      "specSection": "9.4"
     },
     {
       "name": "parses list arrays with deeply nested objects",
@@ -49,7 +49,7 @@
           { "id": 1, "nested": { "x": 1 } }
         ]
       },
-      "specSection": "7"
+      "specSection": "9.4"
     },
     {
       "name": "parses nested tabular arrays as first field on hyphen line",
@@ -65,7 +65,7 @@
           }
         ]
       },
-      "specSection": "7"
+      "specSection": "10"
     },
     {
       "name": "parses objects containing arrays (including empty arrays) in list format",
@@ -75,7 +75,7 @@
           { "name": "test", "data": [] }
         ]
       },
-      "specSection": "7"
+      "specSection": "9.4"
     },
     {
       "name": "parses arrays of arrays within objects",
@@ -85,7 +85,7 @@
           { "matrix": [[1, 2], [3, 4]], "name": "grid" }
         ]
       },
-      "specSection": "7"
+      "specSection": "9.2"
     },
     {
       "name": "parses nested arrays of primitives",
@@ -93,7 +93,7 @@
       "expected": {
         "pairs": [["a", "b"], ["c", "d"]]
       },
-      "specSection": "7.3"
+      "specSection": "9.2"
     },
     {
       "name": "parses quoted strings and mixed lengths in nested arrays",
@@ -101,7 +101,7 @@
       "expected": {
         "pairs": [["a", "b"], ["c,d", "e:f", "true"]]
       },
-      "specSection": "7.3"
+      "specSection": "9.2"
     },
     {
       "name": "parses empty inner arrays",
@@ -109,7 +109,7 @@
       "expected": {
         "pairs": [[], []]
       },
-      "specSection": "7.3"
+      "specSection": "9.2"
     },
     {
       "name": "parses mixed-length inner arrays",
@@ -117,37 +117,37 @@
       "expected": {
         "pairs": [[1], [2, 3]]
       },
-      "specSection": "7.3"
+      "specSection": "9.2"
     },
     {
       "name": "parses root arrays of primitives (inline)",
       "input": "[5]: x,y,\"true\",true,10",
       "expected": ["x", "y", "true", true, 10],
-      "specSection": "7"
+      "specSection": "9.1"
     },
     {
       "name": "parses root arrays of uniform objects in tabular format",
       "input": "[2]{id}:\n  1\n  2",
       "expected": [{ "id": 1 }, { "id": 2 }],
-      "specSection": "7.2"
+      "specSection": "9.3"
     },
     {
       "name": "parses root arrays of non-uniform objects in list format",
       "input": "[2]:\n  - id: 1\n  - id: 2\n    name: Ada",
       "expected": [{ "id": 1 }, { "id": 2, "name": "Ada" }],
-      "specSection": "7"
+      "specSection": "9.4"
     },
     {
       "name": "parses empty root arrays",
       "input": "[0]:",
       "expected": [],
-      "specSection": "7"
+      "specSection": "9.1"
     },
     {
       "name": "parses root arrays of arrays",
       "input": "[2]:\n  - [2]: 1,2\n  - [0]:",
       "expected": [[1, 2], []],
-      "specSection": "7.3"
+      "specSection": "9.2"
     },
     {
       "name": "parses complex mixed object with arrays and nested objects",
@@ -161,7 +161,7 @@
           "prefs": []
         }
       },
-      "specSection": "6"
+      "specSection": "8"
     },
     {
       "name": "parses arrays mixing primitives, objects and strings (list format)",
@@ -169,7 +169,7 @@
       "expected": {
         "items": [1, { "a": 1 }, "text"]
       },
-      "specSection": "7.3"
+      "specSection": "9.4"
     },
     {
       "name": "parses arrays mixing objects and arrays",
@@ -177,7 +177,7 @@
       "expected": {
         "items": [{ "a": 1 }, [1, 2]]
       },
-      "specSection": "7.3"
+      "specSection": "9.4"
     },
     {
       "name": "parses quoted key with list array format",
@@ -188,7 +188,7 @@
           { "id": 2 }
         ]
       },
-      "specSection": "7"
+      "specSection": "9.4"
     }
   ]
 }

package/tests/fixtures/decode/arrays-primitive.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.3",
+  "version": "1.4",
   "category": "decode",
   "description": "Primitive array decoding - inline arrays of strings, numbers, booleans, quoted strings",
   "tests": [
@@ -9,7 +9,7 @@
       "expected": {
         "tags": ["reading", "gaming", "coding"]
       },
-      "specSection": "7.1"
+      "specSection": "9.1"
     },
     {
       "name": "parses number arrays inline",
@@ -17,7 +17,7 @@
       "expected": {
         "nums": [1, 2, 3]
       },
-      "specSection": "7.1"
+      "specSection": "9.1"
     },
     {
       "name": "parses mixed primitive arrays inline",
@@ -25,7 +25,7 @@
       "expected": {
         "data": ["x", "y", true, 10]
       },
-      "specSection": "7.1"
+      "specSection": "9.1"
     },
     {
       "name": "parses empty arrays",
@@ -33,7 +33,7 @@
       "expected": {
         "items": []
       },
-      "specSection": "7.1"
+      "specSection": "9.1"
     },
     {
       "name": "parses single-item array with empty string",
@@ -41,7 +41,7 @@
       "expected": {
         "items": [""]
       },
-      "specSection": "7.1"
+      "specSection": "9.1"
     },
     {
       "name": "parses multi-item array with empty string",
@@ -49,7 +49,7 @@
       "expected": {
         "items": ["a", "", "b"]
       },
-      "specSection": "7.1"
+      "specSection": "9.1"
     },
     {
       "name": "parses whitespace-only strings in arrays",
@@ -57,7 +57,7 @@
       "expected": {
         "items": [" ", "  "]
       },
-      "specSection": "7.1"
+      "specSection": "9.1"
     },
     {
       "name": "parses strings with delimiters in arrays",
@@ -65,7 +65,7 @@
       "expected": {
         "items": ["a", "b,c", "d:e"]
       },
-      "specSection": "7.1"
+      "specSection": "9.1"
     },
     {
       "name": "parses strings that look like primitives when quoted",
@@ -73,7 +73,7 @@
       "expected": {
         "items": ["x", "true", "42", "-3.14"]
       },
-      "specSection": "7.1"
+      "specSection": "9.1"
     },
     {
       "name": "parses strings with structural tokens in arrays",
@@ -81,7 +81,7 @@
       "expected": {
         "items": ["[5]", "- item", "{key}"]
       },
-      "specSection": "7.1"
+      "specSection": "9.1"
     },
     {
       "name": "parses quoted key with inline array",
@@ -89,7 +89,7 @@
       "expected": {
         "my-key": [1, 2, 3]
       },
-      "specSection": "7.1"
+      "specSection": "9.1"
     },
     {
       "name": "parses quoted key containing brackets with inline array",
@@ -97,7 +97,7 @@
       "expected": {
         "key[test]": [1, 2, 3]
       },
-      "specSection": "7.1"
+      "specSection": "9.1"
     },
     {
       "name": "parses quoted key with empty array",
@@ -105,7 +105,7 @@
       "expected": {
         "x-custom": []
       },
-      "specSection": "7.1"
+      "specSection": "9.1"
     }
   ]
 }

package/tests/fixtures/decode/arrays-tabular.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.3",
+  "version": "1.4",
   "category": "decode",
   "description": "Tabular array decoding - parsing arrays of uniform objects with headers",
   "tests": [
@@ -12,7 +12,7 @@
           { "sku": "B2", "qty": 1, "price": 14.5 }
         ]
       },
-      "specSection": "7.2"
+      "specSection": "9.3"
     },
     {
       "name": "parses nulls and quoted values in tabular rows",
@@ -23,7 +23,18 @@
           { "id": 2, "value": "test" }
         ]
       },
-      "specSection": "7.2"
+      "specSection": "9.3"
+    },
+    {
+      "name": "parses quoted colon in tabular row as data",
+      "input": "items[2]{id,note}:\n  1,\"a:b\"\n  2,\"c:d\"",
+      "expected": {
+        "items": [
+          { "id": 1, "note": "a:b" },
+          { "id": 2, "note": "c:d" }
+        ]
+      },
+      "specSection": "9.3"
     },
     {
       "name": "parses quoted header keys in tabular arrays",
@@ -34,7 +45,7 @@
           { "order:id": 2, "full name": "Bob" }
         ]
       },
-      "specSection": "7.2"
+      "specSection": "9.3"
     },
     {
       "name": "parses quoted key with tabular array format",
@@ -45,7 +56,19 @@
           { "id": 2, "name": "Bob" }
         ]
       },
-      "specSection": "7.2"
+      "specSection": "9.3"
+    },
+    {
+      "name": "unquoted colon terminates tabular rows and starts key-value pair",
+      "input": "items[2]{id,name}:\n  1,Alice\n  2,Bob\ncount: 2",
+      "expected": {
+        "items": [
+          { "id": 1, "name": "Alice" },
+          { "id": 2, "name": "Bob" }
+        ],
+        "count": 2
+      },
+      "specSection": "9.3"
     }
   ]
 }

package/tests/fixtures/decode/blank-lines.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.3",
+  "version": "1.4",
   "category": "decode",
   "description": "Blank line handling - strict mode errors on blank lines inside arrays, accepts blank lines outside arrays",
   "tests": [
@@ -11,7 +11,7 @@
       "options": {
         "strict": true
       },
-      "specSection": "9"
+      "specSection": "14.4"
     },
     {
       "name": "throws on blank line inside tabular array",
@@ -21,7 +21,7 @@
       "options": {
         "strict": true
       },
-      "specSection": "9"
+      "specSection": "14.4"
     },
     {
       "name": "throws on multiple blank lines inside array",
@@ -31,7 +31,7 @@
       "options": {
         "strict": true
       },
-      "specSection": "9"
+      "specSection": "14.4"
     },
     {
       "name": "throws on blank line with spaces inside array",
@@ -41,7 +41,7 @@
       "options": {
         "strict": true
       },
-      "specSection": "9"
+      "specSection": "14.4"
     },
     {
       "name": "throws on blank line in nested list array",
@@ -51,7 +51,7 @@
       "options": {
         "strict": true
       },
-      "specSection": "9"
+      "specSection": "14.4"
     },
     {
       "name": "accepts blank line between root-level fields",
@@ -63,7 +63,7 @@
       "options": {
         "strict": true
       },
-      "specSection": "9"
+      "specSection": "12"
     },
     {
       "name": "accepts trailing newline at end of file",
@@ -74,7 +74,7 @@
       "options": {
         "strict": true
       },
-      "specSection": "9"
+      "specSection": "12"
     },
     {
       "name": "accepts multiple trailing newlines",
@@ -85,7 +85,7 @@
       "options": {
         "strict": true
       },
-      "specSection": "9"
+      "specSection": "12"
     },
     {
       "name": "accepts blank line after array ends",
@@ -97,7 +97,7 @@
       "options": {
         "strict": true
       },
-      "specSection": "9"
+      "specSection": "12"
     },
     {
       "name": "accepts blank line between nested object fields",
@@ -111,7 +111,7 @@
       "options": {
         "strict": true
       },
-      "specSection": "9"
+      "specSection": "12"
     },
     {
       "name": "ignores blank lines inside list array when strict=false",
@@ -122,7 +122,7 @@
       "options": {
         "strict": false
       },
-      "specSection": "9"
+      "specSection": "12"
     },
     {
       "name": "ignores blank lines inside tabular array when strict=false",
@@ -136,7 +136,7 @@
       "options": {
         "strict": false
       },
-      "specSection": "9"
+      "specSection": "12"
     },
     {
       "name": "ignores multiple blank lines in arrays when strict=false",
@@ -147,7 +147,7 @@
       "options": {
         "strict": false
       },
-      "specSection": "9"
+      "specSection": "12"
     }
   ]
 }