@toon-format/spec 1.5.1 → 2.0.0

package/CHANGELOG.md

@@ -5,6 +5,28 @@ All notable changes to the TOON specification will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.0] - 2025-11-10
+
+### Breaking Changes
+
+- **Removed:** Length marker (`#`) prefix in array headers has been completely removed from the specification
+- The `[#N]` format is no longer valid syntax. All array headers MUST use `[N]` format only
+- Encoders MUST NOT emit `[#N]` format
+- Decoders MUST NOT accept `[#N]` format (breaking change from v1.5)
+
+### Removed
+
+- All references to length marker from terminology (§1.4), header syntax (§6), ABNF grammar, conformance requirements (§13.2), and parsing helpers (Appendix B)
+- `lengthMarker` encoder option removed from all implementations
+- Length marker test fixtures removed
+
+### Migration from v1.5
+
+- Update decoder implementations to reject `[#N]` syntax
+- Convert any existing `.toon` files using `[#N]` format to `[N]` format
+- Remove `lengthMarker` option from encoder configurations
+- Remove `--length-marker` CLI flags if present
+
 ## [1.5] - 2025-11-08
 
 ### Added

package/CONTRIBUTING.md

@@ -32,7 +32,7 @@ New data type normalization rules
   Example: Handling Map or Set differently
 
 Changing array header syntax
-  Example: Optional length markers becoming required
+  Example: Making field lists mandatory for all arrays
 ```
 
 ### No – Direct PR or Issue First

package/README.md

@@ -1,7 +1,7 @@
 # TOON Format Specification
 
-[![SPEC v1.5](https://img.shields.io/badge/spec-v1.5-lightgrey)](./SPEC.md)
-[![Tests](https://img.shields.io/badge/tests-323-green)](./tests/fixtures/)
+[![SPEC v2.0](https://img.shields.io/badge/spec-v2.0-lightgrey)](./SPEC.md)
+[![Tests](https://img.shields.io/badge/tests-340-green)](./tests/fixtures/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)
 
 This repository contains the official specification for **Token-Oriented Object Notation (TOON)**, a compact, human-readable serialization format designed for passing structured data to Large Language Models with significantly reduced token usage.
@@ -10,24 +10,12 @@ This repository contains the official specification for **Token-Oriented Object
 
 [→ Read the full specification (SPEC.md)](./SPEC.md)
 
-- **Version:** 1.5 (2025-11-10)
+- **Version:** 2.0 (2025-11-10)
 - **Status:** Working Draft
 - **License:** MIT
 
 The specification includes complete grammar (ABNF), encoding rules, validation requirements, and conformance criteria.
 
-### New in v1.5
-
-- **Key Folding** (encode): Collapse nested single-key objects into compact dotted paths
-  - `{"a": {"b": {"c": 1}}}` → `a.b.c: 1`
-  - Opt-in via `keyFolding="safe"` with `flattenDepth` control
-- **Path Expansion** (decode): Expand dotted keys back to nested objects
-  - `a.b.c: 1` → `{"a": {"b": {"c": 1}}}`
-  - Opt-in via `expandPaths="safe"` with deep-merge semantics
-
-> [!NOTE]
-> Both features are opt-in to maintain backward compatibility.
-
 ## What is TOON?
 
 **Token-Oriented Object Notation** is a compact, human-readable serialization format designed for passing structured data to Large Language Models with significantly reduced token usage. It's intended for LLM input, not output.

package/SPEC.md

@@ -2,7 +2,7 @@
 
 ## Token-Oriented Object Notation
 
-**Version:** 1.5
+**Version:** 2.0
 
 **Date:** 2025-11-10
 
@@ -20,9 +20,9 @@ Token-Oriented Object Notation (TOON) is a line-oriented, indentation-based text
 
 ## Status of This Document
 
-This document is a Working Draft v1.4 and may be updated, replaced, or obsoleted. Implementers should monitor the canonical repository at https://github.com/toon-format/spec for changes.
+This document is a Working Draft v2.0 and may be updated, replaced, or obsoleted. Implementers should monitor the canonical repository at https://github.com/toon-format/spec for changes.
 
-This specification is stable for implementation but not yet finalized. Breaking changes are unlikely but possible before v2.0.
+This specification is stable for implementation but not yet finalized. Breaking changes may occur in future major versions.
 
 ## Normative References
 
@@ -165,7 +165,6 @@ Implementations that fail to conform to any MUST or REQUIRED level requirement a
 - Header: The bracketed declaration for arrays, optionally followed by a field list, and terminating with a colon; e.g., key[3]: or items[2]{a,b}:.
 - Field list: Brace-enclosed, delimiter-separated list of field names for tabular arrays: {f1<delim>f2}.
 - List item: A line beginning with "- " at a given depth representing an element in an expanded array.
-- Length marker: Optional "#" prefix for array lengths in headers, e.g., [#3]. Decoders MUST accept and ignore it semantically.
 
 ### 1.5 Delimiter Terms
 
@@ -294,13 +293,12 @@ TOON is a deterministic, line-oriented, indentation-based notation.
 Array headers declare length and active delimiter, and optionally field names.
 
 General forms:
-- Root header (no key): [<marker?>N<delim?>]:
-- With key: key[<marker?>N<delim?>]:
-- Tabular fields: key[<marker?>N<delim?>]{field1<delim>field2<delim>…}:
+- Root header (no key): [N<delim?>]:
+- With key: key[N<delim?>]:
+- Tabular fields: key[N<delim?>]{field1<delim>field2<delim>…}:
 
 Where:
 - N is the non-negative integer length.
-- <marker?> is optional "#"; decoders MUST accept and ignore it semantically.
 - <delim?> is:
   - absent for comma (","),
   - HTAB (U+0009) for tab,
@@ -329,7 +327,7 @@ LF     = %x0A                ; line feed
 SP     = %x20                ; space
 
 ; Header syntax
-bracket-seg   = "[" [ "#" ] 1*DIGIT [ delimsym ] "]"
+bracket-seg   = "[" 1*DIGIT [ delimsym ] "]"
 delimsym      = HTAB / "|"
 ; Field names are keys (quoted/unquoted) separated by the active delimiter
 fields-seg    = "{" fieldname *( delim fieldname ) "}"
@@ -592,7 +590,6 @@ Options:
 - Encoder options:
   - indent (default: 2 spaces)
   - delimiter (document delimiter; default: comma; alternatives: tab, pipe)
-  - lengthMarker (default: disabled)
   - keyFolding (default: `"off"`; alternatives: `"safe"`)
   - flattenDepth (default: Infinity when keyFolding is `"safe"`; non-negative integer ≥ 0; values 0 or 1 have no practical folding effect)
 - Decoder options:
@@ -700,7 +697,6 @@ Conforming decoders MUST:
 - [ ] Unescape quoted strings with only valid escapes (§7.1)
 - [ ] Type unquoted primitives: true/false/null → booleans/null, numeric → number, else → string (§4)
 - [ ] Enforce strict-mode rules when `strict=true` (§14)
-- [ ] Accept and ignore optional # length marker (§6)
 - [ ] Preserve array order and object key order (§2)
 - [ ] When `expandPaths="safe"`, expansion MUST follow §13.4 (IdentifierSegment-only segments, deep merge, conflict rules)
 - [ ] When `expandPaths="safe"` with `strict=true`, MUST error on expansion conflicts per §14.5
@@ -827,7 +823,7 @@ count: 2
 TOON's tabular format generalizes CSV [RFC4180] with several enhancements:
 
 Advantages over CSV:
-- Explicit array length markers enable validation
+- Explicit array length declarations enable validation
 - Field names declared in header (no separate header row)
 - Supports nested structures (CSV is flat-only)
 - Three delimiter options (comma/tab/pipe) vs CSV's comma-only
@@ -853,7 +849,7 @@ Conversion Guidelines:
 - CSV headers map to TOON field names
 - CSV data rows map to TOON tabular rows
 - CSV string escaping (double-quotes) maps to TOON quoting rules
-- CSV row count can be added as array length marker
+- CSV row count can be added as array length declaration
 
 ### 17.3 YAML Interoperability
 
@@ -1007,14 +1003,6 @@ items[2	]{sku	name	qty	price}:
 tags[3|]: reading|gaming|coding
 ```
 
-Length marker:
-```
-tags[#3]: reading,gaming,coding
-pairs[#2]:
-  - [#2]: a,b
-  - [#2]: c,d
-```
-
 Quoted colons and disambiguation (rows continue; colon is inside quotes):
 ```
 links[2]{id,url}:
@@ -1157,12 +1145,11 @@ These sketches illustrate structure and common decoding helpers. They are inform
 ### B.2 Array Header Parsing
 
 - Locate the first "[ … ]" segment on the line; parse:
-  - Optional leading "#" marker (ignored semantically).
   - Length N as decimal integer.
   - Optional delimiter symbol at the end: HTAB or pipe (comma otherwise).
 - If a "{ … }" fields segment occurs between the "]" and the ":", parse field names using the active delimiter; unescape quoted names.
 - Require a colon ":" after the bracket/fields segment.
-- Return the header (key?, length, delimiter, fields?, hasLengthMarker) and any inline values after the colon.
+- Return the header (key?, length, delimiter, fields?) and any inline values after the colon.
 - Absence of a delimiter symbol in the bracket segment ALWAYS means comma for that header (no inheritance).
 
 ### B.3 parseDelimitedValues
@@ -1231,6 +1218,14 @@ Note: Host-type normalization tests (e.g., BigInt, Date, Set, Map) are language-
 
 ## Appendix D: Document Changelog (Informative)
 
+### v2.0 (2025-11-10)
+
+- Breaking change: Length marker (`#`) prefix in array headers has been completely removed from the specification.
+- The `[#N]` format is no longer valid syntax. All array headers MUST use `[N]` format only.
+- Encoders MUST NOT emit `[#N]` format.
+- Decoders MUST NOT accept `[#N]` format (breaking change from v1.5).
+- Removed all references to length marker from terminology, grammar, conformance requirements, and parsing helpers.
+
 ### v1.5 (2025-11-08)
 
 - Added optional key folding for encoders: `keyFolding='safe'` mode with `flattenDepth` control (§13.4).
@@ -1291,7 +1286,6 @@ This specification and reference implementation are released under the MIT Licen
 - The reference encoder/decoder test suites implement:
   - Safe-unquoted string rules and delimiter-aware quoting (document vs active delimiter).
   - Header formation and delimiter-aware parsing with active delimiter scoping.
-  - Length marker propagation (encoding) and acceptance (decoding).
   - Tabular detection requiring uniform keys and primitive-only values.
   - Objects-as-list-items parsing (+2 nested object rule; +1 siblings).
   - Whitespace invariants for encoding and strict-mode indentation enforcement for decoding.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@toon-format/spec",
   "type": "module",
-  "version": "1.5.1",
+  "version": "2.0.0",
   "packageManager": "pnpm@10.19.0",
   "description": "Official specification for Token-Oriented Object Notation (TOON)",
   "author": "Johann Schopplich <hello@johannschopplich.com>",

package/tests/README.md

@@ -92,7 +92,6 @@ All test fixtures follow a standard JSON structure defined in [`fixtures.schema.
 {
   "delimiter": ",",
   "indent": 2,
-  "lengthMarker": "#",
   "keyFolding": "safe",
   "flattenDepth": 3
 }
@@ -100,7 +99,6 @@ All test fixtures follow a standard JSON structure defined in [`fixtures.schema.
 
 - `delimiter`: `","` (comma, default), `"\t"` (tab), or `"|"` (pipe). Affects encoder output; decoders parse the delimiter declared in array headers
 - `indent`: Number of spaces per indentation level (default: `2`)
-- `lengthMarker`: Optional. Set to `"#"` to prefix array lengths (e.g., `[#3]`). Omit this property to disable length markers
 - `keyFolding`: `"off"` (default) or `"safe"`. Enables key folding to collapse single-key object chains into dotted-path notation (v1.5+)
 - `flattenDepth`: Integer. Maximum depth to fold key chains when `keyFolding` is `"safe"` (default: Infinity). Values less than 2 have no practical folding effect (v1.5+)
 
@@ -161,7 +159,6 @@ The fixture format is language-agnostic JSON, so you can load and iterate it usi
 | `arrays-objects.json` | Objects as list items, complex nesting | §9, §10 |
 | `delimiters.json` | Tab and pipe delimiter options | §11 |
 | `whitespace.json` | Formatting invariants and indentation | §12 |
-| `options.json` | Length marker and delimiter option combinations | §3 |
 | `key-folding.json` | Key folding with safe mode, depth control, collision avoidance | §13.4 |
 
 ### Decoding Tests (`fixtures/decode/`)

package/tests/fixtures/decode/delimiters.json

@@ -241,14 +241,6 @@
         "items": [{ "a|b": 1 }, { "a|b": 2 }]
       },
       "specSection": "11"
-    },
-    {
-      "name": "accepts length marker with pipe delimiter",
-      "input": "tags[#3|]: reading|gaming|coding",
-      "expected": {
-        "tags": ["reading", "gaming", "coding"]
-      },
-      "specSection": "6"
     }
   ]
 }

package/tests/fixtures/encode/key-folding.json

@@ -198,17 +198,17 @@
     {
       "name": "encodes folded chains preserving sibling field order",
       "input": {
-        "folded": {
-          "path": {
-            "value": 1
+        "first": {
+          "second": {
+            "third": 1
           }
         },
-        "normal": 2,
-        "nested": {
-          "x": 3
+        "simple": 2,
+        "short": {
+          "path": 3
         }
       },
-      "expected": "folded.path.value: 1\nnormal: 2\nnested.x: 3",
+      "expected": "first.second.third: 1\nsimple: 2\nshort.path: 3",
       "options": {
         "keyFolding": "safe"
       },

package/tests/fixtures.schema.json

@@ -10,7 +10,7 @@
       "type": "string",
       "description": "TOON specification version these tests target",
       "pattern": "^\\d+\\.\\d+$",
-      "examples": ["1.0", "1.3"]
+      "examples": ["1.0", "1.6"]
     },
     "category": {
       "type": "string",
@@ -67,11 +67,6 @@
                 "minimum": 1,
                 "default": 2
               },
-              "lengthMarker": {
-                "type": "string",
-                "const": "#",
-                "description": "Optional marker to prefix array lengths (encode only). Omit to disable the marker."
-              },
               "strict": {
                 "type": "boolean",
                 "description": "Enable strict validation (decode only)",

package/tests/fixtures/encode/options.json

@@ -1,88 +0,0 @@
-{
-  "version": "1.4",
-  "category": "encode",
-  "description": "Encoding options - lengthMarker option and combinations with delimiters",
-  "tests": [
-    {
-      "name": "adds length marker to primitive arrays",
-      "input": {
-        "tags": ["reading", "gaming", "coding"]
-      },
-      "expected": "tags[#3]: reading,gaming,coding",
-      "options": {
-        "lengthMarker": "#"
-      },
-      "specSection": "3"
-    },
-    {
-      "name": "adds length marker to empty arrays",
-      "input": {
-        "items": []
-      },
-      "expected": "items[#0]:",
-      "options": {
-        "lengthMarker": "#"
-      },
-      "specSection": "3"
-    },
-    {
-      "name": "adds length marker to tabular arrays",
-      "input": {
-        "items": [
-          { "sku": "A1", "qty": 2, "price": 9.99 },
-          { "sku": "B2", "qty": 1, "price": 14.5 }
-        ]
-      },
-      "expected": "items[#2]{sku,qty,price}:\n  A1,2,9.99\n  B2,1,14.5",
-      "options": {
-        "lengthMarker": "#"
-      },
-      "specSection": "3"
-    },
-    {
-      "name": "adds length marker to nested arrays",
-      "input": {
-        "pairs": [["a", "b"], ["c", "d"]]
-      },
-      "expected": "pairs[#2]:\n  - [#2]: a,b\n  - [#2]: c,d",
-      "options": {
-        "lengthMarker": "#"
-      },
-      "specSection": "3"
-    },
-    {
-      "name": "combines length marker with pipe delimiter",
-      "input": {
-        "tags": ["reading", "gaming", "coding"]
-      },
-      "expected": "tags[#3|]: reading|gaming|coding",
-      "options": {
-        "lengthMarker": "#",
-        "delimiter": "|"
-      },
-      "specSection": "3"
-    },
-    {
-      "name": "combines length marker with tab delimiter",
-      "input": {
-        "tags": ["reading", "gaming", "coding"]
-      },
-      "expected": "tags[#3\t]: reading\tgaming\tcoding",
-      "options": {
-        "lengthMarker": "#",
-        "delimiter": "\t"
-      },
-      "specSection": "3"
-    },
-    {
-      "name": "default lengthMarker is empty (no marker)",
-      "input": {
-        "tags": ["reading", "gaming", "coding"]
-      },
-      "expected": "tags[3]: reading,gaming,coding",
-      "options": {},
-      "specSection": "3",
-      "note": "Default behavior without lengthMarker option"
-    }
-  ]
-}