@toon-format/spec 1.5.2 → 2.0.1

package/CHANGELOG.md

@@ -5,6 +5,28 @@ All notable changes to the TOON specification will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.0] - 2025-11-10
+
+### Breaking Changes
+
+- **Removed:** Length marker (`#`) prefix in array headers has been completely removed from the specification
+- The `[#N]` format is no longer valid syntax. All array headers MUST use `[N]` format only
+- Encoders MUST NOT emit `[#N]` format
+- Decoders MUST NOT accept `[#N]` format (breaking change from v1.5)
+
+### Removed
+
+- All references to length marker from terminology (§1.4), header syntax (§6), ABNF grammar, conformance requirements (§13.2), and parsing helpers (Appendix B)
+- `lengthMarker` encoder option removed from all implementations
+- Length marker test fixtures removed
+
+### Migration from v1.5
+
+- Update decoder implementations to reject `[#N]` syntax
+- Convert any existing `.toon` files using `[#N]` format to `[N]` format
+- Remove `lengthMarker` option from encoder configurations
+- Remove `--length-marker` CLI flags if present
+
 ## [1.5] - 2025-11-08
 
 ### Added

package/CONTRIBUTING.md

@@ -32,7 +32,7 @@ New data type normalization rules
   Example: Handling Map or Set differently
 
 Changing array header syntax
-  Example: Optional length markers becoming required
+  Example: Making field lists mandatory for all arrays
 ```
 
 ### No – Direct PR or Issue First

package/README.md

@@ -1,67 +1,93 @@
 # TOON Format Specification
 
-[![SPEC v1.5](https://img.shields.io/badge/spec-v1.5-lightgrey)](./SPEC.md)
-[![Tests](https://img.shields.io/badge/tests-348-green)](./tests/fixtures/)
+[![SPEC v2.0](https://img.shields.io/badge/spec-v2.0-lightgrey)](./SPEC.md)
+[![Tests](https://img.shields.io/badge/tests-342-green)](./tests/fixtures/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)
 
-This repository contains the official specification for **Token-Oriented Object Notation (TOON)**, a compact, human-readable serialization format designed for passing structured data to Large Language Models with significantly reduced token usage.
+This repository contains the official specification for **Token-Oriented Object Notation (TOON)**, a compact, human-readable encoding of the JSON data model for LLM prompts. It provides a lossless serialization of the same objects, arrays, and primitives as JSON, but in a syntax that minimizes tokens and makes structure easy for models to follow.
 
 ## 📋 Specification
 
 [→ Read the full specification (SPEC.md)](./SPEC.md)
 
-- **Version:** 1.5 (2025-11-10)
+- **Version:** 2.0 (2025-11-10)
 - **Status:** Working Draft
 - **License:** MIT
 
 The specification includes complete grammar (ABNF), encoding rules, validation requirements, and conformance criteria.
 
-### New in v1.5
-
-- **Key Folding** (encode): Collapse nested single-key objects into compact dotted paths
-  - `{"a": {"b": {"c": 1}}}` → `a.b.c: 1`
-  - Opt-in via `keyFolding="safe"` with `flattenDepth` control
-- **Path Expansion** (decode): Expand dotted keys back to nested objects
-  - `a.b.c: 1` → `{"a": {"b": {"c": 1}}}`
-  - Opt-in via `expandPaths="safe"` with deep-merge semantics
-
-> [!NOTE]
-> Both features are opt-in to maintain backward compatibility.
-
 ## What is TOON?
 
-**Token-Oriented Object Notation** is a compact, human-readable serialization format designed for passing structured data to Large Language Models with significantly reduced token usage. It's intended for LLM input, not output.
-
-TOON's sweet spot is **uniform arrays of objects** – multiple fields per row, same structure across items. It borrows YAML's indentation-based structure for nested objects and CSV's tabular format for uniform data rows, then optimizes both for token efficiency in LLM contexts. For deeply nested or non-uniform data, JSON may be more efficient.
-
-**Key Features:**
+> [!IMPORTANT]
+> For a high-level overview of TOON, its features and benefits, design goals, and comparisons to other formats, see the [`toon-format/toon` repository](https://github.com/toon-format/toon).
 
-- 💸 **Token-efficient:** typically 30–60% fewer tokens than JSON
-- 🤿 **LLM-friendly guardrails:** explicit lengths and fields enable validation
-- 🍱 **Minimal syntax:** removes redundant punctuation (braces, brackets, most quotes)
-- 📐 **Indentation-based structure:** like YAML, uses whitespace instead of braces
-- 🧺 **Tabular arrays:** declare keys once, stream data as rows
+## Serialization Example
 
-## Quick Example
-
-**JSON:**
+<table>
+<tr>
+<th>JSON</th>
+<th>TOON</th>
+</tr>
+<tr>
+<td>
 
 ```json
 {
-  "users": [
-    { "id": 1, "name": "Alice", "role": "admin" },
-    { "id": 2, "name": "Bob", "role": "user" }
+  "context": {
+    "task": "Our favorite hikes together",
+    "location": "Boulder",
+    "season": "spring_2025"
+  },
+  "friends": ["ana", "luis", "sam"],
+  "hikes": [
+    {
+      "id": 1,
+      "name": "Blue Lake Trail",
+      "distanceKm": 7.5,
+      "elevationGain": 320,
+      "companion": "ana",
+      "wasSunny": true
+    },
+    {
+      "id": 2,
+      "name": "Ridge Overlook",
+      "distanceKm": 9.2,
+      "elevationGain": 540,
+      "companion": "luis",
+      "wasSunny": false
+    },
+    {
+      "id": 3,
+      "name": "Wildflower Loop",
+      "distanceKm": 5.1,
+      "elevationGain": 180,
+      "companion": "sam",
+      "wasSunny": true
+    }
   ]
 }
 ```
 
-**TOON:**
+</td>
+<td>
 
+```toon
+context:
+  task: Our favorite hikes together
+  location: Boulder
+  season: spring_2025
+
+friends[3]: ana,luis,sam
+
+hikes[3]{id,name,distanceKm,elevationGain,companion,wasSunny}:
+  1,Blue Lake Trail,7.5,320,ana,true
+  2,Ridge Overlook,9.2,540,luis,false
+  3,Wildflower Loop,5.1,180,sam,true
 ```
-users[2]{id,name,role}:
-  1,Alice,admin
-  2,Bob,user
-```
+
+</td>
+</tr>
+</table>
 
 ## Reference Implementation
 

package/SPEC.md

@@ -2,7 +2,7 @@
 
 ## Token-Oriented Object Notation
 
-**Version:** 1.5
+**Version:** 2.0
 
 **Date:** 2025-11-10
 
@@ -20,9 +20,9 @@ Token-Oriented Object Notation (TOON) is a line-oriented, indentation-based text
 
 ## Status of This Document
 
-This document is a Working Draft v1.4 and may be updated, replaced, or obsoleted. Implementers should monitor the canonical repository at https://github.com/toon-format/spec for changes.
+This document is a Working Draft v2.0 and may be updated, replaced, or obsoleted. Implementers should monitor the canonical repository at https://github.com/toon-format/spec for changes.
 
-This specification is stable for implementation but not yet finalized. Breaking changes are unlikely but possible before v2.0.
+This specification is stable for implementation but not yet finalized. Breaking changes may occur in future major versions.
 
 ## Normative References
 
@@ -165,7 +165,6 @@ Implementations that fail to conform to any MUST or REQUIRED level requirement a
 - Header: The bracketed declaration for arrays, optionally followed by a field list, and terminating with a colon; e.g., key[3]: or items[2]{a,b}:.
 - Field list: Brace-enclosed, delimiter-separated list of field names for tabular arrays: {f1<delim>f2}.
 - List item: A line beginning with "- " at a given depth representing an element in an expanded array.
-- Length marker: Optional "#" prefix for array lengths in headers, e.g., [#3]. Decoders MUST accept and ignore it semantically.
 
 ### 1.5 Delimiter Terms
 
@@ -294,13 +293,12 @@ TOON is a deterministic, line-oriented, indentation-based notation.
 Array headers declare length and active delimiter, and optionally field names.
 
 General forms:
-- Root header (no key): [<marker?>N<delim?>]:
-- With key: key[<marker?>N<delim?>]:
-- Tabular fields: key[<marker?>N<delim?>]{field1<delim>field2<delim>…}:
+- Root header (no key): [N<delim?>]:
+- With key: key[N<delim?>]:
+- Tabular fields: key[N<delim?>]{field1<delim>field2<delim>…}:
 
 Where:
 - N is the non-negative integer length.
-- <marker?> is optional "#"; decoders MUST accept and ignore it semantically.
 - <delim?> is:
   - absent for comma (","),
   - HTAB (U+0009) for tab,
@@ -329,7 +327,7 @@ LF     = %x0A                ; line feed
 SP     = %x20                ; space
 
 ; Header syntax
-bracket-seg   = "[" [ "#" ] 1*DIGIT [ delimsym ] "]"
+bracket-seg   = "[" 1*DIGIT [ delimsym ] "]"
 delimsym      = HTAB / "|"
 ; Field names are keys (quoted/unquoted) separated by the active delimiter
 fields-seg    = "{" fieldname *( delim fieldname ) "}"
@@ -592,7 +590,6 @@ Options:
 - Encoder options:
   - indent (default: 2 spaces)
   - delimiter (document delimiter; default: comma; alternatives: tab, pipe)
-  - lengthMarker (default: disabled)
   - keyFolding (default: `"off"`; alternatives: `"safe"`)
   - flattenDepth (default: Infinity when keyFolding is `"safe"`; non-negative integer ≥ 0; values 0 or 1 have no practical folding effect)
 - Decoder options:
@@ -700,7 +697,6 @@ Conforming decoders MUST:
 - [ ] Unescape quoted strings with only valid escapes (§7.1)
 - [ ] Type unquoted primitives: true/false/null → booleans/null, numeric → number, else → string (§4)
 - [ ] Enforce strict-mode rules when `strict=true` (§14)
-- [ ] Accept and ignore optional # length marker (§6)
 - [ ] Preserve array order and object key order (§2)
 - [ ] When `expandPaths="safe"`, expansion MUST follow §13.4 (IdentifierSegment-only segments, deep merge, conflict rules)
 - [ ] When `expandPaths="safe"` with `strict=true`, MUST error on expansion conflicts per §14.5
@@ -827,7 +823,7 @@ count: 2
 TOON's tabular format generalizes CSV [RFC4180] with several enhancements:
 
 Advantages over CSV:
-- Explicit array length markers enable validation
+- Explicit array length declarations enable validation
 - Field names declared in header (no separate header row)
 - Supports nested structures (CSV is flat-only)
 - Three delimiter options (comma/tab/pipe) vs CSV's comma-only
@@ -853,7 +849,7 @@ Conversion Guidelines:
 - CSV headers map to TOON field names
 - CSV data rows map to TOON tabular rows
 - CSV string escaping (double-quotes) maps to TOON quoting rules
-- CSV row count can be added as array length marker
+- CSV row count can be added as array length declaration
 
 ### 17.3 YAML Interoperability
 
@@ -1007,14 +1003,6 @@ items[2	]{sku	name	qty	price}:
 tags[3|]: reading|gaming|coding
 ```
 
-Length marker:
-```
-tags[#3]: reading,gaming,coding
-pairs[#2]:
-  - [#2]: a,b
-  - [#2]: c,d
-```
-
 Quoted colons and disambiguation (rows continue; colon is inside quotes):
 ```
 links[2]{id,url}:
@@ -1157,12 +1145,11 @@ These sketches illustrate structure and common decoding helpers. They are inform
 ### B.2 Array Header Parsing
 
 - Locate the first "[ … ]" segment on the line; parse:
-  - Optional leading "#" marker (ignored semantically).
   - Length N as decimal integer.
   - Optional delimiter symbol at the end: HTAB or pipe (comma otherwise).
 - If a "{ … }" fields segment occurs between the "]" and the ":", parse field names using the active delimiter; unescape quoted names.
 - Require a colon ":" after the bracket/fields segment.
-- Return the header (key?, length, delimiter, fields?, hasLengthMarker) and any inline values after the colon.
+- Return the header (key?, length, delimiter, fields?) and any inline values after the colon.
 - Absence of a delimiter symbol in the bracket segment ALWAYS means comma for that header (no inheritance).
 
 ### B.3 parseDelimitedValues
@@ -1231,6 +1218,14 @@ Note: Host-type normalization tests (e.g., BigInt, Date, Set, Map) are language-
 
 ## Appendix D: Document Changelog (Informative)
 
+### v2.0 (2025-11-10)
+
+- Breaking change: Length marker (`#`) prefix in array headers has been completely removed from the specification.
+- The `[#N]` format is no longer valid syntax. All array headers MUST use `[N]` format only.
+- Encoders MUST NOT emit `[#N]` format.
+- Decoders MUST NOT accept `[#N]` format (breaking change from v1.5).
+- Removed all references to length marker from terminology, grammar, conformance requirements, and parsing helpers.
+
 ### v1.5 (2025-11-08)
 
 - Added optional key folding for encoders: `keyFolding='safe'` mode with `flattenDepth` control (§13.4).
@@ -1291,7 +1286,6 @@ This specification and reference implementation are released under the MIT Licen
 - The reference encoder/decoder test suites implement:
   - Safe-unquoted string rules and delimiter-aware quoting (document vs active delimiter).
   - Header formation and delimiter-aware parsing with active delimiter scoping.
-  - Length marker propagation (encoding) and acceptance (decoding).
   - Tabular detection requiring uniform keys and primitive-only values.
   - Objects-as-list-items parsing (+2 nested object rule; +1 siblings).
   - Whitespace invariants for encoding and strict-mode indentation enforcement for decoding.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@toon-format/spec",
   "type": "module",
-  "version": "1.5.2",
+  "version": "2.0.1",
   "packageManager": "pnpm@10.19.0",
   "description": "Official specification for Token-Oriented Object Notation (TOON)",
   "author": "Johann Schopplich <hello@johannschopplich.com>",

package/tests/README.md

@@ -92,7 +92,6 @@ All test fixtures follow a standard JSON structure defined in [`fixtures.schema.
 {
   "delimiter": ",",
   "indent": 2,
-  "lengthMarker": "#",
   "keyFolding": "safe",
   "flattenDepth": 3
 }
@@ -100,7 +99,6 @@ All test fixtures follow a standard JSON structure defined in [`fixtures.schema.
 
 - `delimiter`: `","` (comma, default), `"\t"` (tab), or `"|"` (pipe). Affects encoder output; decoders parse the delimiter declared in array headers
 - `indent`: Number of spaces per indentation level (default: `2`)
-- `lengthMarker`: Optional. Set to `"#"` to prefix array lengths (e.g., `[#3]`). Omit this property to disable length markers
 - `keyFolding`: `"off"` (default) or `"safe"`. Enables key folding to collapse single-key object chains into dotted-path notation (v1.5+)
 - `flattenDepth`: Integer. Maximum depth to fold key chains when `keyFolding` is `"safe"` (default: Infinity). Values less than 2 have no practical folding effect (v1.5+)
 
@@ -161,7 +159,6 @@ The fixture format is language-agnostic JSON, so you can load and iterate it usi
 | `arrays-objects.json` | Objects as list items, complex nesting | §9, §10 |
 | `delimiters.json` | Tab and pipe delimiter options | §11 |
 | `whitespace.json` | Formatting invariants and indentation | §12 |
-| `options.json` | Length marker and delimiter option combinations | §3 |
 | `key-folding.json` | Key folding with safe mode, depth control, collision avoidance | §13.4 |
 
 ### Decoding Tests (`fixtures/decode/`)

package/tests/fixtures/decode/arrays-nested.json

@@ -69,10 +69,10 @@
     },
     {
       "name": "parses objects containing arrays (including empty arrays) in list format",
-      "input": "items[1]:\n  - name: test\n    data[0]:",
+      "input": "items[1]:\n  - name: Ada\n    data[0]:",
       "expected": {
         "items": [
-          { "name": "test", "data": [] }
+          { "name": "Ada", "data": [] }
         ]
       },
       "specSection": "9.4"
@@ -120,35 +120,41 @@
       "specSection": "9.2"
     },
     {
-      "name": "parses root arrays of primitives (inline)",
+      "name": "parses root-level primitive array inline",
       "input": "[5]: x,y,\"true\",true,10",
       "expected": ["x", "y", "true", true, 10],
       "specSection": "9.1"
     },
     {
-      "name": "parses root arrays of uniform objects in tabular format",
+      "name": "parses root-level array of uniform objects in tabular format",
       "input": "[2]{id}:\n  1\n  2",
       "expected": [{ "id": 1 }, { "id": 2 }],
       "specSection": "9.3"
     },
     {
-      "name": "parses root arrays of non-uniform objects in list format",
+      "name": "parses root-level array of non-uniform objects in list format",
       "input": "[2]:\n  - id: 1\n  - id: 2\n    name: Ada",
       "expected": [{ "id": 1 }, { "id": 2, "name": "Ada" }],
       "specSection": "9.4"
     },
     {
-      "name": "parses empty root arrays",
-      "input": "[0]:",
-      "expected": [],
-      "specSection": "9.1"
+      "name": "parses root-level array mixing primitive, object, and array of objects in list format",
+      "input": "[3]:\n  - summary\n  - id: 1\n    name: Ada\n  - [2]:\n    - id: 2\n    - status: draft",
+      "expected": ["summary", { "id": 1, "name": "Ada" }, [{ "id": 2 }, { "status": "draft" }]],
+      "specSection": "9.4"
     },
     {
-      "name": "parses root arrays of arrays",
+      "name": "parses root-level array of arrays",
       "input": "[2]:\n  - [2]: 1,2\n  - [0]:",
       "expected": [[1, 2], []],
       "specSection": "9.2"
     },
+    {
+      "name": "parses empty root-level array",
+      "input": "[0]:",
+      "expected": [],
+      "specSection": "9.1"
+    },
     {
       "name": "parses complex mixed object with arrays and nested objects",
       "input": "user:\n  id: 123\n  name: Ada\n  tags[2]: reading,gaming\n  active: true\n  prefs[0]:",
@@ -164,7 +170,7 @@
       "specSection": "8"
     },
     {
-      "name": "parses arrays mixing primitives, objects and strings (list format)",
+      "name": "parses arrays mixing primitives, objects, and strings in list format",
       "input": "items[3]:\n  - 1\n  - a: 1\n  - text",
       "expected": {
         "items": [1, { "a": 1 }, "text"]

package/tests/fixtures/decode/arrays-tabular.json

@@ -59,7 +59,7 @@
       "specSection": "9.3"
     },
     {
-      "name": "unquoted colon terminates tabular rows and starts key-value pair",
+      "name": "treats unquoted colon as terminator for tabular rows and start of key-value pair",
       "input": "items[2]{id,name}:\n  1,Alice\n  2,Bob\ncount: 2",
       "expected": {
         "items": [

package/tests/fixtures/decode/delimiters.json

@@ -66,7 +66,7 @@
       "specSection": "11"
     },
     {
-      "name": "nested arrays inside list items default to comma delimiter",
+      "name": "parses nested arrays inside list items with default comma delimiter",
       "input": "items[1\t]:\n  - tags[3]: a,b,c",
       "expected": {
         "items": [{ "tags": ["a", "b", "c"] }]
@@ -75,7 +75,7 @@
       "note": "Parent uses tab, nested defaults to comma"
     },
     {
-      "name": "nested arrays inside list items default to comma with pipe parent",
+      "name": "parses nested arrays inside list items with default comma delimiter when parent uses pipe",
       "input": "items[1|]:\n  - tags[3]: a,b,c",
       "expected": {
         "items": [{ "tags": ["a", "b", "c"] }]
@@ -83,25 +83,25 @@
       "specSection": "11"
     },
     {
-      "name": "parses root arrays with tab delimiter",
+      "name": "parses root-level array with tab delimiter",
       "input": "[3\t]: x\ty\tz",
       "expected": ["x", "y", "z"],
       "specSection": "11"
     },
     {
-      "name": "parses root arrays with pipe delimiter",
+      "name": "parses root-level array with pipe delimiter",
       "input": "[3|]: x|y|z",
       "expected": ["x", "y", "z"],
       "specSection": "11"
     },
     {
-      "name": "parses root arrays of objects with tab delimiter",
+      "name": "parses root-level array of objects with tab delimiter",
       "input": "[2\t]{id}:\n  1\n  2",
       "expected": [{ "id": 1 }, { "id": 2 }],
       "specSection": "11"
     },
     {
-      "name": "parses root arrays of objects with pipe delimiter",
+      "name": "parses root-level array of objects with pipe delimiter",
       "input": "[2|]{id}:\n  1\n  2",
       "expected": [{ "id": 1 }, { "id": 2 }],
       "specSection": "11"
@@ -241,14 +241,6 @@
         "items": [{ "a|b": 1 }, { "a|b": 2 }]
       },
       "specSection": "11"
-    },
-    {
-      "name": "accepts length marker with pipe delimiter",
-      "input": "tags[#3|]: reading|gaming|coding",
-      "expected": {
-        "tags": ["reading", "gaming", "coding"]
-      },
-      "specSection": "6"
     }
   ]
 }

package/tests/fixtures/decode/indentation-errors.json

@@ -4,7 +4,7 @@
   "description": "Strict mode indentation validation - non-multiple indentation, tab characters, custom indent sizes",
   "tests": [
     {
-      "name": "throws when object field has non-multiple indentation (3 spaces with indent=2)",
+      "name": "throws on object field with non-multiple indentation (3 spaces with indent=2)",
       "input": "a:\n   b: 1",
       "expected": null,
       "shouldError": true,
@@ -15,7 +15,7 @@
       "specSection": "14.3"
     },
     {
-      "name": "throws when list item has non-multiple indentation (3 spaces with indent=2)",
+      "name": "throws on list item with non-multiple indentation (3 spaces with indent=2)",
       "input": "items[2]:\n   - id: 1\n   - id: 2",
       "expected": null,
       "shouldError": true,
@@ -26,7 +26,7 @@
       "specSection": "14.3"
     },
     {
-      "name": "throws with custom indent size when non-multiple (3 spaces with indent=4)",
+      "name": "throws on non-multiple indentation with custom indent=4 (3 spaces)",
       "input": "a:\n   b: 1",
       "expected": null,
       "shouldError": true,
@@ -51,7 +51,7 @@
       "specSection": "12"
     },
     {
-      "name": "throws when tab character used in indentation",
+      "name": "throws on tab character used in indentation",
       "input": "a:\n\tb: 1",
       "expected": null,
       "shouldError": true,
@@ -61,7 +61,7 @@
       "specSection": "14.3"
     },
     {
-      "name": "throws when mixed tabs and spaces in indentation",
+      "name": "throws on mixed tabs and spaces in indentation",
       "input": "a:\n \tb: 1",
       "expected": null,
       "shouldError": true,
@@ -71,7 +71,7 @@
       "specSection": "14.3"
     },
     {
-      "name": "throws when tab at start of line",
+      "name": "throws on tab at start of line",
       "input": "\ta: 1",
       "expected": null,
       "shouldError": true,
@@ -144,7 +144,7 @@
       "specSection": "12"
     },
     {
-      "name": "empty lines do not trigger validation errors",
+      "name": "parses empty lines without validation errors",
       "input": "a: 1\n\nb: 2",
       "expected": {
         "a": 1,
@@ -156,7 +156,7 @@
       "specSection": "12"
     },
     {
-      "name": "root-level content (0 indentation) is always valid",
+      "name": "parses root-level content (0 indentation) as always valid",
       "input": "a: 1\nb: 2\nc: 3",
       "expected": {
         "a": 1,
@@ -169,7 +169,7 @@
       "specSection": "12"
     },
     {
-      "name": "lines with only spaces are not validated if empty",
+      "name": "parses lines with only spaces without validation if empty",
       "input": "a: 1\n   \nb: 2",
       "expected": {
         "a": 1,

package/tests/fixtures/decode/root-form.json

@@ -4,7 +4,7 @@
   "description": "Root form detection - empty document, single primitive, multiple primitives",
   "tests": [
     {
-      "name": "empty document decodes to empty object",
+      "name": "parses empty document as empty object",
       "input": "",
       "expected": {},
       "options": {

package/tests/fixtures/decode/validation-errors.json

@@ -18,14 +18,14 @@
       "specSection": "14.1"
     },
     {
-      "name": "throws when tabular row value count does not match header field count",
+      "name": "throws on tabular row value count mismatch with header field count",
       "input": "items[2]{id,name}:\n  1,Ada\n  2",
       "expected": null,
       "shouldError": true,
       "specSection": "14.1"
     },
     {
-      "name": "throws when tabular row count does not match header length",
+      "name": "throws on tabular row count mismatch with header length",
       "input": "[1]{id}:\n  1\n  2",
       "expected": null,
       "shouldError": true,

package/tests/fixtures/decode/whitespace.json

@@ -49,7 +49,7 @@
       "specSection": "12"
     },
     {
-      "name": "empty tokens decode to empty string",
+      "name": "parses empty tokens as empty string",
       "input": "items[3]: a,,c",
       "expected": {
         "items": ["a", "", "c"]

package/tests/fixtures/encode/arrays-nested.json

@@ -54,10 +54,10 @@
       "specSection": "9.4"
     },
     {
-      "name": "encodes empty root-level array",
-      "input": [],
-      "expected": "[0]:",
-      "specSection": "9.1"
+      "name": "encodes root-level array mixing primitive, object, and array of objects in list format",
+      "input": ["summary", { "id": 1, "name": "Ada" }, [{ "id": 2 }, { "status": "draft" }]],
+      "expected": "[3]:\n  - summary\n  - id: 1\n    name: Ada\n  - [2]:\n    - id: 2\n    - status: draft",
+      "specSection": "9.4"
     },
     {
       "name": "encodes root-level arrays of arrays",
@@ -65,6 +65,12 @@
       "expected": "[2]:\n  - [2]: 1,2\n  - [0]:",
       "specSection": "9.2"
     },
+    {
+      "name": "encodes empty root-level array",
+      "input": [],
+      "expected": "[0]:",
+      "specSection": "9.1"
+    },
     {
       "name": "encodes complex nested structure",
       "input": {

package/tests/fixtures/encode/arrays-objects.json

@@ -27,17 +27,17 @@
     {
       "name": "preserves field order in list items - array first",
       "input": {
-        "items": [{ "nums": [1, 2, 3], "name": "test" }]
+        "items": [{ "nums": [1, 2, 3], "name": "Ada" }]
       },
-      "expected": "items[1]:\n  - nums[3]: 1,2,3\n    name: test",
+      "expected": "items[1]:\n  - nums[3]: 1,2,3\n    name: Ada",
       "specSection": "10"
     },
     {
       "name": "preserves field order in list items - primitive first",
       "input": {
-        "items": [{ "name": "test", "nums": [1, 2, 3] }]
+        "items": [{ "name": "Ada", "nums": [1, 2, 3] }]
       },
-      "expected": "items[1]:\n  - name: test\n    nums[3]: 1,2,3",
+      "expected": "items[1]:\n  - name: Ada\n    nums[3]: 1,2,3",
       "specSection": "10"
     },
     {
@@ -90,10 +90,10 @@
       "name": "encodes objects with empty arrays in list format",
       "input": {
         "items": [
-          { "name": "test", "data": [] }
+          { "name": "Ada", "data": [] }
         ]
       },
-      "expected": "items[1]:\n  - name: test\n    data[0]:",
+      "expected": "items[1]:\n  - name: Ada\n    data[0]:",
       "specSection": "10"
     },
     {
@@ -124,7 +124,7 @@
       "specSection": "9.3"
     },
     {
-      "name": "uses list format when one object has nested column",
+      "name": "uses list format when one object has nested field",
       "input": {
         "items": [
           { "id": 1, "data": "string" },

package/tests/fixtures/encode/arrays-tabular.json

@@ -4,7 +4,7 @@
   "description": "Tabular array encoding - arrays of uniform objects with primitive values",
   "tests": [
     {
-      "name": "encodes arrays of similar objects in tabular format",
+      "name": "encodes arrays of uniform objects in tabular format",
       "input": {
         "items": [
           { "sku": "A1", "qty": 2, "price": 9.99 },

package/tests/fixtures/encode/delimiters.json

@@ -87,7 +87,7 @@
       "specSection": "11"
     },
     {
-      "name": "encodes root arrays with tab delimiter",
+      "name": "encodes root-level array with tab delimiter",
       "input": ["x", "y", "z"],
       "expected": "[3\t]: x\ty\tz",
       "options": {
@@ -96,7 +96,7 @@
       "specSection": "11"
     },
     {
-      "name": "encodes root arrays with pipe delimiter",
+      "name": "encodes root-level array with pipe delimiter",
       "input": ["x", "y", "z"],
       "expected": "[3|]: x|y|z",
       "options": {
@@ -105,7 +105,7 @@
       "specSection": "11"
     },
     {
-      "name": "encodes root arrays of objects with tab delimiter",
+      "name": "encodes root-level array of objects with tab delimiter",
       "input": [{ "id": 1 }, { "id": 2 }],
       "expected": "[2\t]{id}:\n  1\n  2",
       "options": {
@@ -114,7 +114,7 @@
       "specSection": "11"
     },
     {
-      "name": "encodes root arrays of objects with pipe delimiter",
+      "name": "encodes root-level array of objects with pipe delimiter",
       "input": [{ "id": 1 }, { "id": 2 }],
       "expected": "[2|]{id}:\n  1\n  2",
       "options": {

package/tests/fixtures.schema.json

@@ -10,7 +10,7 @@
       "type": "string",
       "description": "TOON specification version these tests target",
       "pattern": "^\\d+\\.\\d+$",
-      "examples": ["1.0", "1.3"]
+      "examples": ["1.0", "1.6"]
     },
     "category": {
       "type": "string",
@@ -67,11 +67,6 @@
                 "minimum": 1,
                 "default": 2
               },
-              "lengthMarker": {
-                "type": "string",
-                "const": "#",
-                "description": "Optional marker to prefix array lengths (encode only). Omit to disable the marker."
-              },
               "strict": {
                 "type": "boolean",
                 "description": "Enable strict validation (decode only)",

package/tests/fixtures/encode/options.json

@@ -1,88 +0,0 @@
-{
-  "version": "1.4",
-  "category": "encode",
-  "description": "Encoding options - lengthMarker option and combinations with delimiters",
-  "tests": [
-    {
-      "name": "adds length marker to primitive arrays",
-      "input": {
-        "tags": ["reading", "gaming", "coding"]
-      },
-      "expected": "tags[#3]: reading,gaming,coding",
-      "options": {
-        "lengthMarker": "#"
-      },
-      "specSection": "3"
-    },
-    {
-      "name": "adds length marker to empty arrays",
-      "input": {
-        "items": []
-      },
-      "expected": "items[#0]:",
-      "options": {
-        "lengthMarker": "#"
-      },
-      "specSection": "3"
-    },
-    {
-      "name": "adds length marker to tabular arrays",
-      "input": {
-        "items": [
-          { "sku": "A1", "qty": 2, "price": 9.99 },
-          { "sku": "B2", "qty": 1, "price": 14.5 }
-        ]
-      },
-      "expected": "items[#2]{sku,qty,price}:\n  A1,2,9.99\n  B2,1,14.5",
-      "options": {
-        "lengthMarker": "#"
-      },
-      "specSection": "3"
-    },
-    {
-      "name": "adds length marker to nested arrays",
-      "input": {
-        "pairs": [["a", "b"], ["c", "d"]]
-      },
-      "expected": "pairs[#2]:\n  - [#2]: a,b\n  - [#2]: c,d",
-      "options": {
-        "lengthMarker": "#"
-      },
-      "specSection": "3"
-    },
-    {
-      "name": "combines length marker with pipe delimiter",
-      "input": {
-        "tags": ["reading", "gaming", "coding"]
-      },
-      "expected": "tags[#3|]: reading|gaming|coding",
-      "options": {
-        "lengthMarker": "#",
-        "delimiter": "|"
-      },
-      "specSection": "3"
-    },
-    {
-      "name": "combines length marker with tab delimiter",
-      "input": {
-        "tags": ["reading", "gaming", "coding"]
-      },
-      "expected": "tags[#3\t]: reading\tgaming\tcoding",
-      "options": {
-        "lengthMarker": "#",
-        "delimiter": "\t"
-      },
-      "specSection": "3"
-    },
-    {
-      "name": "default lengthMarker is empty (no marker)",
-      "input": {
-        "tags": ["reading", "gaming", "coding"]
-      },
-      "expected": "tags[3]: reading,gaming,coding",
-      "options": {},
-      "specSection": "3",
-      "note": "Default behavior without lengthMarker option"
-    }
-  ]
-}