@toon-format/spec 1.3.0

package/VERSIONING.md

@@ -0,0 +1,150 @@
+# TOON Specification Versioning Policy
+
+This document outlines the versioning policy for the TOON specification.
+
+## Semantic Versioning
+
+The TOON specification follows [Semantic Versioning](https://semver.org/) with a `MAJOR.MINOR` format (specifications don't need PATCH versions).
+
+### Version Format
+
+**`MAJOR.MINOR`**
+
+- **MAJOR version** - Incremented for breaking changes that are incompatible with previous versions
+- **MINOR version** - Incremented for backward-compatible additions, clarifications, or non-breaking changes
+
+**Example:** Moving from v1.3 to v1.4 means your implementation keeps working. Moving from v1.3 to v2.0 means you'll likely need to update your code.
+
+## What Constitutes a Breaking Change
+
+Breaking changes (requiring a MAJOR version bump) include:
+
+### Syntax Changes
+
+- Removing or changing existing syntax.
+- Changing the meaning of existing constructs.
+- Adding new reserved characters that could conflict with existing valid TOON documents.
+- Changing encoding/decoding behavior in incompatible ways.
+
+### Semantic Changes
+
+- Changing how valid TOON should be interpreted.
+- Modifying type conversion rules in incompatible ways.
+- Changing quoting rules in ways that break existing documents.
+- Altering delimiter behavior.
+
+### Conformance Changes
+
+- Making previously valid TOON invalid.
+- Adding new MUST requirements that existing implementations don't meet.
+- Changing error handling in ways that break round-trip compatibility.
+
+## What Constitutes a Non-Breaking Change
+
+Non-breaking changes (MINOR version bump) include:
+
+### Clarifications
+
+- Clarifying ambiguous language without changing behavior.
+- Adding examples to illustrate existing rules.
+- Improving specification wording for clarity.
+- Adding informative (non-normative) sections.
+
+### Backward-Compatible Additions
+
+- Adding optional features that don't affect existing documents.
+- Adding new SHOULD or MAY recommendations.
+- Expanding the specification to cover previously undefined behavior (if done in a backward-compatible way).
+- Adding new test cases that existing conformant implementations already pass.
+
+### Documentation Improvements
+
+- Grammar and typo fixes.
+- Reorganizing content for better readability.
+- Adding cross-references.
+- Improving examples.
+
+## Version Lifecycle
+
+### Working Draft
+
+- Current development version.
+- May receive updates without version changes.
+- Indicated by "Status: Working Draft" in the specification.
+
+### Stable Release
+
+- Released versions are immutable.
+- Version number is assigned when changes are merged.
+- Previous versions remain available for reference.
+
+### Deprecation
+
+If we need to make a breaking change (MAJOR version bump):
+
+1. **Announcement:** We add a deprecation notice to the current spec
+2. **Migration Period:** The next MINOR version includes migration guidance
+3. **New Major Version:** Breaking changes are released in the next MAJOR version
+4. **Support:** Previous MAJOR versions remain available – we don't break old links
+
+## Implementation Compatibility
+
+### Specification Version Support
+
+If you're implementing TOON, clearly document which spec version you support:
+
+```json
+{
+  "toon-spec": "1.3"
+}
+```
+
+This helps users know what behavior to expect.
+
+### Forward Compatibility
+
+Your implementation can support multiple spec versions. If you do:
+
+1. Default to the latest supported version
+2. Let users specify which version to target
+3. Document version-specific behavior clearly
+
+### Backward Compatibility
+
+**When a new MINOR version is released:**
+
+- Your implementation stays conformant – no code changes needed
+- Updates may be recommended but aren't required
+
+**When a new MAJOR version is released:**
+
+- You may need updates to support the new version
+- Previous version implementations remain valid
+- We'll provide a migration guide
+
+## Version Numbering Examples
+
+### Current: v1.3
+
+**Next minor update (clarifications, additions):**
+- v1.4
+
+**Next major update (breaking changes):**
+- v2.0
+
+**After v2.0, next minor update:**
+- v2.1
+
+## Version History
+
+See [CHANGELOG.md](./CHANGELOG.md) for detailed version history.
+
+## Questions
+
+Not sure if your proposed change is breaking or non-breaking?
+
+1. Open an issue with the `[RFC]` tag
+2. Describe the proposed change
+3. Ask for feedback from maintainers and community
+
+When in doubt, we err on the side of caution and treat potentially breaking changes as MAJOR version bumps.

package/package.json

@@ -0,0 +1,42 @@
+{
+  "name": "@toon-format/spec",
+  "type": "module",
+  "version": "1.3.0",
+  "packageManager": "pnpm@10.19.0",
+  "description": "Official specification for Token-Oriented Object Notation (TOON)",
+  "author": "Johann Schopplich <hello@johannschopplich.com>",
+  "license": "MIT",
+  "homepage": "https://github.com/toon-format/spec",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/toon-format/spec.git"
+  },
+  "bugs": {
+    "url": "https://github.com/toon-format/spec/issues"
+  },
+  "keywords": [
+    "toon",
+    "format",
+    "specification",
+    "llm",
+    "token-efficiency",
+    "data-format"
+  ],
+  "files": [
+    "CHANGELOG.md",
+    "CONTRIBUTING.md",
+    "SPEC.md",
+    "VERSIONING.md",
+    "tests"
+  ],
+  "scripts": {
+    "lint": "eslint .",
+    "release": "bumpp"
+  },
+  "devDependencies": {
+    "@antfu/eslint-config": "^6.2.0",
+    "bumpp": "^10.3.1",
+    "eslint": "^9.39.0",
+    "vitest": "^4.0.6"
+  }
+}

package/tests/README.md

@@ -0,0 +1,218 @@
+# TOON Test Fixtures
+
+This directory contains **comprehensive language-agnostic JSON test fixtures** for validating TOON implementations against the specification. These fixtures cover all specification requirements and provide a standardized conformance test suite.
+
+## Purpose
+
+The test fixtures serve multiple purposes:
+
+- **Conformance validation:** Verify implementations follow the specification
+- **Regression testing:** Catch behavioral changes across versions
+- **Implementation guide:** Demonstrate expected encoding/decoding behavior
+- **Cross-language consistency:** Ensure all implementations produce identical output
+
+## Directory Structure
+
+```
+tests/
+├── fixtures.schema.json    # JSON Schema for fixture validation
+├── fixtures/
+│   ├── encode/             # Encoding tests (JSON → TOON)
+│   │   ├── primitives.json
+│   │   ├── objects.json
+│   │   ├── arrays-primitive.json
+│   │   ├── arrays-tabular.json
+│   │   ├── arrays-nested.json
+│   │   ├── arrays-objects.json
+│   │   ├── delimiters.json
+│   │   ├── normalization.json
+│   │   ├── whitespace.json
+│   │   └── options.json
+│   └── decode/             # Decoding tests (TOON → JSON)
+│       ├── primitives.json
+│       ├── objects.json
+│       ├── arrays-primitive.json
+│       ├── arrays-tabular.json
+│       ├── arrays-nested.json
+│       ├── delimiters.json
+│       ├── validation-errors.json
+│       ├── indentation-errors.json
+│       └── blank-lines.json
+└── README.md               # This file
+```
+
+## Fixture Format
+
+All test fixtures follow a standard JSON structure defined in [`fixtures.schema.json`](./fixtures.schema.json):
+
+```json
+{
+  "version": "1.3",
+  "category": "encode",
+  "description": "Brief description of test category",
+  "tests": [
+    {
+      "name": "descriptive test name",
+      "input": "JSON value or TOON string",
+      "expected": "TOON string or JSON value",
+      "options": {},
+      "specSection": "7.2",
+      "note": "Optional explanation"
+    }
+  ]
+}
+```
+
+### Field Descriptions
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `version` | Yes | TOON specification version (e.g., `"1.3"`) |
+| `category` | Yes | Test category: `"encode"` or `"decode"` |
+| `description` | Yes | Brief description of what this fixture tests |
+| `tests` | Yes | Array of test cases |
+| `tests[].name` | Yes | Descriptive name explaining what is validated |
+| `tests[].input` | Yes | Input value (JSON for encode, TOON string for decode) |
+| `tests[].expected` | Yes | Expected output (TOON string for encode, JSON for decode) |
+| `tests[].shouldError` | No | If `true`, expects an error (default: `false`) |
+| `tests[].options` | No | Encoder/decoder options (see below) |
+| `tests[].specSection` | No | Reference to specification section (e.g., `"7.2"`, `"§6"`) |
+| `tests[].note` | No | Optional explanation for special cases |
+| `tests[].minSpecVersion` | No | Minimum spec version required (e.g., `"1.3"`) |
+
+### Options
+
+#### Encoding Options
+
+```json
+{
+  "delimiter": ",",
+  "indent": 2,
+  "lengthMarker": ""
+}
+```
+
+- `delimiter`: `","` (comma, default), `"\t"` (tab), or `"|"` (pipe)
+- `indent`: Number of spaces per indentation level (default: `2`)
+- `lengthMarker`: `"#"` to prefix array lengths, or `""` for no marker (default: `""`)
+
+#### Decoding Options
+
+```json
+{
+  "indent": 2,
+  "strict": true
+}
+```
+
+- `indent`: Expected number of spaces per level (default: `2`)
+- `strict`: Enable strict validation (default: `true`)
+
+### Error Tests
+
+Error tests use `shouldError: true` to indicate that the test expects an error to be thrown:
+
+```json
+{
+  "name": "throws on array length mismatch",
+  "input": "tags[3]: a,b",
+  "expected": null,
+  "shouldError": true,
+  "options": { "strict": true }
+}
+```
+
+**Note:** Error tests do not specify expected error messages, as these are implementation-specific and vary across languages.
+
+## Using These Tests
+
+To validate your TOON implementation against these fixtures:
+
+1. **Load a fixture file** from `fixtures/encode/` or `fixtures/decode/`.
+2. **Iterate through the `tests` array** in the fixture.
+3. **For each test case:**
+   - If `shouldError` is `true`: verify your implementation throws an error.
+   - Otherwise: assert that your encoder/decoder produces the `expected` output when given the `input`.
+4. **Pass options** from `test.options` to your encoder/decoder (if present).
+
+The fixture format is language-agnostic JSON, so you can load and iterate it using your language's standard JSON parser and test framework.
+
+## Test Coverage
+
+### Encoding Tests (`fixtures/encode/`)
+
+| File | Description | Spec Sections |
+|------|-------------|---------------|
+| `primitives.json` | String, number, boolean, null encoding and escaping | §5 |
+| `objects.json` | Simple objects, nested objects, key encoding | §6 |
+| `arrays-primitive.json` | Inline primitive arrays, empty arrays | §7.1 |
+| `arrays-tabular.json` | Tabular format with header and rows | §7.2 |
+| `arrays-nested.json` | Arrays of arrays, mixed arrays | §7.3 |
+| `arrays-objects.json` | Objects as list items, complex nesting | §7 |
+| `delimiters.json` | Tab and pipe delimiter options | §8 |
+| `normalization.json` | BigInt, Date, undefined, NaN, Infinity handling | §5 |
+| `whitespace.json` | Formatting invariants and indentation | §4 |
+| `options.json` | Length marker and delimiter option combinations | §3 |
+
+### Decoding Tests (`fixtures/decode/`)
+
+| File | Description | Spec Sections |
+|------|-------------|---------------|
+| `primitives.json` | Parsing primitives, unescaping, ambiguity | §5 |
+| `objects.json` | Parsing objects, keys, nesting | §6 |
+| `arrays-primitive.json` | Inline array parsing | §7.1 |
+| `arrays-tabular.json` | Tabular format parsing | §7.2 |
+| `arrays-nested.json` | Nested and mixed array parsing | §7.3 |
+| `delimiters.json` | Delimiter detection and parsing | §8 |
+| `validation-errors.json` | Syntax errors, length mismatches, malformed input | §9 |
+| `indentation-errors.json` | Strict mode indentation validation | §9 |
+| `blank-lines.json` | Blank line handling in arrays | §9 |
+
+## Validating Fixtures
+
+All fixture files should validate against [`fixtures.schema.json`](./fixtures.schema.json). You can use standard JSON Schema validators:
+
+```bash
+# Using ajv-cli
+npx ajv-cli validate -s fixtures.schema.json -d "fixtures/**/*.json"
+
+# Using check-jsonschema (Python)
+pip install check-jsonschema
+check-jsonschema --schemafile fixtures.schema.json fixtures/**/*.json
+```
+
+## Contributing Test Cases
+
+To contribute new test cases:
+
+1. **Identify the category:** Which fixture file should contain the test?
+2. **Follow the format:** Use the structure defined in `fixtures.schema.json`
+3. **Add spec references:** Link to relevant specification sections
+4. **Validate:** Ensure your fixture validates against the schema
+5. **Test with reference implementation:** Verify expected output is correct
+6. **Submit PR:** Include clear description of what the test validates
+
+See [CONTRIBUTING.md](../CONTRIBUTING.md) for detailed guidelines.
+
+## Reference Implementation
+
+The reference implementation in TypeScript/JavaScript is maintained at: [github.com/toon-format/toon](https://github.com/toon-format/toon)
+
+## Questions or Issues?
+
+If you find:
+
+- Test cases that contradict the specification
+- Missing coverage for edge cases
+- Ambiguous expected outputs
+- Schema validation issues
+
+Please [open an issue](https://github.com/toon-format/spec/issues) with:
+
+- Fixture file and test case name
+- Description of the issue
+- Proposed fix (if applicable)
+
+## License
+
+These test fixtures are released under the MIT License, the same as the specification.

package/tests/fixtures/decode/arrays-nested.json

@@ -0,0 +1,156 @@
+{
+  "version": "1.3",
+  "category": "decode",
+  "description": "Nested and mixed array decoding - list format, arrays of arrays, root arrays, mixed types",
+  "tests": [
+    {
+      "name": "parses list arrays for non-uniform objects",
+      "input": "items[2]:\n  - id: 1\n    name: First\n  - id: 2\n    name: Second\n    extra: true",
+      "expected": {
+        "items": [
+          { "id": 1, "name": "First" },
+          { "id": 2, "name": "Second", "extra": true }
+        ]
+      },
+      "specSection": "7"
+    },
+    {
+      "name": "parses objects with nested values inside list items",
+      "input": "items[1]:\n  - id: 1\n    nested:\n      x: 1",
+      "expected": {
+        "items": [
+          { "id": 1, "nested": { "x": 1 } }
+        ]
+      },
+      "specSection": "7"
+    },
+    {
+      "name": "parses nested tabular arrays as first field on hyphen line",
+      "input": "items[1]:\n  - users[2]{id,name}:\n    1,Ada\n    2,Bob\n    status: active",
+      "expected": {
+        "items": [
+          {
+            "users": [
+              { "id": 1, "name": "Ada" },
+              { "id": 2, "name": "Bob" }
+            ],
+            "status": "active"
+          }
+        ]
+      },
+      "specSection": "7"
+    },
+    {
+      "name": "parses objects containing arrays (including empty arrays) in list format",
+      "input": "items[1]:\n  - name: test\n    data[0]:",
+      "expected": {
+        "items": [
+          { "name": "test", "data": [] }
+        ]
+      },
+      "specSection": "7"
+    },
+    {
+      "name": "parses arrays of arrays within objects",
+      "input": "items[1]:\n  - matrix[2]:\n    - [2]: 1,2\n    - [2]: 3,4\n    name: grid",
+      "expected": {
+        "items": [
+          { "matrix": [[1, 2], [3, 4]], "name": "grid" }
+        ]
+      },
+      "specSection": "7"
+    },
+    {
+      "name": "parses nested arrays of primitives",
+      "input": "pairs[2]:\n  - [2]: a,b\n  - [2]: c,d",
+      "expected": {
+        "pairs": [["a", "b"], ["c", "d"]]
+      },
+      "specSection": "7.3"
+    },
+    {
+      "name": "parses quoted strings and mixed lengths in nested arrays",
+      "input": "pairs[2]:\n  - [2]: a,b\n  - [3]: \"c,d\",\"e:f\",\"true\"",
+      "expected": {
+        "pairs": [["a", "b"], ["c,d", "e:f", "true"]]
+      },
+      "specSection": "7.3"
+    },
+    {
+      "name": "parses empty inner arrays",
+      "input": "pairs[2]:\n  - [0]:\n  - [0]:",
+      "expected": {
+        "pairs": [[], []]
+      },
+      "specSection": "7.3"
+    },
+    {
+      "name": "parses mixed-length inner arrays",
+      "input": "pairs[2]:\n  - [1]: 1\n  - [2]: 2,3",
+      "expected": {
+        "pairs": [[1], [2, 3]]
+      },
+      "specSection": "7.3"
+    },
+    {
+      "name": "parses root arrays of primitives (inline)",
+      "input": "[5]: x,y,\"true\",true,10",
+      "expected": ["x", "y", "true", true, 10],
+      "specSection": "7"
+    },
+    {
+      "name": "parses root arrays of uniform objects in tabular format",
+      "input": "[2]{id}:\n  1\n  2",
+      "expected": [{ "id": 1 }, { "id": 2 }],
+      "specSection": "7.2"
+    },
+    {
+      "name": "parses root arrays of non-uniform objects in list format",
+      "input": "[2]:\n  - id: 1\n  - id: 2\n    name: Ada",
+      "expected": [{ "id": 1 }, { "id": 2, "name": "Ada" }],
+      "specSection": "7"
+    },
+    {
+      "name": "parses empty root arrays",
+      "input": "[0]:",
+      "expected": [],
+      "specSection": "7"
+    },
+    {
+      "name": "parses root arrays of arrays",
+      "input": "[2]:\n  - [2]: 1,2\n  - [0]:",
+      "expected": [[1, 2], []],
+      "specSection": "7.3"
+    },
+    {
+      "name": "parses complex mixed object with arrays and nested objects",
+      "input": "user:\n  id: 123\n  name: Ada\n  tags[2]: reading,gaming\n  active: true\n  prefs[0]:",
+      "expected": {
+        "user": {
+          "id": 123,
+          "name": "Ada",
+          "tags": ["reading", "gaming"],
+          "active": true,
+          "prefs": []
+        }
+      },
+      "specSection": "6"
+    },
+    {
+      "name": "parses arrays mixing primitives, objects and strings (list format)",
+      "input": "items[3]:\n  - 1\n  - a: 1\n  - text",
+      "expected": {
+        "items": [1, { "a": 1 }, "text"]
+      },
+      "specSection": "7.3"
+    },
+    {
+      "name": "parses arrays mixing objects and arrays",
+      "input": "items[2]:\n  - a: 1\n  - [2]: 1,2",
+      "expected": {
+        "items": [{ "a": 1 }, [1, 2]]
+      },
+      "specSection": "7.3"
+    }
+  ]
+}

package/tests/fixtures/decode/arrays-primitive.json

@@ -0,0 +1,87 @@
+{
+  "version": "1.3",
+  "category": "decode",
+  "description": "Primitive array decoding - inline arrays of strings, numbers, booleans, quoted strings",
+  "tests": [
+    {
+      "name": "parses string arrays inline",
+      "input": "tags[3]: reading,gaming,coding",
+      "expected": {
+        "tags": ["reading", "gaming", "coding"]
+      },
+      "specSection": "7.1"
+    },
+    {
+      "name": "parses number arrays inline",
+      "input": "nums[3]: 1,2,3",
+      "expected": {
+        "nums": [1, 2, 3]
+      },
+      "specSection": "7.1"
+    },
+    {
+      "name": "parses mixed primitive arrays inline",
+      "input": "data[4]: x,y,true,10",
+      "expected": {
+        "data": ["x", "y", true, 10]
+      },
+      "specSection": "7.1"
+    },
+    {
+      "name": "parses empty arrays",
+      "input": "items[0]:",
+      "expected": {
+        "items": []
+      },
+      "specSection": "7.1"
+    },
+    {
+      "name": "parses single-item array with empty string",
+      "input": "items[1]: \"\"",
+      "expected": {
+        "items": [""]
+      },
+      "specSection": "7.1"
+    },
+    {
+      "name": "parses multi-item array with empty string",
+      "input": "items[3]: a,\"\",b",
+      "expected": {
+        "items": ["a", "", "b"]
+      },
+      "specSection": "7.1"
+    },
+    {
+      "name": "parses whitespace-only strings in arrays",
+      "input": "items[2]: \" \",\"  \"",
+      "expected": {
+        "items": [" ", "  "]
+      },
+      "specSection": "7.1"
+    },
+    {
+      "name": "parses strings with delimiters in arrays",
+      "input": "items[3]: a,\"b,c\",\"d:e\"",
+      "expected": {
+        "items": ["a", "b,c", "d:e"]
+      },
+      "specSection": "7.1"
+    },
+    {
+      "name": "parses strings that look like primitives when quoted",
+      "input": "items[4]: x,\"true\",\"42\",\"-3.14\"",
+      "expected": {
+        "items": ["x", "true", "42", "-3.14"]
+      },
+      "specSection": "7.1"
+    },
+    {
+      "name": "parses strings with structural tokens in arrays",
+      "input": "items[3]: \"[5]\",\"- item\",\"{key}\"",
+      "expected": {
+        "items": ["[5]", "- item", "{key}"]
+      },
+      "specSection": "7.1"
+    }
+  ]
+}

package/tests/fixtures/decode/arrays-tabular.json

@@ -0,0 +1,40 @@
+{
+  "version": "1.3",
+  "category": "decode",
+  "description": "Tabular array decoding - parsing arrays of uniform objects with headers",
+  "tests": [
+    {
+      "name": "parses tabular arrays of uniform objects",
+      "input": "items[2]{sku,qty,price}:\n  A1,2,9.99\n  B2,1,14.5",
+      "expected": {
+        "items": [
+          { "sku": "A1", "qty": 2, "price": 9.99 },
+          { "sku": "B2", "qty": 1, "price": 14.5 }
+        ]
+      },
+      "specSection": "7.2"
+    },
+    {
+      "name": "parses nulls and quoted values in tabular rows",
+      "input": "items[2]{id,value}:\n  1,null\n  2,\"test\"",
+      "expected": {
+        "items": [
+          { "id": 1, "value": null },
+          { "id": 2, "value": "test" }
+        ]
+      },
+      "specSection": "7.2"
+    },
+    {
+      "name": "parses quoted header keys in tabular arrays",
+      "input": "items[2]{\"order:id\",\"full name\"}:\n  1,Ada\n  2,Bob",
+      "expected": {
+        "items": [
+          { "order:id": 1, "full name": "Ada" },
+          { "order:id": 2, "full name": "Bob" }
+        ]
+      },
+      "specSection": "7.2"
+    }
+  ]
+}