@toon-format/spec 1.3.0 → 1.3.2

package/CHANGELOG.md

@@ -11,7 +11,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Numeric precision requirements: JavaScript implementations SHOULD use `Number.toString()` precision (15-17 digits), all implementations MUST preserve round-trip fidelity (Section 2)
 - RFC 5234 core rules (ALPHA, DIGIT, DQUOTE, HTAB, LF, SP) to ABNF grammar definitions (Section 6)
-- Test case for repeating decimal precision (1/3) to validate round-trip behavior
 
 ## [1.2] - 2025-10-29
 

package/CONTRIBUTING.md

@@ -1,259 +1,106 @@
 # Contributing to TOON Specification
 
-Thanks for your interest in contributing to the TOON specification! Whether you're fixing a typo, proposing a new feature, or clarifying ambiguous wording, your contributions help make TOON better for everyone.
+Thanks for your interest in contributing! This guide shows how to propose changes to the specification.
 
-This document outlines how to propose changes and improvements to the specification.
+## Quick Reference: What Process Do I Need?
 
-## Table of Contents
+| Change Type | Examples | Process |
+| ----------- | -------- | ------- |
+| **Fixes** | Typos, grammar, broken links, clarifying wording | Direct PR |
+| **Test Fixtures** | New test cases, edge case tests, validation tests | Direct PR (see [tests/README.md](./tests/README.md)) |
+| **Minor Changes** | Spec clarifications that may affect implementations | Issue first → PR |
+| **Major Changes** | New syntax, encoding rules, breaking changes | RFC process (see below) |
 
-- [Types of Contributions](#types-of-contributions)
-- [Before You Contribute](#before-you-contribute)
-- [Proposing Changes](#proposing-changes)
-- [RFC Process for Major Changes](#rfc-process-for-major-changes)
-- [Specification Principles](#specification-principles)
-- [Style Guidelines](#style-guidelines)
+## Does My Change Need an RFC?
 
-## Types of Contributions
+### Yes – RFC Required
 
-### Clarifications and Fixes
+Changes that affect compatibility or core behavior need community discussion:
 
-Small improvements that don't change behavior:
-
-- Typo corrections
-- Grammar improvements
-- Clarifying ambiguous language
-- Adding examples
-- Fixing broken links
-
-**Process:** Open a pull request directly.
-
-### Minor Changes
-
-Backward-compatible additions or clarifications that may affect implementations:
-
-- Adding test cases
-- Clarifying edge case behavior
-- Adding informative appendices
-- Expanding examples
-
-**Process:** Open an issue first to discuss, then submit a pull request.
-
-### Contributing Test Fixtures
-
-Test fixtures are language-agnostic JSON files that validate TOON implementations. Contributing test cases helps ensure spec compliance across all implementations.
-
-**Types of test contributions:**
-
-1. **New test cases** - Add tests for uncovered scenarios
-2. **Edge case coverage** - Add tests for corner cases
-3. **Error validation** - Add tests for error conditions
-4. **Spec clarifications** - Add tests when spec behavior is clarified
-
-**Test fixture structure:**
-
-```json
-{
-  "version": "1.3",
-  "category": "encode",
-  "description": "Brief description of test category",
-  "tests": [
-    {
-      "name": "descriptive test name",
-      "input": "JSON value or TOON string",
-      "expected": "TOON string or JSON value",
-      "options": {},
-      "specSection": "7.2"
-    }
-  ]
-}
 ```
+Adding new delimiter types (beyond comma/tab/pipe)
+  Example: Supporting semicolon as delimiter
 
-**Guidelines for test fixtures:**
-
-- One assertion per test - keep tests focused
-- Use descriptive names that explain what's being validated
-- Link to spec sections using `specSection` field
-- Include edge cases and boundary conditions
-- Add error tests in separate files (e.g., `validation-errors.json`)
-- Follow existing test organization patterns
-- Validate your JSON against `tests/fixtures.schema.json`
-
-**Process:**
-
-1. Identify the appropriate fixture file in `tests/fixtures/encode/` or `tests/fixtures/decode/`.
-2. Add your test case following the structure above.
-3. Run validation to ensure your fixture is well-formed.
-4. Submit a PR with clear description of what the test validates.
-
-See [tests/README.md](./tests/README.md) for complete fixture documentation.
-
-### Major Changes
-
-Changes that affect compatibility or core behavior:
-- New syntax features
-- Changes to encoding/decoding rules
-- Modifications to conformance requirements
-- Breaking changes
-
-**Process:** Follow the RFC process (see below).
-
-## Before You Contribute
-
-Quick checklist before opening an issue or PR:
-
-1. **Check existing issues and PRs** – someone may have already proposed your idea
-2. **Review the specification** ([SPEC.md](./SPEC.md)) to understand current behavior
-3. **Check the reference implementation** to see how it handles the case
-4. **Consider backward compatibility** – breaking changes require strong justification
-
-## Proposing Changes
-
-### Opening an Issue
-
-When opening an issue, please include:
-
-1. **Clear description** of the problem or proposed change.
-2. **Current behavior** (with spec references if applicable).
-3. **Proposed behavior** (what should change and why).
-4. **Examples** demonstrating the issue or improvement.
-5. **Impact assessment** on existing implementations.
-6. **Alternatives considered** (if applicable).
+Changing quoting or escape rules
+  Example: Adding \u0000 escape sequences
+  Example: Changing when colons require quotes
 
-### Submitting a Pull Request
+Modifying tabular detection logic
+  Example: Allowing non-uniform objects in tabular format
 
-1. **Fork the repository** and create a branch from `main`.
-2. **Make your changes** following the style guidelines.
-3. **Update examples** if your change affects them.
-4. **Update CHANGELOG.md** with your changes.
-5. **Write a clear PR description** explaining:
-   - What changed and why.
-   - How it affects implementations.
-   - Whether it's backward-compatible.
-6. **Link related issues** in your PR description.
+New data type normalization rules
+  Example: Handling Map or Set differently
 
-### Review Process
-
-- Spec maintainers will review your contribution.
-- There may be discussion and requests for changes.
-- Once approved, maintainers will merge and assign a version number.
-- For breaking changes, the PR will be held until the next major version.
-
-## RFC Process for Major Changes
-
-Major changes require a Request for Comments (RFC) process. This ensures community input and helps us avoid breaking changes that haven't been thoroughly considered.
-
-### 1. Create an RFC Issue
-
-Open an issue with the `[RFC]` prefix and include:
-
-```markdown
-# [RFC] Title of Your Proposal
-
-## Summary
-Brief overview of the change
-
-## Motivation
-Why is this change needed? What problems does it solve?
-
-## Detailed Design
-Technical specification of the proposed change
-
-## Examples
-Code examples showing the change in action
-
-## Drawbacks
-What are the costs and downsides?
-
-## Alternatives
-What other designs were considered?
-
-## Impact on Implementations
-How will this affect existing implementations?
+Changing array header syntax
+  Example: Optional length markers becoming required
+```
 
-## Migration Strategy
-How should implementations adopt this change?
+### No – Direct PR or Issue First
 
-## Unresolved Questions
-What parts need more discussion?
+```
+Typo: "recieves" → "receives"                        → Direct PR
+Adding example for nested arrays                     → Direct PR
+Clarifying ambiguous wording                         → Issue → PR
+Expanding test coverage                              → Direct PR
+Documenting existing edge case behavior              → Issue → PR
 ```
 
-### 2. Discussion Period
-
-- Community members and implementers provide feedback.
-- The RFC is iterated based on feedback.
-- Discussion typically lasts 1-2 weeks minimum.
+## RFC Process
 
-### 3. Decision
+For major changes requiring RFC:
 
-- Maintainers will accept, reject, or request revisions.
-- Accepted RFCs move to implementation (PR phase).
-- Rejected RFCs will be closed with explanation.
+1. **Create RFC Issue** using the Feature Request/RFC template
+2. **Discussion Period** (minimum 1-2 weeks for community feedback)
+3. **Decision** (maintainers accept, reject, or request revisions)
+4. **Implementation** (create PR referencing RFC issue)
 
-### 4. Implementation
+## Contributing Test Fixtures
 
-- Create a PR implementing the RFC.
-- PR must reference the RFC issue.
-- Additional review occurs during PR phase.
+Test fixtures validate TOON implementations across languages. Add your test to `tests/fixtures/encode/` or `tests/fixtures/decode/`, validate against `tests/fixtures.schema.json`, and submit a PR.
 
-## Specification Principles
+See [tests/README.md](./tests/README.md) for fixture structure and guidelines.
 
-When contributing, keep these core principles in mind. They guide every decision we make about TOON:
+## Pull Request Guidelines
 
-### 1. Token Efficiency
-TOON's primary goal is reducing token usage for LLM input. Changes should maintain or improve token efficiency.
+1. Fork the repository and create a feature branch
+2. Make changes following [SPEC.md](./SPEC.md) style (RFC 2119 keywords, examples, precision)
+3. Update `CHANGELOG.md` with your changes
+4. Submit PR using the template and link related issues
 
-### 2. LLM-Friendly Structure
-The format should be easy for LLMs to parse and generate. Explicit length markers and field declarations are features, not bugs.
+**Review process:**
+- Maintainers review and may request changes
+- Breaking changes held until next major version
+- Approved changes merged with version number assigned
 
-### 3. Human Readability
-While optimized for machines, TOON should remain reasonably readable by humans for debugging and validation.
+## Specification Principles
 
-### 4. Simplicity
-Prefer simple, consistent rules over special cases. The spec should be implementable without extensive parsing libraries.
+TOON prioritizes (in order):
 
-### 5. Backward Compatibility
-Breaking changes require very strong justification. Prefer additive changes that maintain compatibility.
+1. **Token Efficiency** – Changes should maintain or improve token usage for LLM input
+2. **LLM-Friendly Structure** – Easy for LLMs to parse and generate (explicit markers are features)
+3. **Simplicity** – Prefer consistent rules over special cases
+4. **Backward Compatibility** – Breaking changes need strong justification
+5. **Interoperability** – Implementations must produce identical output for identical input
+6. **Human Readability** – Debuggable by humans despite machine optimization
 
-### 6. Interoperability
-Multiple implementations should produce identical output for the same input. Ambiguity is not acceptable.
+These principles guide every decision. When in doubt, optimize for token efficiency first.
 
 ## Style Guidelines
 
-### Writing Style
-
-- Use RFC 2119 keywords (MUST, SHOULD, MAY) correctly
-- Be precise and unambiguous
-- Use examples to clarify complex rules
-- Keep language concise but complete
-- Use active voice where possible
+Follow [SPEC.md](./SPEC.md) conventions:
 
-### Formatting
-
-- Use proper markdown formatting
-- Include code blocks with language hints
-- Use section numbering consistently
-- Add cross-references to related sections
-- Keep line length reasonable (80-120 characters)
-
-### Examples
-
-- Show both input and output
-- Include edge cases
-- Demonstrate both valid and invalid cases
-- Add comments explaining non-obvious behavior
+- **RFC 2119 keywords**: Use MUST/SHOULD/MAY correctly (see SPEC.md §1)
+- **Examples over prose**: Show concrete input/output for complex rules
+- **Precision**: Zero ambiguity – multiple implementations must agree
+- **Structure**: Number sections, cross-reference related rules
+- **Line length**: 80-120 characters for readability
 
 ## Questions?
 
-If you have questions about contributing:
-
-1. Check existing issues and discussions.
-2. Open a new issue with the `question` label.
-3. Reach out to maintainers.
-
-## Code of Conduct
-
-Be respectful and constructive in all interactions. We're building this specification together, and diverse perspectives make it better.
+1. Check existing issues and discussions
+2. Open an issue with the `question` label
+3. Reach out to maintainers
 
 ## License
 
-By contributing to this specification, you agree that your contributions will be licensed under the MIT License.
+By contributing, you agree your contributions will be licensed under the MIT License. Be respectful and constructive – diverse perspectives make TOON better.

package/SPEC.md

@@ -20,7 +20,7 @@ Token-Oriented Object Notation (TOON) is a compact, human-readable serialization
 
 ## Status of This Document
 
-This document is a Working Draft v1.3 and may be updated, replaced, or obsoleted. Implementers should monitor the canonical repository at https://github.com/toon-format/toon for changes.
+This document is a Working Draft v1.3 and may be updated, replaced, or obsoleted. Implementers should monitor the canonical repository at https://github.com/toon-format/spec for changes.
 
 This specification is stable for implementation but not yet finalized. Breaking changes are unlikely but possible before v2.0.
 
@@ -335,6 +335,8 @@ Object keys and tabular field names:
 - MAY be unquoted only if they match: ^[A-Za-z_][\w.]*$.
 - Otherwise, they MUST be quoted and escaped per Section 7.1.
 
+Keys requiring quoting per the above rules MUST be quoted in all contexts, including array headers (e.g., "my-key"[N]:).
+
 ### 7.4 Decoding Rules for Strings and Keys (Decoding)
 
 - Quoted strings and keys MUST be unescaped per Section 7.1; any other escape MUST error. Quoted primitives remain strings.
@@ -767,7 +769,7 @@ Intended usage: COMMON (upon standardization)
 
 Restrictions on usage: None
 
-Change controller: Community-maintained. See repository at https://github.com/toon-format/toon
+Change controller: Community-maintained. See repository at https://github.com/toon-format/spec
 
 ### 18.3 Implementation Status
 
@@ -899,6 +901,19 @@ bignum: 9007199254740992
 decimal: 0.3333333333333333
 ```
 
+Quoted keys with arrays (keys requiring quoting per Section 7.3):
+```
+"my-key"[3]: 1,2,3
+
+"x-items"[2]{id,name}:
+  1,Ada
+  2,Bob
+
+"x-items"[2]:
+  - id: 1
+  - id: 2
+```
+
 ## Appendix B: Parsing Helpers (Informative)
 
 These sketches illustrate structure and common decoding helpers. They are informative; normative behavior is defined in Sections 4–12 and 14.
@@ -962,8 +977,8 @@ These sketches illustrate structure and common decoding helpers. They are inform
 
 ### Reference Test Suite
 
-A reference test suite is maintained at:
-https://github.com/toon-format/toon/tree/main/packages/toon/test
+A language-agnostic reference test suite is maintained at:
+https://github.com/toon-format/spec/tree/main/tests
 
 The test suite is versioned alongside this specification. Implementations are encouraged to validate against this test suite, but conformance is determined solely by adherence to the normative requirements in Sections 1-16 and Section 19 of this specification. Test coverage does not define the specification; the specification defines conformance.
 
@@ -984,11 +999,15 @@ The reference test suite covers:
 
 ## Appendix D: Document Changelog (Informative)
 
+### v1.4 (2025-11-02)
+
+- Clarified that keys requiring quoting (Section 7.3) MUST be quoted in all contexts, including array headers (e.g., "my-key"[N]:, "x-custom"[2]{fields}:).
+- No semantic changes to the specification; this is purely clarifying documentation that was already implied by the grammar.
+
 ### v1.3 (2025-10-31)
 
 - Added numeric precision requirements: JavaScript implementations SHOULD use Number.toString() precision (15-17 digits), all implementations MUST preserve round-trip fidelity (Section 2).
 - Added RFC 5234 core rules (ALPHA, DIGIT, DQUOTE, HTAB, LF, SP) to ABNF grammar definitions (Section 6).
-- Added test case for repeating decimal precision (1/3) to validate round-trip behavior.
 
 ### v1.2 (2025-10-29)
 

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@toon-format/spec",
   "type": "module",
-  "version": "1.3.0",
+  "version": "1.3.2",
   "packageManager": "pnpm@10.19.0",
   "description": "Official specification for Token-Oriented Object Notation (TOON)",
   "author": "Johann Schopplich <hello@johannschopplich.com>",

package/tests/fixtures/decode/arrays-nested.json

@@ -14,6 +14,14 @@
       },
       "specSection": "7"
     },
+    {
+      "name": "parses list arrays with empty items",
+      "input": "items[3]:\n  - first\n  - second\n  -",
+      "expected": {
+        "items": ["first", "second", {}]
+      },
+      "specSection": "7.3"
+    },
     {
       "name": "parses objects with nested values inside list items",
       "input": "items[1]:\n  - id: 1\n    nested:\n      x: 1",
@@ -151,6 +159,17 @@
         "items": [{ "a": 1 }, [1, 2]]
       },
       "specSection": "7.3"
+    },
+    {
+      "name": "parses quoted key with list array format",
+      "input": "\"x-items\"[2]:\n  - id: 1\n  - id: 2",
+      "expected": {
+        "x-items": [
+          { "id": 1 },
+          { "id": 2 }
+        ]
+      },
+      "specSection": "7"
     }
   ]
 }

package/tests/fixtures/decode/arrays-primitive.json

@@ -82,6 +82,30 @@
         "items": ["[5]", "- item", "{key}"]
       },
       "specSection": "7.1"
+    },
+    {
+      "name": "parses quoted key with inline array",
+      "input": "\"my-key\"[3]: 1,2,3",
+      "expected": {
+        "my-key": [1, 2, 3]
+      },
+      "specSection": "7.1"
+    },
+    {
+      "name": "parses quoted key containing brackets with inline array",
+      "input": "\"key[test]\"[3]: 1,2,3",
+      "expected": {
+        "key[test]": [1, 2, 3]
+      },
+      "specSection": "7.1"
+    },
+    {
+      "name": "parses quoted key with empty array",
+      "input": "\"x-custom\"[0]:",
+      "expected": {
+        "x-custom": []
+      },
+      "specSection": "7.1"
     }
   ]
 }

package/tests/fixtures/decode/arrays-tabular.json

@@ -35,6 +35,17 @@
         ]
       },
       "specSection": "7.2"
+    },
+    {
+      "name": "parses quoted key with tabular array format",
+      "input": "\"x-items\"[2]{id,name}:\n  1,Ada\n  2,Bob",
+      "expected": {
+        "x-items": [
+          { "id": 1, "name": "Ada" },
+          { "id": 2, "name": "Bob" }
+        ]
+      },
+      "specSection": "7.2"
     }
   ]
 }

package/tests/fixtures/decode/indentation-errors.json

@@ -156,6 +156,27 @@
       },
       "specSection": "9"
     },
+    {
+      "name": "accepts list items with irregular nested indentation when strict=false",
+      "input": "items[2]:\n  - id: 1\n     nested:\n       value: test\n  - id: 2",
+      "expected": {
+        "items": [
+          {
+            "id": 1,
+            "nested": {
+              "value": "test"
+            }
+          },
+          {
+            "id": 2
+          }
+        ]
+      },
+      "options": {
+        "strict": false
+      },
+      "specSection": "4"
+    },
     {
       "name": "empty lines do not trigger validation errors",
       "input": "a: 1\n\nb: 2",