@toon-format/spec 1.3.0

package/CHANGELOG.md

@@ -0,0 +1,42 @@
+# Changelog
+
+All notable changes to the TOON specification will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.3] - 2025-10-31
+
+### Added
+
+- Numeric precision requirements: JavaScript implementations SHOULD use `Number.toString()` precision (15-17 digits), all implementations MUST preserve round-trip fidelity (Section 2)
+- RFC 5234 core rules (ALPHA, DIGIT, DQUOTE, HTAB, LF, SP) to ABNF grammar definitions (Section 6)
+- Test case for repeating decimal precision (1/3) to validate round-trip behavior
+
+## [1.2] - 2025-10-29
+
+### Changed
+
+- Clarified delimiter scoping behavior between array headers
+- Tightened strict-mode indentation requirements: leading spaces MUST be exact multiples of indentSize; tabs in indentation MUST error
+- Defined blank-line and trailing-newline decoding behavior with explicit skipping rules outside arrays
+- Clarified hyphen-based quoting: "-" or any string starting with "-" MUST be quoted
+- Clarified BigInt normalization: values outside safe integer range are converted to quoted decimal strings
+- Clarified row/key disambiguation: uses first unquoted delimiter vs colon position
+
+## [1.1] - 2025-10-29
+
+### Added
+
+- Strict-mode rules
+- Delimiter-aware parsing
+- Decoder options (indent, strict)
+
+## [1.0] - 2025-10-28
+
+### Added
+
+- Initial specification release
+- Encoding normalization rules
+- Decoding interpretation guidelines
+- Conformance requirements

package/CONTRIBUTING.md

@@ -0,0 +1,259 @@
+# Contributing to TOON Specification
+
+Thanks for your interest in contributing to the TOON specification! Whether you're fixing a typo, proposing a new feature, or clarifying ambiguous wording, your contributions help make TOON better for everyone.
+
+This document outlines how to propose changes and improvements to the specification.
+
+## Table of Contents
+
+- [Types of Contributions](#types-of-contributions)
+- [Before You Contribute](#before-you-contribute)
+- [Proposing Changes](#proposing-changes)
+- [RFC Process for Major Changes](#rfc-process-for-major-changes)
+- [Specification Principles](#specification-principles)
+- [Style Guidelines](#style-guidelines)
+
+## Types of Contributions
+
+### Clarifications and Fixes
+
+Small improvements that don't change behavior:
+
+- Typo corrections
+- Grammar improvements
+- Clarifying ambiguous language
+- Adding examples
+- Fixing broken links
+
+**Process:** Open a pull request directly.
+
+### Minor Changes
+
+Backward-compatible additions or clarifications that may affect implementations:
+
+- Adding test cases
+- Clarifying edge case behavior
+- Adding informative appendices
+- Expanding examples
+
+**Process:** Open an issue first to discuss, then submit a pull request.
+
+### Contributing Test Fixtures
+
+Test fixtures are language-agnostic JSON files that validate TOON implementations. Contributing test cases helps ensure spec compliance across all implementations.
+
+**Types of test contributions:**
+
+1. **New test cases** - Add tests for uncovered scenarios
+2. **Edge case coverage** - Add tests for corner cases
+3. **Error validation** - Add tests for error conditions
+4. **Spec clarifications** - Add tests when spec behavior is clarified
+
+**Test fixture structure:**
+
+```json
+{
+  "version": "1.3",
+  "category": "encode",
+  "description": "Brief description of test category",
+  "tests": [
+    {
+      "name": "descriptive test name",
+      "input": "JSON value or TOON string",
+      "expected": "TOON string or JSON value",
+      "options": {},
+      "specSection": "7.2"
+    }
+  ]
+}
+```
+
+**Guidelines for test fixtures:**
+
+- One assertion per test - keep tests focused
+- Use descriptive names that explain what's being validated
+- Link to spec sections using `specSection` field
+- Include edge cases and boundary conditions
+- Add error tests in separate files (e.g., `validation-errors.json`)
+- Follow existing test organization patterns
+- Validate your JSON against `tests/fixtures.schema.json`
+
+**Process:**
+
+1. Identify the appropriate fixture file in `tests/fixtures/encode/` or `tests/fixtures/decode/`.
+2. Add your test case following the structure above.
+3. Run validation to ensure your fixture is well-formed.
+4. Submit a PR with clear description of what the test validates.
+
+See [tests/README.md](./tests/README.md) for complete fixture documentation.
+
+### Major Changes
+
+Changes that affect compatibility or core behavior:
+- New syntax features
+- Changes to encoding/decoding rules
+- Modifications to conformance requirements
+- Breaking changes
+
+**Process:** Follow the RFC process (see below).
+
+## Before You Contribute
+
+Quick checklist before opening an issue or PR:
+
+1. **Check existing issues and PRs** – someone may have already proposed your idea
+2. **Review the specification** ([SPEC.md](./SPEC.md)) to understand current behavior
+3. **Check the reference implementation** to see how it handles the case
+4. **Consider backward compatibility** – breaking changes require strong justification
+
+## Proposing Changes
+
+### Opening an Issue
+
+When opening an issue, please include:
+
+1. **Clear description** of the problem or proposed change.
+2. **Current behavior** (with spec references if applicable).
+3. **Proposed behavior** (what should change and why).
+4. **Examples** demonstrating the issue or improvement.
+5. **Impact assessment** on existing implementations.
+6. **Alternatives considered** (if applicable).
+
+### Submitting a Pull Request
+
+1. **Fork the repository** and create a branch from `main`.
+2. **Make your changes** following the style guidelines.
+3. **Update examples** if your change affects them.
+4. **Update CHANGELOG.md** with your changes.
+5. **Write a clear PR description** explaining:
+   - What changed and why.
+   - How it affects implementations.
+   - Whether it's backward-compatible.
+6. **Link related issues** in your PR description.
+
+### Review Process
+
+- Spec maintainers will review your contribution.
+- There may be discussion and requests for changes.
+- Once approved, maintainers will merge and assign a version number.
+- For breaking changes, the PR will be held until the next major version.
+
+## RFC Process for Major Changes
+
+Major changes require a Request for Comments (RFC) process. This ensures community input and helps us avoid breaking changes that haven't been thoroughly considered.
+
+### 1. Create an RFC Issue
+
+Open an issue with the `[RFC]` prefix and include:
+
+```markdown
+# [RFC] Title of Your Proposal
+
+## Summary
+Brief overview of the change
+
+## Motivation
+Why is this change needed? What problems does it solve?
+
+## Detailed Design
+Technical specification of the proposed change
+
+## Examples
+Code examples showing the change in action
+
+## Drawbacks
+What are the costs and downsides?
+
+## Alternatives
+What other designs were considered?
+
+## Impact on Implementations
+How will this affect existing implementations?
+
+## Migration Strategy
+How should implementations adopt this change?
+
+## Unresolved Questions
+What parts need more discussion?
+```
+
+### 2. Discussion Period
+
+- Community members and implementers provide feedback.
+- The RFC is iterated based on feedback.
+- Discussion typically lasts 1-2 weeks minimum.
+
+### 3. Decision
+
+- Maintainers will accept, reject, or request revisions.
+- Accepted RFCs move to implementation (PR phase).
+- Rejected RFCs will be closed with explanation.
+
+### 4. Implementation
+
+- Create a PR implementing the RFC.
+- PR must reference the RFC issue.
+- Additional review occurs during PR phase.
+
+## Specification Principles
+
+When contributing, keep these core principles in mind. They guide every decision we make about TOON:
+
+### 1. Token Efficiency
+TOON's primary goal is reducing token usage for LLM input. Changes should maintain or improve token efficiency.
+
+### 2. LLM-Friendly Structure
+The format should be easy for LLMs to parse and generate. Explicit length markers and field declarations are features, not bugs.
+
+### 3. Human Readability
+While optimized for machines, TOON should remain reasonably readable by humans for debugging and validation.
+
+### 4. Simplicity
+Prefer simple, consistent rules over special cases. The spec should be implementable without extensive parsing libraries.
+
+### 5. Backward Compatibility
+Breaking changes require very strong justification. Prefer additive changes that maintain compatibility.
+
+### 6. Interoperability
+Multiple implementations should produce identical output for the same input. Ambiguity is not acceptable.
+
+## Style Guidelines
+
+### Writing Style
+
+- Use RFC 2119 keywords (MUST, SHOULD, MAY) correctly
+- Be precise and unambiguous
+- Use examples to clarify complex rules
+- Keep language concise but complete
+- Use active voice where possible
+
+### Formatting
+
+- Use proper markdown formatting
+- Include code blocks with language hints
+- Use section numbering consistently
+- Add cross-references to related sections
+- Keep line length reasonable (80-120 characters)
+
+### Examples
+
+- Show both input and output
+- Include edge cases
+- Demonstrate both valid and invalid cases
+- Add comments explaining non-obvious behavior
+
+## Questions?
+
+If you have questions about contributing:
+
+1. Check existing issues and discussions.
+2. Open a new issue with the `question` label.
+3. Reach out to maintainers.
+
+## Code of Conduct
+
+Be respectful and constructive in all interactions. We're building this specification together, and diverse perspectives make it better.
+
+## License
+
+By contributing to this specification, you agree that your contributions will be licensed under the MIT License.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025-PRESENT Johann Schopplich
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,111 @@
+# TOON Format Specification
+
+[![SPEC v1.3](https://img.shields.io/badge/spec-v1.3-lightgrey)](./SPEC.md)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)
+
+This repository contains the official specification for **Token-Oriented Object Notation (TOON)**, a compact, human-readable serialization format designed for passing structured data to Large Language Models with significantly reduced token usage.
+
+## 📋 Specification
+
+[→ Read the full specification (SPEC.md)](./SPEC.md)
+
+- **Version:** 1.3 (2025-10-31)
+- **Status:** Working Draft
+- **License:** MIT
+
+The specification includes complete grammar (ABNF), encoding rules, validation requirements, and conformance criteria.
+
+## What is TOON?
+
+**Token-Oriented Object Notation** is a compact, human-readable serialization format designed for passing structured data to Large Language Models with significantly reduced token usage. It's intended for LLM input, not output.
+
+TOON's sweet spot is **uniform arrays of objects** – multiple fields per row, same structure across items. It borrows YAML's indentation-based structure for nested objects and CSV's tabular format for uniform data rows, then optimizes both for token efficiency in LLM contexts. For deeply nested or non-uniform data, JSON may be more efficient.
+
+**Key Features:**
+
+- 💸 **Token-efficient:** typically 30–60% fewer tokens than JSON
+- 🤿 **LLM-friendly guardrails:** explicit lengths and fields enable validation
+- 🍱 **Minimal syntax:** removes redundant punctuation (braces, brackets, most quotes)
+- 📐 **Indentation-based structure:** like YAML, uses whitespace instead of braces
+- 🧺 **Tabular arrays:** declare keys once, stream data as rows
+
+## Quick Example
+
+**JSON:**
+
+```json
+{
+  "users": [
+    { "id": 1, "name": "Alice", "role": "admin" },
+    { "id": 2, "name": "Bob", "role": "user" }
+  ]
+}
+```
+
+**TOON:**
+
+```
+users[2]{id,name,role}:
+  1,Alice,admin
+  2,Bob,user
+```
+
+## Reference Implementation
+
+The reference implementation in TypeScript/JavaScript is maintained at: [github.com/toon-format/toon](https://github.com/toon-format/toon)
+
+The reference implementation includes:
+
+- Complete encoder and decoder
+- CLI tools for JSON ↔ TOON conversion
+- Performance benchmarks
+
+## Community Implementations
+
+Official community-driven implementations are currently being developed at the [github.com/toon-format](https://github.com/toon-format) organization.
+
+## Test Fixtures & Conformance
+
+The [tests/fixtures/](./tests/fixtures/) directory contains **language-agnostic JSON test fixtures** for validating TOON implementations. Each fixture file contains test cases with input/output pairs covering all specification requirements.
+
+**What's included:**
+
+- **Encoding tests:** JSON → TOON conversion
+- **Decoding tests:** TOON → JSON parsing
+- **Error cases:** Validation and strict mode checks
+- **Edge cases:** All corner cases from the specification
+
+**For implementers:**
+
+1. Load JSON fixtures from `tests/fixtures/encode/` and `tests/fixtures/decode/`.
+2. Run each test case against your implementation.
+3. Report results using the conformance badge system.
+
+See [tests/README.md](./tests/README.md) for detailed fixture format and usage instructions.
+
+## Contributing
+
+We welcome contributions to improve the specification! Please see [CONTRIBUTING.md](./CONTRIBUTING.md) for:
+
+- How to propose spec changes
+- The RFC process for major changes
+- Guidelines for submitting issues and pull requests
+
+For implementation-specific questions or bugs, please refer to the respective implementation repository.
+
+## Versioning
+
+The TOON specification follows semantic versioning. See [VERSIONING.md](./VERSIONING.md) for our versioning policy and compatibility guarantees.
+
+## Resources
+
+- **Specification:** [SPEC.md](./SPEC.md) - Complete formal specification with ABNF grammar
+- **Examples:** [examples/](./examples/) - Working examples organized by feature
+- **Test Fixtures:** [tests/fixtures/](./tests/fixtures/) - Comprehensive test suite
+- **Changelog:** [CHANGELOG.md](./CHANGELOG.md) - Version history and changes
+- **Reference Implementation:** [github.com/toon-format/toon](https://github.com/toon-format/toon) - TypeScript/JavaScript implementation
+- **Benchmarks:** [Reference repo benchmarks/](https://github.com/toon-format/toon/tree/main/benchmarks) - Token efficiency measurements and accuracy retrieval tests
+
+## License
+
+[MIT](./LICENSE) License © 2025-PRESENT [Johann Schopplich](https://github.com/johannschopplich)