rtoon 0.1.0 → 0.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 89392ee400d56b09518657851bd1f69b63c8ca50f9376da702a541cc69eaefae
-  data.tar.gz: 96c0f189e8dfa1c559dd8b69e58ad351fba7f5c7b92c1c70f42f8432d70c5344
+  metadata.gz: 43281cede188669647dacb5c6bbe76988310a356fed392e00a6e43eb6c7fb237
+  data.tar.gz: b6bf197adc646d47cec17a09a4962fe90a884c9910ccbe261df9e90c210589e6
 SHA512:
-  metadata.gz: c6a0a8e40b860a2b19b3b4060b4f51e1fc430ffc41e3b24dca5907ed16fc9cfcf056763e4184a640104ee61ec7da4b14c0f00e81dcdbabbcfc8e12d04d642a07
-  data.tar.gz: 38033867f87e4c304e8047d82ed7fb2b94ac4fafed741ed571a29e500c4d1e7eb90a55e69cb66d4639914374a234a1d07237fca2ceb9d120e071b118fd21c1e3
+  metadata.gz: ae5b46894a99b025b3884b644e632b901621500f6ea6c9fde203756cf557fe6a33e67882771f8999d35a158690671b158c6c436f0decc11160a0c522a589f9ad
+  data.tar.gz: 7f98aa510a5b9585d9b8b381cd42b97b675a328ac4435117a66b77ca1e3111dccf6d12cfc67d29553839ab87e0a298fc9b3c9524cf6e2cfee26f929425b3fb3d

data/LICENSE.txt

@@ -1,21 +1,15 @@
-MIT License
+Copyright 2025 Zoobean Inc.
 
-Copyright (c) 2025
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software
+and associated documentation files (the “Software”), to deal in the Software without restriction,
+including without limitation the rights to use, copy, modify, merge, publish, distribute,
+sublicense, and/or sell copies of the Software, and to permit persons to whom the Software
+is furnished to do so, subject to the following conditions:
 
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
 
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -1,87 +1,81 @@
-# TOON Parser - Token Object Oriented Notation
+# RTOON - Token Object Oriented Notation Parser
 
-A Ruby gem for parsing Token Object Oriented Notation (TOON), a tabular, schema-based data format with indentation-based structure.
+[![CI](https://github.com/zoobean/rtoon/actions/workflows/ci.yml/badge.svg)](https://github.com/zoobean/rtoon/actions/workflows/ci.yml)
+[![Ruby](https://img.shields.io/badge/ruby-%3E%3D%202.7-red.svg)](https://www.ruby-lang.org/)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE.txt)
+[![Gem Version](https://badge.fury.io/rb/rtoon.svg)](https://badge.fury.io/rb/rtoon)
 
-## What is TOON?
+A Ruby gem for parsing and encoding **Token Object Oriented Notation (TOON)** — a human-readable data format combining tabular structure with JSON-like flexibility.'
 
-TOON is a data serialization format that combines:
-- **Schema declarations** with field definitions
-- **Tabular data** for compact representation
-- **Indentation-based nesting** for hierarchy
-- **Array size hints** for structure clarity
+Inspired by [toon-format/toon](https://github.com/toon-format/toon) which is written in TypeScript and for TypeScript projects.
 
-### Example
 
-```toon
-items[1]{users,status}:
-  users[2]{id,name}:
-    1,Ada
-    2,Bob
-  status: active
-```
+## What is TOON?
 
-This parses to:
+TOON is a data serialization format with schema declarations, tabular rows, and indentation-based nesting.
 
-```ruby
-{
-  "items" => [
-    {
-      "users" => [
-        { "id" => "1", "name" => "Ada" },
-        { "id" => "2", "name" => "Bob" }
-      ],
-      "status" => "active"
-    }
-  ]
-}
-```
+From the original project description:
 
-## Installation
+> Token-Oriented Object Notation is a compact, human-readable serialization format designed for passing structured data to Large Language Models with significantly reduced token usage. It's intended for LLM input as a lossless, drop-in representation of JSON data.
+
+> TOON's sweet spot is uniform arrays of objects – multiple fields per row, same structure across items. It borrows YAML's indentation-based structure for nested objects and CSV's tabular format for uniform data rows, then optimizes both for token efficiency in LLM contexts. For deeply nested or non-uniform data, JSON may be more efficient.
+
+> TOON achieves CSV-like compactness while adding explicit structure that helps LLMs parse and validate data reliably.
 
-Add this line to your application's Gemfile:
+**Example:**
+```toon
+users[2]{id,name}:
+  1,Ada
+  2,Bob
+```
 
+**Parses to:**
 ```ruby
-gem 'rtoon'
+{"users" => [{"id" => "1", "name" => "Ada"}, {"id" => "2", "name" => "Bob"}]}
 ```
 
-Or install it yourself:
+## Installation
 
 ```bash
 gem install rtoon
 ```
 
-## Usage
+Or add to `Gemfile`:
+```ruby
+gem 'rtoon'
+```
 
-### Basic Parsing
+## Quick Start
 
 ```ruby
 require 'rtoon'
 
-toon_string = <<~TOON
-  users[2]{id,name}:
-    1,Ada
-    2,Bob
-TOON
+# Parse TOON → Ruby
+result = Rtoon.parse("users[1]{id,name}:\n  1,Ada\n")
+# => {"users" => [{"id" => "1", "name" => "Ada"}]}
 
-result = Rtoon.parse(toon_string)
-# => {"users" => [{"id" => "1", "name" => "Ada"}, {"id" => "2", "name" => "Bob"}]}
+# Encode Ruby → TOON
+data = {"users" => [{"id" => "1", "name" => "Ada"}]}
+toon = Rtoon.encode(data)
+# => "users[1]{id,name}:\n  1,Ada\n"
 ```
 
-### Schema Declarations
-
-Define data structures with schemas:
+## Syntax
 
+**Schema Declaration:**
 ```toon
-products[3]{id,name,price}:
-  1,Widget,100
-  2,Gadget,200
-  3,Gizmo,300
+identifier[size]{field1,field2}:
+  value1,value2
+  value3,value4
 ```
 
-### Nested Structures
-
-Use indentation for nesting:
+**Field Assignment:**
+```toon
+name: John
+age: 30
+```
 
+**Nested Structure:**
 ```toon
 company[1]{depts,location}:
   depts[2]{name,size}:
@@ -90,209 +84,60 @@ company[1]{depts,location}:
   location: NYC
 ```
 
-### Field Assignments
+## API
 
-Simple key-value pairs:
+### `Rtoon.parse(string)` / `Rtoon.decode(string)`
+Parses TOON-formatted string to Ruby Hash/Array.
 
-```toon
-name: John
-age: 30
-active: true
-```
-
-## TOON Syntax
-
-### Schema Declaration
-
-```
-identifier[size]{field1,field2,...}:
-```
-
-- **identifier**: Name of the data structure
-- **[size]**: Optional array size hint (can be any number)
-- **{fields}**: Comma-separated list of field names
-- **:** Indicates content follows on indented lines
-
-### Data Rows
-
-```
-value1,value2,value3
-```
-
-Comma-separated values matching the schema fields in order.
-
-### Indentation
-
-- Use **2 spaces** for each nesting level
-- Indentation defines the structure hierarchy
-- Empty lines are ignored
-
-### Complete Example
-
-```toon
-database[1]{tables,version}:
-  tables[3]{name,records,indexed}:
-    users,1000,yes
-    posts,5000,yes
-    comments,20000,no
-  version: 3
-```
-
-Parses to:
+### `Rtoon.encode(hash, indent_level = 0)`
+Converts Ruby data structure to TOON format.
 
+### Error Handling
 ```ruby
-{
-  "database" => [
-    {
-      "tables" => [
-        { "name" => "users", "records" => "1000", "indexed" => "yes" },
-        { "name" => "posts", "records" => "5000", "indexed" => "yes" },
-        { "name" => "comments", "records" => "20000", "indexed" => "no" }
-      ],
-      "version" => "3"
-    }
-  ]
-}
+begin
+  result = Rtoon.parse(invalid_toon)
+rescue Rtoon::Parser::ParseError => e
+  puts "Parse error: #{e.message}"
+end
 ```
 
-## API Reference
-
-### `Rtoon.parse(string)`
-
-Parses a TOON string and returns a Ruby hash/array structure.
-
-**Parameters:**
-- `string` (String): TOON formatted string
-
-**Returns:**
-- Hash or Array with parsed data
-
-**Raises:**
-- `RtoonParser::ParseError`: If the string contains invalid TOON syntax
-
-### `Rtoon.decode(string)`
-
-Alias for `Rtoon.parse(string)`.
-
-## Features
-
-✅ Schema-based declarations
-✅ Tabular data rows
-✅ Indentation-based nesting
-✅ Array size hints
-✅ Field assignments
-✅ Multi-level nesting
-✅ Empty line handling
-
-## Grammar
-
-TOON uses a context-free grammar parsed with Racc (Ruby parser generator):
-
-- **Statements**: Schema blocks or field assignments
-- **Schema Block**: Header + indented content
-- **Schema Header**: `name[size]{fields}:`
-- **Data Rows**: Comma-separated values
-- **Field Assignment**: `name: value`
-- **Indentation**: INDENT/DEDENT tokens (2 spaces)
-
 ## Development
 
-### Prerequisites
-
-```bash
-# Ruby 2.7 or higher
-ruby --version
-
-# Racc (included with Ruby)
-ruby -e "require 'racc/parser'; puts 'Racc available'"
-```
-
-### Building from Source
-
 ```bash
-# Clone or extract the gem
+# Clone and setup
+git clone https://github.com/zoobean/rtoon.git
 cd rtoon
-
-# Compile grammar
-racc -o lib/rtoon.tab.rb lib/rtoon.y
+bundle install
 
 # Run tests
 ruby test/toon_test.rb
 
-# Build gem
-gem build rtoon.gemspec
+# Compile grammar (if modified)
+racc -o lib/rtoon/parser.tab.rb lib/rtoon/parser.y
 ```
 
-### Running Tests
-
-```bash
-ruby test/toon_test.rb
-```
-
-Expected output:
-```
-11 runs, 39 assertions, 0 failures, 0 errors, 0 skips
-```
-
-## Architecture
-
-```
-TOON String
-    ↓
-RtoonLexer (tokenization + indentation tracking)
-    ↓
-RtoonParser (Racc-generated parser)
-    ↓
-Ruby Hash/Array
-```
-
-### Components
-
-- **lib/rtoon/lexer.rb**: Indentation-aware lexer
-- **lib/rtoon/parser.y**: Racc grammar definition
-- **lib/rtoon/parser.tab.rb**: Generated parser (don't edit!)
-- **lib/rtoon/encoder.rb**: TOON encoder (Ruby → TOON)
-- **lib/rtoon.rb**: Main interface
-
-## Use Cases
-
-TOON is ideal for:
-
-- **Configuration files** with tabular data
-- **Database seeds** with schemas and rows
-- **API responses** with structured tables
-- **Data exports** in readable format
-- **Test fixtures** with clear structure
-
 ## Limitations
 
-- Values are currently returned as strings (no automatic type conversion)
-- No support for nested arrays in data rows
+- Values returned as strings (no automatic type conversion)
 - No string escaping (commas in values not supported)
-- Numbers are treated as identifiers/strings
-
-## Future Enhancements
-
-- Type inference/conversion (numbers, booleans)
-- String literals with quotes
-- Escape sequences for commas in values
-- Comments support
-- Encoder (Ruby → TOON)
+- Fixed 2-space indentation
+- No comment syntax
 
 ## Contributing
 
 1. Fork the repository
-2. Create your feature branch
-3. Add tests for your changes
-4. Ensure all tests pass
-5. Submit a pull request
+2. Create a feature branch
+3. Add tests for new features
+4. Submit a Pull Request
+
+See [ARCHITECTURE.md](ARCHITECTURE.md) for implementation details.
 
 ## License
 
-MIT License - See LICENSE.txt
+MIT License - See [LICENSE.txt](LICENSE.txt)
+
+## Author
 
-## Resources
+**Antonio Chavez** - [Zoobean](https://github.com/zoobean)
 
-- Racc: https://github.com/ruby/racc
-- Parser theory: https://en.wikipedia.org/wiki/Parsing
-- Indentation-based parsing: https://en.wikipedia.org/wiki/Off-side_rule
+Report bugs: [GitHub Issues](https://github.com/zoobean/rtoon/issues)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rtoon
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.1
 platform: ruby
 authors:
 - Antonio Chavez
@@ -61,7 +61,6 @@ extra_rdoc_files: []
 files:
 - LICENSE.txt
 - README.md
-- TOON_SPEC.md
 - lib/rtoon.rb
 - lib/rtoon/encoder.rb
 - lib/rtoon/lexer.rb

data/TOON_SPEC.md

@@ -1,61 +0,0 @@
-# TOON Format Specification
-
-## Overview
-Token Object Oriented Notation (TOON) is a tabular, schema-based data format with indentation-based structure.
-
-## Syntax Elements
-
-### Schema Declaration
-```
-identifier[size]{field1,field2,...}:
-```
-
-- **identifier**: Name of the data structure
-- **[size]**: Optional array size hint
-- **{fields}**: Comma-separated list of field names
-- **:** Indicates content follows
-
-### Data Rows
-```
-value1,value2,value3
-```
-
-Comma-separated values corresponding to the schema fields.
-
-### Nesting
-Uses indentation (2 spaces) to indicate nested structures.
-
-### Example
-```
-items[1]{users,status}:
-  users[2]{id,name}:
-    1,Ada
-    2,Bob
-  status: active
-```
-
-## Parsed Structure
-The above should parse to:
-```ruby
-{
-  "items" => [
-    {
-      "users" => [
-        { "id" => "1", "name" => "Ada" },
-        { "id" => "2", "name" => "Bob" }
-      ],
-      "status" => "active"
-    }
-  ]
-}
-```
-
-## Grammar Rules
-
-1. **Top-level**: Schema declaration or field assignment
-2. **Schema**: `name[size]{fields}:` followed by indented content
-3. **Field**: `name: value` (simple assignment)
-4. **Data rows**: Comma-separated values matching schema fields
-5. **Indentation**: 2 spaces per level
-6. **Arrays**: Created from schema declarations with size hint
-7. **Objects**: Created from schema fields