toon_my_json 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 9e4ebddf5d14d6d65b6205dd7bd42d14e1010e83c4312f5dbf4a6fd913c2012b
+  data.tar.gz: e4c5e7d10c381d40f73cfb6d7883b72758cf39bb2ac6c989726c71a2c92b673d
+SHA512:
+  metadata.gz: ffbeb4e4fcd13b5ceb76dda9718bbaf4e1cf7d9ad7fcaa1a31fc1ab4b8a36827aab7f1f80995966b84458bdc41e3248f7e64c738e074e1e12cf2a3bb9c455338
+  data.tar.gz: 8c94d7b7006f63204ec1eef550528bbd66fecad2ba3e1fea530d0aaa5be84af1f63866942e081f1e00080fac7000250ac377ade6615036a63290b2d34b6e9e53

data/CHANGELOG.md

@@ -0,0 +1,33 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2025-11-13
+
+### Added
+- Initial release of toon_my_json gem
+- `ToonMyJson.encode` - Convert JSON/Ruby objects to TOON format
+- `ToonMyJson.decode` - Convert TOON format back to JSON/Ruby objects
+- Bidirectional conversion support (JSON ↔ TOON)
+- Tabular format for uniform arrays (30-60% space savings)
+- Smart string quoting (only when necessary)
+- Support for nested structures (objects and arrays)
+- Lossless roundtrip conversions
+- Command-line interface (`toon` command)
+  - `--encode` flag for JSON to TOON conversion (default)
+  - `--decode` flag for TOON to JSON conversion
+  - `--indent` option for custom indentation
+  - `--delimiter` option for custom field delimiters
+  - `--no-length-marker` option to disable array length markers
+
+### Features
+- Automatic detection of uniform arrays for tabular formatting
+- Handles primitives, objects, arrays, and nested structures
+- Multiple input types: JSON strings, Ruby objects, TOON strings
+- Customizable encoding options (indent, delimiter, length markers)
+- Ruby API and CLI support
+
+[0.1.0]: https://github.com/mykyta/toon-my-json/releases/tag/v0.1.0

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Mykyta
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,329 @@
+# ToonMyJson
+
+A Ruby gem for bidirectional conversion between JSON and TOON (Token-Oriented Object Notation) format. TOON is a compact serialization format designed for Large Language Models that reduces token usage by 30-60% compared to JSON.
+
+## What is TOON?
+
+TOON is a compact, human-readable format that combines the best of YAML's indentation-based structure with CSV's tabular format for arrays. It minimizes syntax overhead by removing redundant punctuation like braces, brackets, and unnecessary quotes.
+
+### Format Comparison
+
+**JSON (verbose):**
+```json
+{
+  "users": [
+    { "id": 1, "name": "Alice", "role": "admin" },
+    { "id": 2, "name": "Bob", "role": "user" }
+  ]
+}
+```
+
+**TOON (compact):**
+```
+users:
+[2]{id,name,role}:
+  1,Alice,admin
+  2,Bob,user
+```
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'toon_my_json'
+```
+
+And then execute:
+
+```bash
+bundle install
+```
+
+Or install it yourself as:
+
+```bash
+gem install toon_my_json
+```
+
+## Requirements
+
+- Ruby >= 3.0.0
+- JSON gem (~> 2.0)
+
+## Quick Start
+
+```ruby
+require 'toon_my_json'
+
+# Encode JSON to TOON
+data = { "users" => [{ "id" => 1, "name" => "Alice" }] }
+toon = ToonMyJson.encode(data)
+# => "users:\n[1]{id,name}:\n  1,Alice"
+
+# Decode TOON back to JSON
+restored = ToonMyJson.decode(toon)
+# => {"users"=>[{"id"=>1, "name"=>"Alice"}]}
+```
+
+## Usage
+
+### Ruby API
+
+#### Encoding (JSON → TOON)
+
+```ruby
+require 'toon_my_json'
+
+# Convert a Ruby hash to TOON
+data = { "name" => "Alice", "age" => 30 }
+ToonMyJson.encode(data)
+# => "name: Alice\nage: 30"
+
+# Convert a JSON string to TOON
+json = '{"name":"Alice","age":30}'
+ToonMyJson.encode(json)
+# => "name: Alice\nage: 30"
+
+# Arrays automatically use tabular format for uniform data
+data = [
+  { "id" => 1, "name" => "Alice", "role" => "admin" },
+  { "id" => 2, "name" => "Bob", "role" => "user" }
+]
+ToonMyJson.encode(data)
+# => "[2]{id,name,role}:\n1,Alice,admin\n2,Bob,user"
+```
+
+#### Decoding (TOON → JSON)
+
+```ruby
+# Convert TOON back to Ruby objects
+toon = "name: Alice\nage: 30"
+ToonMyJson.decode(toon)
+# => {"name"=>"Alice", "age"=>30}
+
+# Get JSON string output instead of Ruby object
+ToonMyJson.decode(toon, json: true)
+# => "{\n  \"name\": \"Alice\",\n  \"age\": 30\n}"
+```
+
+#### Roundtrip Conversion
+
+```ruby
+# Perfect lossless conversion
+original = { "company" => "TechCorp", "year" => 2020 }
+toon = ToonMyJson.encode(original)
+restored = ToonMyJson.decode(toon)
+# => {"company"=>"TechCorp", "year"=>2020}
+original == restored  # => true
+```
+
+### Configuration Options
+
+#### Encoding Options
+
+```ruby
+# Custom indentation (default: 2)
+ToonMyJson.encode(data, indent: 4)
+
+# Custom delimiter for arrays (default: ',')
+ToonMyJson.encode(data, delimiter: '|')
+
+# Disable length markers (default: true)
+ToonMyJson.encode(data, length_marker: false)
+```
+
+### Decoding Options
+
+```ruby
+# Custom indentation for JSON output (default: 2)
+ToonMyJson.decode(toon, indent: 4)
+
+# Custom delimiter (must match what was used in encoding)
+ToonMyJson.decode(toon, delimiter: '|')
+
+# Get JSON string output instead of Ruby object
+ToonMyJson.decode(toon, json: true)
+```
+
+### Command Line Interface
+
+The gem includes a `toon` CLI tool for converting between JSON and TOON formats:
+
+```bash
+# Encode JSON to TOON (default)
+$ toon input.json
+$ echo '{"name":"Alice","age":30}' | toon
+
+# Decode TOON to JSON
+$ toon --decode input.toon
+$ echo -e 'name: Alice\nage: 30' | toon --decode
+
+# Roundtrip conversion
+$ echo '{"name":"Alice"}' | toon | toon --decode
+
+# Options
+$ toon --indent 4 --delimiter '|' input.json      # Custom formatting
+$ toon --no-length-marker input.json              # Disable array length markers
+$ toon --decode --delimiter '|' input.toon        # Decode with custom delimiter
+
+# Help and version
+$ toon --help
+$ toon --version
+```
+
+## Features
+
+- **Bidirectional Conversion**: Encode JSON to TOON and decode TOON back to JSON
+- **Tabular Format**: Automatically detects uniform arrays of objects and converts them to compact tabular format
+- **Smart Quoting**: Only adds quotes when necessary (special characters, reserved words, etc.)
+- **Nested Structures**: Handles deeply nested objects and arrays
+- **Lossless Roundtrips**: Encode and decode without data loss
+- **Flexible Options**: Customize indentation, delimiters, and length markers
+- **CLI Tool**: Convert files from the command line with full encode/decode support
+- **Multiple Input Types**: Accepts JSON strings, Ruby objects, or TOON strings
+
+## Advanced Examples
+
+### Complex Nested Structure
+
+```ruby
+data = {
+  "company" => "TechCorp",
+  "employees" => [
+    { "id" => 1, "name" => "Alice", "department" => "Engineering" },
+    { "id" => 2, "name" => "Bob", "department" => "Sales" }
+  ],
+  "metadata" => {
+    "founded" => 2020,
+    "location" => "San Francisco"
+  }
+}
+
+puts ToonMyJson.encode(data)
+```
+
+**Output:**
+```
+company: TechCorp
+employees:
+[2]{id,name,department}:
+  1,Alice,Engineering
+  2,Bob,Sales
+metadata:
+  founded: 2020
+  location: San Francisco
+```
+
+### Primitive Arrays
+
+```ruby
+data = { "colors" => ["red", "green", "blue"] }
+ToonMyJson.encode(data)
+# => "colors: red,green,blue"
+```
+
+### Mixed Arrays
+
+```ruby
+data = ["string", 42, { "key" => "value" }]
+ToonMyJson.encode(data)
+# => "- string\n- 42\n- key: value"
+```
+
+### Decoding Examples
+
+```ruby
+# Decode simple hash
+toon = <<~TOON
+  name: Alice
+  age: 30
+TOON
+ToonMyJson.decode(toon)
+# => {"name"=>"Alice", "age"=>30}
+
+# Decode tabular array
+toon = <<~TOON
+  [2]{id,name,role}:
+    1,Alice,admin
+    2,Bob,user
+TOON
+ToonMyJson.decode(toon)
+# => [{"id"=>1, "name"=>"Alice", "role"=>"admin"}, {"id"=>2, "name"=>"Bob", "role"=>"user"}]
+
+# Decode complex nested structure
+toon = <<~TOON
+  company: TechCorp
+  employees:
+  [2]{id,name,department}:
+    1,Alice,Engineering
+    2,Bob,Sales
+  metadata:
+    founded: 2020
+    location: San Francisco
+TOON
+result = ToonMyJson.decode(toon)
+# => {"company"=>"TechCorp", "employees"=>[...], "metadata"=>{...}}
+```
+
+## Development
+
+After checking out the repo, run `bundle install` to install dependencies:
+
+```bash
+bundle install
+```
+
+Run the test suite:
+
+```bash
+bundle exec rspec
+# or
+bundle exec rake spec
+```
+
+This represents complete test coverage for production Ruby code, ensuring all code paths and conditional branches are thoroughly tested.
+
+Run performance benchmarks (performance can be improved):
+
+```bash
+# Run all benchmark tests
+bundle exec rspec --tag benchmark
+
+# Or with environment variable
+BENCHMARK=1 bundle exec rspec
+
+# Run only the benchmark-ips comparison (shows iterations/second)
+bundle exec rspec --tag ips
+```
+
+Performance benchmarks validate:
+- Encoding 1000 records completes in under 10ms
+- Decoding 1000 records completes in under 50ms
+- Roundtrip conversion completes in under 60ms
+- Iterations per second for common operations
+
+Install the gem locally:
+
+```bash
+bundle exec rake install
+```
+
+Build the gem:
+
+```bash
+bundle exec rake build
+```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/mykbren/toon-my-json.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## References
+
+- [TOON Format Specification](https://github.com/toon-format/toon)
+- [Original TypeScript Implementation](https://github.com/toon-format/toon)

data/Rakefile

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec
+task test: :spec

data/bin/toon

@@ -0,0 +1,79 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require_relative '../lib/toon_my_json'
+require 'optparse'
+
+options = {
+  indent: 2,
+  delimiter: ',',
+  length_marker: true,
+  mode: :encode
+}
+
+OptionParser.new do |opts|
+  opts.banner = 'Usage: toon [options] [file]'
+  opts.separator ''
+  opts.separator 'Convert between JSON and TOON formats'
+  opts.separator ''
+  opts.separator 'Options:'
+
+  opts.on('-e', '--encode', 'Encode JSON to TOON (default)') do
+    options[:mode] = :encode
+  end
+
+  opts.on('-D', '--decode', 'Decode TOON to JSON') do
+    options[:mode] = :decode
+  end
+
+  opts.on('-i', '--indent N', Integer, 'Indentation spaces (default: 2)') do |n|
+    options[:indent] = n
+  end
+
+  opts.on('-d', '--delimiter CHAR', String, 'Field delimiter (default: ,)') do |d|
+    options[:delimiter] = d
+  end
+
+  opts.on('--no-length-marker', 'Disable array length markers (encode only)') do
+    options[:length_marker] = false
+  end
+
+  opts.on('-h', '--help', 'Show this help message') do
+    puts opts
+    exit
+  end
+
+  opts.on('-v', '--version', 'Show version') do
+    puts "toon_my_json version #{ToonMyJson::VERSION}"
+    exit
+  end
+end.parse!
+
+begin
+  # Read from file or stdin
+  input = if ARGV.empty?
+            $stdin.read
+          else
+            File.read(ARGV[0])
+          end
+
+  # Convert and output
+  mode = options.delete(:mode)
+  result = if mode == :decode
+             # Remove encode-only options
+             decode_options = options.slice(:indent, :delimiter)
+             ToonMyJson.decode(input, **decode_options, json: true)
+           else
+             ToonMyJson.encode(input, **options)
+           end
+  puts result
+rescue JSON::ParserError => e
+  warn "Error: Invalid JSON - #{e.message}"
+  exit 1
+rescue Errno::ENOENT => e
+  warn "Error: File not found - #{e.message}"
+  exit 1
+rescue StandardError => e
+  warn "Error: #{e.message}"
+  exit 1
+end

data/lib/toon_my_json/decoder.rb

@@ -0,0 +1,286 @@
+# frozen_string_literal: true
+
+module ToonMyJson
+  # Decodes TOON format back to Ruby objects
+  class Decoder
+    attr_reader :lines, :current_line, :delimiter
+
+    def initialize(indent: 2, delimiter: ',')
+      @indent_size = indent
+      @delimiter = delimiter
+    end
+
+    def decode(toon_string)
+      @lines = toon_string.split("\n")
+      @current_line = 0
+
+      # Detect if it's a single line
+      if @lines.length == 1
+        content = @lines[0].strip
+
+        # Check if it's a key-value (check this first!)
+        return parse_hash(0) if key_value_line?(content)
+
+        # Check if it's a primitive array (contains delimiter but not quotes around everything)
+        return parse_primitive_array(content) if content.include?(@delimiter) && !content.match(/^".*"$/)
+
+        # Otherwise it's a single primitive
+        return parse_primitive(content)
+      end
+
+      # Multi-line parsing
+      parse_value(0)
+    end
+
+    private
+
+    def parse_value(expected_indent)
+      return nil if @current_line >= @lines.length
+
+      line = @lines[@current_line]
+      indent = get_indent(line)
+
+      return nil if indent < expected_indent
+
+      content = line.strip
+
+      # Check for tabular array header [N]{fields}: or {fields}:
+      return parse_tabular_array(indent) if content.match(/^(?:\[\d+\])?\{[^}]+\}:$/)
+
+      # Check for list array (lines starting with -)
+      return parse_list_array(indent) if content.start_with?('-')
+
+      # Check if this looks like a hash (has key-value pairs)
+      return parse_hash(indent) if key_value_line?(content)
+
+      # Single primitive
+      @current_line += 1
+      parse_primitive(content)
+    end
+
+    def key_value_line?(line)
+      # A key-value line has a colon, but the colon should not be inside quotes
+      # Use split_key_value to check and avoid duplicate logic
+      _, value = split_key_value(line)
+      !value.nil?
+    end
+
+    def parse_hash(expected_indent)
+      hash = {}
+
+      while @current_line < @lines.length
+        line = @lines[@current_line]
+        indent = get_indent(line)
+
+        break if indent < expected_indent
+
+        content = line.strip
+        break if content.empty?
+
+        # Check if it's a tabular array header (not a key-value pair)
+        break if content.match(/^(?:\[\d+\])?\{[^}]+\}:$/)
+
+        # Check if it's a list array item
+        break if content.start_with?('-')
+
+        # Parse key-value pair
+        key, value_part = split_key_value(content)
+        if value_part.nil?
+          # Not a valid key-value line (no unquoted colon), stop parsing hash
+          break
+        end
+
+        key = parse_string(key.strip)
+
+        if value_part.strip.empty?
+          # Value on next lines (nested)
+          @current_line += 1
+
+          # Check if next line is a tabular array header (can be at any indent)
+          if @current_line < @lines.length
+            next_line = @lines[@current_line].strip
+            if next_line.match(/^(?:\[\d+\])?\{[^}]+\}:$/)
+              # Parse tabular array regardless of indent
+              hash[key] = parse_tabular_array(get_indent(@lines[@current_line]))
+              next
+            end
+          end
+
+          # For nested values, accept same indent or greater
+          hash[key] = parse_value(expected_indent)
+        else
+          # Value on same line
+          value_part = value_part.strip
+          @current_line += 1
+
+          hash[key] = case value_part
+                      when '[]'
+                        []
+                      when '{}'
+                        {}
+                      else
+                        # Could be primitive, primitive array, or inline object
+                        if value_part.include?(@delimiter) && !value_part.match(/^".*"$/)
+                          # Primitive array
+                          parse_primitive_array(value_part)
+                        else
+                          parse_primitive(value_part)
+                        end
+                      end
+        end
+      end
+
+      hash
+    end
+
+    def split_key_value(line)
+      # Split by first colon not in quotes
+      in_quotes = false
+      line.each_char.with_index do |char, i|
+        if char == '"' && (i.zero? || line[i - 1] != '\\')
+          in_quotes = !in_quotes
+        elsif char == ':' && !in_quotes
+          return [line[0...i], line[(i + 1)..]]
+        end
+      end
+      [line, nil]
+    end
+
+    def parse_tabular_array(expected_indent)
+      line = @lines[@current_line].strip
+
+      # Parse header: [N]{field1,field2,...}: or {field1,field2,...}:
+      match = line.match(/^(?:\[\d+\])?\{([^}]+)\}:$/)
+      return [] unless match
+
+      fields = match[1].split(@delimiter).map(&:strip)
+      @current_line += 1
+
+      array = []
+      while @current_line < @lines.length
+        line = @lines[@current_line]
+        indent = get_indent(line)
+
+        break if indent <= expected_indent
+
+        content = line.strip
+        break if content.empty?
+
+        # Stop if we hit a key-value line (next section)
+        break if key_value_line?(content) && !content.match(/^(?:\[\d+\])?\{[^}]+\}:$/)
+
+        # Parse row
+        values = parse_csv_line(content)
+        row = {}
+        fields.each_with_index do |field, i|
+          row[field] = values[i] if i < values.length
+        end
+        array << row
+        @current_line += 1
+      end
+
+      array
+    end
+
+    def parse_list_array(expected_indent)
+      array = []
+
+      while @current_line < @lines.length
+        line = @lines[@current_line]
+        indent = get_indent(line)
+
+        break if indent < expected_indent
+
+        content = line.strip
+        break unless content.start_with?('-')
+
+        # Remove leading dash and space
+        item_content = content[1..].strip
+
+        @current_line += 1
+        array << if item_content.empty?
+                   # Multi-line item (next line)
+                   parse_value(expected_indent + @indent_size)
+                 else
+                   # Inline item
+                   parse_primitive(item_content)
+                 end
+      end
+
+      array
+    end
+
+    def parse_primitive_array(content)
+      parse_csv_line(content)
+    end
+
+    def parse_csv_line(line)
+      values = []
+      current = String.new # Pre-allocate mutable string
+      in_quotes = false
+      i = 0
+
+      while i < line.length
+        char = line[i]
+
+        if char == '"' && (i.zero? || line[i - 1] != '\\')
+          in_quotes = !in_quotes
+          current << char
+        elsif char == @delimiter && !in_quotes
+          values << parse_primitive(current.strip)
+          current.clear
+        else
+          current << char
+        end
+
+        i += 1
+      end
+
+      values << parse_primitive(current.strip) unless current.strip.empty?
+      values
+    end
+
+    def parse_primitive(value)
+      value = value.strip
+
+      # Handle quoted strings
+      if value.start_with?('"') && value.end_with?('"') && value.length > 1
+        # Remove quotes and unescape in single pass
+        return unescape_string(value[1...-1])
+      end
+
+      # Handle special values
+      case value
+      when 'null'
+        nil
+      when 'true'
+        true
+      when 'false'
+        false
+      when /^-?\d+$/
+        value.to_i
+      when /^-?\d+\.\d+$/
+        value.to_f
+      else
+        value
+      end
+    end
+
+    def parse_string(value)
+      if value.start_with?('"') && value.end_with?('"') && value.length > 1
+        unescape_string(value[1...-1])
+      else
+        value
+      end
+    end
+
+    def unescape_string(str)
+      # Unescape only the specific escape sequences we support: \\ and \"
+      str.gsub(/\\\\|\\"/) { |match| match == '\\\\' ? '\\' : '"' }
+    end
+
+    def get_indent(line)
+      line.match(/^(\s*)/)[1].length
+    end
+  end
+end

data/lib/toon_my_json/encoder.rb

@@ -0,0 +1,183 @@
+# frozen_string_literal: true
+
+module ToonMyJson
+  # Encodes Ruby objects to TOON format
+  class Encoder
+    RESERVED_CHARS = /[,:\[\]{}#\n\r\t]/
+    NEEDS_QUOTES = /\A\s|\s\z|#{RESERVED_CHARS}/
+
+    attr_reader :indent_size, :delimiter, :length_marker
+
+    def initialize(indent: 2, delimiter: ',', length_marker: true)
+      @indent_size = indent
+      @delimiter = delimiter
+      @length_marker = length_marker
+    end
+
+    def encode(value, depth = 0)
+      case value
+      when Hash
+        encode_hash(value, depth)
+      when Array
+        encode_array(value, depth)
+      when nil
+        'null'
+      when true, false
+        value.to_s
+      when Numeric
+        value.to_s
+      when String
+        encode_string(value)
+      else
+        encode_string(value.to_s)
+      end
+    end
+
+    private
+
+    def encode_hash(hash, depth)
+      return '{}' if hash.empty?
+
+      lines = []
+      hash.each do |key, value|
+        encoded_value = encode_value_for_hash(value, depth)
+        lines << "#{indent(depth)}#{encode_key(key)}:#{encoded_value}"
+      end
+      lines.join("\n")
+    end
+
+    def encode_value_for_hash(value, depth)
+      case value
+      when Hash
+        if value.empty?
+          ' {}'
+        else
+          "\n#{encode_hash(value, depth + 1)}"
+        end
+      when Array
+        if value.empty?
+          ' []'
+        elsif uniform_array?(value) && value.first.is_a?(Hash)
+          "\n#{encode_tabular_array(value, depth + 1)}"
+        elsif primitive_array?(value)
+          " #{encode_primitive_array(value)}"
+        else
+          "\n#{encode_list_array(value, depth + 1)}"
+        end
+      else
+        " #{encode(value, depth)}"
+      end
+    end
+
+    def encode_array(array, depth)
+      return '[]' if array.empty?
+
+      if uniform_array?(array) && array.first.is_a?(Hash)
+        encode_tabular_array(array, depth)
+      elsif primitive_array?(array)
+        encode_primitive_array(array)
+      else
+        encode_list_array(array, depth)
+      end
+    end
+
+    def encode_tabular_array(array, depth)
+      return '[]' if array.empty?
+
+      # Get all unique keys across all objects
+      keys = array.flat_map(&:keys).uniq
+
+      # Build header
+      length_prefix = @length_marker ? "[#{array.length}]" : ''
+      header = "#{length_prefix}{#{keys.join(delimiter)}}"
+
+      # Build rows
+      rows = array.map do |item|
+        row_values = keys.map { |key| encode(item[key] || item[key.to_sym], depth) }
+        "#{indent(depth)}#{row_values.join(delimiter)}"
+      end
+
+      "#{header}:\n#{rows.join("\n")}"
+    end
+
+    def encode_primitive_array(array)
+      array.map { |v| encode(v, 0) }.join(delimiter)
+    end
+
+    def encode_list_array(array, depth)
+      lines = array.map do |item|
+        case item
+        when Hash, Array
+          encoded = encode(item, depth + 1)
+          # If multiline, indent the nested structure
+          if encoded.include?("\n")
+            "#{indent(depth)}-\n#{indent_multiline(encoded, depth + 1)}"
+          else
+            "#{indent(depth)}- #{encoded}"
+          end
+        else
+          "#{indent(depth)}- #{encode(item, depth)}"
+        end
+      end
+      lines.join("\n")
+    end
+
+    def encode_key(key)
+      key_str = key.to_s
+      # Keys generally don't need quotes unless they contain special chars
+      key_str.match?(NEEDS_QUOTES) ? encode_string(key_str) : key_str
+    end
+
+    def encode_string(str)
+      return '""' if str.empty?
+
+      # Check if string needs quotes
+      if str.match?(NEEDS_QUOTES) || looks_like_number?(str) || looks_like_boolean?(str)
+        # Escape quotes and backslashes
+        escaped = str.gsub('\\', '\\\\\\\\').gsub('"', '\\"')
+        "\"#{escaped}\""
+      else
+        str
+      end
+    end
+
+    def uniform_array?(array)
+      return false if array.empty? || !array.first.is_a?(Hash)
+
+      # Check if all elements are hashes with similar structure
+      first_keys = array.first.keys.sort
+      min_overlap = (first_keys.length * 0.8).ceil
+
+      array.all? do |item|
+        next false unless item.is_a?(Hash)
+
+        # Count matching keys without sorting every time
+        overlap = 0
+        item_keys = item.keys
+        first_keys.each { |key| overlap += 1 if item_keys.include?(key) }
+        overlap >= min_overlap
+      end
+    end
+
+    def primitive_array?(array)
+      array.all? { |v| v.is_a?(String) || v.is_a?(Numeric) || v == true || v == false || v.nil? }
+    end
+
+    def looks_like_number?(str)
+      str.match?(/\A-?\d+(\.\d+)?\z/)
+    end
+
+    def looks_like_boolean?(str)
+      %w[true false null].include?(str)
+    end
+
+    def indent(depth)
+      ' ' * (depth * @indent_size)
+    end
+
+    def indent_multiline(text, depth)
+      indent_str = indent(depth)
+      text.gsub(/^/, indent_str)
+    end
+  end
+end

data/lib/toon_my_json/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module ToonMyJson
+  VERSION = '0.1.0'
+end

data/lib/toon_my_json.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+require_relative 'toon_my_json/version'
+require_relative 'toon_my_json/encoder'
+require_relative 'toon_my_json/decoder'
+require 'json'
+
+# ToonMyJson provides bidirectional conversion between JSON and TOON format.
+# TOON (Token-Oriented Object Notation) is a compact serialization format
+# designed for LLMs that reduces token usage by 30-60% compared to JSON.
+module ToonMyJson
+  class Error < StandardError; end
+
+  # Convert a Ruby object or JSON string to TOON format
+  #
+  # @param input [String, Hash, Array, Object] JSON string or Ruby object
+  # @param options [Hash] Encoding options
+  # @option options [Integer] :indent Number of spaces per indentation level (default: 2)
+  # @option options [String] :delimiter Field delimiter for arrays (',', '\t', or '|') (default: ',')
+  # @option options [Boolean] :length_marker Include array length markers (default: true)
+  # @return [String] TOON formatted string
+  def self.encode(input, **options)
+    data = if input.is_a?(String) && (input.start_with?('{', '[') || input.strip.start_with?('{', '['))
+             begin
+               JSON.parse(input)
+             rescue JSON::ParserError
+               input
+             end
+           else
+             input
+           end
+    Encoder.new(**options).encode(data)
+  end
+
+  # Alias for encode
+  def self.convert(input, **options)
+    encode(input, **options)
+  end
+
+  # Convert TOON format string to Ruby object
+  #
+  # @param toon_string [String] TOON formatted string
+  # @param options [Hash] Decoding options
+  # @option options [Integer] :indent Number of spaces per indentation level (default: 2)
+  # @option options [String] :delimiter Field delimiter for arrays (',', '\t', or '|') (default: ',')
+  # @option options [Boolean] :json Return as JSON string instead of Ruby object (default: false)
+  # @return [Hash, Array, Object, String] Ruby object or JSON string
+  def self.decode(toon_string, **options)
+    json_output = options.delete(:json)
+    result = Decoder.new(**options).decode(toon_string)
+    json_output ? JSON.pretty_generate(result) : result
+  end
+end

metadata

@@ -0,0 +1,144 @@
+--- !ruby/object:Gem::Specification
+name: toon_my_json
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- mykbren
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2025-11-14 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: json
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: benchmark-ips
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: simplecov
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.22'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.22'
+description: A Ruby gem for converting between JSON and TOON format. TOON is a compact
+  serialization format designed for LLMs that reduces token usage by 30-60% compared
+  to JSON. Supports bidirectional conversion, tabular arrays, nested structures, and
+  lossless roundtrips.
+email:
+- myk.bren@gmail.com
+executables:
+- toon
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/toon
+- lib/toon_my_json.rb
+- lib/toon_my_json/decoder.rb
+- lib/toon_my_json/encoder.rb
+- lib/toon_my_json/version.rb
+homepage: https://github.com/mykbren/toon-my-json
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/mykbren/toon-my-json
+  source_code_uri: https://github.com/mykbren/toon-my-json
+  changelog_uri: https://github.com/mykbren/toon-my-json/blob/main/CHANGELOG.md
+  rubygems_mfa_required: 'true'
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.0.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.4.6
+signing_key:
+specification_version: 4
+summary: Bidirectional JSON - TOON (Token-Oriented Object Notation) converter
+test_files: []