toon-fu 4.1.0 → 4.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6eda50e6a13e61ee805fbaf15f417176c2463648bc226af942aaa383ee50cc5c
-  data.tar.gz: 65406f2ccd9d8e643c9f26109d16d52d3361c92e200323e235e52ec572669461
+  metadata.gz: e095746e545c92bf6411c36b2042dde6d305deb9a1363d500d48877f09939b63
+  data.tar.gz: 937a2796b6a6f04afa2469994c94527473e93c66f575747adefea369691628d4
 SHA512:
-  metadata.gz: ef2e69c9a75e69600957f6ab92e139ee522d960cafc2793aa7621854fa9c5bf667b7c197f2b32820b361af3070b68769dfc86c8e639a428c8bec8ad61ee6ee99
-  data.tar.gz: 8d5ced2be98d6bba1e423a75aac5eb22992bbfa8e76b1cf5cc2a33f09a0b1eea96590c4a8b33970768cd9885d30a39e0fc8db8522c263c60d93130d1c7043112
+  metadata.gz: 979b0210dfed7025dc7be6ba5c32362d7293ea394cb9fbf495374a17d136a9e81d9f8af0419278b891d39bde60c731624050efd4bf5c6ee3dfcc7858d7347b5e
+  data.tar.gz: 481896ac9ff0f47a297e794e90ebd3aae04d3dfbe1267a0e5b8c264f5ef4385a1816b0d241d0bd2ace3047d7748d9b4eefa2487cc251b3f264c8054967552d2c

data/README.md

@@ -14,17 +14,23 @@ TOON is a compact, readable encoding of the JSON data model for LLM prompts: ind
 
 Every Ruby TOON gem on RubyGems was published in late 2025 and stopped at spec 1.2, three major versions behind. They leave strings starting with `#` or `+` unquoted, which a current reader takes for a comment or a number, and shift dates by a day east of Greenwich. toon-fu tracks the spec: its version is the spec version, and a daily check flags a newer spec.
 
+Yes, we know:
+
+[![xkcd 927: Standards](https://imgs.xkcd.com/comics/standards.png)](https://xkcd.com/927/)
+
+<sub>[xkcd #927 "Standards"](https://xkcd.com/927/) by Randall Munroe, [CC BY-NC 2.5](https://creativecommons.org/licenses/by-nc/2.5/).</sub>
+
 ## Getting started
 
-```bash
-gem install toon-fu
+Add it to your Gemfile:
+
+```ruby
+gem "toon-fu"
 ```
 
-In a Gemfile, pin the spec line you speak — see [Versioning](#versioning).
+In a plain script: `gem install toon-fu` and `require "toon_fu"`. To pin the spec version, see [Versioning](#versioning).
 
 ```ruby
-require "toon_fu"
-
 ToonFu.encode({users: [{id: 1, name: "Ada", role: "admin"}, {id: 2, name: "Bob", role: "user"}]})
 ```
 
@@ -78,7 +84,7 @@ ToonFu.encode({price: Money.new(1999, "EUR")})
 #   currency: EUR
 ```
 
-Anything toon-fu does not know raises `ToonFu::Error` instead of guessing:
+Any other value raises `ToonFu::Error`:
 
 ```ruby
 ToonFu.encode({at: Object.new})
@@ -103,16 +109,48 @@ For ActiveRecord models, pass `record.as_json` (or define `as_toon`).
 | objects with `to_hash`, `to_ary`, `to_str` | the value they convert to |
 | objects with `as_toon` | whatever `as_toon` returns, encoded in turn |
 
-Everything else raises `ToonFu::Error` — including a `Struct` or `Data` without `as_toon`, and circular references.
+Everything else raises `ToonFu::Error` — including a `Struct` or `Data` without `as_toon`, circular references, and nesting too deep for the stack.
+
+## Compared with other Ruby TOON gems
+
+The only one that passes every spec fixture: toon-fu passes all 179 encode fixtures; the table compares the 154 that use default options, which every gem can run.
+
+| Gem | Spec fixtures passed | Speed vs toon-fu |
+|---|---:|---:|
+| **toon-fu** | **154 / 154** | **1.00×** |
+| sorbet-toon 0.1.0 | 119 / 154 | 0.59× |
+| toon-ruby 0.1.1 | 117 / 154 | 0.62× |
+| toon_my_json 0.1.0 | 57 / 154 | 1.68× |
+| toon-format 0.1.2 | 45 / 154 | 1.18× |
+
+The Ruby TOON encoders with more than 10,000 downloads, measured by [`benchmark/run.rb`](benchmark/run.rb) on Ruby 3.4.10 (2026-09-24). **Spec fixtures** are the spec's own encode fixtures that use default options. **Speed** is the geometric mean of encodes per second over five workloads — tables of 100 and 1000 rows, nested objects, a list of mixed objects, strings needing quotes — relative to toon-fu.
+
+What falls through the gaps:
+
+- **sorbet-toon, toon-ruby** — `#tag` and `+1` go out unquoted, so a current reader sees a comment and a number; arrays of objects with nested columns lose their table form; no keyed tables. Unknown objects slip through instead of raising: toon-ruby writes `null`, sorbet-toon `"#<Foo:0x…>"`. toon-ruby also moves a `Date` a day back east of Greenwich.
+- **toon_my_json, toon-format** — output a TOON reader cannot read: the table header on its own line, a `[2,]` length, rows at the wrong depth, nested objects in cells as broken text; toon_my_json also writes `false` as `null`. They do less work, and it shows in both columns.
+
+Rerun it:
+
+```bash
+BUNDLE_GEMFILE=benchmark/Gemfile bundle install
+BUNDLE_GEMFILE=benchmark/Gemfile bundle exec ruby benchmark/run.rb
+```
+
+To see where toon-fu itself spends time and allocates, profile one workload (CPU by stackprof, allocation sites by memory_profiler):
+
+```bash
+BUNDLE_GEMFILE=benchmark/Gemfile bundle exec ruby benchmark/profile.rb "table, 1000 rows"
+```
 
 ## Versioning
 
 The gem version tracks the TOON specification it implements:
 
 - `MAJOR.MINOR` is the spec version: `X.Y.Z` speaks TOON `X.Y`.
-- `PATCH` is the gem's own: fixes and improvements that do not change the dialect.
+- `PATCH` is the gem's own: fixes and improvements within the same dialect.
 
-Pin the spec line, not just the major: `gem "toon-fu", "~> X.Y.0"` takes our fixes and never moves you to a new dialect. The gem badge above shows the current release; the spec-drift badge turns red when a newer spec is released and toon-fu has not caught up yet.
+Pin the spec line with `gem "toon-fu", "~> X.Y.0"`: it takes our fixes and keeps you on the dialect you speak. The gem badge above shows the current release; the spec-drift badge turns red when a newer spec is released and toon-fu has not caught up yet.
 
 Release notes: [GitHub releases](https://github.com/hoblin/toon-fu/releases).
 

data/lib/toon-fu.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+require_relative "toon_fu"

data/lib/toon_fu/encoder.rb

@@ -44,9 +44,11 @@ module ToonFu
     #   Symbol or Integer, and keys that collide once converted to strings;
     #   for strings that are not valid UTF-8; for circular references,
     #   including an +as_toon+ or implicit conversion that leads back to its
-    #   own object
+    #   own object, and for nesting too deep for the stack
     def encode(value)
       Writer.new(@delimiter, @indent).write(Normalizer.new.call(value))
+    rescue SystemStackError
+      raise Error, "cannot encode a circular reference or nesting too deep"
     end
   end
 end

data/lib/toon_fu/fields.rb

@@ -23,12 +23,15 @@ module ToonFu
       new(columns)
     end
 
+    attr_reader :paths
+
     def initialize(columns)
       @columns = columns
+      @paths = columns.flat_map { |key, nested| nested ? nested.paths.map { |path| [key, *path] } : [[key]] }
     end
 
     def cells(row)
-      @columns.flat_map { |key, nested| nested ? nested.cells(row[key]) : [row[key]] }
+      @paths.map { |path| path.reduce(row) { |node, key| node[key] } }
     end
   end
 end

data/lib/toon_fu/float_literal.rb

@@ -4,6 +4,15 @@ module ToonFu
   class FloatLiteral
     DECIMAL_RANGE = (1e-6...1e21)
 
+    def self.format(value)
+      plain = value.to_s
+      return new(value).to_s if plain.include?("e") || !value.finite?
+
+      return "0" if value.zero?
+
+      plain.end_with?(".0") ? plain.delete_suffix(".0") : plain
+    end
+
     def initialize(value)
       @value = value
     end

data/lib/toon_fu/normalizer.rb

@@ -4,15 +4,11 @@ module ToonFu
   class Normalizer
     TRAILING_FRACTION_ZEROS = /\.?0+\z/
 
-    def initialize
-      @path = {}.compare_by_identity
-    end
-
     def call(value)
       raise Error, "cannot encode a BasicObject" unless Kernel === value
       return core(value) unless value.respond_to?(:as_toon)
 
-      within(value) { call(value.as_toon) }
+      call(value.as_toon)
     end
 
     private
@@ -22,8 +18,9 @@ module ToonFu
       when nil, true, false, Integer, Float then value
       when String then utf8(value)
       when Symbol then utf8(value.name)
-      when Hash then within(value) { object(value) }
-      when Array, Set then within(value) { value.map { |element| call(element) } }
+      when Hash then object(value)
+      when Array then array(value)
+      when Set then value.map { |element| call(element) }
       when Time then timestamp(value)
       when DateTime then date_time(value)
       when Date then value.iso8601
@@ -33,14 +30,50 @@ module ToonFu
 
     def convert(value)
       if defined?(BigDecimal) && value.is_a?(BigDecimal) then DecimalLiteral.new(value)
-      elsif value.respond_to?(:to_hash) then within(value) { call(value.to_hash) }
-      elsif value.respond_to?(:to_ary) then within(value) { call(value.to_ary) }
-      elsif value.respond_to?(:to_str) then within(value) { call(value.to_str) }
+      elsif value.respond_to?(:to_hash) then call(value.to_hash)
+      elsif value.respond_to?(:to_ary) then call(value.to_ary)
+      elsif value.respond_to?(:to_str) then call(value.to_str)
       else raise Error, "cannot encode #{value.class}; convert it first or define #as_toon"
       end
     end
 
+    def array(values)
+      return values.map { |element| call(element) } unless values.instance_of?(Array)
+
+      copy = nil
+      index = 0
+      while index < values.size
+        element = values[index]
+        normal = call(element)
+        unless copy.nil? && normal.equal?(element)
+          copy ||= values.first(index)
+          copy << normal
+        end
+        index += 1
+      end
+      copy || values
+    end
+
     def object(hash)
+      return rebuild(hash) unless plain?(hash)
+
+      copy = nil
+      hash.each do |key, value|
+        normal = call(value)
+        next if copy.nil? && normal.equal?(value)
+
+        copy ||= hash.take_while { |pair_key, _| !pair_key.equal?(key) }.to_h
+        copy[key] = normal
+      end
+      copy || hash
+    end
+
+    def plain?(hash)
+      hash.instance_of?(Hash) && !hash.compare_by_identity? &&
+        hash.all? { |key, _| key.instance_of?(String) && key.encoding == Encoding::UTF_8 && key.valid_encoding? }
+    end
+
+    def rebuild(hash)
       hash.each_with_object({}) do |(key, value), result|
         name = key_name(key)
         raise Error, "duplicate key #{name.inspect} after converting keys to strings" if result.key?(name)
@@ -58,15 +91,6 @@ module ToonFu
       end
     end
 
-    def within(container)
-      raise Error, "cannot encode a circular reference through #{container.class}" if @path.key?(container)
-
-      @path[container] = true
-      result = yield
-      @path.delete(container)
-      result
-    end
-
     def timestamp(time)
       moment = time.strftime("%Y-%m-%dT%H:%M:%S.%9N").sub(TRAILING_FRACTION_ZEROS, "")
       "#{moment}#{time.utc? ? "Z" : time.strftime("%:z")}"
@@ -78,6 +102,8 @@ module ToonFu
     end
 
     def utf8(string)
+      return string if string.encoding == Encoding::UTF_8 && string.valid_encoding?
+
       string = string.dup.force_encoding(Encoding::UTF_8) if string.encoding == Encoding::BINARY
       string = string.encode(Encoding::UTF_8) unless string.encoding == Encoding::UTF_8
       raise Error, "cannot encode a string that is not valid UTF-8: #{string.inspect}" unless string.valid_encoding?

data/lib/toon_fu/string_literal.rb

@@ -4,7 +4,7 @@ module ToonFu
   class StringLiteral
     READS_AS_LITERAL = /\A(?:true|false|null|[+-]?[0-9]+(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?)\z/
     UNSAFE = Encoder::DELIMITERS.to_h do |delimiter|
-      [delimiter, /\A[ \t#-]|[ \t]\z|[:"\\\[\]{}\x00-\x1f#{Regexp.escape(delimiter)}]/]
+      [delimiter, /\A[ \t#-]|[ \t]\z|[:"\\\[\]{}\x00-\x1f#{Regexp.escape(delimiter) if delimiter.ord > 0x1f}]/]
     end.freeze
     BARE_KEY = /\A[A-Za-z_][A-Za-z0-9_.]*\z/
     ESCAPABLE = /["\\\x00-\x1f]/

data/lib/toon_fu/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ToonFu
-  VERSION = "4.1.0"
+  VERSION = "4.1.2"
 end

data/lib/toon_fu/writer.rb

@@ -118,7 +118,7 @@ module ToonFu
       case value
       when nil then "null"
       when true, false, Integer then value.to_s
-      when Float then FloatLiteral.new(value).to_s
+      when Float then FloatLiteral.format(value)
       when DecimalLiteral then value.to_s
       when String then @strings.encode(value)
       else raise Error, "cannot encode #{value.class}"

data/lib/toon_fu.rb

@@ -30,11 +30,19 @@ module ToonFu
   #   ToonFu.encode([{id: 1}, {id: 2}])     # => "[2]{id}:\n  1\n  2"
   #   ToonFu.encode({a: {x: 1}, b: {x: 2}}) # => "[2:]{x}:\n  a: 1\n  b: 2"
   #
-  # @param value [Object] see {Encoder#encode} for the accepted types
+  # @param value [Object] see {Encoder#encode} for the accepted types; a Hash
+  #   needs braces, since bare +key: value+ pairs are Ruby keyword arguments
   # @param options [Hash] see {Encoder#initialize}
   # @return [String]
   # @raise [Error] see {Encoder#encode}
-  def self.encode(value, **options)
-    Encoder.new(**options).encode(value)
+  # @raise [ArgumentError] when no value is given, as with a Hash written
+  #   without braces
+  def self.encode(value = (missing = true), **options)
+    raise ArgumentError, "ToonFu.encode takes the value as its first argument; wrap a Hash in braces: ToonFu.encode({key: value})" if missing
+
+    (options.empty? ? DEFAULT_ENCODER : Encoder.new(**options)).encode(value)
   end
+
+  DEFAULT_ENCODER = Ractor.make_shareable(Encoder.new)
+  private_constant :DEFAULT_ENCODER
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: toon-fu
 version: !ruby/object:Gem::Version
-  version: 4.1.0
+  version: 4.1.2
 platform: ruby
 authors:
 - Yevhenii Hurin
@@ -9,9 +9,11 @@ bindir: bin
 cert_chain: []
 date: 1980-01-02 00:00:00.000000000 Z
 dependencies: []
-description: A Ruby implementation of TOON, the token-efficient serialization format
-  for LLM input. The gem's MAJOR.MINOR tracks the TOON specification version; PATCH
-  is the gem's own.
+description: 'Encodes Ruby hashes, arrays, dates and your own objects into TOON, the
+  token-efficient format for LLM prompts: tables for uniform arrays, quotes only where
+  needed. The only Ruby TOON gem that passes all of the current spec''s encode fixtures;
+  its version tracks the spec it implements, and anything it cannot represent raises
+  an error so your data arrives complete.'
 email:
 - evgeny.gurin@gmail.com
 executables: []
@@ -20,6 +22,7 @@ extra_rdoc_files: []
 files:
 - LICENSE
 - README.md
+- lib/toon-fu.rb
 - lib/toon_fu.rb
 - lib/toon_fu/decimal_literal.rb
 - lib/toon_fu/encodable.rb
@@ -36,6 +39,8 @@ licenses:
 metadata:
   source_code_uri: https://github.com/hoblin/toon-fu
   changelog_uri: https://github.com/hoblin/toon-fu/releases
+  documentation_uri: https://rubydoc.info/gems/toon-fu
+  bug_tracker_uri: https://github.com/hoblin/toon-fu/issues
   rubygems_mfa_required: 'true'
 rdoc_options: []
 require_paths:
@@ -53,6 +58,6 @@ required_rubygems_version: !ruby/object:Gem::Requirement
 requirements: []
 rubygems_version: 3.6.9
 specification_version: 4
-summary: TOON (Token-Oriented Object Notation) for Ruby, versioned by the spec it
-  implements
+summary: TOON encoder for Ruby that passes every spec fixture — compact, token-efficient
+  LLM input from your Ruby data
 test_files: []