cton 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: d011537d6c7b854d8ffffb3dc1e7848534a7919c12f4c1cc4ba84db338cb1669
+  data.tar.gz: c96a91aae6acc9dd37e3df0b8e0fe28acfbdb6c20ea1f22a946676962f7bf7ac
+SHA512:
+  metadata.gz: 77be2aa2db0a728eaa6835be6a66456fefcd12ca27f32d3fbb3a6d77d91f4b109eca64eb958f24bb63b7ae2e193f389029fc6f1986cd590b708da3e3820b288e
+  data.tar.gz: b09740b530363e012c7f00ba6b33aaec366503310756e328a00d9def849fcb13158e911265cf57dcf6a8b613be6cd3e0e6bf164e1cfa480983b73c29a53e8cbb

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/.rubocop.yml

@@ -0,0 +1,8 @@
+AllCops:
+  TargetRubyVersion: 3.1
+
+Style/StringLiterals:
+  EnforcedStyle: double_quotes
+
+Style/StringLiteralsInInterpolation:
+  EnforcedStyle: double_quotes

data/CHANGELOG.md

@@ -0,0 +1,25 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2025-11-18
+
+### Added
+
+- **Initial Release**: First public version of the `cton` gem.
+- **CTON Encoder**: `Cton.dump` (aliased as `Cton.generate`) to convert a Ruby `Hash` into a CTON string.
+  - Encodes objects, arrays, strings, numbers, booleans, and `nil`.
+  - Automatic table detection for arrays of uniform hashes, creating a highly compact representation: `key[N]{h1,h2}=v1,v2;...`.
+  - Smart string quoting: only quotes strings containing special characters, whitespace, or those that could be misinterpreted as numbers, booleans, or null.
+  - Number normalization: canonicalizes floats and `BigDecimal` to a clean, exponent-free format. `NaN` and `Infinity` are converted to `null` for safety.
+  - Configurable separator (`\n` by default) between top-level key-value pairs.
+- **CTON Decoder**: `Cton.load` (aliased as `Cton.parse`) to parse a CTON string back into a Ruby object.
+  - Handles all CTON structures: objects `()`, arrays `[]`, and compact tables `[]{}`.
+  - Supports an optional `symbolize_names: true` argument to convert all hash keys to symbols.
+  - Robust parsing of scalars, including quoted and unquoted strings.
+- **Error Handling**: `Cton::EncodeError` for unsupported Ruby types during encoding and `Cton::ParseError` for malformed CTON input.
+- **Documentation**: `README.md` with a format overview, usage examples, and rationale.
+- **Testing**: Comprehensive RSpec suite ensuring round-trip integrity and correct handling of edge cases.

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,132 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or advances of
+  any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official email address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[INSERT CONTACT METHOD].
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or permanent
+ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2025 Davide Santangelo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,109 @@
+# CTON
+
+CTON (Compact Token-Oriented Notation) is an aggressively minified, JSON-compatible wire format that keeps prompts short without giving up schema hints. It is shape-preserving (objects, arrays, scalars, table-like arrays) and deterministic, so you can safely round-trip between Ruby hashes and compact strings that work well in LLM prompts.
+
+## Why another format?
+
+- **Less noise than YAML/JSON**: no indentation, no braces around the root, and optional quoting.
+- **Schema guardrails**: arrays carry their length (`friends[3]`) and table headers (`{id,name,...}`) so downstream parsing can verify shape.
+- **LLM-friendly**: works as a single string you can embed in a prompt together with short parsing instructions.
+- **Token savings**: CTON compounds the JSON → TOON savings; see the section below for concrete numbers.
+
+## Token savings vs JSON & TOON
+
+- **JSON → TOON**: The [TOON benchmarks](https://toonformat.dev) report roughly 40% fewer tokens than plain JSON on mixed-structure prompts while retaining accuracy due to explicit array lengths and headers.
+- **TOON → CTON**: By stripping indentation and forcing everything inline, CTON cuts another ~20–40% of characters. The sample above is ~350 characters as TOON and ~250 as CTON (~29% fewer), and larger tabular datasets show similar reductions.
+- **Net effect**: In practice you can often reclaim 50–60% of the token budget versus raw JSON, leaving more room for instructions or reasoning steps while keeping a deterministic schema.
+
+## Format at a glance
+
+```
+context(task="Our favorite hikes together",location=Boulder,season=spring_2025)
+friends[3]=ana,luis,sam
+hikes[3]{id,name,distanceKm,elevationGain,companion,wasSunny}=1,"Blue Lake Trail",7.5,320,ana,true;2,"Ridge Overlook",9.2,540,luis,false;3,"Wildflower Loop",5.1,180,sam,true
+```
+
+- Objects use parentheses and `key=value` pairs separated by commas.
+- Arrays encode their length: `[N]=...`. When every element is a flat hash with the same keys, they collapse into a compact table: `[N]{key1,key2}=row1;row2`.
+- Scalars (numbers, booleans, `null`) keep their JSON text. Strings only need quotes when they contain whitespace or reserved punctuation.
+- For parsing safety the Ruby encoder inserts a single `\n` between top-level segments. You can override this if you truly need a fully inline document (see options below).
+
+## Installation
+
+Add the gem to your application:
+
+```bash
+bundle add cton
+```
+
+Or install it directly:
+
+```bash
+gem install cton
+```
+
+## Usage
+
+```ruby
+require "cton"
+
+payload = {
+	"context" => {
+		"task" => "Our favorite hikes together",
+		"location" => "Boulder",
+		"season" => "spring_2025"
+	},
+	"friends" => %w[ana luis sam],
+	"hikes" => [
+		{ "id" => 1, "name" => "Blue Lake Trail", "distanceKm" => 7.5, "elevationGain" => 320, "companion" => "ana", "wasSunny" => true },
+		{ "id" => 2, "name" => "Ridge Overlook", "distanceKm" => 9.2, "elevationGain" => 540, "companion" => "luis", "wasSunny" => false },
+		{ "id" => 3, "name" => "Wildflower Loop", "distanceKm" => 5.1, "elevationGain" => 180, "companion" => "sam", "wasSunny" => true }
+	]
+}
+
+cton = Cton.dump(payload)
+# => "context(... )\nfriends[3]=ana,luis,sam\nhikes[3]{...}"
+
+round_tripped = Cton.load(cton)
+# => original hash
+
+# Need symbols?
+symbolized = Cton.load(cton, symbolize_names: true)
+
+# Want a truly inline document? Opt in explicitly (decoding becomes unsafe for ambiguous cases).
+inline = Cton.dump(payload, separator: "")
+```
+
+### Table detection
+
+Whenever an array is made of hashes that all expose the same scalar keys, the encoder flattens it into a table to save tokens. Mixed or nested arrays fall back to `[N]=(value1,value2,...)`.
+
+### Separators & ambiguity
+
+Removing every newline makes certain inputs ambiguous because `sam` and the next key `hikes` can merge into `samhikes`. The default `separator: "\n"` avoids that by inserting a single newline between root segments. You may pass `separator: ""` to `Cton.dump` for maximum compactness, but decoding such strings is only safe if you can guarantee extra quoting or whitespace between segments.
+
+### Literal safety & number normalization
+
+Following the TOON specification's guardrails, the encoder now:
+
+- Auto-quotes strings that would otherwise be parsed as booleans, `null`, or numbers (e.g., `"true"`, `"007"`, `"1e6"`, `"-5"`) so they round-trip as strings without extra work.
+- Canonicalizes float/BigDecimal output: no exponent notation, no trailing zeros, and `-0` collapses to `0`.
+- Converts `NaN` and `±Infinity` inputs to `null`, matching TOON's normalization guidance so downstream decoders don't explode on non-finite numbers.
+
+## Development
+
+```bash
+bin/setup   # install dependencies
+bundle exec rspec
+bin/console # interactive playground
+```
+
+To release a new version, bump `Cton::VERSION` and run `bundle exec rake release`.
+
+## Contributing
+
+Bug reports and pull requests are welcome at https://github.com/davidesantangelo/cton. Please follow the [Code of Conduct](CODE_OF_CONDUCT.md).
+
+## License
+
+MIT © Davide Santangelo

data/Rakefile

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "rubocop/rake_task"
+
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]

data/lib/cton/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module Cton
+  VERSION = "0.1.0"
+end

data/lib/cton.rb

@@ -0,0 +1,581 @@
+# frozen_string_literal: true
+
+require "bigdecimal"
+require_relative "cton/version"
+
+module Cton
+  class Error < StandardError; end
+  class EncodeError < Error; end
+  class ParseError < Error; end
+
+  module_function
+
+  def dump(payload, options = {})
+    separator = options.fetch(:separator, "\n")
+    Encoder.new(separator: separator).encode(payload)
+  end
+  alias generate dump
+
+  def load(cton_string, symbolize_names: false)
+    Decoder.new(symbolize_names: symbolize_names).decode(cton_string)
+  end
+  alias parse load
+
+  class Encoder
+    SAFE_TOKEN = /\A[0-9A-Za-z_.:-]+\z/.freeze
+    NUMERIC_TOKEN = /\A-?(?:\d+)(?:\.\d+)?(?:[eE][+-]?\d+)?\z/.freeze
+    RESERVED_LITERALS = %w[true false null].freeze
+
+    def initialize(separator: "\n")
+      @separator = separator || ""
+    end
+
+    def encode(payload)
+      encode_root(payload)
+    end
+
+    private
+
+    attr_reader :separator
+
+    def encode_root(value)
+      case value
+      when Hash
+        value.map { |key, nested| encode_top_level_pair(key, nested) }.join(separator)
+      else
+        encode_value(value, context: :standalone)
+      end
+    end
+
+    def encode_top_level_pair(key, value)
+      "#{format_key(key)}#{encode_value(value, context: :top_pair)}"
+    end
+
+    def encode_value(value, context:)
+      case value
+      when Hash
+        encode_object(value)
+      when Array
+        encode_array(value)
+      else
+        prefix = context == :top_pair ? "=" : ""
+        "#{prefix}#{encode_scalar(value)}"
+      end
+    end
+
+    def encode_object(hash)
+      return "()" if hash.empty?
+
+      pairs = hash.map do |key, value|
+        "#{format_key(key)}=#{encode_value(value, context: :object)}"
+      end
+      "(#{pairs.join(',')})"
+    end
+
+    def encode_array(list)
+      length = list.length
+      return "[0]=" if length.zero?
+
+      if table_candidate?(list)
+        "[#{length}]#{encode_table(list)}"
+      else
+        body = if list.all? { |value| scalar?(value) }
+                 list.map { |value| encode_scalar(value) }.join(",")
+               else
+                 list.map { |value| encode_array_element(value) }.join(",")
+               end
+        "[#{length}]=#{body}"
+      end
+    end
+
+    def encode_table(rows)
+      header = rows.first.keys
+      header_token = "{#{header.map { |key| format_key(key) }.join(',')}}"
+      table_rows = rows.map do |row|
+        header.map { |field| encode_scalar(row.fetch(field)) }.join(",")
+      end
+      "#{header_token}=#{table_rows.join(';')}"
+    end
+
+    def encode_array_element(value)
+      encode_value(value, context: :array)
+    end
+
+    def encode_scalar(value)
+      case value
+      when String
+        encode_string(value)
+      when TrueClass, FalseClass
+        value ? "true" : "false"
+      when NilClass
+        "null"
+      when Numeric
+        format_number(value)
+      else
+        raise EncodeError, "Unsupported value: #{value.class}"
+      end
+    end
+
+    def encode_string(value)
+      return '""' if value.empty?
+
+      string_needs_quotes?(value) ? quote_string(value) : value
+    end
+
+    FLOAT_DECIMAL_PRECISION = Float::DIG
+
+    def format_number(value)
+      case value
+      when Float
+        return "null" if value.nan? || value.infinite?
+
+        normalize_decimal_string(float_decimal_string(value))
+      when Integer
+        value.to_s
+      else
+        if defined?(BigDecimal) && value.is_a?(BigDecimal)
+          normalize_decimal_string(value.to_s("F"))
+        else
+          value.to_s
+        end
+      end
+    end
+
+    def normalize_decimal_string(string)
+      stripped = string.start_with?("+") ? string[1..-1] : string
+      return "0" if zero_string?(stripped)
+
+      if stripped.include?(".")
+        stripped = stripped.sub(/0+\z/, "")
+        stripped = stripped.sub(/\.\z/, "")
+      end
+
+      stripped
+    end
+
+    def zero_string?(string)
+      string.match?(/\A-?0+(?:\.0+)?\z/)
+    end
+
+    def float_decimal_string(value)
+      if defined?(BigDecimal)
+        BigDecimal(value.to_s).to_s("F")
+      else
+        Kernel.format("%.#{FLOAT_DECIMAL_PRECISION}f", value)
+      end
+    end
+
+    def format_key(key)
+      key_string = key.to_s
+      unless SAFE_TOKEN.match?(key_string)
+        raise EncodeError, "Invalid key: #{key_string.inspect}"
+      end
+      key_string
+    end
+
+    def string_needs_quotes?(value)
+      return true unless SAFE_TOKEN.match?(value)
+      RESERVED_LITERALS.include?(value) || numeric_like?(value)
+    end
+
+    def numeric_like?(value)
+      NUMERIC_TOKEN.match?(value)
+    end
+
+    def quote_string(value)
+      "\"#{escape_string(value)}\""
+    end
+
+    def escape_string(value)
+      value.gsub(/["\\\n\r\t]/) do |char|
+        case char
+        when "\n" then "\\n"
+        when "\r" then "\\r"
+        when "\t" then "\\t"
+        else
+          "\\#{char}"
+        end
+      end
+    end
+
+    def scalar?(value)
+      value.is_a?(String) || value.is_a?(Numeric) || value == true || value == false || value.nil?
+    end
+
+    def table_candidate?(rows)
+      return false if rows.empty?
+
+      first = rows.first
+      return false unless first.is_a?(Hash) && !first.empty?
+
+      keys = first.keys
+      rows.all? do |row|
+        row.is_a?(Hash) && row.keys == keys && row.values.all? { |val| scalar?(val) }
+      end
+    end
+  end
+
+  class Decoder
+    TERMINATORS = [",", ";", ")", "]", "}"].freeze
+
+    def initialize(symbolize_names: false)
+      @symbolize_names = symbolize_names
+    end
+
+    def decode(cton)
+      @source = cton.to_s
+      @index = 0
+      skip_ws
+
+      value = if key_ahead?(@index)
+                parse_document
+              else
+                parse_value(allow_key_boundary: true)
+              end
+
+      skip_ws
+      raise ParseError, "Unexpected trailing data" unless eof?
+
+      value
+    end
+
+    private
+
+    attr_reader :symbolize_names
+
+    def parse_document
+      result = {}
+      until eof?
+        key = parse_key_name
+        value = parse_value_for_key
+        assign_pair(result, key, value)
+        skip_ws
+      end
+      result
+    end
+
+    def parse_value_for_key
+      skip_ws
+      char = current_char
+      case char
+      when "("
+        parse_object
+      when "["
+        parse_array
+      when "="
+        advance
+        parse_scalar(allow_key_boundary: true)
+      else
+        raise ParseError, "Unexpected token #{char.inspect} while reading value"
+      end
+    end
+
+    def parse_object
+      expect!("(")
+      skip_ws
+      if current_char == ")"
+        expect!(")")
+        return {}
+      end
+
+      pairs = {}
+      loop do
+        key = parse_key_name
+        expect!("=")
+        value = parse_value
+        assign_pair(pairs, key, value)
+        skip_ws
+        break if current_char == ")"
+        expect!(",")
+        skip_ws
+      end
+      expect!(")")
+      pairs
+    end
+
+    def parse_array
+      expect!("[")
+      length = parse_integer_literal
+      expect!("]")
+      skip_ws
+
+      header = parse_header if current_char == "{"
+
+      expect!("=")
+      return [] if length.zero?
+
+      header ? parse_table_rows(length, header) : parse_array_elements(length)
+    end
+
+    def parse_header
+      expect!("{")
+      fields = []
+      loop do
+        fields << parse_key_name
+        break if current_char == "}"
+        expect!(",")
+      end
+      expect!("}")
+      fields
+    end
+
+    def parse_table_rows(length, header)
+      rows = []
+      length.times do |row_index|
+        row = {}
+        header.each_with_index do |field, column_index|
+          allow_boundary = row_index == length - 1 && column_index == header.length - 1
+          row[field] = parse_scalar(allow_key_boundary: allow_boundary)
+          expect!(",") if column_index < header.length - 1
+        end
+        rows << symbolize_keys(row)
+        expect!(";") if row_index < length - 1
+      end
+      rows
+    end
+
+    def parse_array_elements(length)
+      values = []
+      length.times do |index|
+        allow_boundary = index == length - 1
+        values << parse_value(allow_key_boundary: allow_boundary)
+        expect!(",") if index < length - 1
+      end
+      values
+    end
+
+    def parse_value(allow_key_boundary: false)
+      skip_ws
+      char = current_char
+      raise ParseError, "Unexpected end of input" if char.nil?
+
+      case char
+      when "("
+        parse_object
+      when "["
+        parse_array
+      when '"'
+        parse_string
+      else
+        parse_scalar(allow_key_boundary: allow_key_boundary)
+      end
+    end
+
+    def parse_scalar(terminators: TERMINATORS, allow_key_boundary: false)
+      skip_ws
+      return parse_string if current_char == '"'
+
+      start = @index
+      limit_index = allow_key_boundary ? next_key_index(@index) : nil
+      exit_reason = nil
+
+      while !eof?
+        if limit_index && @index >= limit_index
+          exit_reason = :boundary
+          break
+        end
+
+        char = current_char
+
+        if char.nil?
+          exit_reason = :eof
+          break
+        elsif terminators.include?(char)
+          exit_reason = :terminator
+          break
+        elsif whitespace?(char)
+          exit_reason = :whitespace
+          break
+        elsif "()[]{}".include?(char)
+          exit_reason = :structure
+          break
+        end
+
+        @index += 1
+      end
+
+      token = if exit_reason == :boundary && limit_index
+                @source[start...limit_index]
+              else
+                @source[start...@index]
+              end
+
+      raise ParseError, "Empty value" if token.nil? || token.empty?
+
+      convert_scalar(token)
+    end
+
+    def convert_scalar(token)
+      case token
+      when "true" then true
+      when "false" then false
+      when "null" then nil
+      else
+        if integer?(token)
+          token.to_i
+        elsif float?(token)
+          token.to_f
+        else
+          token
+        end
+      end
+    end
+
+    def parse_string
+      expect!("\"")
+      buffer = +""
+      while !eof?
+        char = current_char
+        raise ParseError, "Unterminated string" if char.nil?
+
+        if char == '\\'
+          @index += 1
+          escaped = current_char
+          raise ParseError, "Invalid escape sequence" if escaped.nil?
+          buffer << case escaped
+                    when 'n' then "\n"
+                    when 'r' then "\r"
+                    when 't' then "\t"
+                    when '"', '\\' then escaped
+                    else
+                      raise ParseError, "Unsupported escape sequence"
+                    end
+        elsif char == '"'
+          break
+        else
+          buffer << char
+        end
+        @index += 1
+      end
+      expect!("\"")
+      buffer
+    end
+
+    def parse_key_name
+      skip_ws
+      start = @index
+      while !eof? && safe_key_char?(current_char)
+        @index += 1
+      end
+      token = @source[start...@index]
+      raise ParseError, "Invalid key" if token.nil? || token.empty?
+      symbolize_names ? token.to_sym : token
+    end
+
+    def parse_integer_literal
+      start = @index
+      while !eof? && current_char =~ /\d/
+        @index += 1
+      end
+      token = @source[start...@index]
+      raise ParseError, "Expected digits" if token.nil? || token.empty?
+      Integer(token, 10)
+    rescue ArgumentError
+      raise ParseError, "Invalid length literal"
+    end
+
+    def assign_pair(hash, key, value)
+      hash[key] = value
+    end
+
+    def symbolize_keys(row)
+      symbolize_names ? row.transform_keys(&:to_sym) : row
+    end
+
+    def expect!(char)
+      skip_ws
+      actual = current_char
+      raise ParseError, "Expected #{char.inspect}, got #{actual.inspect}" unless actual == char
+      @index += 1
+    end
+
+    def skip_ws
+      @index += 1 while !eof? && whitespace?(current_char)
+    end
+
+    def whitespace?(char)
+      char == " " || char == "\t" || char == "\n" || char == "\r"
+    end
+
+    def eof?
+      @index >= @source.length
+    end
+
+    def current_char
+      @source[@index]
+    end
+
+    def advance
+      @index += 1
+    end
+
+    def key_ahead?(offset)
+      idx = offset
+      idx += 1 while idx < @source.length && whitespace?(@source[idx])
+      start = idx
+      while idx < @source.length && safe_key_char?(@source[idx])
+        idx += 1
+      end
+      return false if idx == start
+      next_char = @source[idx]
+      ["(", "[", "="].include?(next_char)
+    end
+
+    def safe_key_char?(char)
+      !char.nil? && char.match?(/[0-9A-Za-z_.:-]/)
+    end
+
+    def integer?(token)
+      token.match?(/\A-?(?:0|[1-9]\d*)\z/)
+    end
+
+    def float?(token)
+      token.match?(/\A-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?\z/)
+    end
+
+    def next_key_index(from_index)
+      idx = from_index
+      in_string = false
+
+      while idx < @source.length
+        char = @source[idx]
+
+        if in_string
+          if char == '\\'
+            idx += 2
+            next
+          elsif char == '"'
+            in_string = false
+            idx += 1
+            next
+          else
+            idx += 1
+            next
+          end
+        else
+          case char
+          when '"'
+            in_string = true
+            idx += 1
+            next
+          else
+            if safe_key_char?(char)
+              start = idx
+              idx += 1 while idx < @source.length && safe_key_char?(@source[idx])
+              next_char = @source[idx]
+              if start > from_index && ["(", "[", "="].include?(next_char)
+                return start
+              end
+              idx = start + 1
+              next
+            end
+            idx += 1
+          end
+        end
+      end
+
+      nil
+    end
+  end
+end
+

data/sig/cton.rbs

@@ -0,0 +1,11 @@
+module Cton
+  VERSION: String
+  class Error < ::StandardError; end
+  class EncodeError < Error; end
+  class ParseError < Error; end
+
+  def self.dump: (untyped, ?Hash[Symbol, untyped]) -> String
+  def self.generate: (untyped, ?Hash[Symbol, untyped]) -> String
+  def self.load: (String, ?symbolize_names: bool) -> untyped
+  def self.parse: (String, ?symbolize_names: bool) -> untyped
+end

metadata

@@ -0,0 +1,57 @@
+--- !ruby/object:Gem::Specification
+name: cton
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Davide Santangelo
+autorequire:
+bindir: exe
+cert_chain: []
+date: 2025-11-18 00:00:00.000000000 Z
+dependencies: []
+description: CTON provides a JSON-compatible, token-efficient text representation
+  optimized for LLM prompts.
+email:
+- davide.santangelo@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".rspec"
+- ".rubocop.yml"
+- CHANGELOG.md
+- CODE_OF_CONDUCT.md
+- LICENSE.txt
+- README.md
+- Rakefile
+- lib/cton.rb
+- lib/cton/version.rb
+- sig/cton.rbs
+homepage: https://github.com/davidesantangelo/cton
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/davidesantangelo/cton
+  source_code_uri: https://github.com/davidesantangelo/cton
+  changelog_uri: https://github.com/davidesantangelo/cton/blob/master/CHANGELOG.md
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.1.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.3.26
+signing_key:
+specification_version: 4
+summary: Compact Token-Oriented Notation encoder/decoder.
+test_files: []