cton 0.1.1 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '0295922a011dd898278f9f57de2f48f2acbe0ce3363f263f4e2b2753993ebdea'
-  data.tar.gz: 33bb13ee23584ff6cd51bf6bc6c1a869d02f206ff0792a172c9aa7cdc5547977
+  metadata.gz: 1c9161ae830ba6b01d3ec94d1170fc4295aaacfa839c869aaa6adefe2711cc2d
+  data.tar.gz: 80c4ba30abbf8a562bde581f26e7dc5529aa46275b61930fe28370f34156db61
 SHA512:
-  metadata.gz: 9dff47df67680eabf6fb7ac05dac606e969df0a0d31d575318fa4a72c51c8fe85d38b671f4e2b8b8caa0e969184c044e06547b135b9a5b5b0baa4c3e28232322
-  data.tar.gz: be363392d2305b6940e46060310a908922bbf822ae487e9d4b20441e4cc51e5e8161332cd5d8c0846743663d1251ff7b1b5480f21f36f9b52c0aafb0404f4b74
+  metadata.gz: 914196284081bacd5b7f5f6ac9a1b246ea8924eddbb26cd28796b12a2ee2156718a9ff3b795b86188d483da4fc294b002a93e935815f692b432dac78b5304dcf
+  data.tar.gz: d9b1bfb1f7de402fe9de0d7da90f750d490dbdcb4e572e9283bc3ed15b1b43e3f3f9b44a4031f80ecf5d610a73d64719f3c78f170060de78035e04dab3a9d663

data/CHANGELOG.md

@@ -5,6 +5,39 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.0] - 2025-11-20
+
+### Added
+
+- **Performance tunables**: `Cton.dump` now accepts `decimal_mode: :fast | :precise`, allowing callers to trade float-format determinism for lower allocation pressure. Specs cover both modes.
+- **Benchmark harness**: New `bench/encode_decode_bench.rb` script (wired into the README/Development docs) exercises encode/decode hot paths and prints comparative JSON vs CTON timings. On Ruby 3.1.4/macOS the fast encoder completes 1,000 iterations in ~0.63s and the new inline decoder stress test wraps 400 concatenated documents in ~4.14s.
+- **Regression tests**: Added specs for streaming documents without separators plus validation around the new decimal mode toggle.
+
+### Changed
+
+- **Encoder**: Memoizes table schemas per array instance, adds a fast-path for homogeneous scalar lists, and reduces float/BigDecimal copying by favoring Ruby's native float formatting before falling back to `BigDecimal`. Unsupported `decimal_mode` values now raise immediately.
+- **Decoder**: Replaces high-allocation `StringScanner` tokenization with raw string slicing, improves key-boundary detection for inline payloads, and keeps symbolization logic untouched. Boundary heuristics now prefer alphabetic key starts to avoid splitting numeric payloads.
+- **Documentation**: README now calls out the tuning flags, inline caveats, and benchmark instructions; Development workflow highlights how to rerun the perf suite.
+
+### Fixed
+
+- **Inline parsing**: Eliminated the runaway allocations and incorrect key splits when processing long documents with `separator: ""`.
+- **Float normalization**: Restored canonical `9.2`-style output in fast mode while keeping the new perf optimizations.
+
+## [0.2.0] - 2025-11-19
+
+### Added
+
+- **CLI Tool**: New `bin/cton` executable for converting between JSON and CTON from the command line. Supports auto-detection, pretty printing, and file I/O.
+- **Streaming IO**: `Cton.dump` now accepts an `IO` object as the second argument (or via `io:` keyword), allowing direct writing to files or sockets without intermediate string allocation.
+- **Pretty Printing**: Added `pretty: true` option to `Cton.dump` to format output with indentation and newlines for better readability.
+- **Extended Types**: Native support for `Time`, `Date` (ISO8601), `Set` (as Array), and `OpenStruct` (as Object).
+- **Enhanced Error Reporting**: `ParseError` now includes line and column numbers to help locate syntax errors in large documents.
+
+### Changed
+
+- **Ruby 3 Compatibility**: Improved argument handling in `Cton.dump` to robustly support Ruby 3 keyword arguments when passing hashes.
+
 ## [0.1.1] - 2025-11-18
 
 ### Changed

data/README.md

@@ -1,32 +1,115 @@
 # CTON
 
-CTON (Compact Token-Oriented Notation) is an aggressively minified, JSON-compatible wire format that keeps prompts short without giving up schema hints. It is shape-preserving (objects, arrays, scalars, table-like arrays) and deterministic, so you can safely round-trip between Ruby hashes and compact strings that work well in LLM prompts.
+[![Gem Version](https://badge.fury.io/rb/cton.svg)](https://badge.fury.io/rb/cton)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/davidesantangelo/cton/blob/master/LICENSE.txt)
+
+**CTON** (Compact Token-Oriented Notation) is an aggressively minified, JSON-compatible wire format that keeps prompts short without giving up schema hints. It is shape-preserving (objects, arrays, scalars, table-like arrays) and deterministic, so you can safely round-trip between Ruby hashes and compact strings that work well in LLM prompts.
+
+---
+
+## 📖 Table of Contents
+
+- [What is CTON?](#what-is-cton)
+- [Why another format?](#why-another-format)
+- [Examples](#examples)
+- [Token Savings](#token-savings-vs-json--toon)
+- [Installation](#installation)
+- [Usage](#usage)
+- [Performance & Benchmarks](#performance--benchmarks)
+- [Teaching CTON to LLMs](#teaching-cton-to-llms)
+- [Development](#development)
+- [Contributing](#contributing)
+- [License](#license)
+
+---
+
+## What is CTON?
+
+CTON is designed to be the most efficient way to represent structured data for Large Language Models (LLMs). It strips away the "syntactic sugar" of JSON that humans like (indentation, excessive quoting, braces) but machines don't strictly need, while adding "structural hints" that help LLMs generate valid output.
+
+### Key Concepts
+
+1.  **Root is Implicit**: No curly braces `{}` wrapping the entire document.
+2.  **Minimal Punctuation**:
+    *   Objects use `key=value`.
+    *   Nested objects use parentheses `(key=value)`.
+    *   Arrays use brackets with length `[N]=item1,item2`.
+3.  **Table Compression**: If an array contains objects with the same keys, CTON automatically converts it into a table format `[N]{header1,header2}=val1,val2;val3,val4`. This is a massive token saver for datasets.
+
+---
+
+## Examples
+
+### Simple Key-Value Pairs
+
+**JSON**
+```json
+{
+  "task": "planning",
+  "urgent": true,
+  "id": 123
+}
+```
+
+**CTON**
+```text
+task=planning,urgent=true,id=123
+```
+
+### Nested Objects
+
+**JSON**
+```json
+{
+  "user": {
+    "name": "Davide",
+    "settings": {
+      "theme": "dark"
+    }
+  }
+}
+```
+
+**CTON**
+```text
+user(name=Davide,settings(theme=dark))
+```
+
+### Arrays and Tables
+
+**JSON**
+```json
+{
+  "tags": ["ruby", "gem", "llm"],
+  "files": [
+    { "name": "README.md", "size": 1024 },
+    { "name": "lib/cton.rb", "size": 2048 }
+  ]
+}
+```
+
+**CTON**
+```text
+tags[3]=ruby,gem,llm
+files[2]{name,size}=README.md,1024;lib/cton.rb,2048
+```
+
+---
 
 ## Why another format?
 
 - **Less noise than YAML/JSON**: no indentation, no braces around the root, and optional quoting.
 - **Schema guardrails**: arrays carry their length (`friends[3]`) and table headers (`{id,name,...}`) so downstream parsing can verify shape.
 - **LLM-friendly**: works as a single string you can embed in a prompt together with short parsing instructions.
-- **Token savings**: CTON compounds the JSON → TOON savings; see the section below for concrete numbers.
+- **Token savings**: CTON compounds the JSON → TOON savings.
 
-## Token savings vs JSON & TOON
+### Token savings vs JSON & TOON
 
 - **JSON → TOON**: The [TOON benchmarks](https://toonformat.dev) report roughly 40% fewer tokens than plain JSON on mixed-structure prompts while retaining accuracy due to explicit array lengths and headers.
-- **TOON → CTON**: By stripping indentation and forcing everything inline, CTON cuts another ~20–40% of characters. The sample above is ~350 characters as TOON and ~250 as CTON (~29% fewer), and larger tabular datasets show similar reductions.
-- **Net effect**: In practice you can often reclaim 50–60% of the token budget versus raw JSON, leaving more room for instructions or reasoning steps while keeping a deterministic schema.
+- **TOON → CTON**: By stripping indentation and forcing everything inline, CTON cuts another ~20–40% of characters.
+- **Net effect**: In practice you can often reclaim **50–60% of the token budget** versus raw JSON, leaving more room for instructions or reasoning steps while keeping a deterministic schema.
 
-## Format at a glance
-
-```
-context(task="Our favorite hikes together",location=Boulder,season=spring_2025)
-friends[3]=ana,luis,sam
-hikes[3]{id,name,distanceKm,elevationGain,companion,wasSunny}=1,"Blue Lake Trail",7.5,320,ana,true;2,"Ridge Overlook",9.2,540,luis,false;3,"Wildflower Loop",5.1,180,sam,true
-```
-
-- Objects use parentheses and `key=value` pairs separated by commas.
-- Arrays encode their length: `[N]=...`. When every element is a flat hash with the same keys, they collapse into a compact table: `[N]{key1,key2}=row1;row2`.
-- Scalars (numbers, booleans, `null`) keep their JSON text. Strings only need quotes when they contain whitespace or reserved punctuation.
-- For parsing safety the Ruby encoder inserts a single `\n` between top-level segments. You can override this if you truly need a fully inline document (see options below).
+---
 
 ## Installation
 
@@ -42,28 +125,32 @@ Or install it directly:
 gem install cton
 ```
 
+---
+
 ## Usage
 
 ```ruby
 require "cton"
 
 payload = {
-	"context" => {
-		"task" => "Our favorite hikes together",
-		"location" => "Boulder",
-		"season" => "spring_2025"
-	},
-	"friends" => %w[ana luis sam],
-	"hikes" => [
-		{ "id" => 1, "name" => "Blue Lake Trail", "distanceKm" => 7.5, "elevationGain" => 320, "companion" => "ana", "wasSunny" => true },
-		{ "id" => 2, "name" => "Ridge Overlook", "distanceKm" => 9.2, "elevationGain" => 540, "companion" => "luis", "wasSunny" => false },
-		{ "id" => 3, "name" => "Wildflower Loop", "distanceKm" => 5.1, "elevationGain" => 180, "companion" => "sam", "wasSunny" => true }
-	]
+  "context" => {
+    "task" => "Our favorite hikes together",
+    "location" => "Boulder",
+    "season" => "spring_2025"
+  },
+  "friends" => %w[ana luis sam],
+  "hikes" => [
+    { "id" => 1, "name" => "Blue Lake Trail", "distanceKm" => 7.5, "elevationGain" => 320, "companion" => "ana", "wasSunny" => true },
+    { "id" => 2, "name" => "Ridge Overlook", "distanceKm" => 9.2, "elevationGain" => 540, "companion" => "luis", "wasSunny" => false },
+    { "id" => 3, "name" => "Wildflower Loop", "distanceKm" => 5.1, "elevationGain" => 180, "companion" => "sam", "wasSunny" => true }
+  ]
 }
 
+# Encode to CTON
 cton = Cton.dump(payload)
 # => "context(... )\nfriends[3]=ana,luis,sam\nhikes[3]{...}"
 
+# Decode back to Hash
 round_tripped = Cton.load(cton)
 # => original hash
 
@@ -72,24 +159,159 @@ symbolized = Cton.load(cton, symbolize_names: true)
 
 # Want a truly inline document? Opt in explicitly (decoding becomes unsafe for ambiguous cases).
 inline = Cton.dump(payload, separator: "")
+
+# Pretty print for human readability
+pretty = Cton.dump(payload, pretty: true)
+
+# Stream to an IO object (file, socket, etc.)
+File.open("data.cton", "w") do |f|
+  Cton.dump(payload, f)
+end
+
+# Toggle float normalization strategies
+fast  = Cton.dump(payload) # default :fast mode
+strict = Cton.dump(payload, decimal_mode: :precise)
 ```
 
-### Table detection
+### CLI Tool
 
-Whenever an array is made of hashes that all expose the same scalar keys, the encoder flattens it into a table to save tokens. Mixed or nested arrays fall back to `[N]=(value1,value2,...)`.
+CTON comes with a command-line tool for quick conversions:
 
-### Separators & ambiguity
+```bash
+# Convert JSON to CTON
+echo '{"hello": "world"}' | cton
+# => hello=world
 
-Removing every newline makes certain inputs ambiguous because `sam` and the next key `hikes` can merge into `samhikes`. The default `separator: "\n"` avoids that by inserting a single newline between root segments. You may pass `separator: ""` to `Cton.dump` for maximum compactness, but decoding such strings is only safe if you can guarantee extra quoting or whitespace between segments.
+# Convert CTON to JSON
+echo 'hello=world' | cton --to-json
+# => {"hello":"world"}
 
-### Literal safety & number normalization
+# Pretty print
+cton --pretty input.json
+```
 
-Following the TOON specification's guardrails, the encoder now:
+### Advanced Features
+
+#### Extended Types
+CTON natively supports serialization for:
+- `Time` and `Date` (ISO8601 strings)
+- `Set` (converted to Arrays)
+- `OpenStruct` (converted to Objects)
+
+#### Table detection
+Whenever an array is made of hashes that all expose the same scalar keys, the encoder flattens it into a table to save tokens. Mixed or nested arrays fall back to `[N]=(value1,value2,...)`.
+
+#### Separators & ambiguity
+Removing every newline makes certain inputs ambiguous because `sam` and the next key `hikes` can merge into `samhikes`. The default `separator: "\n"` avoids that by inserting a single newline between root segments. You may pass `separator: ""` to `Cton.dump` for maximum compactness, but decoding such strings is only safe if you can guarantee extra quoting or whitespace between segments. When you intentionally omit separators, keep next-level keys alphabetic (e.g., `payload`, `k42`) so the decoder's boundary heuristic can split `...1payload...` without misclassifying numeric prefixes.
 
+#### Literal safety & number normalization
+Following the TOON specification's guardrails, the encoder now:
 - Auto-quotes strings that would otherwise be parsed as booleans, `null`, or numbers (e.g., `"true"`, `"007"`, `"1e6"`, `"-5"`) so they round-trip as strings without extra work.
 - Canonicalizes float/BigDecimal output: no exponent notation, no trailing zeros, and `-0` collapses to `0`.
 - Converts `NaN` and `±Infinity` inputs to `null`, matching TOON's normalization guidance so downstream decoders don't explode on non-finite numbers.
 
+#### Decimal normalization modes
+- `decimal_mode: :fast` (default) prefers Ruby's native float representation and only falls back to `BigDecimal` when scientific notation is detected, minimizing allocations on tight loops.
+- `decimal_mode: :precise` forces the legacy `BigDecimal` path for every float, which is slower but useful for audit-grade dumps where you want deterministic decimal expansion.
+- Both modes share the same trailing-zero stripping and `-0 → 0` normalization, so switching modes never affects integer formatting.
+
+---
+
+## Performance & Benchmarks
+
+CTON focuses on throughput: encoder table schemas are memoized, scalar list encoding keeps a reusable buffer, floats avoid `BigDecimal` when they can, and the decoder slices straight from the raw string to sidestep `StringScanner` allocations. You can reproduce the numbers below with the bundled script:
+
+```bash
+bundle exec ruby bench/encode_decode_bench.rb
+# customize input size / iterations
+ITERATIONS=2000 STREAM_SIZE=400 bundle exec ruby bench/encode_decode_bench.rb
+```
+
+Latest results on Ruby 3.1.4/macOS (M-series), 1,000 iterations, `STREAM_SIZE=200`:
+
+| Benchmark | Time (s) |
+| --- | --- |
+| `cton dump` (:fast) | 0.626 |
+| `cton dump` (:precise) | 0.658 |
+| `json generate` | 0.027 |
+| `cton load` | 2.067 |
+| `json parse` | 0.045 |
+| `cton inline load` (separator=`""`, double payload) | 4.140 |
+
+`cton inline load` deliberately concatenates documents without separators to stress the new boundary detector; it now finishes without the runaway allocations seen in earlier releases.
+
+---
+
+## Teaching CTON to LLMs
+
+Use this system prompt to teach an LLM how to understand and generate CTON:
+
+````markdown
+You are an expert in data serialization and specifically in CTON (Compact Token-Oriented Notation). CTON is a token-efficient data format optimized for LLMs that serves as a compact alternative to JSON.
+
+Your task is to interpret CTON input and convert it to JSON, or convert JSON input into valid CTON format, following the specification below.
+
+### CTON Specification
+
+CTON minimizes syntax characters (braces, quotes) while preserving structure and type safety.
+
+**1. Basic Structure (Key-Value)**
+- **Rule:** Do not use outer curly braces `{}` for the root object.
+- **Rule:** Use `=` to separate keys and values.
+- **Rule:** Use `,` to separate fields.
+- **Rule:** Do not use quotes around "safe" strings (alphanumeric, simple text).
+- **Example:** - JSON: `{"task": "planning", "urgent": true}`
+  - CTON: `task=planning,urgent=true`
+
+**2. Nested Objects**
+- **Rule:** Use parentheses `()` to denote a nested object instead of `{}`.
+- **Example:**
+  - JSON: `{"context": {"user": "Davide", "theme": "dark"}}`
+  - CTON: `context(user=Davide,theme=dark)`
+
+**3. Arrays of Objects (Table Compression)**
+- **Rule:** Use the syntax `key[count]{columns}=values` for arrays of objects to avoid repeating keys.
+- **Structure:** `key[Length]{col1,col2}=val1,val2;val1,val2`
+- **Details:** - `[N]` denotes the number of items in the array.
+  - `{col1,col2}` defines the schema headers.
+  - `;` separates distinct objects (rows).
+  - `,` separates values within an object.
+- **Example:**
+
+JSON:
+```json
+{
+  "files": [
+    { "name": "README.md", "size": 1024 },
+    { "name": "lib.rb", "size": 2048 }
+  ]
+}
+```
+
+CTON: `files[2]{name,size}=README.md,1024;lib.rb,2048`
+
+**4. Type Safety & Literals**
+- **Booleans/Null:** `true`, `false`, and `null` are preserved as literals (unquoted).
+- **Numbers:** Integers and floats are written as is (e.g., `1024`, `3.14`).
+- **Escaping:** If a string value looks like a boolean, number, or contains reserved characters (like `,`, `;`, `=`, `(`, `)`), it must be wrapped in double quotes (e.g., `"true"`).
+
+### Examples for Training
+
+**Input (JSON):**
+```json
+{
+  "id": 123,
+  "active": true,
+  "metadata": {
+    "created_at": "2023-01-01",
+    "tags": "admin"
+  }
+}
+```
+````
+
+---
+
 ## Type Safety
 
 CTON ships with RBS signatures (`sig/cton.rbs`) to support type checking and IDE autocompletion.
@@ -100,6 +322,7 @@ CTON ships with RBS signatures (`sig/cton.rbs`) to support type checking and IDE
 bin/setup        # install dependencies
 bundle exec rake # run tests and rubocop
 bin/console      # interactive playground
+bundle exec ruby bench/encode_decode_bench.rb # performance smoke test
 ```
 
 To release a new version, bump `Cton::VERSION` and run `bundle exec rake release`.
@@ -110,4 +333,4 @@ Bug reports and pull requests are welcome at https://github.com/davidesantangelo
 
 ## License
 
-MIT © Davide Santangelo
+MIT © [Davide Santangelo](https://github.com/davidesantangelo)

data/bench/encode_decode_bench.rb

@@ -0,0 +1,65 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "benchmark"
+require "json"
+require_relative "../lib/cton"
+
+ITERATIONS = Integer(ENV.fetch("ITERATIONS", 1_000))
+STREAM_SIZE = Integer(ENV.fetch("STREAM_SIZE", 200))
+
+sample_payload = {
+  "context" => {
+    "task" => "Our favorite hikes together",
+    "location" => "Boulder",
+    "season" => "spring_2025"
+  },
+  "friends" => %w[ana luis sam],
+  "hikes" => Array.new(STREAM_SIZE) do |idx|
+    {
+      "id" => idx + 1,
+      "name" => "Trail ##{idx + 1}",
+      "distanceKm" => (6.0 + ((idx % 5) * 0.5)),
+      "elevationGain" => 250 + ((idx % 3) * 50),
+      "companion" => %w[ana luis sam][idx % 3],
+      "wasSunny" => idx.even?
+    }
+  end
+}
+
+warm_cton = Cton.dump(sample_payload)
+warm_json = JSON.generate(sample_payload)
+
+puts "\nEncoding benchmarks (iterations=#{ITERATIONS}, stream_size=#{STREAM_SIZE})"
+Benchmark.bm(25) do |bm|
+  bm.report("cton dump fast") do
+    ITERATIONS.times { Cton.dump(sample_payload) }
+  end
+
+  bm.report("cton dump precise") do
+    ITERATIONS.times { Cton.dump(sample_payload, decimal_mode: :precise) }
+  end
+
+  bm.report("json generate") do
+    ITERATIONS.times { JSON.generate(sample_payload) }
+  end
+end
+
+puts "\nDecoding benchmarks"
+Benchmark.bm(25) do |bm|
+  bm.report("cton load") do
+    ITERATIONS.times { Cton.load(warm_cton) }
+  end
+
+  bm.report("json parse") do
+    ITERATIONS.times { JSON.parse(warm_json) }
+  end
+end
+
+puts "\nStreaming decode stress (#{STREAM_SIZE * 2} documents, separator=\"\")"
+inline_blob = warm_cton.delete("\n") * 2
+Benchmark.bm(25) do |bm|
+  bm.report("cton inline load") do
+    ITERATIONS.times { Cton.load(inline_blob) }
+  end
+end