cton 0.3.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1c9161ae830ba6b01d3ec94d1170fc4295aaacfa839c869aaa6adefe2711cc2d
-  data.tar.gz: 80c4ba30abbf8a562bde581f26e7dc5529aa46275b61930fe28370f34156db61
+  metadata.gz: a1d32c4a3d5f55726071135d05705be7084d7c8332a1f03a5a724523981a7b81
+  data.tar.gz: '0813fb27119f36ab0117e7ca1401fe33d8826bc7bfbe09b68589dece8ae2f376'
 SHA512:
-  metadata.gz: 914196284081bacd5b7f5f6ac9a1b246ea8924eddbb26cd28796b12a2ee2156718a9ff3b795b86188d483da4fc294b002a93e935815f692b432dac78b5304dcf
-  data.tar.gz: d9b1bfb1f7de402fe9de0d7da90f750d490dbdcb4e572e9283bc3ed15b1b43e3f3f9b44a4031f80ecf5d610a73d64719f3c78f170060de78035e04dab3a9d663
+  metadata.gz: e536052cfe992ab7a5844640b622093a5d6f091edfa6af6ce456761e229fa1eb949ce50f103e9aadca54fa6abce66b0fc535fb5335c884e5de5bd2b29408d747
+  data.tar.gz: 66be8d4e9ff12a3e99dd138c82aa676cdc6e8f9e61a07183137b24de07b2f2d177c9a11b6e27e2407476dc45a19e3b012b63be87f942981a34002a6a292f791e

data/CHANGELOG.md

@@ -5,6 +5,68 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.0.0] - 2026-01-17
+
+### Added
+
+- **Schema Validation DSL**: Define schemas via `Cton.schema` and validate data with `Cton.validate_schema` for LLM-safe outputs.
+- **Streaming APIs**: `Cton.load_stream`, `Cton.dump_stream`, plus `StreamReader`/`StreamWriter` for newline-delimited documents.
+- **CTON-B Binary Mode**: Optional binary envelope with compression via `Cton.dump_binary`/`Cton.load_binary`.
+- **CLI Enhancements**: `--schema`, `--stream`, `--to-binary`, and `--from-binary` support.
+
+### Changed
+
+- **Performance**: Faster scalar scans in the decoder and reusable scalar buffers in the encoder.
+- **Docs**: README refocused on LLM usage, schema validation, and streaming workflows.
+
+## [0.4.0] - 2025-11-26
+
+### Added
+
+- **Comment Support**: CTON now supports single-line comments using `#` syntax. Comments are ignored during parsing, allowing for annotated data files.
+  - Decoder skips comments (from `#` to end of line) during parsing
+  - Encoder can emit comments via new `comments:` option: `Cton.dump(data, comments: { "key" => "description" })`
+
+- **Validation API**: New methods to validate CTON syntax without full parsing:
+  - `Cton.valid?(string)` returns `true` or `false`
+  - `Cton.validate(string)` returns a `ValidationResult` object with detailed error information
+  - `ValidationResult` includes `valid?`, `errors`, and `to_s` methods
+  - `ValidationError` includes `message`, `line`, `column`, and `source_excerpt`
+
+- **Token Statistics API**: Analyze and compare CTON vs JSON token efficiency:
+  - `Cton.stats(data)` returns a `Stats` object with comprehensive metrics
+  - `Cton.stats_hash(data)` returns stats as a Hash
+  - `Stats` includes `json_chars`, `cton_chars`, `savings_percent`, `estimated_json_tokens`, `estimated_cton_tokens`
+  - `Stats.compare(data)` compares multiple format variants (CTON, CTON inline, CTON pretty, JSON, JSON pretty)
+
+- **Custom Type Registry**: Register custom serializers for domain objects:
+  - `Cton.register_type(klass, as: :object) { |value| ... }` registers a type handler
+  - `Cton.unregister_type(klass)` removes a handler
+  - `Cton.clear_type_registry!` clears all handlers
+  - Supports `:object`, `:array`, and `:scalar` modes
+
+- **Enhanced CLI**: New command-line options:
+  - `--stats` / `-s`: Show token savings statistics comparing JSON vs CTON
+  - `--validate`: Validate CTON syntax without conversion
+  - `--minify` / `-m`: Output CTON without separators (fully inline)
+  - Improved error messages with line/column information and colored output
+
+- **Enhanced Error Reporting**: `ParseError` now includes structured location information:
+  - `line` and `column` attributes for precise error location
+  - `source_excerpt` showing context around the error
+  - `suggestions` array for helpful hints
+  - `to_h` method for programmatic error handling
+
+### Changed
+
+- **Decoder optimizations**: Pre-compiled frozen regex patterns (`SAFE_KEY_PATTERN`, `INTEGER_PATTERN`, `FLOAT_PATTERN`) for faster matching
+- **Encoder**: Now uses frozen regex constants for `SAFE_TOKEN` and `NUMERIC_TOKEN`
+- **RBS signatures**: Comprehensive type signatures for all new APIs
+
+### Fixed
+
+- **Comment handling**: Whitespace and comments are now properly skipped in all parsing contexts
+
 ## [0.3.0] - 2025-11-20
 
 ### Added

data/README.md

@@ -3,92 +3,73 @@
 [![Gem Version](https://badge.fury.io/rb/cton.svg)](https://badge.fury.io/rb/cton)
 [![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/davidesantangelo/cton/blob/master/LICENSE.txt)
 
-**CTON** (Compact Token-Oriented Notation) is an aggressively minified, JSON-compatible wire format that keeps prompts short without giving up schema hints. It is shape-preserving (objects, arrays, scalars, table-like arrays) and deterministic, so you can safely round-trip between Ruby hashes and compact strings that work well in LLM prompts.
+CTON (Compact Token-Oriented Notation) is a token-efficient, JSON-compatible wire format built for LLM prompts. It keeps structure explicit (objects, arrays, table arrays) while removing syntactic noise, so prompts are shorter and outputs are easier to validate. CTON is deterministic and round-trippable, making it safe for LLM workflows.
+
+**CTON is designed to be the reference language for LLM data exchange**: short, deterministic, schema-aware.
 
 ---
 
-## 📖 Table of Contents
+## Quickstart
 
-- [What is CTON?](#what-is-cton)
-- [Why another format?](#why-another-format)
-- [Examples](#examples)
-- [Token Savings](#token-savings-vs-json--toon)
-- [Installation](#installation)
-- [Usage](#usage)
-- [Performance & Benchmarks](#performance--benchmarks)
-- [Teaching CTON to LLMs](#teaching-cton-to-llms)
-- [Development](#development)
-- [Contributing](#contributing)
-- [License](#license)
+```bash
+bundle add cton
+```
 
----
+```ruby
+require "cton"
 
-## What is CTON?
+payload = {
+  "user" => { "id" => 42, "name" => "Ada" },
+  "tags" => ["llm", "compact"],
+  "events" => [
+    { "id" => 1, "action" => "login" },
+    { "id" => 2, "action" => "upload" }
+  ]
+}
 
-CTON is designed to be the most efficient way to represent structured data for Large Language Models (LLMs). It strips away the "syntactic sugar" of JSON that humans like (indentation, excessive quoting, braces) but machines don't strictly need, while adding "structural hints" that help LLMs generate valid output.
+cton = Cton.dump(payload)
+# => user(id=42,name=Ada)
+# => tags[2]=llm,compact
+# => events[2]{id,action}=1,login;2,upload
 
-### Key Concepts
+round_trip = Cton.load(cton)
+# => same as payload
+```
 
-1.  **Root is Implicit**: No curly braces `{}` wrapping the entire document.
-2.  **Minimal Punctuation**:
-    *   Objects use `key=value`.
-    *   Nested objects use parentheses `(key=value)`.
-    *   Arrays use brackets with length `[N]=item1,item2`.
-3.  **Table Compression**: If an array contains objects with the same keys, CTON automatically converts it into a table format `[N]{header1,header2}=val1,val2;val3,val4`. This is a massive token saver for datasets.
+```bash
+# CLI usage
+cton input.json
+cton --to-json data.cton
+cton --stats input.json
+```
 
 ---
 
-## Examples
+## Why CTON for LLMs?
 
-### Simple Key-Value Pairs
+- **Shorter prompts**: CTON removes braces, indentation, and repeated keys.
+- **Schema hints built-in**: arrays include length and tables include headers.
+- **Deterministic output**: round-trip safe and validates structure.
+- **LLM-friendly**: small grammar + clear guardrails for generation.
 
-**JSON**
-```json
-{
-  "task": "planning",
-  "urgent": true,
-  "id": 123
-}
-```
+---
+
+## CTON in 60 seconds
+
+### Objects & Scalars
 
-**CTON**
 ```text
 task=planning,urgent=true,id=123
 ```
 
 ### Nested Objects
 
-**JSON**
-```json
-{
-  "user": {
-    "name": "Davide",
-    "settings": {
-      "theme": "dark"
-    }
-  }
-}
-```
-
-**CTON**
 ```text
-user(name=Davide,settings(theme=dark))
+user(name=Ada,settings(theme=dark))
 ```
 
-### Arrays and Tables
+### Arrays & Tables
 
-**JSON**
-```json
-{
-  "tags": ["ruby", "gem", "llm"],
-  "files": [
-    { "name": "README.md", "size": 1024 },
-    { "name": "lib/cton.rb", "size": 2048 }
-  ]
-}
-```
-
-**CTON**
 ```text
 tags[3]=ruby,gem,llm
 files[2]{name,size}=README.md,1024;lib/cton.rb,2048
@@ -96,236 +77,146 @@ files[2]{name,size}=README.md,1024;lib/cton.rb,2048
 
 ---
 
-## Why another format?
-
-- **Less noise than YAML/JSON**: no indentation, no braces around the root, and optional quoting.
-- **Schema guardrails**: arrays carry their length (`friends[3]`) and table headers (`{id,name,...}`) so downstream parsing can verify shape.
-- **LLM-friendly**: works as a single string you can embed in a prompt together with short parsing instructions.
-- **Token savings**: CTON compounds the JSON → TOON savings.
+## LLM Prompt Kit (Recommended)
 
-### Token savings vs JSON & TOON
+System prompt template:
 
-- **JSON → TOON**: The [TOON benchmarks](https://toonformat.dev) report roughly 40% fewer tokens than plain JSON on mixed-structure prompts while retaining accuracy due to explicit array lengths and headers.
-- **TOON → CTON**: By stripping indentation and forcing everything inline, CTON cuts another ~20–40% of characters.
-- **Net effect**: In practice you can often reclaim **50–60% of the token budget** versus raw JSON, leaving more room for instructions or reasoning steps while keeping a deterministic schema.
+```markdown
+You are an expert in CTON (Compact Token-Oriented Notation). Convert between JSON and CTON following the rules below and preserve the schema exactly.
 
----
-
-## Installation
-
-Add the gem to your application:
-
-```bash
-bundle add cton
+Rules:
+1. Do not wrap the root in `{}`.
+2. Objects use `key=value` and nested objects use `key(...)`.
+3. Arrays are `key[N]=v1,v2` and table arrays are `key[N]{k1,k2}=v1,v2;v1,v2`.
+4. Use unquoted literals for `true`, `false`, and `null`.
+5. Quote strings containing reserved characters (`,`, `;`, `=`, `(`, `)`) or whitespace.
+6. Always keep array length and table headers accurate.
 ```
 
-Or install it directly:
+Few-shot example:
 
-```bash
-gem install cton
+```text
+JSON: {"team":[{"id":1,"name":"Ada"},{"id":2,"name":"Lin"}]}
+CTON: team[2]{id,name}=1,Ada;2,Lin
 ```
 
 ---
 
-## Usage
-
-```ruby
-require "cton"
-
-payload = {
-  "context" => {
-    "task" => "Our favorite hikes together",
-    "location" => "Boulder",
-    "season" => "spring_2025"
-  },
-  "friends" => %w[ana luis sam],
-  "hikes" => [
-    { "id" => 1, "name" => "Blue Lake Trail", "distanceKm" => 7.5, "elevationGain" => 320, "companion" => "ana", "wasSunny" => true },
-    { "id" => 2, "name" => "Ridge Overlook", "distanceKm" => 9.2, "elevationGain" => 540, "companion" => "luis", "wasSunny" => false },
-    { "id" => 3, "name" => "Wildflower Loop", "distanceKm" => 5.1, "elevationGain" => 180, "companion" => "sam", "wasSunny" => true }
-  ]
-}
-
-# Encode to CTON
-cton = Cton.dump(payload)
-# => "context(... )\nfriends[3]=ana,luis,sam\nhikes[3]{...}"
-
-# Decode back to Hash
-round_tripped = Cton.load(cton)
-# => original hash
-
-# Need symbols?
-symbolized = Cton.load(cton, symbolize_names: true)
-
-# Want a truly inline document? Opt in explicitly (decoding becomes unsafe for ambiguous cases).
-inline = Cton.dump(payload, separator: "")
+## Schema Validation (1.0.0)
 
-# Pretty print for human readability
-pretty = Cton.dump(payload, pretty: true)
+CTON ships with a schema DSL for validation inside your LLM pipeline.
 
-# Stream to an IO object (file, socket, etc.)
-File.open("data.cton", "w") do |f|
-  Cton.dump(payload, f)
+```ruby
+schema = Cton.schema do
+  object do
+    key "user" do
+      object do
+        key "id", integer
+        key "name", string
+        optional "role", enum("admin", "viewer")
+      end
+    end
+    key "tags", array(of: string)
+  end
 end
 
-# Toggle float normalization strategies
-fast  = Cton.dump(payload) # default :fast mode
-strict = Cton.dump(payload, decimal_mode: :precise)
+result = Cton.validate_schema(payload, schema)
+puts result.valid? # true/false
 ```
 
-### CLI Tool
+Schema files can be used from the CLI as well:
 
-CTON comes with a command-line tool for quick conversions:
+```ruby
+# schema.rb
+CTON_SCHEMA = Cton.schema do
+  object do
+    key "user", object { key "id", integer }
+  end
+end
+```
 
 ```bash
-# Convert JSON to CTON
-echo '{"hello": "world"}' | cton
-# => hello=world
-
-# Convert CTON to JSON
-echo 'hello=world' | cton --to-json
-# => {"hello":"world"}
-
-# Pretty print
-cton --pretty input.json
+cton --schema schema.rb input.cton
 ```
 
-### Advanced Features
-
-#### Extended Types
-CTON natively supports serialization for:
-- `Time` and `Date` (ISO8601 strings)
-- `Set` (converted to Arrays)
-- `OpenStruct` (converted to Objects)
+---
 
-#### Table detection
-Whenever an array is made of hashes that all expose the same scalar keys, the encoder flattens it into a table to save tokens. Mixed or nested arrays fall back to `[N]=(value1,value2,...)`.
+## Streaming IO (1.0.0)
 
-#### Separators & ambiguity
-Removing every newline makes certain inputs ambiguous because `sam` and the next key `hikes` can merge into `samhikes`. The default `separator: "\n"` avoids that by inserting a single newline between root segments. You may pass `separator: ""` to `Cton.dump` for maximum compactness, but decoding such strings is only safe if you can guarantee extra quoting or whitespace between segments. When you intentionally omit separators, keep next-level keys alphabetic (e.g., `payload`, `k42`) so the decoder's boundary heuristic can split `...1payload...` without misclassifying numeric prefixes.
+Handle newline-delimited CTON streams efficiently:
 
-#### Literal safety & number normalization
-Following the TOON specification's guardrails, the encoder now:
-- Auto-quotes strings that would otherwise be parsed as booleans, `null`, or numbers (e.g., `"true"`, `"007"`, `"1e6"`, `"-5"`) so they round-trip as strings without extra work.
-- Canonicalizes float/BigDecimal output: no exponent notation, no trailing zeros, and `-0` collapses to `0`.
-- Converts `NaN` and `±Infinity` inputs to `null`, matching TOON's normalization guidance so downstream decoders don't explode on non-finite numbers.
+```ruby
+io = File.open("events.cton", "r")
+Cton.load_stream(io).each do |event|
+  # process event
+end
+```
 
-#### Decimal normalization modes
-- `decimal_mode: :fast` (default) prefers Ruby's native float representation and only falls back to `BigDecimal` when scientific notation is detected, minimizing allocations on tight loops.
-- `decimal_mode: :precise` forces the legacy `BigDecimal` path for every float, which is slower but useful for audit-grade dumps where you want deterministic decimal expansion.
-- Both modes share the same trailing-zero stripping and `-0 → 0` normalization, so switching modes never affects integer formatting.
+```ruby
+io = File.open("events.cton", "w")
+Cton.dump_stream(events, io)
+```
 
 ---
 
-## Performance & Benchmarks
+## CTON-B (Binary Mode)
 
-CTON focuses on throughput: encoder table schemas are memoized, scalar list encoding keeps a reusable buffer, floats avoid `BigDecimal` when they can, and the decoder slices straight from the raw string to sidestep `StringScanner` allocations. You can reproduce the numbers below with the bundled script:
+CTON-B is an optional binary envelope for compact transport (with optional compression):
 
-```bash
-bundle exec ruby bench/encode_decode_bench.rb
-# customize input size / iterations
-ITERATIONS=2000 STREAM_SIZE=400 bundle exec ruby bench/encode_decode_bench.rb
+```ruby
+binary = Cton.dump_binary(payload)
+round_trip = Cton.load_binary(binary)
 ```
 
-Latest results on Ruby 3.1.4/macOS (M-series), 1,000 iterations, `STREAM_SIZE=200`:
+CLI:
 
-| Benchmark | Time (s) |
-| --- | --- |
-| `cton dump` (:fast) | 0.626 |
-| `cton dump` (:precise) | 0.658 |
-| `json generate` | 0.027 |
-| `cton load` | 2.067 |
-| `json parse` | 0.045 |
-| `cton inline load` (separator=`""`, double payload) | 4.140 |
+```bash
+cton --to-binary input.json > output.ctonb
+cton --from-binary output.ctonb
+```
 
-`cton inline load` deliberately concatenates documents without separators to stress the new boundary detector; it now finishes without the runaway allocations seen in earlier releases.
+Note: `--stream` with binary assumes newline-delimited binary frames.
 
 ---
 
-## Teaching CTON to LLMs
-
-Use this system prompt to teach an LLM how to understand and generate CTON:
-
-````markdown
-You are an expert in data serialization and specifically in CTON (Compact Token-Oriented Notation). CTON is a token-efficient data format optimized for LLMs that serves as a compact alternative to JSON.
-
-Your task is to interpret CTON input and convert it to JSON, or convert JSON input into valid CTON format, following the specification below.
-
-### CTON Specification
-
-CTON minimizes syntax characters (braces, quotes) while preserving structure and type safety.
-
-**1. Basic Structure (Key-Value)**
-- **Rule:** Do not use outer curly braces `{}` for the root object.
-- **Rule:** Use `=` to separate keys and values.
-- **Rule:** Use `,` to separate fields.
-- **Rule:** Do not use quotes around "safe" strings (alphanumeric, simple text).
-- **Example:** - JSON: `{"task": "planning", "urgent": true}`
-  - CTON: `task=planning,urgent=true`
+## Performance & Benchmarks
 
-**2. Nested Objects**
-- **Rule:** Use parentheses `()` to denote a nested object instead of `{}`.
-- **Example:**
-  - JSON: `{"context": {"user": "Davide", "theme": "dark"}}`
-  - CTON: `context(user=Davide,theme=dark)`
+CTON focuses on throughput: memoized table schemas, low-allocation scalar streams, and fast boundary detection for inline docs.
 
-**3. Arrays of Objects (Table Compression)**
-- **Rule:** Use the syntax `key[count]{columns}=values` for arrays of objects to avoid repeating keys.
-- **Structure:** `key[Length]{col1,col2}=val1,val2;val1,val2`
-- **Details:** - `[N]` denotes the number of items in the array.
-  - `{col1,col2}` defines the schema headers.
-  - `;` separates distinct objects (rows).
-  - `,` separates values within an object.
-- **Example:**
+Run benchmarks:
 
-JSON:
-```json
-{
-  "files": [
-    { "name": "README.md", "size": 1024 },
-    { "name": "lib.rb", "size": 2048 }
-  ]
-}
+```bash
+bundle exec ruby bench/encode_decode_bench.rb
+ITERATIONS=2000 STREAM_SIZE=400 bundle exec ruby bench/encode_decode_bench.rb
 ```
 
-CTON: `files[2]{name,size}=README.md,1024;lib.rb,2048`
-
-**4. Type Safety & Literals**
-- **Booleans/Null:** `true`, `false`, and `null` are preserved as literals (unquoted).
-- **Numbers:** Integers and floats are written as is (e.g., `1024`, `3.14`).
-- **Escaping:** If a string value looks like a boolean, number, or contains reserved characters (like `,`, `;`, `=`, `(`, `)`), it must be wrapped in double quotes (e.g., `"true"`).
+---
 
-### Examples for Training
+## CLI Reference
 
-**Input (JSON):**
-```json
-{
-  "id": 123,
-  "active": true,
-  "metadata": {
-    "created_at": "2023-01-01",
-    "tags": "admin"
-  }
-}
+```bash
+cton [input]                 # auto-detect JSON/CTON
+cton --to-json input.cton     # CTON → JSON
+cton --to-cton input.json     # JSON → CTON
+cton --to-binary input.json   # JSON → CTON-B
+cton --from-binary input.ctonb
+cton --minify input.json      # no separators
+cton --pretty input.json
+cton --stream input.ndjson
+cton --schema schema.rb input.cton
 ```
-````
 
 ---
 
-## Type Safety
-
-CTON ships with RBS signatures (`sig/cton.rbs`) to support type checking and IDE autocompletion.
-
 ## Development
 
 ```bash
 bin/setup        # install dependencies
 bundle exec rake # run tests and rubocop
 bin/console      # interactive playground
-bundle exec ruby bench/encode_decode_bench.rb # performance smoke test
 ```
 
-To release a new version, bump `Cton::VERSION` and run `bundle exec rake release`.
+---
 
 ## Contributing
 

data/lib/cton/binary.rb

@@ -0,0 +1,74 @@
+# frozen_string_literal: true
+
+require "zlib"
+
+module Cton
+  module Binary
+    MAGIC = "CTON".b
+    VERSION = 1
+    FLAG_COMPRESSED = 1
+
+    module_function
+
+    def dump(data, compress: true, **options)
+      payload = Cton.dump(data, **options).b
+      flags = 0
+
+      if compress
+        payload = Zlib.deflate(payload)
+        flags |= FLAG_COMPRESSED
+      end
+
+      header = MAGIC + [VERSION, flags].pack("CC")
+      header + encode_varint(payload.bytesize) + payload
+    end
+
+    def load(binary)
+      source = binary.to_s.b
+      raise Cton::Error, "Invalid CTON-B header" unless source.start_with?(MAGIC)
+
+      version = source.getbyte(4)
+      flags = source.getbyte(5)
+      raise Cton::Error, "Unsupported CTON-B version" unless version == VERSION
+
+      length, consumed = decode_varint(source, 6)
+      payload_start = 6 + consumed
+      payload = source.byteslice(payload_start, length)
+      raise Cton::Error, "Invalid CTON-B payload length" if payload.nil? || payload.bytesize < length
+
+      payload = Zlib.inflate(payload) if (flags & FLAG_COMPRESSED).positive?
+
+      Cton.load(payload)
+    end
+
+    def encode_varint(value)
+      bytes = []
+      remaining = value
+      while remaining >= 0x80
+        bytes << ((remaining & 0x7f) | 0x80)
+        remaining >>= 7
+      end
+      bytes << remaining
+      bytes.pack("C*")
+    end
+
+    def decode_varint(source, offset)
+      result = 0
+      shift = 0
+      index = offset
+
+      loop do
+        byte = source.getbyte(index)
+        raise Cton::Error, "Invalid CTON-B varint" unless byte
+
+        result |= (byte & 0x7f) << shift
+        index += 1
+        break if (byte & 0x80).zero?
+
+        shift += 7
+      end
+
+      [result, index - offset]
+    end
+  end
+end