cton 0.3.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1c9161ae830ba6b01d3ec94d1170fc4295aaacfa839c869aaa6adefe2711cc2d
-  data.tar.gz: 80c4ba30abbf8a562bde581f26e7dc5529aa46275b61930fe28370f34156db61
+  metadata.gz: 4e7225331f668eaed1f9b9e85e4cfc6fd183b997a8a6a89b69a5300ad2fbc54c
+  data.tar.gz: 20cdff68722bc7dc8d7629b95c5ae47d6d0eb21a8be26ebc93cc32605a5ed329
 SHA512:
-  metadata.gz: 914196284081bacd5b7f5f6ac9a1b246ea8924eddbb26cd28796b12a2ee2156718a9ff3b795b86188d483da4fc294b002a93e935815f692b432dac78b5304dcf
-  data.tar.gz: d9b1bfb1f7de402fe9de0d7da90f750d490dbdcb4e572e9283bc3ed15b1b43e3f3f9b44a4031f80ecf5d610a73d64719f3c78f170060de78035e04dab3a9d663
+  metadata.gz: 53f865fe83082953fae7cfa899c1c08014a21b0662d774253dc929c091eb4925c838007abfaeebdbacc3219a59751be0de1990d2230c9347a4d4e87ed9cc9753
+  data.tar.gz: b34df4bb7946e32feb693086892a410d9b440ba6374b3d3673e125b20ab7403c69bded90a43f5dd7683c3c54b889449965cfd136a7b180811a7ccb0a1ab185b9

data/CHANGELOG.md

@@ -5,6 +5,54 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.0] - 2025-11-26
+
+### Added
+
+- **Comment Support**: CTON now supports single-line comments using `#` syntax. Comments are ignored during parsing, allowing for annotated data files.
+  - Decoder skips comments (from `#` to end of line) during parsing
+  - Encoder can emit comments via new `comments:` option: `Cton.dump(data, comments: { "key" => "description" })`
+
+- **Validation API**: New methods to validate CTON syntax without full parsing:
+  - `Cton.valid?(string)` returns `true` or `false`
+  - `Cton.validate(string)` returns a `ValidationResult` object with detailed error information
+  - `ValidationResult` includes `valid?`, `errors`, and `to_s` methods
+  - `ValidationError` includes `message`, `line`, `column`, and `source_excerpt`
+
+- **Token Statistics API**: Analyze and compare CTON vs JSON token efficiency:
+  - `Cton.stats(data)` returns a `Stats` object with comprehensive metrics
+  - `Cton.stats_hash(data)` returns stats as a Hash
+  - `Stats` includes `json_chars`, `cton_chars`, `savings_percent`, `estimated_json_tokens`, `estimated_cton_tokens`
+  - `Stats.compare(data)` compares multiple format variants (CTON, CTON inline, CTON pretty, JSON, JSON pretty)
+
+- **Custom Type Registry**: Register custom serializers for domain objects:
+  - `Cton.register_type(klass, as: :object) { |value| ... }` registers a type handler
+  - `Cton.unregister_type(klass)` removes a handler
+  - `Cton.clear_type_registry!` clears all handlers
+  - Supports `:object`, `:array`, and `:scalar` modes
+
+- **Enhanced CLI**: New command-line options:
+  - `--stats` / `-s`: Show token savings statistics comparing JSON vs CTON
+  - `--validate`: Validate CTON syntax without conversion
+  - `--minify` / `-m`: Output CTON without separators (fully inline)
+  - Improved error messages with line/column information and colored output
+
+- **Enhanced Error Reporting**: `ParseError` now includes structured location information:
+  - `line` and `column` attributes for precise error location
+  - `source_excerpt` showing context around the error
+  - `suggestions` array for helpful hints
+  - `to_h` method for programmatic error handling
+
+### Changed
+
+- **Decoder optimizations**: Pre-compiled frozen regex patterns (`SAFE_KEY_PATTERN`, `INTEGER_PATTERN`, `FLOAT_PATTERN`) for faster matching
+- **Encoder**: Now uses frozen regex constants for `SAFE_TOKEN` and `NUMERIC_TOKEN`
+- **RBS signatures**: Comprehensive type signatures for all new APIs
+
+### Fixed
+
+- **Comment handling**: Whitespace and comments are now properly skipped in all parsing contexts
+
 ## [0.3.0] - 2025-11-20
 
 ### Added

data/README.md

@@ -35,6 +35,10 @@ CTON is designed to be the most efficient way to represent structured data for L
     *   Nested objects use parentheses `(key=value)`.
     *   Arrays use brackets with length `[N]=item1,item2`.
 3.  **Table Compression**: If an array contains objects with the same keys, CTON automatically converts it into a table format `[N]{header1,header2}=val1,val2;val3,val4`. This is a massive token saver for datasets.
+4.  **Comments**: Single-line comments with `#` for annotating data.
+5.  **Validation API**: Check CTON syntax without full parsing for quick validation.
+6.  **Token Statistics**: Built-in measurement of token efficiency vs JSON.
+7.  **Custom Type Registry**: Register serializers for domain objects.
 
 ---
 
@@ -188,10 +192,125 @@ echo 'hello=world' | cton --to-json
 
 # Pretty print
 cton --pretty input.json
+
+# Minify (fully inline, no separators)
+cton --minify input.json
+
+# Validate CTON syntax
+cton --validate input.cton
+# => ✓ Valid CTON
+
+# Show token savings statistics
+echo '{"name": "test", "items": [1,2,3]}' | cton --stats
+# => JSON:  33 chars / 33 bytes (~9 tokens)
+# => CTON:  26 chars / 26 bytes (~7 tokens)
+# => Saved: 21.2% (7 chars, ~2 tokens)
 ```
 
 ### Advanced Features
 
+#### Comments
+
+CTON supports single-line comments using the `#` character:
+
+```ruby
+cton_with_comments = <<~CTON
+  # User configuration
+  user(
+    name=Alice,
+    # Age is optional
+    age=30
+  )
+CTON
+
+Cton.load(cton_with_comments)
+# => {"user" => {"name" => "Alice", "age" => 30}}
+
+# Add comments when encoding
+Cton.dump(data, comments: { "user" => "User configuration" })
+```
+
+#### Validation API
+
+Validate CTON syntax without full parsing:
+
+```ruby
+# Quick validity check
+Cton.valid?("key=value")  # => true
+Cton.valid?("key=(broken")  # => false
+
+# Detailed validation with error info
+result = Cton.validate("key=(broken")
+result.valid?  # => false
+result.errors.first.message  # => "Expected '=' in object"
+result.errors.first.line     # => 1
+result.errors.first.column   # => 5
+```
+
+#### Token Statistics
+
+Measure CTON's token efficiency compared to JSON:
+
+```ruby
+stats = Cton.stats(data)
+puts stats.savings_percent    # => 45.5
+puts stats.estimated_token_savings  # => 12
+
+# Full comparison
+puts stats.to_s
+# => JSON:  100 chars / 100 bytes (~25 tokens)
+# => CTON:   55 chars /  55 bytes (~14 tokens)
+# => Saved: 45.0% (45 chars, ~11 tokens)
+
+# Compare all format variants
+Cton::Stats.compare(data)
+# => { cton: {...}, cton_inline: {...}, json: {...}, ... }
+```
+
+#### Custom Type Registry
+
+Register custom serializers for your domain objects:
+
+```ruby
+class Money
+  attr_reader :cents, :currency
+  def initialize(cents, currency)
+    @cents = cents
+    @currency = currency
+  end
+end
+
+# Register as object
+Cton.register_type(Money) do |money|
+  { amount: money.cents, currency: money.currency }
+end
+
+Cton.dump("price" => Money.new(1999, "USD"))
+# => "price(amount=1999,currency=USD)"
+
+# Register as scalar
+Cton.register_type(UUID, as: :scalar) { |uuid| uuid.to_s }
+
+# Unregister when done
+Cton.unregister_type(Money)
+```
+
+#### Enhanced Error Reporting
+
+Parse errors include detailed context for debugging:
+
+```ruby
+begin
+  Cton.load("user(name=Alice,invalid")
+rescue Cton::ParseError => e
+  puts e.message       # => "Unterminated object at line 1, column 20"
+  puts e.line          # => 1
+  puts e.column        # => 20
+  puts e.source_excerpt  # => "...name=Alice,invalid"
+  puts e.suggestions   # => ["Did you forget a closing ')'?"]
+end
+```
+
 #### Extended Types
 CTON natively supports serialization for:
 - `Time` and `Date` (ISO8601 strings)

data/lib/cton/decoder.rb

@@ -6,6 +6,9 @@ module Cton
   class Decoder
     TERMINATORS = [",", ";", ")", "]", "}"].freeze
     KEY_VALUE_BOUNDARY_TOKENS = ["(", "[", "="].freeze
+    SAFE_KEY_PATTERN = /[0-9A-Za-z_.:-]+/
+    INTEGER_PATTERN = /\A-?(?:0|[1-9]\d*)\z/
+    FLOAT_PATTERN = /\A-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?\z/
 
     def initialize(symbolize_names: false)
       @symbolize_names = symbolize_names
@@ -14,7 +17,7 @@ module Cton
     def decode(cton)
       @raw_string = cton.to_s
       @scanner = StringScanner.new(@raw_string)
-      skip_ws
+      skip_ws_and_comments
 
       value = if key_ahead?
                 parse_document
@@ -22,7 +25,7 @@ module Cton
                 parse_value(allow_key_boundary: true)
               end
 
-      skip_ws
+      skip_ws_and_comments
       raise_error("Unexpected trailing data") unless @scanner.eos?
 
       value
@@ -32,9 +35,16 @@ module Cton
 
     attr_reader :symbolize_names, :scanner, :raw_string
 
-    def raise_error(message)
+    def raise_error(message, suggestions: nil)
       line, col = calculate_location(@scanner.pos)
-      raise ParseError, "#{message} at line #{line}, column #{col}"
+      excerpt = extract_source_excerpt(@scanner.pos)
+      raise ParseError.new(
+        message,
+        line: line,
+        column: col,
+        source_excerpt: excerpt,
+        suggestions: suggestions
+      )
     end
 
     def calculate_location(pos)
@@ -46,19 +56,31 @@ module Cton
       [line, col]
     end
 
+    def extract_source_excerpt(pos, length: 30)
+      start = [pos - 10, 0].max
+      finish = [pos + length, raw_string.length].min
+      excerpt = raw_string[start...finish]
+      excerpt = "...#{excerpt}" if start.positive?
+      excerpt = "#{excerpt}..." if finish < raw_string.length
+      excerpt.gsub(/\s+/, " ")
+    end
+
     def parse_document
       result = {}
       until @scanner.eos?
+        skip_ws_and_comments
+        break if @scanner.eos?
+
         key = parse_key_name
         value = parse_value_for_key
         result[key] = value
-        skip_ws
+        skip_ws_and_comments
       end
       result
     end
 
     def parse_value_for_key
-      skip_ws
+      skip_ws_and_comments
       if @scanner.scan("(")
         parse_object
       elsif @scanner.scan("[")
@@ -71,7 +93,7 @@ module Cton
     end
 
     def parse_object
-      skip_ws
+      skip_ws_and_comments
       return {} if @scanner.scan(")")
 
       pairs = {}
@@ -80,11 +102,11 @@ module Cton
         expect!("=")
         value = parse_value
         pairs[key] = value
-        skip_ws
+        skip_ws_and_comments
         break if @scanner.scan(")")
 
         expect!(",")
-        skip_ws
+        skip_ws_and_comments
       end
       pairs
     end
@@ -92,7 +114,7 @@ module Cton
     def parse_array
       length = parse_integer_literal
       expect!("]")
-      skip_ws
+      skip_ws_and_comments
 
       header = parse_header if @scanner.peek(1) == "{"
 
@@ -140,7 +162,7 @@ module Cton
     end
 
     def parse_value(allow_key_boundary: false)
-      skip_ws
+      skip_ws_and_comments
       if @scanner.scan("(")
         parse_object
       elsif @scanner.scan("[")
@@ -153,7 +175,7 @@ module Cton
     end
 
     def parse_scalar(allow_key_boundary: false)
-      skip_ws
+      skip_ws_and_comments
       return parse_string if @scanner.peek(1) == '"'
 
       @scanner.pos
@@ -283,8 +305,8 @@ module Cton
     end
 
     def parse_key_name
-      skip_ws
-      token = @scanner.scan(/[0-9A-Za-z_.:-]+/)
+      skip_ws_and_comments
+      token = @scanner.scan(SAFE_KEY_PATTERN)
       raise_error("Invalid key") if token.nil?
       symbolize_names ? token.to_sym : token
     end
@@ -302,7 +324,7 @@ module Cton
     end
 
     def expect!(char)
-      skip_ws
+      skip_ws_and_comments
       return if @scanner.scan(Regexp.new(Regexp.escape(char)))
 
       raise_error("Expected #{char.inspect}, got #{@scanner.peek(1).inspect}")
@@ -312,16 +334,23 @@ module Cton
       @scanner.skip(/\s+/)
     end
 
+    def skip_ws_and_comments
+      loop do
+        @scanner.skip(/\s+/)
+        break unless @scanner.scan(/#[^\n]*\n?/)
+      end
+    end
+
     def whitespace?(char)
       [" ", "\t", "\n", "\r"].include?(char)
     end
 
     def key_ahead?
       pos = @scanner.pos
-      skip_ws
+      skip_ws_and_comments
 
-      if @scanner.scan(/[0-9A-Za-z_.:-]+/)
-        skip_ws
+      if @scanner.scan(SAFE_KEY_PATTERN)
+        skip_ws_and_comments
         next_char = @scanner.peek(1)
         result = ["(", "[", "="].include?(next_char)
         @scanner.pos = pos
@@ -337,11 +366,11 @@ module Cton
     end
 
     def integer?(token)
-      token.match?(/\A-?(?:0|[1-9]\d*)\z/)
+      token.match?(INTEGER_PATTERN)
     end
 
     def float?(token)
-      token.match?(/\A-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?\z/)
+      token.match?(FLOAT_PATTERN)
     end
   end
 end

data/lib/cton/encoder.rb

@@ -11,10 +11,11 @@ module Cton
     RESERVED_LITERALS = %w[true false null].freeze
     FLOAT_DECIMAL_PRECISION = Float::DIG
 
-    def initialize(separator: "\n", pretty: false, decimal_mode: :fast)
+    def initialize(separator: "\n", pretty: false, decimal_mode: :fast, comments: nil)
       @separator = separator || ""
       @pretty = pretty
       @decimal_mode = decimal_mode
+      @comments = comments || {}
       raise ArgumentError, "decimal_mode must be :fast or :precise" unless %i[fast precise].include?(@decimal_mode)
 
       @indent_level = 0
@@ -29,7 +30,7 @@ module Cton
 
     private
 
-    attr_reader :separator, :io, :pretty, :indent_level, :decimal_mode
+    attr_reader :separator, :io, :pretty, :indent_level, :decimal_mode, :comments
 
     def encode_root(value)
       case value
@@ -37,6 +38,7 @@ module Cton
         first = true
         value.each do |key, nested|
           io << separator unless first
+          emit_comment_for(key.to_s)
           encode_top_level_pair(key, nested)
           first = false
         end
@@ -51,6 +53,9 @@ module Cton
     end
 
     def encode_value(value, context:)
+      # Check type registry first for custom transformations
+      value = Cton.type_registry.transform(value) if Cton.type_registry.registered?(value.class)
+
       if defined?(Set) && value.is_a?(Set)
         value = value.to_a
       elsif defined?(OpenStruct) && value.is_a?(OpenStruct)
@@ -373,5 +378,14 @@ module Cton
     def newline
       io << "\n" << ("  " * indent_level)
     end
+
+    def emit_comment_for(key)
+      comment = comments[key] || comments[key.to_sym]
+      return unless comment
+
+      comment.to_s.each_line do |line|
+        io << "# " << line.chomp << "\n"
+      end
+    end
   end
 end

data/lib/cton/stats.rb

@@ -0,0 +1,123 @@
+# frozen_string_literal: true
+
+require "json"
+
+module Cton
+  # Token statistics for comparing JSON vs CTON efficiency
+  class Stats
+    # Rough estimate: GPT models average ~4 characters per token
+    CHARS_PER_TOKEN = 4
+
+    attr_reader :data, :json_string, :cton_string
+
+    def initialize(data, cton_string: nil, json_string: nil)
+      @data = data
+      @json_string = json_string || JSON.generate(data)
+      @cton_string = cton_string || Cton.dump(data)
+    end
+
+    def json_chars
+      json_string.length
+    end
+
+    def cton_chars
+      cton_string.length
+    end
+
+    def json_bytes
+      json_string.bytesize
+    end
+
+    def cton_bytes
+      cton_string.bytesize
+    end
+
+    def savings_chars
+      json_chars - cton_chars
+    end
+
+    def savings_bytes
+      json_bytes - cton_bytes
+    end
+
+    def savings_percent
+      return 0.0 if json_chars.zero?
+
+      ((1 - (cton_chars.to_f / json_chars)) * 100).round(1)
+    end
+
+    def estimated_json_tokens
+      (json_chars / CHARS_PER_TOKEN.to_f).ceil
+    end
+
+    def estimated_cton_tokens
+      (cton_chars / CHARS_PER_TOKEN.to_f).ceil
+    end
+
+    def estimated_token_savings
+      estimated_json_tokens - estimated_cton_tokens
+    end
+
+    def to_h
+      {
+        json_chars: json_chars,
+        cton_chars: cton_chars,
+        json_bytes: json_bytes,
+        cton_bytes: cton_bytes,
+        savings_chars: savings_chars,
+        savings_bytes: savings_bytes,
+        savings_percent: savings_percent,
+        estimated_tokens: {
+          json: estimated_json_tokens,
+          cton: estimated_cton_tokens,
+          savings: estimated_token_savings
+        }
+      }
+    end
+
+    def to_s
+      <<~STATS
+        JSON:  #{json_chars} chars / #{json_bytes} bytes (~#{estimated_json_tokens} tokens)
+        CTON:  #{cton_chars} chars / #{cton_bytes} bytes (~#{estimated_cton_tokens} tokens)
+        Saved: #{savings_percent}% (#{savings_chars} chars, ~#{estimated_token_savings} tokens)
+      STATS
+    end
+
+    # Compare multiple encoding options
+    def self.compare(data, options: {})
+      results = {}
+
+      # Standard CTON
+      results[:cton] = new(data).to_h
+
+      # Inline CTON (no separators)
+      inline_cton = Cton.dump(data, separator: "")
+      results[:cton_inline] = {
+        chars: inline_cton.length,
+        bytes: inline_cton.bytesize
+      }
+
+      # Pretty CTON
+      pretty_cton = Cton.dump(data, pretty: true)
+      results[:cton_pretty] = {
+        chars: pretty_cton.length,
+        bytes: pretty_cton.bytesize
+      }
+
+      # JSON variants
+      json = JSON.generate(data)
+      results[:json] = {
+        chars: json.length,
+        bytes: json.bytesize
+      }
+
+      pretty_json = JSON.pretty_generate(data)
+      results[:json_pretty] = {
+        chars: pretty_json.length,
+        bytes: pretty_json.bytesize
+      }
+
+      results
+    end
+  end
+end

data/lib/cton/type_registry.rb

@@ -0,0 +1,137 @@
+# frozen_string_literal: true
+
+module Cton
+  # Registry for custom type serializers
+  # Allows users to define how custom classes should be encoded to CTON
+  class TypeRegistry
+    # Handler wraps a serialization block with metadata
+    Handler = Struct.new(:klass, :mode, :block, keyword_init: true)
+
+    def initialize
+      @handlers = {}
+    end
+
+    # Register a custom type handler
+    #
+    # @param klass [Class] The class to handle
+    # @param as [Symbol] How to serialize: :object (Hash), :array, or :scalar
+    # @param block [Proc] Transformation block receiving the value
+    #
+    # @example Register a Money class
+    #   Cton.register_type(Money) do |money|
+    #     { amount: money.cents, currency: money.currency }
+    #   end
+    #
+    # @example Register a UUID as scalar
+    #   Cton.register_type(UUID, as: :scalar) do |uuid|
+    #     uuid.to_s
+    #   end
+    #
+    def register(klass, as: :object, &block)
+      raise ArgumentError, "Block required for type registration" unless block_given?
+      raise ArgumentError, "as must be :object, :array, or :scalar" unless %i[object array scalar].include?(as)
+
+      @handlers[klass] = Handler.new(klass: klass, mode: as, block: block)
+    end
+
+    # Unregister a type handler
+    #
+    # @param klass [Class] The class to unregister
+    def unregister(klass)
+      @handlers.delete(klass)
+    end
+
+    # Check if a handler exists for a class
+    #
+    # @param klass [Class] The class to check
+    # @return [Boolean]
+    def registered?(klass)
+      @handlers.key?(klass) || find_handler_for_ancestors(klass)
+    end
+
+    # Transform a value using its registered handler
+    # Returns the value unchanged if no handler is registered
+    #
+    # @param value [Object] The value to transform
+    # @return [Object] The transformed value
+    def transform(value)
+      handler = find_handler(value.class)
+      return value unless handler
+
+      handler.block.call(value)
+    end
+
+    # Get the handler for a class
+    #
+    # @param klass [Class] The class to look up
+    # @return [Handler, nil]
+    def handler_for(klass)
+      find_handler(klass)
+    end
+
+    # List all registered types
+    #
+    # @return [Array<Class>]
+    def registered_types
+      @handlers.keys
+    end
+
+    # Clear all registered handlers
+    def clear!
+      @handlers.clear
+    end
+
+    private
+
+    def find_handler(klass)
+      # Direct match
+      return @handlers[klass] if @handlers.key?(klass)
+
+      # Check ancestors (for inheritance support)
+      find_handler_for_ancestors(klass)
+    end
+
+    def find_handler_for_ancestors(klass)
+      klass.ancestors.each do |ancestor|
+        return @handlers[ancestor] if @handlers.key?(ancestor)
+      end
+      nil
+    end
+  end
+
+  # Global type registry instance
+  @type_registry = TypeRegistry.new
+
+  class << self
+    # Access the global type registry
+    #
+    # @return [TypeRegistry]
+    attr_reader :type_registry
+
+    # Register a custom type handler
+    #
+    # @param klass [Class] The class to handle
+    # @param as [Symbol] How to serialize: :object, :array, or :scalar
+    # @param block [Proc] Transformation block
+    #
+    # @example
+    #   Cton.register_type(Money) do |money|
+    #     { amount: money.cents, currency: money.currency }
+    #   end
+    def register_type(klass, as: :object, &block)
+      type_registry.register(klass, as: as, &block)
+    end
+
+    # Unregister a custom type handler
+    #
+    # @param klass [Class] The class to unregister
+    def unregister_type(klass)
+      type_registry.unregister(klass)
+    end
+
+    # Clear all custom type handlers
+    def clear_type_registry!
+      type_registry.clear!
+    end
+  end
+end