parsanol 3.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: '01840daa27539f714d2270ca715a635a783f72adcd4370d3e1196fbcd05015c3'
+  data.tar.gz: a54323f2b2da9bbc65f8a984411eb67c44495af2530f19e5b57a845e832d7648
+SHA512:
+  metadata.gz: debaafa0bf490766dc6d9b4e871c444c532b9882a9074e2603d779339c4706b68afc9b94d6a476a8d84ea3c458ebe1538fb67e9bd9ff0c153a8c64eda71387ee
+  data.tar.gz: 5efa3a1e216953644811211e48bb0aeaed5ef2106b6a387f3a8b27a9a858e60b95c23f063c549ba236fc5e5ef8b75f7de0e1a98f61836624a2533ea7c0335f74

data/HISTORY.txt

@@ -0,0 +1,25 @@
+# Changelog
+
+## Unreleased
+
+## 3.0.0 (2025-02-28)
+
+### Added
+- Updated to use parsanol 0.1.2 from crates.io
+- Fixed repetition with separator pattern in Parslet compatibility layer
+- Added spec for repetition pattern handling
+
+### Changes
+- Updated Cargo.toml to use published parsanol crate instead of local path
+
+## 2.0.0
+
+### Added
+- Initial release as Parsanol (renamed from Parslet)
+- Ruby native extension support
+- Multiple parser modes: RubyTransform, JsonOutput, DirectObjects
+
+## 1.0.0
+
+### Added
+- Original Parslet parser library

data/LICENSE

@@ -0,0 +1,23 @@
+Copyright (c) 2025 Ribose Inc.
+Copyright (c) 2010-2018 Kaspar Schiess
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

data/README.adoc

@@ -0,0 +1,643 @@
+= Parsanol
+
+image:https://img.shields.io/gem/v/parsanol.svg[RubyGems Version]
+image:https://img.shields.io/github/license/parsanol/parsanol-ruby.svg[License]
+image:https://github.com/parsanol/parsanol-ruby/actions/workflows/test.yml/badge.svg["Build", link="https://github.com/parsanol/parsanol-ruby/actions/workflows/test.yml"]
+
+A high-performance PEG (Parsing Expression Grammar) parser construction library for Ruby, with support for static frozen parsers and dynamic parsers.
+
+== Purpose
+
+Parsanol provides a DSL for constructing parsers using PEG semantics. It offers excellent error reporting and supports both pure Ruby parsing and optional Rust native extensions for maximum performance.
+
+== Features
+
+* PEG-based parser construction
+* Great error reporting with detailed parse failure information
+* Optional Rust native extension for up to 29x faster parsing
+* **Slice support** - preserves source positions for linters and IDEs
+* WebAssembly support for browser/Node.js environments
+* Tree transformation for converting parse results to AST
+* Grammar optimization passes
+* Memory-efficient parsing with object pooling
+* **Streaming Builder API** - single-pass parsing with Ruby callbacks
+* **Parallel Parsing** - batch processing with multi-core speedup
+* **Infix Expression Parsing** - built-in operator precedence support
+* **Security Features** - input size and recursion limits for untrusted data
+* **Debug Tools** - tracing and grammar visualization
+
+== Installation
+
+[source,ruby]
+----
+gem install parsanol
+----
+
+Or add to your Gemfile:
+
+[source,ruby]
+----
+gem 'parsanol'
+----
+
+== Usage
+
+=== Basic Parser
+
+[source,ruby]
+----
+require 'parsanol'
+
+class MyParser < Parsanol::Parser
+  rule(:keyword) { str('if') | str('while') }
+  rule(:expression) { keyword >> str('(') >> expression >> str(')') }
+  root(:expression)
+end
+
+parser = MyParser.new
+result = parser.parse('if(x)')
+----
+
+=== Parslet Compatibility
+
+Parsanol is fully compatible with the Parslet API:
+
+[source,ruby]
+----
+require 'parslet'  # Works exactly like original Parslet
+
+class MyParser < Parslet::Parser
+  rule(:hello) { str('hello') }
+  root(:hello)
+end
+----
+
+=== Transformation
+
+[source,ruby]
+----
+class MyTransform < Parsanol::Transform
+  rule(keyword: simple(:k)) { Keyword.new(k) }
+end
+
+MyTransform.apply(parse_tree)
+----
+
+== Migrating from Parslet
+
+Parsanol provides full Parslet API compatibility with two migration modes.
+
+=== Mode 1: Drop-in Replacement (Zero Code Changes)
+
+Simply replace the parslet gem with parsanol in your Gemfile:
+
+[source,ruby]
+----
+# Gemfile
+- gem 'parslet'
++ gem 'parsanol'
+----
+
+Your existing code works without modification:
+
+[source,ruby]
+----
+# No changes needed!
+require 'parslet'  # Parsanol aliases itself
+
+class MyParser < Parslet::Parser
+  rule(:number) { match('[0-9]').repeat(1) }
+  root(:number)
+end
+
+parser = MyParser.new
+parser.parse('123')  # Works exactly the same
+----
+
+*What you get:*
+- Up to 29x performance improvement (measured with Expressir)
+- Slice support for source position tracking
+- 99.5% fewer allocations
+- Same error messages
+- 100% API compatibility (including Parslet::Slice)
+
+=== Mode 2: Native Parsanol API (Enhanced Features)
+
+For maximum performance and new features:
+
+[source,ruby]
+----
+require 'parsanol'  # Use Parsanol namespace
+
+class MyParser < Parsanol::Parser
+  rule(:number) { match('[0-9]').repeat(1) }
+  root(:number)
+end
+----
+
+*Additional features in native mode:*
+- Direct Rust FFI (fastest)
+- JSON grammar export
+- Expression parser with precedence climbing
+- Streaming parsing (planned)
+
+=== API Compatibility Matrix
+
+[cols="2,1,3"]
+|===
+| Parslet API | Status | Notes
+
+| `str('foo')` | ✅ | Literal string match
+| `match('[0-9]')` | ✅ | Character class
+| `any` | ✅ | Any single character
+| `>>` (sequence) | ✅ | Sequential composition
+| `\|` (choice) | ✅ | Ordered choice
+| `.repeat(n, m)` | ✅ | Repetition with bounds
+| `.repeat(1)` | ✅ | One or more
+| `.repeat` | ✅ | Zero or more
+| `.maybe` | ✅ | Optional
+| `.as(:name)` | ✅ | Label capture
+| `.absent?` | ✅ | Negative lookahead
+| `.present?` | ✅ | Positive lookahead
+| `infix_expression` | ✅ | Precedence climbing
+| `parse()` | ✅ | Parse and return tree
+| `parse_with_debug()` | ✅ | Parse with error output
+| `Parslet::Transform` | ✅ | Tree transformation
+| `simple(:x)` | ✅ | Match simple value
+| `sequence(:x)` | ✅ | Match array of values
+| `subtree(:x)` | ✅ | Match any subtree
+| `Parslet::Slice` | ✅ | Parsanol::Slice compatible
+|===
+
+=== Migration Checklist
+
+. **Backup your project**
+
+. **Update Gemfile**
++
+[source,ruby]
+----
+gem 'parsanol'  # Replace parslet
+----
+
+. **Run tests**
++
+[source,shell]
+----
+bundle install
+bundle exec rspec
+----
+
+. **Verify performance** (optional)
++
+[source,ruby]
+----
+require 'benchmark'
+
+input = "your test input"
+Benchmark.bm do |x|
+  x.report("parse") { 1000.times { parser.parse(input) } }
+end
+----
+
+=== Common Gotchas
+
+. **Native extension requires Rust**
++
+Parsanol falls back to pure Ruby if Rust is unavailable. For maximum performance, install Rust:
++
+[source,shell]
+----
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+----
+
+. **Error message formatting**
++
+Error messages have slightly different formatting but contain the same information.
+
+. **Grammar introspection**
++
+Works the same way in both modes.
+
+=== Performance Comparison
+
+[cols="3,2,2,2"]
+|===
+| Operation | Parslet | Parsanol (Ruby) | Parsanol (ZeroCopy+Slice)
+
+| EXPRESS parsing (22KB) | 3036 ms | - | 106 ms (28.7x)
+| JSON parsing | 1x | 1x | 29x
+| Calculator | 1x | 1x | 25x
+| Memory allocations | 100% | 100% | 0.5%
+|===
+
+== Performance
+
+Parsanol offers 5 different approaches for parsing in Ruby, each moving more work from Ruby to Rust:
+
+=== The 5 Approaches
+
+[cols="5,2,2,3"]
+|===
+| Approach | Speed | Use Case | How It Works
+
+| 1. parslet-ruby | 1x (baseline) | Compatibility, debugging | Pure Ruby parsing
+| 2. parsanol-ruby | ~1x | Learning, prototyping | Parsanol Ruby backend
+| 3. parsanol-native (Batch) | ~20x | Need Ruby objects | Rust parsing, AST via u64
+| 4. parsanol-native (ZeroCopy) | ~25x | Maximum performance | Direct FFI construction
+| 5. parsanol-native (ZeroCopy + Slice) | ~29x | Linters, IDEs, Expressir | Zero-copy + source positions
+|===
+
+=== Slice Support (New)
+
+The ZeroCopy + Slice mode preserves source position information for each parsed element:
+
+[source,ruby]
+----
+# Before (plain strings - no position info):
+[{"word"=>"hello"}, " ", {"name"=>"world"}]
+
+# After (Slice objects with position info):
+[{"word"=>"hello"@0}, " "@5, {"name"=>"world"@6}]
+
+# The @N notation shows the byte offset in the original input
+# Parsanol::Slice is compatible with Parslet::Slice
+----
+
+This is essential for tools like linters, IDEs, and Expressir that need to track where in the source code each element came from.
+
+=== Evidence-Based Results
+
+Actual benchmark results from Expressir parsing EXPRESS schemas (22KB file, 733 lines):
+
+[cols="3,2,2,3"]
+|===
+| Mode | Time | Speedup | Notes
+
+| Ruby (Parslet) | 3036 ms | 1x (baseline) | Pure Ruby parsing
+| Native Batch (u64) | 153 ms | 19.9x faster | AST via u64 array transfer
+| Native ZeroCopy (Slice) | 106 ms | 28.7x faster | Zero-copy with source positions
+|===
+
+*Run `bundle exec ruby benchmark/run_all.rb` to see results on YOUR machine.*
+
+=== Running Benchmarks
+
+Verify these results yourself:
+
+[source,shell]
+----
+cd parsanol-ruby
+bundle install
+bundle exec rake compile  # Build native extension
+bundle exec ruby benchmark/run_all.rb --quick
+----
+
+See `benchmark/APPROACHES.md` for detailed diagrams explaining each approach.
+
+== Architecture
+
+Parsanol consists of two main components:
+
+* *parsanol-ruby*: The Ruby gem with parser DSL and transformation engine
+* *parsanol-rs*: Rust crate providing native parsing acceleration
+
+=== Three Transformation Modes
+
+Parsanol supports three transformation modes:
+
+1. **Ruby Transform** (Parslet-compatible): Parse in Rust, transform in Ruby
+2. **Serialized Output**: Parse and transform in Rust, return JSON
+3. **Native FFI**: Parse, transform, and return Ruby objects directly
+
+=== Static vs Dynamic Parsers
+
+Parsanol supports two parser modes:
+
+* *Dynamic Parsers*: Can be modified during parsing, more flexible but slower
+* *Static/Frozen Parsers*: Pre-compiled grammar, fastest path for production use
+
+=== Streaming Builder API
+
+For maximum performance, use the streaming builder API which eliminates intermediate AST construction. Your Ruby callbacks receive parsed values directly during parsing:
+
+[IMPORTANT]
+====
+The streaming builder API requires the native extension. If the native extension is not available, use the pure Ruby parser instead.
+====
+
+[source,ruby]
+----
+require 'parsanol'
+
+# Define a custom builder by including Parsanol::BuilderCallbacks
+class StringCollector
+  include Parsanol::BuilderCallbacks
+
+  def initialize
+    @strings = []
+  end
+
+  def on_string(value, offset, length)
+    @strings << value
+  end
+
+  def finish
+    @strings
+  end
+end
+
+# Serialize grammar from a parser
+grammar = Parsanol::Native.serialize_grammar(MyParser.new.root)
+
+# Parse with the builder
+builder = StringCollector.new
+result = Parsanol::Native.parse_with_builder(grammar, input, builder)
+# result: ["hello", "world"]
+----
+
+==== Available Callback Methods
+
+Include `Parsanol::BuilderCallbacks` in your class and override these methods:
+
+[cols="1,3,2"]
+|===
+| Method | Description | Default
+
+| `on_start(input)` | Parsing started | No-op
+| `on_success` | Parsing succeeded | No-op
+| `on_error(message)` | Parsing failed | No-op
+| `on_string(value, offset, length)` | String/slice matched | No-op
+| `on_int(value)` | Integer matched | No-op
+| `on_float(value)` | Float matched | No-op
+| `on_bool(value)` | Boolean matched | No-op
+| `on_nil` | Nil matched | No-op
+| `on_hash_start(size)` | Entering a hash/object | No-op
+| `on_hash_key(key)` | Hash key encountered | No-op
+| `on_hash_value(key)` | About to parse hash value | No-op
+| `on_hash_end(size)` | Exiting a hash/object | No-op
+| `on_array_start(size)` | Entering an array | No-op
+| `on_array_element(index)` | About to parse array element | No-op
+| `on_array_end(size)` | Exiting an array | No-op
+| `on_named_start(name)` | Starting named rule | No-op
+| `on_named_end(name)` | Finished named rule | No-op
+| `finish` | Parsing complete, return result | Returns nil
+|===
+
+The `size` parameter in `on_array_start`, `on_hash_start`, `on_array_end`, and `on_hash_end` indicates the number of elements (may be `nil` for start callbacks).
+
+==== Built-in Builders
+
+Parsanol provides three built-in builders for common use cases:
+
+* `Parsanol::Builders::DebugBuilder` - Collects all parsing events as strings for debugging
+* `Parsanol::Builders::StringCollector` - Collects all string values into an array
+* `Parsanol::Builders::NodeCounter` - Counts nodes by type (strings, ints, arrays, hashes, etc.)
+
+[source,ruby]
+----
+# DebugBuilder - see all parsing events
+debug = Parsanol::Builders::DebugBuilder.new
+Parsanol::Native.parse_with_builder(grammar, input, debug)
+puts debug.events  # => ["start: ...", "string: ...", "hash_start", ...]
+
+# StringCollector - extract all strings
+collector = Parsanol::Builders::StringCollector.new
+Parsanol::Native.parse_with_builder(grammar, input, collector)
+puts collector.strings  # => ["hello", "world", ...]
+
+# NodeCounter - count node types
+counter = Parsanol::Builders::NodeCounter.new
+Parsanol::Native.parse_with_builder(grammar, input, counter)
+puts counter.counts  # => {:string=>5, :int=>3, :hash=>2, :array=>1}
+----
+
+==== Advanced Example: Building Custom Objects
+
+[source,ruby]
+----
+class JsonBuilder
+  include Parsanol::BuilderCallbacks
+
+  def initialize
+    @stack = []
+    @current_key = nil
+  end
+
+  def on_string(value, offset, length)
+    add_value(value)
+  end
+
+  def on_int(value)
+    add_value(value)
+  end
+
+  def on_float(value)
+    add_value(value)
+  end
+
+  def on_bool(value)
+    add_value(value)
+  end
+
+  def on_nil
+    add_value(nil)
+  end
+
+  def on_hash_start(size = nil)
+    @stack.push({})
+  end
+
+  def on_hash_key(key)
+    @current_key = key
+  end
+
+  def on_hash_end(size)
+    finished = @stack.pop
+    add_value(finished) unless @stack.empty?
+    @result = finished if @stack.empty?
+  end
+
+  def on_array_start(size = nil)
+    @stack.push([])
+  end
+
+  def on_array_element(index)
+    # Called before each array element
+  end
+
+  def on_array_end(size)
+    finished = @stack.pop
+    add_value(finished) unless @stack.empty?
+    @result = finished if @stack.empty?
+  end
+
+  def on_named_start(name)
+    # Called when entering a named rule (e.g., .as(:name))
+  end
+
+  def on_named_end(name)
+    # Called when exiting a named rule
+  end
+
+  def finish
+    @result
+  end
+
+  private
+
+  def add_value(value)
+    return if @stack.empty?
+    case @stack.last
+    when Hash
+      @stack.last[@current_key] = value
+      @current_key = nil
+    when Array
+      @stack.last << value
+    end
+  end
+end
+
+builder = JsonBuilder.new
+result = Parsanol::Native.parse_with_builder(grammar, input, builder)
+# result: Parsed JSON as Ruby objects
+----
+
+=== Parallel Parsing
+
+Parse multiple inputs in parallel using all CPU cores:
+
+[source,ruby]
+----
+require 'parsanol/parallel'
+
+grammar = MyParser.new.serialize_grammar
+inputs = Dir.glob("*.json").map { |f| File.read(f) }
+
+# Parse all files in parallel (8x faster on 8 cores)
+results = Parsanol::Parallel.parse_batch(grammar, inputs)
+
+# With configuration
+config = Parsanol::Parallel::Config.new
+  .with_num_threads(4)
+  .with_min_chunk_size(50)
+
+results = Parsanol::Parallel.parse_batch(grammar, inputs, config: config)
+----
+
+=== Infix Expression Parsing
+
+Built-in support for parsing infix expressions with operator precedence:
+
+[source,ruby]
+----
+class CalculatorParser < Parsanol::Parser
+  rule(:number) { match('[0-9]').repeat(1).as(:int) }
+  rule(:primary) { number | str('(') >> expr >> str(')') }
+
+  # Define operators with precedence and associativity
+  rule(:expr) {
+    infix_expression(primary,
+      [str('*'), 2, :left],
+      [str('/'), 2, :left],
+      [str('+'), 1, :left],
+      [str('-'), 1, :left],
+      [str('^'), 3, :right]  # Right-associative
+    )
+  }
+  root(:expr)
+end
+----
+
+=== Security Features
+
+For parsing untrusted input, use built-in limits:
+
+[source,ruby]
+----
+# Configure limits for untrusted input
+result = Parsanol::Native.parse_with_limits(
+  grammar_json,
+  untrusted_input,
+  max_input_size: 10 * 1024 * 1024,  # 10 MB max
+  max_recursion_depth: 100            # Limit recursion
+)
+----
+
+=== Debug Tools
+
+Enable tracing for debugging grammars:
+
+[source,ruby]
+----
+# Parse with trace
+result, trace = Parsanol::Native.parse_with_trace(grammar_json, input)
+puts trace
+
+# Generate grammar visualization
+mermaid = Parsanol::Native.grammar_to_mermaid(grammar_json)
+dot = Parsanol::Native.grammar_to_dot(grammar_json)
+----
+
+== Development
+
+=== Setup
+
+[source,shell]
+----
+bundle install
+----
+
+=== Testing
+
+[source,shell]
+----
+# Run all tests
+bundle exec rake spec
+
+# Run unit tests only
+bundle exec rake spec:unit
+
+# Run specific test file
+bundle exec rspec spec/parsanol/atoms/str_spec.rb
+----
+
+=== Compiling Native Extension
+
+The native extension requires Rust 1.75+:
+
+[source,shell]
+----
+# Install Rust (if not already installed)
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+
+# Compile the native extension
+bundle exec rake compile
+
+# Verify native extension is working
+ruby -I lib -e "require 'parsanol'; puts Parsanol::Native.available?"
+# => true
+----
+
+=== Running Benchmarks
+
+[source,shell]
+----
+# Quick benchmarks
+bundle exec rake benchmark
+
+# Comprehensive benchmark suite
+bundle exec rake benchmark:all
+
+# Run specific benchmark
+bundle exec ruby benchmark/run_all.rb --quick
+----
+
+== License
+
+MIT License - see LICENSE file for details.
+
+== Resources
+
+* https://github.com/parsanol/parsanol-ruby[GitHub Repository]
+* https://github.com/parsanol/parsanol-rs[Rust Crate]