rng 0.1.2 → 0.3.3

data/CHANGELOG.md

@@ -0,0 +1,317 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Fixed
+- **Critical**: Parser baseline bug preventing simple RNC patterns from parsing
+  - Added `standalone_pattern` rule to allow simple patterns at grammar level
+  - Fixed `grammar` rule to accept references, text, empty, and other standalone patterns
+  - Resolved fundamental parser limitation where `foo` and similar simple patterns failed
+  - Improved valid RNC parsing from 17/53 (32.1%) to 23/53 (43.4%) - +6 tests (+35%)
+  - Maintained invalid rejection at 29/31 (93.5%) and Metanorma at 21/21 (100%)
+- **Critical**: Include processor string literal extraction bug
+  - Fixed `extract_string_literal()` to properly handle `[:string_parts]` structure
+  - Resolved 18 of 21 Metanorma schema parsing failures
+  - Improved test suite compliance from 34% to 49% (+8 tests)
+- Parser whitespace handling to allow files starting with documentation comments
+- Parser choice ordering to prioritize element_def over named_pattern
+
+### Added
+- Documentation comments support (`##` syntax)
+  - Model classes now have `documentation` attribute (Element, Attribute, Define, Start)
+  - RncBuilder generates `##` comments from documentation
+  - RncParser properly handles leading documentation comments
+  - Full round-trip preservation (RNC → RNG → RNC)
+  - RNG XML uses `<a:documentation>` elements in annotations namespace
+- Annotation support infrastructure (ForeignAttribute, ForeignElement model classes)
+  - Parser rules for annotation blocks `[ns:attr = "val"]`
+  - Processor extracts annotations from parse tree
+  - Converter generates XML foreign attributes/elements
+
+### Added
+
+- **Namespace Support (Complete)**: Enhanced namespace handling with multiple declaration types
+  - Full support for prefixed namespace declarations (`namespace prefix = "uri"`)
+  - Support for default namespace with prefix (`default namespace prefix = "uri"`)
+  - Support for multiple namespace and datatype declarations in schema preamble
+  - New model classes for structured namespace handling:
+    - `Rng::NamespaceDeclaration` - Represents namespace declarations with OOP API
+    - `Rng::DatatypeDeclaration` - Represents datatype library declarations
+    - `Rng::SchemaPreamble` - Container for preamble declarations
+  - Clean separation of preamble parsing from grammar tree building
+  - Namespace prefix resolution in RNG XML output (prefixes resolved to URIs)
+
+### Changed
+
+- **Parser architecture enhanced** for namespace support
+  - Parse tree structure now includes `:preamble_data` for namespace/datatype declarations
+  - ParseTreeProcessor extracts preamble into `SchemaPreamble` objects
+  - Grammar tree receives structured metadata (`:default_namespace`, `:namespace_map`, `:datatype_map`)
+  - RncToRngConverter resolves namespace prefixes to URIs in element and attribute names
+- **Backward compatibility maintained** - Both legacy and new namespace formats work seamlessly
+  - Legacy format: `default namespace = "uri"` (still fully supported)
+  - New formats: `namespace prefix = "uri"` and `default namespace prefix = "uri"`
+  - Converter handles both old and new metadata formats transparently
+
+### Fixed
+
+- Fixed syntax error in ParseTreeProcessor (extraneous `end` statement removed)
+- Fixed namespace prefix resolution in RNG XML generation (prefixes now map to URIs)
+
+### Architecture
+
+- Implemented model-driven approach for namespace declarations (OOP classes instead of Hash objects)
+- Applied separation of concerns: preamble metadata stored separately from grammar tree
+- Followed Open/Closed Principle: new functionality added without modifying existing code paths
+- Maintained MECE structure: each component has one clear responsibility
+
+### Testing
+
+- Created comprehensive test suite for namespace support (18 examples, all passing)
+  - Legacy compatibility tests (3 tests)
+  - New namespace declaration tests (6 tests)
+  - Datatype library tests (3 tests)
+  - Combined declarations tests (3 tests)
+  - Edge case tests (3 tests)
+- All unit tests pass for new model classes (21 examples)
+- Zero regressions in existing test suite
+- Namespace prefixes correctly resolve to URIs in generated RNG XML
+
+## [0.3.0] - 2025-11-28
+
+### Added
+
+- **Documentation Comments Support**: Full support for `##` syntax with round-trip conversion (RNC ↔ RNG ↔ RNC)
+  - Parse documentation comments from RNC files
+  - Generate `<a:documentation>` elements in RNG XML
+  - Regenerate `##` comments when converting back to RNC
+  - Supported contexts: element, attribute, define, start patterns
+  - Preserves multi-line documentation through all transformations
+
+- **String Concatenation Support**: Parse-time string joining with `~` operator
+  - Concatenate string literals in all contexts: namespaces, URIs, values, parameters
+  - Multi-part concatenation: `"a" ~ "b" ~ "c"` joins to `"abc"`
+  - Supports whitespace around operator for readability
+  - Transparent concatenation - final schema contains joined strings
+
+- **Escape Sequence Support**: RELAX NG Compact Syntax escape sequences
+  - Unicode code points in identifiers: `\x{HHHHHH}` syntax (1-6 hex digits)
+  - Unicode code points in string literals: `\x{HHHHHH}` syntax
+  - Character escapes in strings: `\"`, `\\`, `\n`, `\r`, `\t`
+  - Escaped backslash: `\\x{...}` stays literal (not converted)
+  - Backward compatible: regular identifiers work unchanged
+
+### Changed
+
+- **Parse tree structure** for identifiers and strings (backward compatible)
+  - Identifiers and strings now parsed as character arrays to support escapes
+  - Old format: `{identifier: "foo"}`, `{string: "hello"}`
+  - New format: `{identifier_parts: [{char: "f"}, ...]}`, `{string_parts: [...]}`
+  - Converter transparently handles both old and new formats
+- ParseTreeProcessor now normalizes `:patterns` to `:definitions` in flat grammars
+
+### Fixed
+
+- **Critical**: Restored documentation comment parsing rules that were lost in previous versions
+  - Recovered 5 regressed tests (tests #47-51)
+  - Baseline restored from 13/53 to 18/53 (24.5% → 34.0%)
+
+### Technical Details
+
+- **Test Results**: 17/53 valid RNC parsing (32.1%), 27/31 invalid rejection (87.1%)
+- **Production Support**: 100% Metanorma schema support maintained (21/21 passing)
+- **Files Modified**:
+  - `lib/rng/rnc_parser.rb` - Added escape sequence grammar rules
+  - `lib/rng/rnc_to_rng_converter.rb` - Added escape sequence processing
+  - `lib/rng/parse_tree_processor.rb` - Fixed flat grammar normalization
+- **Minor Regression**: One test regression (18→17) due to parse tree structure changes
+- **Escape Sequences**: Core functionality fully working (Unicode, character escapes)
+
+### Documentation
+
+- Updated README.adoc with comprehensive documentation for both features
+- Added syntax examples, usage patterns, and API documentation
+- Created `IMPLEMENTATION_STATUS_PHASE4B_COMPLETE.md` with technical details
+- Created `CONTINUATION_PLAN_PHASE4_COMPLETE.md` for future roadmap
+
+## [0.2.0] - 2025-11-24
+
+### Added
+
+#### Major Features
+
+- **RNC Compact Syntax Parser** - Complete implementation of RELAX NG Compact syntax parser
+  - Parses RNC schemas to internal object model
+  - Converts RNC to RNG XML format
+  - Supports all basic RNC patterns and constructs
+
+- **RNC Generator** - Generate RNC compact syntax from object model
+  - `Rng.to_rnc()` method for converting schemas to compact syntax
+  - Clean, readable output formatting
+  - Round-trip conversion support (RNC → RNG → RNC)
+
+- **Augmentation Operators** - Support for RELAX NG pattern augmentation
+  - Choice augmentation with `|=` operator
+  - Interleave augmentation with `&=` operator
+  - Generates proper `combine="choice"` and `combine="interleave"` attributes
+  - Works both inside and outside grammar blocks
+
+- **Datatype Parameters** - Full support for XML Schema datatype constraints
+  - Pattern constraints: `xsd:string { pattern = "..." }`
+  - Range constraints: `xsd:int { minInclusive = "0" maxInclusive = "120" }`
+  - Length constraints: `xsd:string { length = "4" }`
+  - Multiple parameters per datatype
+  - All standard XML Schema parameters supported
+
+#### RNC Parser Features
+
+- Comment support (`#` line comments)
+- Element and attribute definitions
+- Named pattern definitions and references
+- Start pattern declarations
+- Occurrence markers (`?`, `*`, `+`)
+- Choice operator (`|`)
+- Sequence operator (`,`)
+- Group patterns with parentheses
+- Text and empty patterns
+- Value literals
+- Namespace declarations
+- Datatype library support
+- Mixed content patterns
+
+#### API Enhancements
+
+- `Rng.parse_rnc(rnc_string)` - Parse RNC compact syntax
+- `Rng.to_rnc(schema)` - Convert schema to RNC format
+- `Rng::RncParser.parse()` - Low-level RNC parsing
+- `Rng::RncParser.to_rnc()` - Low-level RNC generation
+
+### Fixed
+
+- Fixed gem loading with autoload to resolve circular dependencies
+- Fixed Text element rendering in nested XML structures (upgraded lutaml-model to 0.7.7)
+- Fixed attribute special value handling (empty, omitted, nil)
+- Fixed element ordering in round-trip conversions
+- Fixed pattern reference handling in choice and sequence contexts
+- Fixed empty array detection in RNC builder
+- Fixed occurrence marker duplication in content generation
+- Fixed group definition parsing with proper typo fix
+- **Fixed Nokogiri adapter auto-configuration** - The XML adapter is now automatically configured when the gem is loaded, eliminating the need for manual setup
+
+### Changed
+
+- Upgraded lutaml-model dependency from 0.7.3 to 0.7.7
+- Improved RNG to RNC conversion logic
+- Enhanced error handling in parser
+- Updated test suite with 78 additional tests
+- Reorganized parser grammar rules for clarity
+
+### Documentation
+
+- Added comprehensive README.adoc sections for new features
+- Added augmentation operators documentation with examples
+- Added datatype parameters documentation with examples
+- Created TEST_RESULTS_PHASE7.md with detailed test analysis
+- Updated IMPLEMENTATION_STATUS.md with phase completion details
+
+### Testing
+
+- Added Metanorma schema test suite (63 tests)
+- Added complex pattern tests (11 tests)
+- Added error handling tests (7 tests)
+- Added performance benchmarks (2 tests)
+- All basic functionality tests passing (100%)
+- Round-trip conversion tests passing
+- **Verified 100% success rate parsing all 19 Metanorma RNG files**
+- **2 out of 21 Metanorma RNC files parse successfully** (standalone schemas only)
+
+### Known Limitations
+
+The following RNC features are not yet implemented (planned for future releases):
+
+**CRITICAL (Blocks real-world usage):**
+
+- **`include` directive** - External file inclusion (blocks ~90% of production RNC schemas)
+  - Affects: 19 out of 21 Metanorma RNC schemas fail due to this
+  - Workaround: Use RNG XML format or manually inline included content
+  - Status: **Planned for v0.3.0 with HIGH priority**
+
+**MEDIUM Priority:**
+
+- `div` elements - Organizational sections
+- `externalRef` - External grammar references
+
+**LOW Priority:**
+
+- `parentRef` - Parent grammar references
+- Annotations - `[ ... ]` metadata blocks
+- Advanced pattern combinations (interleave, list, etc. from RNC source)
+
+**Round-trip notes:**
+
+- XML comments are not preserved (Lutaml::Model limitation)
+- Attribute ordering may change (not semantically significant)
+- Namespace prefixes may be reassigned (URIs preserved)
+
+These limitations affect parsing of complex real-world RNC schemas, but:
+- ✅ All RNG XML schemas parse perfectly (100% Metanorma compatibility)
+- ✅ Basic to moderate RNC schemas work correctly
+- ✅ RNC generation from object model works for all supported patterns
+
+## [0.1.2] - 2025-11-23
+
+### Initial Release
+
+Basic RELAX NG XML (RNG) support:
+- Parse RNG XML schemas
+- Object model for all RELAX NG patterns
+- Round-trip RNG XML conversion
+- Integration with Lutaml ecosystem
+
+---
+
+## Release Notes
+
+### v0.2.0 - RNC Compact Syntax Support
+
+This release adds comprehensive support for RELAX NG Compact syntax (RNC), making it much easier to work with RELAX NG schemas in Ruby. You can now:
+
+1. **Parse RNC schemas** directly with `Rng.parse_rnc()`
+2. **Generate RNC syntax** from object model with `Rng.to_rnc()`
+3. **Use augmentation operators** to extend pattern definitions
+4. **Constrain datatypes** with parameters like pattern, range, and length
+5. **Convert between formats** seamlessly (RNC ↔ RNG)
+
+The implementation is production-ready for basic to moderate complexity schemas. Complex schemas using advanced features like `div`, `externalRef`, and `parentRef` are not yet supported but are planned for v0.3.0.
+
+### Migration Guide from 0.1.x
+
+No breaking changes. All existing code continues to work. New features are purely additive:
+
+```ruby
+# New in 0.2.0: Parse RNC
+schema = Rng.parse_rnc(File.read('schema.rnc'))
+
+# New in 0.2.0: Generate RNC
+rnc = Rng.to_rnc(schema)
+
+# Existing: Parse RNG (still works)
+schema = Rng.parse(File.read('schema.rng'))
+```
+
+### Performance
+
+- RNC parsing: ~2.27ms average for moderate schemas
+- Round-trip conversion: <5ms for most schemas
+- No performance regressions in existing RNG parsing
+
+### Credits
+
+- Development: Rng team
+- Testing: Metanorma schema test suite
+- Dependencies: lutaml-model 0.7.7+, parslet

data/CLAUDE.md

@@ -0,0 +1,139 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+RNG is a Ruby gem for parsing, manipulating, and converting RELAX NG schemas (both RNG XML and RNC compact syntax). It uses Lutaml::Model for the object model, Parslet for RNC parsing, and Nokogiri for XML processing.
+
+## Commands
+
+```bash
+# Install dependencies
+bundle install
+
+# Run all tests
+bundle exec rspec
+
+# Run a specific test file
+bundle exec rspec spec/rng/rnc_parser_spec.rb
+
+# Run a specific test by line number
+bundle exec rspec spec/rng/rnc_parser_spec.rb:42
+
+# Run the official test suite (Jing-Trang compacttest.xml)
+bundle exec rspec spec/rng/compacttest_spec.rb
+
+# Run linter with auto-fix
+bundle exec rubocop -A
+
+# Run default task (tests + rubocop)
+bundle exec rake
+```
+
+### CLI Commands
+
+```bash
+# Validate a schema
+rng validate schema.rng
+
+# Validate an XML document against a schema
+rng validate schema.rng document.xml
+
+# Convert between RNG and RNC formats
+rng convert schema.rng -o schema.rnc
+rng convert schema.rnc -o schema.rng
+
+# Parse and display schema structure
+rng parse schema.rng
+
+# Show schema information
+rng info --statistics schema.rng
+```
+
+## Architecture
+
+The library has two parsing paths and one generation path:
+
+### RNG XML Parsing
+`Rng.parse()` → `Grammar.from_xml()` → Lutaml::Model with Nokogiri adapter
+
+All RNG model classes inherit from `Lutaml::Model::Serializable` and define XML mappings via the `xml do` block.
+
+### RNC Compact Parsing
+`Rng.parse_rnc()` → `RncParser.parse()` → `ParseTreeProcessor.normalize()` → `RncToRngConverter.convert()` → `Grammar.from_xml()`
+
+The RNC parser is a Parslet-based PEG parser in `lib/rng/rnc_parser.rb`. Key supporting classes:
+- `ParseTreeProcessor` (`lib/rng/parse_tree_processor.rb`) - Normalizes parse trees into consistent grammar structures
+- `RncToRngConverter` (`lib/rng/rnc_to_rng_converter.rb`) - Converts parse trees to RNG XML using Nokogiri builder
+- `IncludeProcessor` (`lib/rng/include_processor.rb`) - Handles file I/O and include directive resolution
+
+### RNG to RNC Generation
+`Rng.to_rnc()` → `ToRnc.convert()` → `RncParser.to_rnc()` → `RncBuilder.build()`
+
+`RncBuilder` (`lib/rng/rnc_builder.rb`) traverses the object model and generates RNC text.
+
+## Key Dependencies
+
+- **lutaml-model** - Object model and XML serialization
+- **nokogiri** - XML parsing and building
+- **parslet** - RNC compact syntax parser
+- **canon** - XML comparison matchers for tests (`be_xml_equivalent_with`)
+
+## Object Model Structure
+
+The object model mirrors RELAX NG concepts:
+- `Grammar` - Root container (can have start, define, element, include)
+- `Start` - Entry point definition
+- `Define` - Named pattern definitions
+- `Element` / `Attribute` - XML structures
+- Pattern classes: `Choice`, `Group`, `Interleave`, `Mixed`, `Optional`, `ZeroOrMore`, `OneOrMore`, `Text`, `Empty`, `Value`, `Data`, `List`
+- Reference classes: `Ref`, `ParentRef`, `ExternalRef`
+- Name classes: `Name`, `AnyName`, `NsName`, `Except`
+- `Div` - Documentation and grouping container
+
+## Design Decisions
+
+- **Foreign elements/attributes are NOT supported**: The RELAX NG spec allows elements and attributes from non-RNG namespaces as annotations. This library does not preserve or round-trip them. They are silently dropped during XML parsing and not stored in the object model. Tests containing foreign elements/attributes are skipped with an explicit message. Do not add `foreign_elements` or `foreign_attributes` attributes to model classes.
+
+## External Href Resolution
+
+The library supports resolving external references via `Rng.parse(rng_xml, location: path, resolve_external: true)`:
+
+- **`<include href="uri"/>`** at grammar level - merges definitions from external grammar
+- **`<externalRef href="uri"/>`** at pattern level - replaces ref with content from external grammar's start pattern
+
+### Setting Up Test Fixtures from Jing-Trang
+
+The spectest_spec.rb has 22 pending tests that require external resources from Jing-Trang's test suite. To enable these tests:
+
+1. Ensure Jing-Trang is checked out at `~/src/external/jing-trang`
+
+2. Extract test fixtures:
+   ```bash
+   bundle exec rake fixtures:extract_spectest
+   # or
+   ruby scripts/extract_spectest_resources.rb
+   ```
+
+   This creates `spec/fixtures/spectest_external/` with 20 test cases, each in its own `case_N/` subdirectory.
+
+3. Note: Each spectest.xml test case has isolated resources (virtual file system). The Jing-Trang framework runs each test with its own set of resources. To fully enable these tests, spectest_spec.rb would need to be updated to copy resources for each test case before running.
+
+## Important Notes
+
+- The Nokogiri adapter must be configured at load time: `Lutaml::Model::Config.configure { |c| c.xml_adapter = Lutaml::Model::Xml::NokogiriAdapter }`
+- Ruby 3.0.0+ required
+- The `RNG_VERBOSE=1` environment variable enables parser warnings
+- The official test suite (`spec/rng/compacttest_spec.rb`) uses Jing-Trang's `compacttest.xml` with 87 test cases
+
+## Documentation Site
+
+The gem has a Jekyll-based documentation site in `docs/`:
+- `docs/index.adoc` - Home page
+- `docs/getting-started/` - Installation and quick start
+- `docs/guides/` - Parsing, conversion, validation guides
+- `docs/reference/` - API and CLI reference
+- `docs/understanding/` - Architecture and format comparison
+
+Build docs locally: `cd docs && bundle exec jekyll serve`

data/Gemfile

@@ -1,18 +1,17 @@
 # frozen_string_literal: true
 
-source "https://rubygems.org"
+source 'https://rubygems.org'
 
 # Specify your gem's dependencies in rng.gemspec
 gemspec
 
-gem "diffy" # For generating human-readable diffs
-gem "equivalent-xml"
-gem "lutaml-model"
-gem "nokogiri"
-gem "rake", "~> 13.0"
-gem "rspec", "~> 3.0"
-gem "rubocop", "~> 1.21"
-gem "rubocop-performance", require: false
-gem "rubocop-rake", require: false
-gem "rubocop-rspec", require: false
-gem "xml-c14n"
+gem 'benchmark'
+gem 'canon'
+gem 'lutaml-model', github: 'lutaml/lutaml-model', branch: 'main'
+gem 'nokogiri'
+gem 'rake'
+gem 'rspec'
+gem 'rubocop'
+gem 'rubocop-performance'
+gem 'rubocop-rake'
+gem 'rubocop-rspec'