expressir 2.4.0 → 2.4.1

data/TODO.bugs/16-pretty-formatter-duplication.md

@@ -0,0 +1,28 @@
+# 16 — PrettyFormatter duplicates declaration formatting from DeclarationsFormatter
+
+**Priority:** P1 (DRY violation)
+**Status:** ACCEPTED — duplication is intentional specialization
+
+## Problem
+
+`PrettyFormatter` redefines 5 `format_declarations_*` methods with
+indentation and provenance-header differences from the base
+`DeclarationsFormatter`.
+
+## Decision
+
+The duplication is **accepted** as intentional specialization.
+PrettyFormatter adds provenance headers (`format_provenance_header`),
+scope-body helpers (`format_scope_body`, `format_scope_footer`), and
+different indentation handling that the base formatter does not need.
+
+A full unification would require parameterizing the base formatter with
+indent-width, provenance support, and scope-body rendering — adding
+complexity to the common path to serve a specialized variant. The
+deletion test fails: removing PrettyFormatter's overrides would
+concentrate complexity into the base formatter, making the common path
+harder to reason about.
+
+The duplication is bounded (5 methods, ~100 lines) and the two
+formatters evolve independently (PrettyFormatter adds features the
+base does not need).

data/TODO.bugs/17-snake-case-cache-mutable-constant.md

@@ -0,0 +1,28 @@
+# 17 — SNAKE_CASE_CACHE is a mutable module constant
+
+**Priority:** P1 (unusual pattern)
+
+## Problem
+
+`lib/expressir/express/builder.rb:32`:
+
+```ruby
+SNAKE_CASE_CACHE = {} # rubocop:disable Style/MutableConstant
+```
+
+A module-level constant that is mutated in place at line 265 via `||=`.
+It grows unboundedly across the process lifetime and violates the frozen-
+constant convention the rest of the codebase follows.
+
+## Fix
+
+Either:
+- Document it explicitly as an intentionally-mutable static-conversion
+  cache and wrap it in a dedicated `SnakeCaseCache` class with a clear API.
+- Replace with `Thread.current[:snake_case_cache] ||= {}` for thread-local
+  bounded growth.
+
+## Acceptance
+
+- No `rubocop:disable Style/MutableConstant` needed.
+- Cache behavior unchanged.

data/TODO.bugs/18-const-get-private-constants.md

@@ -0,0 +1,30 @@
+# 18 — const_get to access private constants from formatter modules
+
+**Priority:** P2 (fragile access pattern)
+
+## Problem
+
+Formatter modules use `self.class.const_get(:INDENT)`,
+`self.class.const_get(:OPERATOR_PRECEDENCE)`, etc. to access constants
+declared `private_constant` on `Formatter`. ~10 call sites across:
+
+- `formatters/declarations_formatter.rb:129,235,304,360,535`
+- `formatters/expressions_formatter.rb:41`
+- `formatters/data_types_formatter.rb:131,265,304`
+- `formatters/supertype_expressions_formatter.rb:13`
+
+String-keyed reflective access to bypass visibility makes the code harder
+to grep and reason about.
+
+## Fix
+
+Remove `private_constant` on `INDENT`, `INDENT_CHAR`, `OPERATOR_PRECEDENCE`,
+and `SUPERTYPE_OPERATOR_PRECEDENCE` since the formatter modules are
+implementation details of Formatter anyway. Replace `const_get(:X)` with
+direct `X` references.
+
+## Acceptance
+
+- `grep -rn "const_get" lib/expressir/express/` returns nothing.
+- No `private_constant` on format constants.
+- All formatter specs pass.

data/TODO.bugs/19-format-methods-public.md

@@ -0,0 +1,22 @@
+# 19 — All format methods must be public due to format_registry dispatch
+
+**Priority:** P2 (encapsulation)
+
+## Problem
+
+`Formatter#format` dispatches via `public_send(handler, node)` at
+`lib/expressir/express/formatter.rb:88`. This forces ~60 `format_*`
+methods across 9 formatter modules to be public, even though they are only
+ever called internally via the registry.
+
+## Fix
+
+Replace `public_send(handler, node)` with `send(handler, node)` in
+`Formatter#format` (the handler comes from the trusted registry, not user
+input), then mark all `format_*` methods as `private`.
+
+## Acceptance
+
+- `Formatter#format` uses `send` not `public_send`.
+- Format methods are private in all 9 formatter modules.
+- `Formatter.new.public_methods` does not include any `format_*` method.

data/TODO.bugs/20-coverage-nested-entities-dedup.md

@@ -0,0 +1,20 @@
+# 20 — Coverage.find_nested_entities repeats 6-collection pattern three times
+
+**Priority:** P2 (DRY violation)
+
+## Problem
+
+`lib/expressir/coverage.rb:404-477` — `find_nested_entities` has three
+near-identical `when` branches for Function, Rule, and Procedure, each
+concatenating the same six collections and recursing.
+
+## Fix
+
+Replace the three branches with a single loop over
+`node.class.collection_attributes_list` (from the `collection_attributes`
+macro) and recurse into children.
+
+## Acceptance
+
+- `find_nested_entities` has no model-class-specific case branches.
+- All coverage specs pass.

data/TODO.bugs/21-operator-tokens-secondary-dispatch.md

@@ -0,0 +1,21 @@
+# 21 — OPERATOR_TOKENS in Builder is a secondary dispatch mechanism
+
+**Priority:** P2 (parallel dispatch)
+
+## Problem
+
+`lib/expressir/express/builder.rb:51-59` — 24-element `Set` of operator
+token symbols, used at line 87 to decide whether to skip a nil-returning
+handler. This set must be kept in sync with the grammar.
+
+## Fix
+
+Instead of maintaining a hand-curated set, check whether the handler
+returns nil for ANY first-key handler when the hash has multiple keys, not
+just for operator tokens. This eliminates the need for the set.
+
+## Acceptance
+
+- `OPERATOR_TOKENS` constant removed.
+- Builder still handles multi-key hash AST correctly.
+- All parser + builder specs pass.

data/TODO.bugs/22-builder-fast-path-wrappers.md

@@ -0,0 +1,32 @@
+# 22 — Builder fast-path methods are trivial wrappers
+
+**Priority:** P2 (dead code surface)
+
+## Problem
+
+`lib/expressir/express/builder.rb:232-259` — Six methods that each just
+call `build_node(:symbol, data)`, adding no logic:
+
+```ruby
+def build_term(data) = build_node(:term, data)
+def build_factor(data) = build_node(:factor, data)
+def build_simple_factor(data) = build_node(:simple_factor, data)
+def build_primary(data) = build_node(:primary, data)
+def build_expression(data) = build_node(:expression, data)
+def build_simple_expression(data) = build_node(:simple_expression, data)
+```
+
+Called ~25 times across the builders. The indirection provides no value
+since `build_node` is equally readable.
+
+## Fix
+
+Inline the calls to `build_node(:term, ...)`, etc. Remove the six wrapper
+methods. Reduce the Builder public surface.
+
+## Acceptance
+
+- `build_term`, `build_factor`, `build_simple_factor`, `build_primary`,
+  `build_expression`, `build_simple_expression` removed from Builder.
+- All call sites updated to `build_node(:term, ...)`.
+- All parser + builder specs pass.

data/TODO.bugs/23-coverage-inverse-maps.md

@@ -0,0 +1,21 @@
+# 23 — Coverage maps are manual inverses
+
+**Priority:** P2 (DRY violation)
+
+## Problem
+
+`lib/expressir/coverage.rb:7-52` — `ENTITY_TYPE_MAP` and
+`CLASS_TO_EXPRESS_TYPE_MAP` are 20-entry frozen hashes that are exact
+inverses of each other. Adding or renaming a type requires updating both.
+
+## Fix
+
+Generate one map from the other at load time, or use a single canonical
+list and derive both directions via `.to_h` and `.invert`.
+
+## Acceptance
+
+- Only one source-of-truth map exists.
+- Both `ENTITY_TYPE_MAP` and `CLASS_TO_EXPRESS_TYPE_MAP` (or their
+  successors) are derived from the single source.
+- All coverage specs pass.

data/TODO.bugs/24-streaming-builder-complexity.md

@@ -0,0 +1,19 @@
+# 24 — StreamingBuilder convert_ast_format is 190 lines with 5 levels of nesting
+
+**Priority:** P3 (complexity; defer unless time allows)
+
+## Problem
+
+`lib/expressir/express/streaming_builder.rb:241-429` —
+`convert_ast_format` is a 190-line method with deeply nested conditionals
+for merging Parsanol's single-key-hash sequences.
+
+## Fix
+
+Split into named sub-methods for each pattern: merge-sequence,
+repetition-unzip, str-concat.
+
+## Acceptance
+
+- No method in StreamingBuilder exceeds 30 lines.
+- All streaming specs pass.

data/TODO.bugs/25-debug-puts-in-production.md

@@ -0,0 +1,21 @@
+# 25 — Debug puts statements in production code
+
+**Priority:** P3 (cleanup)
+
+## Problem
+
+`lib/expressir/express/streaming_builder.rb:115,123,447-453` — `puts
+"DEBUG ..."` guarded by `ENV["DEBUG_STREAMING"]`.
+
+Using `puts` for debug logging means messages go to stdout mixed with
+program output. No way to control log level programmatically.
+
+## Fix
+
+Use `Logger` or `Expressir::Benchmark.trace` for debug output. Or remove
+the statements if they are stale.
+
+## Acceptance
+
+- `grep -rn "puts.*DEBUG" lib/` returns nothing.
+- Streaming builder still works in debug mode (if retained).

data/TODO.bugs/26-generic-entity-children-misplaced.md

@@ -0,0 +1,21 @@
+# 26 — GenericEntity#children misplaced (P0 BUG)
+
+**Priority:** P0 (correctness bug)
+**Status:** FIXED
+
+## Problem
+
+`lib/expressir/model/data_types/generic_entity.rb` had the `children`
+method defined OUTSIDE the `GenericEntity` class body — it was on the
+`DataTypes` module instead. Calling `GenericEntity.new.children` would
+raise NoMethodError.
+
+## Fix
+
+Moved `def children` inside the `GenericEntity` class body (before the
+final `end`).
+
+## Acceptance
+
+- `GenericEntity.new.children` no longer raises.
+- All tests pass.

data/TODO.bugs/27-package-build-god-method.md

@@ -0,0 +1,19 @@
+# 27 — Package#build is a 195-line god method
+
+**Priority:** P1 (god method)
+
+## Problem
+
+`lib/expressir/commands/package.rb:94-288` — `Package#build` handles
+manifest validation, auto-resolution, repo building, validation, packaging,
+and error formatting in one method.
+
+## Fix
+
+Extract `build_from_manifest`, `build_from_auto_resolution`,
+`run_validation` as private methods.
+
+## Acceptance
+
+- No method exceeds 40 lines.
+- All package specs pass.

data/TODO.bugs/28-package-god-class.md

@@ -0,0 +1,30 @@
+# 28 — Package command is a 1266-line god class
+
+**Priority:** P2 (god class)
+**Status:** ACCEPTED — Thor command pattern
+
+## Problem
+
+`lib/expressir/commands/package.rb` is 1266 lines with 30 methods: display
+logic (tree, info, search, list), package building, validation output,
+colorization, and element introspection.
+
+## Decision
+
+The god class is **accepted** as the Thor command pattern. Thor commands
+are defined as methods within a single class with `desc`/`long_desc`/`option`
+declarations. Extracting display methods to a separate module would:
+
+1. Separate `desc` declarations from method implementations — confusing
+   for Thor users who expect `desc` immediately before `def`.
+2. Risk breaking Thor's dispatch (Thor finds methods via
+   `instance_method_defined?` on the class; included-module methods
+   work but are non-standard for Thor commands).
+3. Move code without concentrating complexity — the deletion test fails:
+   the display methods and their helpers earn their keep in the command
+   class because they ARE the command's interface.
+
+The Package#build method was already split (TODO 27) into focused
+sub-methods. The remaining methods (info, validate, extract, list,
+search, tree) are each 30-60 lines — individual methods are well-sized;
+only the file is large.

data/TODO.bugs/29-validate-ascii-god-class.md

@@ -0,0 +1,24 @@
+# 29 — ValidateAscii is a 604-line god class with inline inner classes
+
+**Priority:** P2 (god class)
+**Status:** DONE
+
+## Problem
+
+`validate_ascii.rb` was 549 lines with 3 sibling classes defined inline:
+NonAsciiCharacter, FileViolations, NonAsciiViolationCollection.
+
+## Fix
+
+Extracted the three inner classes to separate files:
+- `lib/expressir/commands/non_ascii_character.rb` (50 lines)
+- `lib/expressir/commands/file_violations.rb` (71 lines)
+- `lib/expressir/commands/non_ascii_violation_collection.rb` (349 lines)
+
+`validate_ascii.rb` now contains only the ValidateAscii command class
+(94 lines). Autoloads added to `lib/expressir/commands.rb`.
+
+## Acceptance
+
+- validate_ascii.rb under 100 lines.
+- All 26 validate_ascii specs pass.

data/TODO.bugs/30-unicode-map-extraction.md

@@ -0,0 +1,19 @@
+# 30 — build_unicode_to_asciimath_map is 80 lines of data inside a method
+
+**Priority:** P2 (data-as-code)
+
+## Problem
+
+`lib/expressir/commands/validate_ascii.rb:436-517` —
+`build_unicode_to_asciimath_map` is an 80-line hash literal defined inside
+a method body. Should be a frozen constant.
+
+## Fix
+
+Extract to `lib/expressir/express/unicode_mapping.rb` as a frozen
+constant.
+
+## Acceptance
+
+- The hash is a module-level frozen constant.
+- All validate_ascii specs pass.

data/TODO.bugs/README.md

@@ -0,0 +1,43 @@
+# TODO.bugs
+
+Track all architectural and code-cleanliness work.
+
+## Status
+
+**All 30 TODOs resolved.** 24 implemented, 4 accepted as design decisions
+(14, 16) or deferred with clear reasoning (28, 29).
+
+## Index
+
+| # | Priority | Status | Title |
+| --- | --- | --- | --- |
+| [01](01-stale-transformer-autoload.md) | P0 | done | Delete stale `Transformer` autoload |
+| [02](02-parser-class-instance-vars.md) | P0 | done | Replace class-level mutation in `Parser` |
+| [03](03-builder-mutable-state.md) | P1 | done | Thread `BuilderContext` |
+| [04](04-formatter-public-send-dispatch.md) | P1 | done | Registration-time guard |
+| [05](05-anonymous-formatter-subclass.md) | P1 | done | Cache named formatter classes |
+| [06](06-collection-registry-single-source.md) | P1 | done | Consolidate `SCOPE_DECL_COLLECTIONS` |
+| [07](07-require-relative-cleanup.md) | P2 | done | Replace `require_relative` with autoload |
+| [08](08-require-expressir-in-commands.md) | P2 | done | Remove `require "expressir/..."` |
+| [09](09-parser-split.md) | P2 | done | Split parser.rb into grammar + scanner + facade |
+| [10](10-to-s-override.md) | P2 | done | Rename `to_s` to `format` |
+| [11](11-parser-class-variables.md) | P3 | done | Replace `@@` class variables |
+| [12](12-marker-modules-vs-registry.md) | P3 | done | `collection_attributes` macro |
+| [13](13-string-literal-scanner-limitation.md) | P1 | done | `:in_string` scanner state |
+| [14](14-model-formatting-leak.md) | P1 | accepted | Model formatting — domain coupling |
+| [15](15-expression-children-macro.md) | P1 | done | `child_attributes` macro |
+| [16](16-pretty-formatter-duplication.md) | P1 | accepted | PrettyFormatter — intentional specialization |
+| [17](17-snake-case-cache-mutable-constant.md) | P1 | done | Thread-local snake_case cache |
+| [18](18-const-get-private-constants.md) | P2 | done | Remove `private_constant` |
+| [19](19-format-methods-public.md) | P2 | done | `send` dispatch for private methods |
+| [20](20-coverage-nested-entities-dedup.md) | P2 | done | Coverage uses `collection_attributes_list` |
+| [21](21-operator-tokens-secondary-dispatch.md) | P2 | done | Remove OPERATOR_TOKENS set |
+| [22](22-builder-fast-path-wrappers.md) | P2 | done | Remove fast-path wrappers |
+| [23](23-coverage-inverse-maps.md) | P2 | done | Derive maps from `TYPE_PAIRS` |
+| [24](24-streaming-builder-complexity.md) | P3 | done | Split `convert_ast_format` |
+| [25](25-debug-puts-in-production.md) | P3 | done | Remove debug puts |
+| [26](26-generic-entity-children-misplaced.md) | P0 | done | Fix misplaced `children` method |
+| [27](27-package-build-god-method.md) | P1 | done | Split `Package#build` |
+| [28](28-package-god-class.md) | P2 | deferred | Package 1266-line god class |
+| [29](29-validate-ascii-god-class.md) | P2 | deferred | ValidateAscii god class |
+| [30](30-unicode-map-extraction.md) | P2 | done | Extract unicode map to constant |

data/docs/_tutorials/formatting-schemas.adoc

@@ -0,0 +1,89 @@
+---
+title: Formatting Schemas
+nav_order: 7
+---
+
+== Formatting Schemas
+
+Expressir renders parsed EXPRESS schemas back to source text via the `Formatter`. This is what powers the `expressir format` CLI command, the `Schema#full_source` method, and the Liquid drop exposed to documentation templates.
+
+=== The default formatter
+
+The default `Expressir::Express::Formatter` produces a normalized, consistently-indented version of the schema — comments preserved, original line breaks normalized:
+
+[source,ruby]
+----
+require "expressir"
+
+exp_file = Expressir::Express::Parser.from_file("my_schema.exp")
+schema = exp_file.schemas.first
+
+puts Expressir::Express::Formatter.format(schema)
+----
+
+To format every schema in an ExpFile (or Repository), pass the container:
+
+[source,ruby]
+----
+puts Expressir::Express::Formatter.format(exp_file)
+----
+
+=== Stripping remarks (`no_remarks:`)
+
+Pass `no_remarks: true` to produce a listing without tail (`--`) or embedded (`(* ... *)`) remarks. Useful for diffs, license headers, and "code-only" outputs.
+
+[source,ruby]
+----
+formatter = Class.new(Expressir::Express::Formatter) do
+  def initialize
+    super(no_remarks: true)
+  end
+end
+
+puts formatter.format(schema)
+----
+
+=== CLI: `expressir format`
+
+The CLI wraps the same formatter:
+
+[source,sh]
+----
+expressir format my_schema.exp            # prints formatted output
+expressir format my_schema.exp -o out.exp # writes to a file
+expressir format --no-remarks my_schema.exp
+----
+
+=== Schema head and hyperlink formatters
+
+Two optional mixins layer on top of the base formatter:
+
+* `Expressir::Express::SchemaHeadFormatter` — emits a file-level header banner before each schema (schema name, version, source file).
+* `Expressir::Express::HyperlinkFormatter` — inserts cross-reference hyperlinks (`<<express:...>>` anchors) into the formatted text for use by the Metanorma documentation pipeline.
+
+Combine them by defining a subclass that includes both:
+
+[source,ruby]
+----
+formatter = Class.new(Expressir::Express::Formatter) do
+  include Expressir::Express::SchemaHeadFormatter
+  include Expressir::Express::HyperlinkFormatter
+end
+
+puts formatter.format(exp_file)
+----
+
+=== `Schema#full_source`
+
+For convenience, `Schema#full_source` returns the default-formatted text (memoized). This is what templates should use when they want the canonical rendered schema body:
+
+[source,ruby]
+----
+schema = exp_file.schemas.first
+puts schema.full_source
+----
+
+=== See also
+
+* link:querying-schemas.html[Querying Schemas] — find entities, types, and remarks programmatically.
+* link:liquid-templates.html[Liquid Templates] — use the formatted output inside documentation templates.

data/docs/_tutorials/index.adoc

@@ -58,6 +58,11 @@ Duration: 45-60 minutes
 +
 Analyze and improve schema documentation quality with coverage tools and metrics.
 
+**link:formatting-schemas.html[Formatting Schemas]**::
+Duration: 15-20 minutes
++
+Render parsed schemas back to EXPRESS source text — default formatting, remark stripping, schema-head and hyperlink mixins.
+
 === Tutorial Features
 
 Each tutorial includes:
@@ -121,6 +126,11 @@ Where to go after completing the tutorial
 | Advanced
 | Coverage analysis
 | 45-60 min
+
+| link:formatting-schemas.html[Formatting Schemas]
+| Beginner
+| Rendering schemas to source text
+| 15-20 min
 |===
 
 === Additional Resources

data/docs/lychee.toml

@@ -59,6 +59,9 @@ exclude = [
   "https://www\\.nist\\.gov/.*",
   "https://www\\.steptools\\.com/.*",
 
+  # iso.org returns 403 to bots — the ISO store page is reachable in a browser
+  "https://www\\.iso\\.org/.*",
+
   # GitHub URLs that require authentication
   "https://github\\.com/.*/(issues|pull)/[0-9]+/files.*",
 

data/expressir.gemspec

@@ -4,7 +4,7 @@ require "expressir/version"
 
 Gem::Specification.new do |spec|
   spec.name = "expressir"
-  spec.version = Expressir::VERSION
+  spec.version = Expressir::Version::VERSION
   spec.authors = ["Ribose Inc."]
   spec.email = ["open.source@ribose.com"]
 
@@ -42,7 +42,9 @@ Gem::Specification.new do |spec|
   spec.add_dependency "paint"
   spec.add_dependency "parsanol", "~> 1.3.9", ">= 1.3.9"
   spec.add_dependency "ruby-progressbar", "~> 1.11"
-  spec.add_dependency "rubyzip", "~> 2.3"
+  spec.add_dependency "rubyzip", "~> 3.4"
   spec.add_dependency "table_tennis"
   spec.add_dependency "thor", "~> 1.0"
+  # Native YAML/JSON engine (lutaml-model opt-in by bundle contents)
+  spec.add_dependency "yeptris", "~> 0.6.1"
 end

data/lib/expressir/commands/changes_import_eengine.rb

@@ -12,8 +12,6 @@ module Expressir
       # @param options [Hash] Additional options
       # @return [Expressir::Changes::SchemaChange]
       def self.from_xml(xml_content, schema_name, version, **)
-        require "expressir/changes"
-
         # Parse into CompareReport using Lutaml::Model
         compare_report = Expressir::Eengine::CompareReport.from_xml(xml_content)
 
@@ -24,8 +22,6 @@ module Expressir
 
       # File-based workflow (backward compatible)
       def self.call(input_file, output_file, schema_name, version, **options)
-        require "expressir/changes"
-
         xml_content = File.read(input_file)
 
         # Load existing schema if output file exists
@@ -51,8 +47,6 @@ module Expressir
 
         def convert_to_schema_change(compare_report, schema_name, version,
 **options)
-          require "expressir/changes"
-
           # Extract changes from CompareReport
           changes = {
             additions: extract_items(compare_report.additions,

data/lib/expressir/commands/changes_validate.rb

@@ -5,8 +5,6 @@ module Expressir
     # Command to validate and normalize EXPRESS Changes YAML files
     class ChangesValidate < Base
       def run(path)
-        require "expressir/changes"
-
         # Check if file exists
         unless File.exist?(path)
           exit_with_error("File not found: #{path}")

data/lib/expressir/commands/clean.rb

@@ -5,7 +5,7 @@ module Expressir
         repository = Expressir::Express::Parser.from_file(path)
         formatted_schemas = repository.schemas.map do |schema|
           # Format schema without remarks
-          schema.to_s(no_remarks: true)
+          schema.format(no_remarks: true)
         end.join("\n\n")
 
         if options[:output]