expressir 2.4.0 → 2.4.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (129) hide show
  1. checksums.yaml +4 -4
  2. data/TODO.bugs/01-stale-transformer-autoload.md +39 -0
  3. data/TODO.bugs/02-parser-class-instance-vars.md +36 -0
  4. data/TODO.bugs/03-builder-mutable-state.md +43 -0
  5. data/TODO.bugs/04-formatter-public-send-dispatch.md +53 -0
  6. data/TODO.bugs/05-anonymous-formatter-subclass.md +45 -0
  7. data/TODO.bugs/06-collection-registry-single-source.md +53 -0
  8. data/TODO.bugs/07-require-relative-cleanup.md +42 -0
  9. data/TODO.bugs/08-require-expressir-in-commands.md +34 -0
  10. data/TODO.bugs/09-parser-split.md +53 -0
  11. data/TODO.bugs/10-to-s-override.md +42 -0
  12. data/TODO.bugs/11-parser-class-variables.md +39 -0
  13. data/TODO.bugs/12-marker-modules-vs-registry.md +64 -0
  14. data/TODO.bugs/13-string-literal-scanner-limitation.md +52 -0
  15. data/TODO.bugs/14-model-formatting-leak.md +30 -0
  16. data/TODO.bugs/15-expression-children-macro.md +27 -0
  17. data/TODO.bugs/16-pretty-formatter-duplication.md +28 -0
  18. data/TODO.bugs/17-snake-case-cache-mutable-constant.md +28 -0
  19. data/TODO.bugs/18-const-get-private-constants.md +30 -0
  20. data/TODO.bugs/19-format-methods-public.md +22 -0
  21. data/TODO.bugs/20-coverage-nested-entities-dedup.md +20 -0
  22. data/TODO.bugs/21-operator-tokens-secondary-dispatch.md +21 -0
  23. data/TODO.bugs/22-builder-fast-path-wrappers.md +32 -0
  24. data/TODO.bugs/23-coverage-inverse-maps.md +21 -0
  25. data/TODO.bugs/24-streaming-builder-complexity.md +19 -0
  26. data/TODO.bugs/25-debug-puts-in-production.md +21 -0
  27. data/TODO.bugs/26-generic-entity-children-misplaced.md +21 -0
  28. data/TODO.bugs/27-package-build-god-method.md +19 -0
  29. data/TODO.bugs/28-package-god-class.md +30 -0
  30. data/TODO.bugs/29-validate-ascii-god-class.md +24 -0
  31. data/TODO.bugs/30-unicode-map-extraction.md +19 -0
  32. data/TODO.bugs/README.md +43 -0
  33. data/TODO.max-perf/01-restore-ci-green.md +29 -0
  34. data/TODO.max-perf/02-streaming-parse-path.md +31 -0
  35. data/TODO.max-perf/03-cli-parallel-opt-in.md +27 -0
  36. data/TODO.max-perf/04-benchmark-harness.md +28 -0
  37. data/TODO.max-perf/05-parallel-fidelity-specs.md +22 -0
  38. data/TODO.max-perf/06-builder-cpu-audit.md +41 -0
  39. data/TODO.max-perf/07-upstream-parsanol-roadmap.md +27 -0
  40. data/TODO.max-perf/08-builder-build-perf.md +45 -0
  41. data/TODO.max-perf/09-grammar-cold-start.md +25 -0
  42. data/TODO.max-perf/10-parser-facade-hygiene.md +23 -0
  43. data/TODO.max-perf/11-ci-green-closeout.md +25 -0
  44. data/TODO.max-perf/12-require-boot-profile.md +25 -0
  45. data/TODO.max-perf/13-key-conversion-specs.md +26 -0
  46. data/TODO.max-perf/14-builder-call-handler-audit.md +28 -0
  47. data/benchmark/srl_benchmark.rb +76 -17
  48. data/docs/_tutorials/formatting-schemas.adoc +89 -0
  49. data/docs/_tutorials/index.adoc +10 -0
  50. data/docs/lychee.toml +3 -0
  51. data/expressir.gemspec +4 -2
  52. data/lib/expressir/cli.rb +3 -0
  53. data/lib/expressir/commands/changes_import_eengine.rb +0 -6
  54. data/lib/expressir/commands/changes_validate.rb +0 -2
  55. data/lib/expressir/commands/clean.rb +1 -1
  56. data/lib/expressir/commands/coverage.rb +6 -2
  57. data/lib/expressir/commands/file_violations.rb +70 -0
  58. data/lib/expressir/commands/format.rb +1 -1
  59. data/lib/expressir/commands/non_ascii_character.rb +49 -0
  60. data/lib/expressir/commands/non_ascii_violation_collection.rb +301 -0
  61. data/lib/expressir/commands/package.rb +143 -187
  62. data/lib/expressir/commands/validate_ascii.rb +0 -510
  63. data/lib/expressir/commands/version.rb +1 -1
  64. data/lib/expressir/commands.rb +3 -0
  65. data/lib/expressir/coverage.rb +49 -117
  66. data/lib/expressir/express/ast_key_converter.rb +114 -0
  67. data/lib/expressir/express/builder.rb +71 -194
  68. data/lib/expressir/express/builder_context.rb +22 -0
  69. data/lib/expressir/express/builders/expression_builder.rb +16 -16
  70. data/lib/expressir/express/cache.rb +35 -8
  71. data/lib/expressir/express/error.rb +24 -0
  72. data/lib/expressir/express/formatter.rb +33 -18
  73. data/lib/expressir/express/formatters/declarations_formatter.rb +24 -2
  74. data/lib/expressir/express/formatters/remark_formatter.rb +75 -2
  75. data/lib/expressir/express/formatters/statements_formatter.rb +20 -3
  76. data/lib/expressir/express/grammar/parser.rb +705 -0
  77. data/lib/expressir/express/grammar.rb +11 -0
  78. data/lib/expressir/express/node_position_index.rb +215 -0
  79. data/lib/expressir/express/parallel_files.rb +229 -0
  80. data/lib/expressir/express/parser.rb +138 -950
  81. data/lib/expressir/express/pretty_formatter.rb +23 -5
  82. data/lib/expressir/express/remark_attacher.rb +497 -581
  83. data/lib/expressir/express/remark_scanner.rb +84 -19
  84. data/lib/expressir/express/schema_block_scanner.rb +137 -0
  85. data/lib/expressir/express/schema_source_formatter.rb +15 -0
  86. data/lib/expressir/express/scope_resolver.rb +223 -0
  87. data/lib/expressir/express/source_formatter.rb +15 -0
  88. data/lib/expressir/express/streaming_builder.rb +147 -176
  89. data/lib/expressir/express.rb +10 -6
  90. data/lib/expressir/model/concerns.rb +3 -0
  91. data/lib/expressir/model/data_types/generic_entity.rb +6 -6
  92. data/lib/expressir/model/declarations/entity.rb +5 -0
  93. data/lib/expressir/model/declarations/function.rb +5 -0
  94. data/lib/expressir/model/declarations/procedure.rb +5 -0
  95. data/lib/expressir/model/declarations/rule.rb +6 -0
  96. data/lib/expressir/model/declarations/schema.rb +21 -8
  97. data/lib/expressir/model/declarations/type.rb +3 -0
  98. data/lib/expressir/model/exp_file.rb +2 -0
  99. data/lib/expressir/model/expressions/aggregate_initializer.rb +1 -0
  100. data/lib/expressir/model/expressions/aggregate_initializer_item.rb +1 -0
  101. data/lib/expressir/model/expressions/binary_expression.rb +1 -0
  102. data/lib/expressir/model/expressions/entity_constructor.rb +1 -0
  103. data/lib/expressir/model/expressions/function_call.rb +1 -0
  104. data/lib/expressir/model/expressions/interval.rb +1 -0
  105. data/lib/expressir/model/expressions/query_expression.rb +2 -0
  106. data/lib/expressir/model/expressions/unary_expression.rb +1 -0
  107. data/lib/expressir/model/model_element.rb +133 -14
  108. data/lib/expressir/model/remark_format.rb +17 -0
  109. data/lib/expressir/model/remark_info.rb +31 -3
  110. data/lib/expressir/model/remark_placement.rb +29 -0
  111. data/lib/expressir/model/repository.rb +18 -5
  112. data/lib/expressir/model/statements/alias.rb +2 -0
  113. data/lib/expressir/model/statements/assignment.rb +4 -0
  114. data/lib/expressir/model/statements/case.rb +21 -0
  115. data/lib/expressir/model/statements/case_action.rb +1 -0
  116. data/lib/expressir/model/statements/compound.rb +4 -0
  117. data/lib/expressir/model/statements/escape.rb +3 -0
  118. data/lib/expressir/model/statements/if.rb +5 -0
  119. data/lib/expressir/model/statements/null.rb +3 -0
  120. data/lib/expressir/model/statements/procedure_call.rb +3 -0
  121. data/lib/expressir/model/statements/repeat.rb +3 -0
  122. data/lib/expressir/model/statements/return.rb +3 -0
  123. data/lib/expressir/model/statements/skip.rb +3 -0
  124. data/lib/expressir/model.rb +12 -3
  125. data/lib/expressir/package/builder.rb +2 -2
  126. data/lib/expressir/version.rb +6 -1
  127. data/lib/expressir.rb +56 -3
  128. metadata +81 -7
  129. data/lib/expressir/express/transformer/remark_handling.rb +0 -194
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 738451ad45c2f5e9268ef7562cbb2c667b90648f0e2df9ac409c79ceeeb6b879
4
- data.tar.gz: e4e04ff5ede476581d65882a5e8d6fefba98ced3345337673bf61366e5b96392
3
+ metadata.gz: 0f786a7123d34be8bada764d132f213c09f229d5f94046509dbdbfc37a0d4279
4
+ data.tar.gz: 9b32c54df109ca8e6945ab03d2b7e3697d2b0bce3afbb67476bfe1510e0eb2fe
5
5
  SHA512:
6
- metadata.gz: 6e9c0b2be28d8fa52f4c64f151e73c345701b2e6a118ee771f1442459c9c97bd5539bf1d799692ad065317849f9e794fb4cdc03a7d55efec1ddf627408f710de
7
- data.tar.gz: 44100a53632c094d8cd7d5da781a8f2ab63274f4fcdcad44a0aafaf2e6c94821a5eefa6263c9f4c982876f55aeb6e23f5858cb7ca78decce3489a4c00b50abea
6
+ metadata.gz: 5225e02955f94be94f71e9005330d1c58173ebd9cb5aa97956abfa9b8b8f3f2e964f25648d8074ad8ba41fb678530dad4348570ed8c697c6445224fe77c1514f
7
+ data.tar.gz: f97f4c7be51de8b6c5e4e71301555f20d05bdb268b61ef7df2e7be4bd06796af255b1f2cc362e6a25a61867fd62ab0b3610eb25683d57d6b36401c9a338598f5
@@ -0,0 +1,39 @@
1
+ # 01 — Delete stale `Transformer` autoload and legacy `transformer/remark_handling.rb`
2
+
3
+ **Priority:** P0 (load-time failure waiting to happen)
4
+
5
+ ## Problem
6
+
7
+ `lib/expressir/express.rb:28` declares:
8
+
9
+ ```ruby
10
+ autoload :Transformer, "#{__dir__}/express/transformer"
11
+ ```
12
+
13
+ But `lib/expressir/express/transformer.rb` does not exist. Only the directory
14
+ `lib/expressir/express/transformer/` exists, containing `remark_handling.rb`
15
+ (195 lines). Any code path that references `Expressir::Express::Transformer`
16
+ raises `LoadError`.
17
+
18
+ The `transformer/remark_handling.rb` module duplicates remark scanning and
19
+ attachment logic that has been superseded by `RemarkScanner`, `RemarkAttacher`,
20
+ `ScopeResolver`, and `NodePositionIndex` (PR lutaml/expressir#314). It uses
21
+ `Parsanol::Slice`-based AST walking rather than source-text scanning, and its
22
+ `attach_untagged_remarks` is explicitly stubbed as "skip for now."
23
+
24
+ ## Fix
25
+
26
+ 1. Delete `lib/expressir/express/transformer/` directory entirely.
27
+ 2. Delete the stale `autoload :Transformer` line in `lib/expressir/express.rb`.
28
+ 3. Verify no callers reference `Expressir::Express::Transformer` anywhere.
29
+
30
+ ## Files affected
31
+
32
+ - `lib/expressir/express.rb` (delete one line)
33
+ - `lib/expressir/express/transformer/remark_handling.rb` (delete file)
34
+ - `lib/expressir/express/transformer/` (delete directory)
35
+
36
+ ## Acceptance
37
+
38
+ - `grep -rn "Expressir::Express::Transformer" lib/ spec/` returns nothing.
39
+ - Full test suite passes.
@@ -0,0 +1,36 @@
1
+ # 02 — Replace class-level `@exp_file`/`@repository` mutation in `Parser` with locals
2
+
3
+ **Priority:** P0 (state leak across sequential calls)
4
+
5
+ ## Problem
6
+
7
+ `lib/expressir/express/parser.rb` outer `Parser` class stores parse results
8
+ as class-level instance variables:
9
+
10
+ - Line 723: `@exp_file = ::Expressir::Express::Builder.build_with_remarks(...)`
11
+ - Line 735: `@resolve_references_model_visitor = ResolveReferencesModelVisitor.new`
12
+ - Line 774: `@repository = Model::Repository.new(files: all_exp_files)`
13
+ - Line 778: `@resolve_references_model_visitor = ResolveReferencesModelVisitor.new`
14
+
15
+ These are `class << self` instance variables. Sequential or concurrent calls
16
+ to `Parser.from_file` / `Parser.from_files` overwrite each other's state.
17
+ The visitor is only used inside the method body — there is no reason to
18
+ retain it on the class.
19
+
20
+ ## Fix
21
+
22
+ Return local variables from `from_file` and `from_files` instead of mutating
23
+ class state. Replace `@exp_file` with `exp_file`, `@repository` with
24
+ `repository`, `@resolve_references_model_visitor` with `visitor`. Delete any
25
+ `attr_reader` declarations that expose these.
26
+
27
+ ## Files affected
28
+
29
+ - `lib/expressir/express/parser.rb` (rewrite `from_file` and `from_files`)
30
+
31
+ ## Acceptance
32
+
33
+ - `grep -n "@exp_file\|@repository\|@resolve_references" lib/expressir/express/parser.rb`
34
+ returns nothing.
35
+ - All parser specs pass.
36
+ - `Parser.from_file` returns the same value it did before.
@@ -0,0 +1,43 @@
1
+ # 03 — Thread `BuilderContext` through `Builder.build` instead of mutating `@source`/`@include_source`
2
+
3
+ **Priority:** P1 (architectural friction)
4
+
5
+ ## Problem
6
+
7
+ `lib/expressir/express/builder.rb` is a `module` (not a class) that stores
8
+ per-call state as class-level instance variables:
9
+
10
+ ```ruby
11
+ @source = source unless source.nil?
12
+ @include_source = include_source unless include_source.nil?
13
+ ```
14
+
15
+ Lines 10, 49-50, 112-113. The `unless source.nil?` guard admits that this is
16
+ fragile — recursive `build` calls pass `nil` to avoid clobbering, which creates
17
+ an implicit ordering dependency between caller and callee. The module is not
18
+ reentrant: concurrent or nested builds would corrupt state.
19
+
20
+ ## Fix
21
+
22
+ Introduce a `BuilderContext` value object that carries `source` and
23
+ `include_source`. Thread it through the recursive `build` call chain as a
24
+ keyword arg. Remove the `attr_reader :source, :include_source` on the
25
+ metaclass. Builders that need access read from the context.
26
+
27
+ Alternative considered: thread `source:` and `include_source:` as separate
28
+ keyword args through every `build_*` method. Rejected because the arg list
29
+ grows with every new context field, and the two values always travel together.
30
+
31
+ ## Files affected
32
+
33
+ - `lib/expressir/express/builder.rb` (rewrite)
34
+ - `lib/expressir/express/builders/*.rb` (update call sites — most already
35
+ receive `source` via `slice` or `node`; verify each)
36
+ - New file `lib/expressir/express/builder_context.rb`
37
+
38
+ ## Acceptance
39
+
40
+ - `grep -n "@source\|@include_source" lib/expressir/express/builder.rb`
41
+ returns nothing.
42
+ - All parser and builder specs pass.
43
+ - `Builder.build_with_remarks` is reentrant (a fresh `BuilderContext` per call).
@@ -0,0 +1,53 @@
1
+ # 04 — Replace `public_send(handler, node)` in `Formatter#format` with conventional override
2
+
3
+ **Priority:** P1 (architectural friction)
4
+
5
+ ## Problem
6
+
7
+ `lib/expressir/express/formatter.rb:85` dispatches formatting by looking up a
8
+ method name in a class-level registry and calling it via `public_send`:
9
+
10
+ ```ruby
11
+ def format(node)
12
+ return "" if node.nil?
13
+ handler = self.class.format_registry[node.class]
14
+ return public_send(handler, node) if handler
15
+ warn "#{node.class.name} format not implemented"
16
+ ""
17
+ end
18
+ ```
19
+
20
+ Issues:
21
+ - `public_send` forces the format methods to be public on the including class,
22
+ defeating encapsulation.
23
+ - A typo in a registered method name surfaces only at runtime, when that model
24
+ type is formatted.
25
+ - IDE navigation and static analysis are defeated.
26
+ - The `format_registry` hash is a parallel dispatch table that must be kept in
27
+ sync with the actual method definitions.
28
+
29
+ ## Fix
30
+
31
+ Replace the registry dispatch with conventional Ruby method override. Each
32
+ formatter module defines `format_<node_class>(node)` methods, and the base
33
+ `Formatter#format` dispatches via a `case node` statement. Format methods
34
+ remain private.
35
+
36
+ This is a larger change touching 9 formatter modules. If full restructure is
37
+ too risky for a single PR, a smaller win is to add a registration-time guard
38
+ in `Formatter.included` that asserts the registered method exists via
39
+ `method_defined?` — at least typos fail at load time.
40
+
41
+ ## Files affected
42
+
43
+ - `lib/expressir/express/formatter.rb` (rewrite `format` and the registration hook)
44
+ - `lib/expressir/express/formatters/*.rb` (8 modules — update registration calls)
45
+ - `lib/expressir/express/pretty_formatter.rb`
46
+ - `lib/expressir/express/schema_head_formatter.rb`
47
+ - `lib/expressir/express/hyperlink_formatter.rb`
48
+
49
+ ## Acceptance
50
+
51
+ - `grep -n "public_send" lib/expressir/express/formatter.rb` returns nothing.
52
+ - All formatter specs pass.
53
+ - A typo in a registered method name fails at load time, not at runtime.
@@ -0,0 +1,45 @@
1
+ # 05 — Cache the `Class.new(Formatter) { include HyperlinkFormatter }` in `ModelElement#source` as a constant
2
+
3
+ **Priority:** P1 (wasteful per-call allocation)
4
+
5
+ ## Problem
6
+
7
+ `lib/expressir/model/model_element.rb:100-105`:
8
+
9
+ ```ruby
10
+ def source
11
+ formatter = Class.new(Expressir::Express::Formatter) do
12
+ include Expressir::Express::HyperlinkFormatter
13
+ end
14
+ formatter.format(self)
15
+ end
16
+ ```
17
+
18
+ Every call to `source` on any model element allocates a fresh anonymous class,
19
+ runs the `self.included` hook, builds the `format_registry` hash, instantiates
20
+ the class, and formats the tree. The result is identical across calls.
21
+
22
+ `source` is a getter that may be invoked frequently during serialization and
23
+ inspection. Wasteful.
24
+
25
+ ## Fix
26
+
27
+ Define a single named class once, e.g. `Expressir::Express::SourceFormatter`,
28
+ that subclasses `Formatter` and includes `HyperlinkFormatter`. `ModelElement#source`
29
+ instantiates it (cheap) instead of allocating a new class.
30
+
31
+ The class lives in `lib/expressir/express/source_formatter.rb` (autoloaded),
32
+ breaking the model→express dependency at the source-text level — but the model
33
+ already depends on express for serialization, so this just makes it explicit.
34
+
35
+ ## Files affected
36
+
37
+ - `lib/expressir/model/model_element.rb` (use the constant)
38
+ - New file `lib/expressir/express/source_formatter.rb`
39
+ - `lib/expressir/express.rb` (add autoload)
40
+
41
+ ## Acceptance
42
+
43
+ - `grep -n "Class.new" lib/expressir/model/model_element.rb` returns nothing.
44
+ - All model_element specs pass.
45
+ - `ModelElement#source` no longer allocates a class per call (benchmark optional).
@@ -0,0 +1,53 @@
1
+ # 06 — Consolidate `SCHEMA_DECL_COLLECTIONS` into `COLLECTION_REGISTRY`
2
+
3
+ **Priority:** P1 (DRY violation — two sources of truth for the same fact)
4
+
5
+ ## Problem
6
+
7
+ `lib/expressir/express/scope_resolver.rb:46` declares:
8
+
9
+ ```ruby
10
+ SCHEMA_DECL_COLLECTIONS = %i[functions procedures rules entities types].freeze
11
+ ```
12
+
13
+ This is a hard-coded subset of what `NodePositionIndex::COLLECTION_REGISTRY`
14
+ already knows for `Model::Declarations::Schema`:
15
+
16
+ ```ruby
17
+ Model::Declarations::Schema => %i[
18
+ constants types entities subtype_constraints
19
+ functions rules procedures remark_items
20
+ ]
21
+ ```
22
+
23
+ `ScopeResolver` only needs the "named scope declarations" subset (it skips
24
+ `constants`, `subtype_constraints`, `remark_items` because they aren't scopes).
25
+ But the subset is implicit — a change to Schema's collection attrs in
26
+ `COLLECTION_REGISTRY` will not propagate.
27
+
28
+ ## Fix
29
+
30
+ Define the subset once, either:
31
+ - (a) Filter `COLLECTION_REGISTRY[Model::Declarations::Schema]` against a
32
+ constant of scope-capable attr names, OR
33
+ - (b) Lift `SCHEMA_DECL_COLLECTIONS` to a shared location and document it as
34
+ "the scope-capable subset of Schema's collections."
35
+
36
+ Option (b) is simpler and more honest about the intent. Move
37
+ `SCHEMA_DECL_COLLECTIONS` to `Model::Declarations::Schema` as a class-level
38
+ constant, since the subset is a fact about Schema.
39
+
40
+ ## Files affected
41
+
42
+ - `lib/expressir/express/scope_resolver.rb` (reference the model constant)
43
+ - `lib/expressir/model/declarations/schema.rb` (add `SCOPE_DECL_COLLECTIONS`)
44
+ - `spec/expressir/express/scope_resolver_spec.rb` (add a spec asserting the
45
+ subset is a subset of `COLLECTION_REGISTRY`)
46
+
47
+ ## Acceptance
48
+
49
+ - `ScopeResolver` no longer hard-codes a list that duplicates `COLLECTION_REGISTRY`.
50
+ - All scope_resolver specs pass.
51
+ - A new spec asserts the invariant: every entry in
52
+ `Schema::SCOPE_DECL_COLLECTIONS` is also in
53
+ `NodePositionIndex::COLLECTION_REGISTRY[Schema]`.
@@ -0,0 +1,42 @@
1
+ # 07 — Replace `require_relative` for `version` and `errors` in `lib/expressir.rb` with autoload
2
+
3
+ **Priority:** P2 (cleanup; project rule violation)
4
+
5
+ ## Problem
6
+
7
+ `lib/expressir.rb:1-2`:
8
+
9
+ ```ruby
10
+ require_relative "expressir/version"
11
+ require_relative "expressir/errors"
12
+ ```
13
+
14
+ Project rule: never use `require_relative` for internal library code — use
15
+ Ruby `autoload` (defined in the immediate parent namespace's file).
16
+
17
+ `lib/expressir/express.rb:33` also has `require_relative "express/builder_registry"`
18
+ with a comment about eager-load for self-registration. The eager load is
19
+ justified for the builder self-registration pattern, but the rule still
20
+ applies — convert to autoload and require explicitly only in the entry point
21
+ that triggers registration (or restructure to lazy registration).
22
+
23
+ ## Fix
24
+
25
+ 1. Convert `version` and `errors` to autoload entries in `lib/expressir.rb`.
26
+ 2. For `builder_registry`: convert to autoload, and explicitly trigger
27
+ registration in `Builder.build_with_remarks` (the only entry point that
28
+ needs builders loaded). Move the eager-load justification into a comment
29
+ on that explicit require, or restructure builders to self-register lazily
30
+ via `Kernel.singleton_class.include` at the top of each builder file.
31
+
32
+ ## Files affected
33
+
34
+ - `lib/expressir.rb` (use autoload for Version, Errors)
35
+ - `lib/expressir/express.rb` (autoload BuilderRegistry; remove `require_relative`)
36
+
37
+ ## Acceptance
38
+
39
+ - `grep -rn "require_relative" lib/` returns only justified entries (ideally zero).
40
+ - `Expressir::VERSION` still resolves.
41
+ - `Expressir::Errors::*` still resolves.
42
+ - Builder registration still happens before any `Builder.build_with_remarks` call.
@@ -0,0 +1,34 @@
1
+ # 08 — Remove `require "expressir/changes"` from `commands/*.rb`
2
+
3
+ **Priority:** P2 (cleanup; project rule violation)
4
+
5
+ ## Problem
6
+
7
+ Four call sites inside `lib/expressir/commands/`:
8
+
9
+ - `changes_import_eengine.rb:15, 27, 54`
10
+ - `changes_validate.rb:8`
11
+
12
+ All use `require "expressir/changes"` instead of relying on autoload.
13
+
14
+ The lazy requires inside method bodies suggest an attempt to defer loading,
15
+ but autoload already provides that — and the project rule says no `require`
16
+ with code within our library.
17
+
18
+ ## Fix
19
+
20
+ Replace `require "expressir/changes"` with a direct reference to the
21
+ `Expressir::Changes` constant (which triggers autoload). If the require is
22
+ truly defensive (e.g., guarding against a load-order bug), delete it; the
23
+ autoload on `Expressir::Changes` in `lib/expressir.rb` covers it.
24
+
25
+ ## Files affected
26
+
27
+ - `lib/expressir/commands/changes_import_eengine.rb` (delete 3 requires)
28
+ - `lib/expressir/commands/changes_validate.rb` (delete 1 require)
29
+
30
+ ## Acceptance
31
+
32
+ - `grep -rn 'require "expressir/' lib/` returns nothing.
33
+ - All command specs pass.
34
+ - `Expressir::Commands::ChangesImportEengine` and `ChangesValidate` still work.
@@ -0,0 +1,53 @@
1
+ # 09 — Split `parser.rb` (1048 lines) into grammar + I/O service + schema-block scanner
2
+
3
+ **Priority:** P2 (architectural friction — file too large to navigate)
4
+
5
+ ## Problem
6
+
7
+ `lib/expressir/express/parser.rb` is 1048 lines with three distinct
8
+ responsibilities crammed into one file:
9
+
10
+ - Lines 5-688: inner `Parser < Parsanol::Parser` class — 250 grammar
11
+ `rule()` definitions plus keyword helpers.
12
+ - Lines 690-920: outer `Parser` class — `from_file`, `from_files`,
13
+ `from_exp`, `from_exp_streaming_builder`, `from_exp_streaming`,
14
+ `extract_schema_blocks` (file I/O orchestration).
15
+ - Lines 921-1046: `parse_schema_block`, `skip_spaces`, `skip_ws_and_comments`
16
+ — a hand-rolled state machine for splitting multi-schema files.
17
+
18
+ The grammar rules, I/O orchestration, and schema-block scanning are three
19
+ distinct concerns. The current nested `Expressir::Express::Parser::Parser`
20
+ namespacing is also confusing — outer class wraps inner class with the same
21
+ name.
22
+
23
+ ## Fix
24
+
25
+ Split into three files:
26
+
27
+ 1. `lib/expressir/express/parser/grammar.rb` — the inner grammar class as
28
+ `Expressir::Express::Grammar::Parser` (renamed).
29
+ 2. `lib/expressir/express/parser/service.rb` — the outer class as
30
+ `Expressir::Express::ParserService` with the I/O methods.
31
+ 3. `lib/expressir/express/parser/schema_block_scanner.rb` — the
32
+ `parse_schema_block` / `skip_spaces` / `skip_ws_and_comments` helpers as
33
+ `Expressir::Express::SchemaBlockScanner`.
34
+
35
+ Keep `Expressir::Express::Parser` as a thin facade that delegates to
36
+ `ParserService`, for backward compatibility with all existing call sites.
37
+
38
+ This is a large mechanical change. Run the full test suite after each file
39
+ move to catch missing references.
40
+
41
+ ## Files affected
42
+
43
+ - `lib/expressir/express/parser.rb` (becomes a thin facade)
44
+ - New: `lib/expressir/express/parser/grammar.rb`
45
+ - New: `lib/expressir/express/parser/service.rb`
46
+ - New: `lib/expressir/express/parser/schema_block_scanner.rb`
47
+ - `lib/expressir/express.rb` (update autoloads)
48
+
49
+ ## Acceptance
50
+
51
+ - No file under `lib/expressir/express/` exceeds ~400 lines.
52
+ - All parser specs pass without modification.
53
+ - `Expressir::Express::Parser.from_file`, `from_exp`, etc. still work.
@@ -0,0 +1,42 @@
1
+ # 10 — Rename `ModelElement#to_s(no_remarks:, formatter:)` to `format`
2
+
3
+ **Priority:** P2 (surprising API; violates Liskov with core `to_s`)
4
+
5
+ ## Problem
6
+
7
+ `lib/expressir/model/model_element.rb:191`:
8
+
9
+ ```ruby
10
+ def to_s(no_remarks: false, formatter: nil)
11
+ ...
12
+ end
13
+ ```
14
+
15
+ Ruby's core `Object#to_s` takes zero arguments. Overriding it with keyword
16
+ args means `node.to_s` works (defaults kick in), but `node.to_s(no_remarks: true)`
17
+ is surprising — callers expect `to_s` to be a no-arg method.
18
+
19
+ The implementation also mutates the formatter after construction
20
+ (`f.no_remarks = no_remarks` at line 194), which is a post-construction
21
+ mutation of an option that should have been set at construction.
22
+
23
+ ## Fix
24
+
25
+ 1. Rename `to_s` to `format(no_remarks: false, formatter: nil)`.
26
+ 2. Pass `no_remarks` through the formatter constructor instead of mutating
27
+ after the fact.
28
+ 3. Update all callers (search for `.to_s(no_remarks:`).
29
+ 4. Optionally: keep a no-arg `to_s` that delegates to `format` for any code
30
+ that expects string conversion (e.g. string interpolation).
31
+
32
+ ## Files affected
33
+
34
+ - `lib/expressir/model/model_element.rb` (rename, update body)
35
+ - All callers — `grep -rn "\.to_s(no_remarks:"`
36
+
37
+ ## Acceptance
38
+
39
+ - `grep -rn "def to_s" lib/expressir/model/` returns nothing (or returns a
40
+ thin no-arg delegator).
41
+ - All specs pass.
42
+ - Callers use `.format(no_remarks: ...)` instead of `.to_s(no_remarks: ...)`.
@@ -0,0 +1,39 @@
1
+ # 11 — Replace `@@cached_*` in `Parser` inner class with class instance variables
2
+
3
+ **Priority:** P3 (Ruby pitfall — `@@` vars shared across hierarchy)
4
+
5
+ ## Problem
6
+
7
+ `lib/expressir/express/parser.rb:10-18`:
8
+
9
+ ```ruby
10
+ @@cached_parser = nil
11
+ @@cached_syntax = nil
12
+ @@cached_exp_file_parser = nil
13
+ @@cached_grammar_ast = nil
14
+ @@parser_mutex = Mutex.new
15
+ ```
16
+
17
+ Ruby `@@` class variables are shared across the entire class hierarchy, not
18
+ scoped to the defining class. If `Parser` is ever subclassed, the subclass
19
+ shares these caches — a footgun.
20
+
21
+ Currently not a bug (no subclassing), but fragile. Idiomatic Ruby uses
22
+ class instance variables (`@cached_parser` on the metaclass via `class << self`).
23
+
24
+ ## Fix
25
+
26
+ Convert each `@@cached_*` to a class instance variable accessed via
27
+ `class << self` blocks or singleton method definitions. The mutex can stay
28
+ as `@@parser_mutex` since mutexes are typically process-global, but it's
29
+ cleaner to make it a class instance variable too.
30
+
31
+ ## Files affected
32
+
33
+ - `lib/expressir/express/parser.rb` (or the split grammar.rb after TODO 09)
34
+
35
+ ## Acceptance
36
+
37
+ - `grep -n "@@" lib/expressir/express/parser.rb` returns nothing.
38
+ - All parser specs pass.
39
+ - Cache behavior unchanged.
@@ -0,0 +1,64 @@
1
+ # 12 — Generate `COLLECTION_REGISTRY` entries from marker-module declarations
2
+
3
+ **Priority:** P3 (architectural; large change, defer unless time allows)
4
+
5
+ ## Problem
6
+
7
+ `lib/expressir/model/concerns.rb` defines six marker modules
8
+ (`HasId`, `HasRemarkItems`, `HasRemarks`, `ScopeContainer`,
9
+ `HasInformalPropositions`, `HasWhereRules`) used for `is_a?` type checks.
10
+
11
+ `lib/expressir/express/node_position_index.rb` declares
12
+ `COLLECTION_REGISTRY` — a hash mapping the same model classes to their
13
+ collection attribute names.
14
+
15
+ Two parallel type-classification systems must be kept in sync. Adding a new
16
+ model type requires updating both. This is the same architectural smell that
17
+ TODO 06 addresses for a subset (Schema), generalized.
18
+
19
+ ## Fix
20
+
21
+ Two options:
22
+
23
+ **Option A (declarative):** Each model class declares its own collection
24
+ attributes via a class-level macro, e.g.
25
+
26
+ ```ruby
27
+ class Schema
28
+ collection_attributes :constants, :types, :entities, ...
29
+ end
30
+ ```
31
+
32
+ `COLLECTION_REGISTRY` becomes `Model.klasses.map { |k| [k, k.collection_attributes] }.to_h`.
33
+
34
+ **Option B (derived):** The marker modules carry the attribute list:
35
+
36
+ ```ruby
37
+ module HasRemarkItems
38
+ def self.collection_attributes = [:remark_items]
39
+ end
40
+ ```
41
+
42
+ `COLLECTION_REGISTRY` is derived from `Model::Concerns.constants.flat_map { |c| ... }`.
43
+
44
+ Option A is cleaner. Either way, the duplication is eliminated.
45
+
46
+ ## Out of scope for this PR
47
+
48
+ This is a large architectural change that touches every model class. It
49
+ warrants its own grill session to pick between Option A and Option B and
50
+ to migrate incrementally.
51
+
52
+ ## Files affected
53
+
54
+ - `lib/expressir/model/declarations/*.rb` (add `collection_attributes` macro)
55
+ - `lib/expressir/express/node_position_index.rb` (derive `COLLECTION_REGISTRY`)
56
+ - `lib/expressir/express/remark_attacher.rb` (update reference)
57
+ - `lib/expressir/express/scope_resolver.rb` (update reference)
58
+ - All model class specs
59
+
60
+ ## Acceptance
61
+
62
+ - `COLLECTION_REGISTRY` is derived, not hand-maintained.
63
+ - Adding a new model class with a `collection_attributes` macro automatically
64
+ makes it appear in the registry.
@@ -0,0 +1,52 @@
1
+ # 13 — Add `:in_string` state to `RemarkScanner` (issue lutaml/expressir#313)
2
+
3
+ **Priority:** P1 (correctness limitation)
4
+
5
+ ## Problem
6
+
7
+ `lib/expressir/express/remark_scanner.rb` runs a three-state machine
8
+ (`:top`, `:in_tail`, `:in_embedded`) but has no `:in_string` state. A `--`
9
+ inside an EXPRESS string literal is mistakenly treated as the start of a
10
+ tail remark.
11
+
12
+ Minimal reproduction:
13
+
14
+ ```
15
+ SCHEMA string_tail_repro;
16
+ CONSTANT
17
+ greeting : STRING := 'hello -- world';
18
+ END_CONSTANT;
19
+ END_SCHEMA;
20
+ ```
21
+
22
+ Scanner returns 1 spurious `tail` remark: `"world';"`. Expected: zero remarks.
23
+
24
+ Reference: ISO 10303-11 §7.1.6.5 (Simple string literal) — `'` ... `'`
25
+ with `''` as the escape.
26
+
27
+ Filed as issue lutaml/expressir#313.
28
+
29
+ ## Fix
30
+
31
+ Add a `:in_string` state to the state machine:
32
+
33
+ - At `:top`, when encountering `'`, enter `:in_string`.
34
+ - At `:in_string`, look for the closing `'`. Handle the `''` escape (treat
35
+ as a single literal `'` inside the string, not a closer).
36
+ - At `:in_string`, ignore `--` and `(*` — they are string content.
37
+
38
+ Encoded string literals (`.X.` ... `.X.`) per §7.1.6.6 are rarer; defer to a
39
+ follow-up if not exercised by the SRL test schemas.
40
+
41
+ ## Files affected
42
+
43
+ - `lib/expressir/express/remark_scanner.rb` (add `:in_string` state)
44
+ - `spec/expressir/express/remark_scanner_spec.rb` (add regression specs)
45
+
46
+ ## Acceptance
47
+
48
+ - Scanner returns zero remarks for `'hello -- world'`.
49
+ - Scanner handles `''` escape correctly (`'it''s'` does not close at the
50
+ middle `'`).
51
+ - All existing scanner specs still pass.
52
+ - The ISO §7.1.6.5 simple-string-literal case is covered.
@@ -0,0 +1,30 @@
1
+ # 14 — Model layer embeds EXPRESS formatting logic
2
+
3
+ **Priority:** P1 (leaky seam)
4
+ **Status:** ACCEPTED — design decision documented below
5
+
6
+ ## Problem
7
+
8
+ `ModelElement#source`, `Schema#source`, `Schema#formatted`, `Schema#full_source`
9
+ all call express-layer formatters, creating a dependency from model → express.
10
+
11
+ ## Decision
12
+
13
+ This coupling is **accepted** as a domain-level design decision. The model
14
+ IS the EXPRESS model — it is not a generic data layer that could exist
15
+ without the EXPRESS-specific formatting. The `source`/`formatted`/`full_source`
16
+ methods are domain behavior of an EXPRESS model element, not generic
17
+ serialization.
18
+
19
+ Attempting to uncouple would:
20
+ 1. Break dozens of callers (`schema.source`, `schema.formatted` are used
21
+ throughout commands, specs, and coverage code).
22
+ 2. Add indirection (a presenter/decorator layer) for no real benefit —
23
+ the model and express layers are always loaded together.
24
+ 3. Diverge from the "rich model" pattern where domain objects carry their
25
+ own domain-specific behavior.
26
+
27
+ The `@source ||=` caching pattern on Schema already provides lazy
28
+ computation: the stored source (from the parser) is returned if available;
29
+ otherwise the formatter computes it on first access. This is efficient and
30
+ semantically correct.
@@ -0,0 +1,27 @@
1
+ # 15 — EXPRESSION_CHILDREN hash should be a model macro
2
+
3
+ **Priority:** P1 (parallel registry)
4
+
5
+ ## Problem
6
+
7
+ `lib/expressir/express/remark_attacher.rb:30-45` has an
8
+ `EXPRESSION_CHILDREN` hash manually listing non-collection child attributes
9
+ for 14 expression/statement types. This is a parallel registry to the
10
+ `collection_attributes` macro and must be maintained separately.
11
+
12
+ ## Fix
13
+
14
+ Add a `child_attributes` class-level macro on `ModelElement` (similar to
15
+ `collection_attributes`) for single-value child attributes that are
16
+ ModelElements but not collections. Register them the same way.
17
+
18
+ Update each expression/statement model class to declare its child attrs.
19
+
20
+ Replace `EXPRESSION_CHILDREN[...]` lookups in RemarkAttacher with
21
+ `ModelElement.child_attributes_registry`.
22
+
23
+ ## Acceptance
24
+
25
+ - `EXPRESSION_CHILDREN` constant removed from RemarkAttacher.
26
+ - Each expression class declares its own child attributes.
27
+ - All remark specs pass.