token-resolver 1.0.0 → 1.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9aafcdcd6d3f7a05af0fbb5c0a59c9e146e6dae06167d58377e4a45cb7413411
-  data.tar.gz: 1cc7453a96d191312242f4534cc49601e2df38c6121a5b75b88fa999a3c6936c
+  metadata.gz: 66d9dd5b6e50ded563ba876174ef20a039d7c074193136becdb5c49117b34039
+  data.tar.gz: 24245d7bd943145c2c7096931d3d2b5c514b226538e2c1f0f35dc408c72a9cb8
 SHA512:
-  metadata.gz: 7406a221baac8a84c873818ea242f38622d3290fe2b51cc252f1653d396225c89d6e1bc1571911e38092b035a7269db1cf22568a5e6fb0b4eda3935db8614cfa
-  data.tar.gz: 3db97ee6943027bf55cf212a572f4c7f73de2018073f5262dc601b17c75fab5fc1d0aed0b1488176464a04ad9fa17bf4ca2d817b3d5bd2e128ab8ec5b8ffcd5f
+  metadata.gz: 178cc7d07c311b8c8c66b022c71a6ef0e6fa6e903b00abe2c815d217ff03129ff546b66ea02f956e85c71149a56c02340e4f0fdff7c6acb02fe44e6d736736cb
+  data.tar.gz: 9acb5dd248d648b4db87d34fa39f34401e365904b899736a0ef97f21dca47beb8446b02673e9ce7dd7bdca5045a909127b2765e18bc99a5a8ffd8d079cfd917e

data/CHANGELOG.md

@@ -30,6 +30,33 @@ Please file a bug if you notice a violation of semantic versioning.
 
 ### Security
 
+## [1.0.1] - 2026-02-22
+
+- TAG: [v1.0.1][1.0.1t]
+- COVERAGE: 98.13% -- 263/268 lines in 10 files
+- BRANCH COVERAGE: 91.18% -- 62/68 branches in 10 files
+- 96.77% documented
+
+### Added
+
+- `Config#segment_pattern` option — a parslet character class constraining which characters
+  are valid inside token segments (default: `"[A-Za-z0-9_]"`). This prevents false positive
+  token matches against Ruby block parameters (`{ |x| expr }`), shell variable expansion
+  (`${VAR:+val}`), and other syntax that structurally resembles tokens but contains spaces
+  or punctuation in the "segments".
+- `Resolve#resolve` now validates replacement keys against the config's `segment_pattern` and
+  raises `ArgumentError` if a key contains characters that the grammar would never parse.
+
+### Fixed
+
+- **False positive token matches** — the grammar previously used `any` (match any character)
+  for segment content, which allowed spaces, operators, and punctuation inside token segments.
+  This caused Ruby block syntax like `{ |fp| File.exist?(fp) }` and shell expansion like
+  `${CLASSPATH:+:$CLASSPATH}` to be incorrectly parsed as tokens. With multi-separator configs
+  (`["|", ":"]`), the second `|` was reconstructed as `:` during `on_missing: :keep`
+  roundtripping, silently corrupting source files. The grammar now uses
+  `match(segment_pattern)` instead of `any`, limiting segments to word characters by default.
+
 ## [1.0.0] - 2026-02-21
 
 - TAG: [v1.0.0][1.0.0t]
@@ -43,6 +70,8 @@ Please file a bug if you notice a violation of semantic versioning.
 
 ### Security
 
-[Unreleased]: https://github.com/kettle-rb/token-resolver/compare/v1.0.0...HEAD
+[Unreleased]: https://github.com/kettle-rb/token-resolver/compare/v1.0.1...HEAD
+[1.0.1]: https://github.com/kettle-rb/token-resolver/compare/v1.0.0...v1.0.1
+[1.0.1t]: https://github.com/kettle-rb/token-resolver/releases/tag/v1.0.1
 [1.0.0]: https://github.com/kettle-rb/ast-merge/compare/e0e299cad6e6914d512845c71df6b7ac8009e5ac...v1.0.0
 [1.0.0t]: https://github.com/kettle-rb/ast-merge/tags/v1.0.0

data/README.md

@@ -157,13 +157,36 @@ NOTE: Be prepared to track down certs for signed gems and add them the same way
 
 ### Token Config Options
 
-| Option | Default | Description |
-|--------|---------|-------------|
-| `pre` | `"{"` | Opening delimiter |
-| `post` | `"}"` | Closing delimiter |
-| `separators` | `["\|"]` | Segment separators (sequential; last repeats) |
-| `min_segments` | `2` | Minimum segments for a valid token |
-| `max_segments` | `nil` | Maximum segments (`nil` = unlimited) |
+| Option            | Default             | Description                                        |
+|-------------------|---------------------|----------------------------------------------------|
+| `pre`             | `"{"`               | Opening delimiter                                  |
+| `post`            | `"}"`               | Closing delimiter                                  |
+| `separators`      | `["|"]` (pipe) | Segment separators (sequential; last repeats)      |
+| `min_segments`    | `2`                 | Minimum segments for a valid token                 |
+| `max_segments`    | `nil`               | Maximum segments (`nil` = unlimited)               |
+| `segment_pattern` | `"[A-Za-z0-9_]"`    | Parslet character class for valid segment content  |
+
+### Segment Character Constraints
+
+Token segments (the parts between delimiters and separators) only match characters that
+conform to the `segment_pattern`. By default, this is word characters: uppercase and
+lowercase letters, digits, and underscores.
+
+This prevents false positives with syntax that structurally resembles tokens but isn't:
+
+```ruby
+# These are NOT parsed as tokens (spaces, punctuation disqualify them):
+"items.map { |x| x.to_s }"                 # Ruby block parameters
+"${CLASSPATH:+:$CLASSPATH}"                 # Shell variable expansion
+"cert_chain.select! { |fp| File.exist? }"  # Ruby block with expressions
+```
+
+If you need different characters in your token segments, provide a custom pattern:
+
+```ruby
+# Allow hyphens in segments: {NS|my-key}
+config = Token::Resolver::Config.new(segment_pattern: "[A-Za-z0-9_-]")
+```
 
 ## 🔧 Basic Usage
 
@@ -263,6 +286,14 @@ infinite loops and ensures predictable behavior when replacement values contain
 If the input doesn't contain the `pre` delimiter at all, the parser fast-paths and returns
 a single Text node without invoking parslet.
 
+### False Positive Prevention
+
+The grammar constrains segment content to the configured `segment_pattern` (default: word
+characters). This ensures that syntax using the same delimiter characters — such as Ruby
+block parameters (`{ |x| expr }`) or shell variable expansion (`${VAR:+val}`) — is never
+mistakenly parsed as a token. Replacement keys that contain characters outside the
+`segment_pattern` are rejected with an `ArgumentError` at resolve time.
+
 
 ## 🦷 FLOSS Funding
 
@@ -623,7 +654,7 @@ Thanks for RTFM. ☺️
 [📌gitmoji]: https://gitmoji.dev
 [📌gitmoji-img]: https://img.shields.io/badge/gitmoji_commits-%20%F0%9F%98%9C%20%F0%9F%98%8D-34495e.svg?style=flat-square
 [🧮kloc]: https://www.youtube.com/watch?v=dQw4w9WgXcQ
-[🧮kloc-img]: https://img.shields.io/badge/KLOC-0.258-FFDD67.svg?style=for-the-badge&logo=YouTube&logoColor=blue
+[🧮kloc-img]: https://img.shields.io/badge/KLOC-0.268-FFDD67.svg?style=for-the-badge&logo=YouTube&logoColor=blue
 [🔐security]: SECURITY.md
 [🔐security-img]: https://img.shields.io/badge/security-policy-259D6C.svg?style=flat
 [📄copyright-notice-explainer]: https://opensource.stackexchange.com/questions/5778/why-do-licenses-such-as-the-mit-license-specify-a-single-year

data/lib/token/resolver/config.rb

@@ -44,6 +44,11 @@ module Token
       # @return [Integer, nil] Maximum number of segments (nil = unlimited)
       attr_reader :max_segments
 
+      # @return [String] Parslet-compatible character class for segment content.
+      #   Only characters matching this pattern are valid inside a token segment.
+      #   Default: `"[A-Za-z0-9_]"` (word characters — no spaces, no operators).
+      attr_reader :segment_pattern
+
       # Create a new Config.
       #
       # @param pre [String] Opening delimiter (default: "{")
@@ -51,9 +56,11 @@ module Token
       # @param separators [Array<String>] Segment separators (default: ["|"])
       # @param min_segments [Integer] Minimum segment count (default: 2)
       # @param max_segments [Integer, nil] Maximum segment count (default: nil)
+      # @param segment_pattern [String] Parslet match() character class for valid segment
+      #   characters (default: "[A-Za-z0-9_]")
       #
       # @raise [ArgumentError] If any delimiter is empty or constraints are invalid
-      def initialize(pre: "{", post: "}", separators: ["|"], min_segments: 2, max_segments: nil)
+      def initialize(pre: "{", post: "}", separators: ["|"], min_segments: 2, max_segments: nil, segment_pattern: "[A-Za-z0-9_]")
         validate!(pre, post, separators, min_segments, max_segments)
 
         @pre = pre.dup.freeze
@@ -61,6 +68,7 @@ module Token
         @separators = separators.map { |s| s.dup.freeze }.freeze
         @min_segments = min_segments
         @max_segments = max_segments
+        @segment_pattern = segment_pattern.dup.freeze
 
         freeze
       end
@@ -85,7 +93,8 @@ module Token
           post == other.post &&
           separators == other.separators &&
           min_segments == other.min_segments &&
-          max_segments == other.max_segments
+          max_segments == other.max_segments &&
+          segment_pattern == other.segment_pattern
       end
 
       alias_method :==, :eql?
@@ -94,7 +103,7 @@ module Token
       #
       # @return [Integer]
       def hash
-        [pre, post, separators, min_segments, max_segments].hash
+        [pre, post, separators, min_segments, max_segments, segment_pattern].hash
       end
 
       # Get the separator for a given boundary index.

data/lib/token/resolver/grammar.rb

@@ -53,6 +53,7 @@ module Token
           separators = config.separators
           min_segs = config.min_segments
           max_segs = config.max_segments
+          seg_pattern = config.segment_pattern
 
           Class.new(Parslet::Parser) do
             # A segment is one or more characters that are not a separator or post delimiter.
@@ -62,10 +63,11 @@ module Token
             # Build the set of strings that terminate a segment
             terminators = ([post_str] + separators).uniq
 
-            # segment: one or more chars that aren't any terminator
+            # segment: one or more chars that match the segment_pattern and
+            #          aren't any terminator string.
             rule(:segment) {
               terminator_absent = terminators.map { |t| str(t).absent? }.reduce(:>>)
-              (terminator_absent >> any).repeat(1)
+              (terminator_absent >> match(seg_pattern)).repeat(1)
             }
 
             # token: pre + segment + (sep + segment).repeat + post

data/lib/token/resolver/resolve.rb

@@ -50,16 +50,19 @@ module Token
       # @return [String] Resolved text
       #
       # @raise [UnresolvedTokenError] If on_missing is :raise and a token has no replacement
+      # @raise [ArgumentError] If a replacement key contains characters outside the config's segment_pattern
       def resolve(document_or_nodes, replacements)
-        nodes = case document_or_nodes
+        nodes, config = case document_or_nodes
         when Document
-          document_or_nodes.nodes
+          [document_or_nodes.nodes, document_or_nodes.config]
         when Array
-          document_or_nodes
+          [document_or_nodes, nil]
         else
           raise ArgumentError, "Expected Document or Array of nodes, got #{document_or_nodes.class}"
         end
 
+        validate_replacement_keys!(replacements, config) if config && !replacements.empty?
+
         result = +""
         nodes.each do |node|
           if node.token?
@@ -88,6 +91,28 @@ module Token
           # emit nothing
         end
       end
+
+      # Validate that all replacement keys only contain characters allowed by the config.
+      # Each key is composed of segments (matching segment_pattern) joined by separators.
+      #
+      # @param replacements [Hash{String => String}]
+      # @param config [Config]
+      # @raise [ArgumentError] If any key contains invalid characters
+      def validate_replacement_keys!(replacements, config)
+        # Build a regex that matches a valid key: segment (sep segment)*
+        seg = config.segment_pattern
+        seps = config.separators.map { |s| Regexp.escape(s) }.join("|")
+        valid_key_re = /\A#{seg}+((?:#{seps})#{seg}+)*\z/
+
+        replacements.each_key do |key|
+          unless valid_key_re.match?(key)
+            raise ArgumentError,
+              "Invalid replacement key: #{key.inspect}. " \
+                "Key segments must match #{config.segment_pattern.inspect} " \
+                "and be separated by one of #{config.separators.inspect}."
+          end
+        end
+      end
     end
   end
 end

data/lib/token/resolver/version.rb

@@ -5,7 +5,7 @@ module Token
     # Version information for Token::Resolver
     module Version
       # Current version of the token-resolver gem
-      VERSION = "1.0.0"
+      VERSION = "1.0.1"
     end
     VERSION = Version::VERSION # traditional location
   end

data.tar.gz.sig

@@ -1,2 +1,4 @@
-?<"�˽��(���S��wzӈ�_�B�+7�i�H�nxv���ۨ��t"��Y9��u��0Ti��P�]/�Z�03�vh���ڸ�B�
8*�v�@�
-�v-�Y5[�Q�
3fG�)8T{)+��
��C[2������ъ�ߌ�q��a��_��9���,��:�D�6���A����i�5i�@�
�Uv��e�?�����d��$�Bí�%y ���d�2/9�
+iد�쾰�'Ctۨfi��(�t!��I�S���?��o�
+�}l���1�%��p=>U
+�#F��h<cg0G��,*	A�4)8��h�h`]��X�ۛ��M�<���-M�!���Z�txKFl�CJ�ˇ�d^	� �6�;+�ԝ�Q3�m�{���X
+���ѫ1ى�p����[z{v�r&2��	N�,���9�k��Q攛�Տݰr졽�^��"uQI	���+r����F��Vm��Uxw&`�	��Rނ��������[]Z��x����j���b

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: token-resolver
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.0.1
 platform: ruby
 authors:
 - Peter H. Boling
@@ -274,10 +274,10 @@ licenses:
 - MIT
 metadata:
   homepage_uri: https://token-resolver.galtzo.com/
-  source_code_uri: https://github.com/kettle-rb/token-resolver/tree/v1.0.0
-  changelog_uri: https://github.com/kettle-rb/token-resolver/blob/v1.0.0/CHANGELOG.md
+  source_code_uri: https://github.com/kettle-rb/token-resolver/tree/v1.0.1
+  changelog_uri: https://github.com/kettle-rb/token-resolver/blob/v1.0.1/CHANGELOG.md
   bug_tracker_uri: https://github.com/kettle-rb/token-resolver/issues
-  documentation_uri: https://www.rubydoc.info/gems/token-resolver/1.0.0
+  documentation_uri: https://www.rubydoc.info/gems/token-resolver/1.0.1
   funding_uri: https://github.com/sponsors/pboling
   wiki_uri: https://github.com/kettle-rb/token-resolver/wiki
   news_uri: https://www.railsbling.com/tags/token-resolver