token-resolver 1.0.0

data/RUBOCOP.md

@@ -0,0 +1,71 @@
+# RuboCop Usage Guide
+
+## Overview
+
+A tale of two RuboCop plugin gems.
+
+### RuboCop Gradual
+
+This project uses `rubocop_gradual` instead of vanilla RuboCop for code style checking. The `rubocop_gradual` tool allows for gradual adoption of RuboCop rules by tracking violations in a lock file.
+
+### RuboCop LTS
+
+This project uses `rubocop-lts` to ensure, on a best-effort basis, compatibility with Ruby >= 1.9.2.
+RuboCop rules are meticulously configured by the `rubocop-lts` family of gems to ensure that a project is compatible with a specific version of Ruby. See: https://rubocop-lts.gitlab.io for more.
+
+## Checking RuboCop Violations
+
+To check for RuboCop violations in this project, always use:
+
+```bash
+bundle exec rake rubocop_gradual:check
+```
+
+**Do not use** the standard RuboCop commands like:
+- `bundle exec rubocop`
+- `rubocop`
+
+## Understanding the Lock File
+
+The `.rubocop_gradual.lock` file tracks all current RuboCop violations in the project. This allows the team to:
+
+1. Prevent new violations while gradually fixing existing ones
+2. Track progress on code style improvements
+3. Ensure CI builds don't fail due to pre-existing violations
+
+## Common Commands
+
+- **Check violations**
+    - `bundle exec rake rubocop_gradual`
+    - `bundle exec rake rubocop_gradual:check`
+- **(Safe) Autocorrect violations, and update lockfile if no new violations**
+  - `bundle exec rake rubocop_gradual:autocorrect`
+- **Force update the lock file (w/o autocorrect) to match violations present in code**
+  - `bundle exec rake rubocop_gradual:force_update`
+
+## Workflow
+
+1. Before submitting a PR, run `bundle exec rake rubocop_gradual:autocorrect`
+   a. or just the default `bundle exec rake`, as autocorrection is a pre-requisite of the default task.
+2. If there are new violations, either:
+   - Fix them in your code
+   - Run `bundle exec rake rubocop_gradual:force_update` to update the lock file (only for violations you can't fix immediately)
+3. Commit the updated `.rubocop_gradual.lock` file along with your changes
+
+## Never add inline RuboCop disables
+
+Do not add inline `rubocop:disable` / `rubocop:enable` comments anywhere in the codebase (including specs, except when following the few existing `rubocop:disable` patterns for a rule already being disabled elsewhere in the code). We handle exceptions in two supported ways:
+
+- Permanent/structural exceptions: prefer adjusting the RuboCop configuration (e.g., in `.rubocop.yml`) to exclude a rule for a path or file pattern when it makes sense project-wide.
+- Temporary exceptions while improving code: record the current violations in `.rubocop_gradual.lock` via the gradual workflow:
+  - `bundle exec rake rubocop_gradual:autocorrect` (preferred; will autocorrect what it can and update the lock only if no new violations were introduced)
+  - If needed, `bundle exec rake rubocop_gradual:force_update` (as a last resort when you cannot fix the newly reported violations immediately)
+
+In general, treat the rules as guidance to follow; fix violations rather than ignore them. For example, RSpec conventions in this project expect `described_class` to be used in specs that target a specific class under test.
+
+## Benefits of rubocop_gradual
+
+- Allows incremental adoption of code style rules
+- Prevents CI failures due to pre-existing violations
+- Provides a clear record of code style debt
+- Enables focused efforts on improving code quality over time

data/SECURITY.md

@@ -0,0 +1,21 @@
+# Security Policy
+
+## Supported Versions
+
+| Version  | Supported |
+|----------|-----------|
+| 1.latest | ✅         |
+
+## Security contact information
+
+To report a security vulnerability, please use the
+[Tidelift security contact](https://tidelift.com/security).
+Tidelift will coordinate the fix and disclosure.
+
+## Additional Support
+
+If you are interested in support for versions older than the latest release,
+please consider sponsoring the project / maintainer @ https://liberapay.com/pboling/donate,
+or find other sponsorship links in the [README].
+
+[README]: README.md

data/lib/token/resolver/config.rb

@@ -0,0 +1,134 @@
+# frozen_string_literal: true
+
+module Token
+  module Resolver
+    # Configuration object defining the token structure for parsing.
+    #
+    # A Config describes what tokens look like: their opening/closing delimiters,
+    # segment separators, and segment count constraints. Configs are frozen after
+    # initialization and implement #hash/#eql? for grammar caching.
+    #
+    # @example Default config (tokens like {X|Y})
+    #   config = Token::Resolver::Config.default
+    #   config.pre          # => "{"
+    #   config.post         # => "}"
+    #   config.separators   # => ["|"]
+    #   config.min_segments # => 2
+    #
+    # @example Custom config (tokens like <<X:Y>>)
+    #   config = Token::Resolver::Config.new(
+    #     pre: "<<",
+    #     post: ">>",
+    #     separators: [":"],
+    #   )
+    #
+    # @example Multi-separator config (tokens like {KJ|SECTION:NAME})
+    #   config = Token::Resolver::Config.new(
+    #     separators: ["|", ":"],
+    #   )
+    #   # First boundary uses "|", second uses ":", rest repeat ":"
+    #
+    class Config
+      # @return [String] Opening delimiter for tokens
+      attr_reader :pre
+
+      # @return [String] Closing delimiter for tokens
+      attr_reader :post
+
+      # @return [Array<String>] Separators between segments (used sequentially; last repeats)
+      attr_reader :separators
+
+      # @return [Integer] Minimum number of segments for a valid token
+      attr_reader :min_segments
+
+      # @return [Integer, nil] Maximum number of segments (nil = unlimited)
+      attr_reader :max_segments
+
+      # Create a new Config.
+      #
+      # @param pre [String] Opening delimiter (default: "{")
+      # @param post [String] Closing delimiter (default: "}")
+      # @param separators [Array<String>] Segment separators (default: ["|"])
+      # @param min_segments [Integer] Minimum segment count (default: 2)
+      # @param max_segments [Integer, nil] Maximum segment count (default: nil)
+      #
+      # @raise [ArgumentError] If any delimiter is empty or constraints are invalid
+      def initialize(pre: "{", post: "}", separators: ["|"], min_segments: 2, max_segments: nil)
+        validate!(pre, post, separators, min_segments, max_segments)
+
+        @pre = pre.dup.freeze
+        @post = post.dup.freeze
+        @separators = separators.map { |s| s.dup.freeze }.freeze
+        @min_segments = min_segments
+        @max_segments = max_segments
+
+        freeze
+      end
+
+      class << self
+        # Default config suitable for kettle-jem style tokens like {KJ|GEM_NAME}.
+        #
+        # @return [Config]
+        def default
+          @default ||= new # rubocop:disable ThreadSafety/ClassInstanceVariable
+        end
+      end
+
+      # Equality based on all attributes.
+      #
+      # @param other [Object]
+      # @return [Boolean]
+      def eql?(other)
+        return false unless other.is_a?(Config)
+
+        pre == other.pre &&
+          post == other.post &&
+          separators == other.separators &&
+          min_segments == other.min_segments &&
+          max_segments == other.max_segments
+      end
+
+      alias_method :==, :eql?
+
+      # Hash based on all attributes (for use as Hash key / grammar cache).
+      #
+      # @return [Integer]
+      def hash
+        [pre, post, separators, min_segments, max_segments].hash
+      end
+
+      # Get the separator for a given boundary index.
+      #
+      # When there are more segment boundaries than separators, the last separator repeats.
+      #
+      # @param index [Integer] Zero-based boundary index
+      # @return [String] The separator to use
+      def separator_at(index)
+        if index < separators.length
+          separators[index]
+        else
+          separators.last
+        end
+      end
+
+      private
+
+      def validate!(pre, post, separators, min_segments, max_segments)
+        raise ArgumentError, "pre must be a non-empty String" unless pre.is_a?(String) && !pre.empty?
+        raise ArgumentError, "post must be a non-empty String" unless post.is_a?(String) && !post.empty?
+        raise ArgumentError, "separators must be a non-empty Array" unless separators.is_a?(Array) && !separators.empty?
+
+        separators.each_with_index do |sep, i|
+          raise ArgumentError, "separators[#{i}] must be a non-empty String" unless sep.is_a?(String) && !sep.empty?
+        end
+
+        raise ArgumentError, "min_segments must be a positive Integer" unless min_segments.is_a?(Integer) && min_segments >= 1
+
+        if max_segments
+          raise ArgumentError, "max_segments must be a positive Integer" unless max_segments.is_a?(Integer) && max_segments >= 1
+          raise ArgumentError, "max_segments (#{max_segments}) must be >= min_segments (#{min_segments})" if max_segments < min_segments
+        end
+      end
+    end
+  end
+end

data/lib/token/resolver/document.rb

@@ -0,0 +1,82 @@
+# frozen_string_literal: true
+
+module Token
+  module Resolver
+    # Parses input text and provides access to the resulting text and token nodes.
+    #
+    # Document is the primary public API for parsing. It uses Grammar to parse
+    # the input and Transform to convert the parse tree into node objects.
+    #
+    # @example Basic usage
+    #   doc = Token::Resolver::Document.new("Hello {KJ|NAME}!")
+    #   doc.nodes       # => [Text("Hello "), Token(["KJ", "NAME"]), Text("!")]
+    #   doc.tokens      # => [Token(["KJ", "NAME"])]
+    #   doc.token_keys  # => ["KJ|NAME"]
+    #   doc.to_s        # => "Hello {KJ|NAME}!"
+    #   doc.text_only?  # => false
+    #
+    # @example Fast-path for text without tokens
+    #   doc = Token::Resolver::Document.new("No tokens here")
+    #   doc.text_only?  # => true
+    #
+    class Document
+      # @return [Array<Node::Text, Node::Token>] Parsed nodes
+      attr_reader :nodes
+
+      # @return [Config] The config used for parsing
+      attr_reader :config
+
+      # Parse input text into a Document.
+      #
+      # @param input [String] Text to parse for tokens
+      # @param config [Config] Token configuration (default: Config.default)
+      def initialize(input, config: Config.default)
+        @config = config
+        @input = input
+        @nodes = parse(input)
+      end
+
+      # Return only the Token nodes.
+      #
+      # @return [Array<Node::Token>]
+      def tokens
+        @tokens ||= @nodes.select(&:token?)
+      end
+
+      # Return the unique token keys found in the input.
+      #
+      # @return [Array<String>]
+      def token_keys
+        @token_keys ||= tokens.map(&:key).uniq
+      end
+
+      # Reconstruct the original input from nodes (roundtrip fidelity).
+      #
+      # @return [String]
+      def to_s
+        @nodes.map(&:to_s).join
+      end
+
+      # Whether the input contains no tokens.
+      #
+      # @return [Boolean]
+      def text_only?
+        tokens.empty?
+      end
+
+      private
+
+      def parse(input)
+        # Fast-path: if input doesn't contain the pre delimiter, no tokens possible
+        return [Node::Text.new(input)] if input.nil? || input.empty? || !input.include?(@config.pre)
+
+        parser_class = Grammar.build(@config)
+        tree = parser_class.new.parse(input)
+        Transform.apply(tree, @config)
+      rescue Parslet::ParseFailed
+        # Grammar should never fail, but if it does, treat entire input as text
+        [Node::Text.new(input)]
+      end
+    end
+  end
+end

data/lib/token/resolver/grammar.rb

@@ -0,0 +1,122 @@
+# frozen_string_literal: true
+
+require "parslet"
+
+module Token
+  module Resolver
+    # Dynamically builds a Parslet::Parser subclass from a Config.
+    #
+    # The grammar recognizes structured tokens (e.g., `{KJ|GEM_NAME}`) within
+    # arbitrary text. It is designed to **never fail** — any input is valid.
+    # Unrecognized content (including incomplete tokens) becomes text nodes.
+    #
+    # @example Building and using a grammar
+    #   parser_class = Token::Resolver::Grammar.build(Config.default)
+    #   tree = parser_class.new.parse("Hello {KJ|NAME}!")
+    #   # => [{:text=>"H"@0}, {:text=>"e"@1}, ..., {:token=>{:segments=>[...]}}, ...]
+    #
+    # @note The raw parslet tree contains one :text entry per character.
+    #   Use Transform to coalesce these into proper Text nodes.
+    #
+    class Grammar
+      # Cache of built parser classes, keyed by Config.
+      # Config is frozen and implements #hash/#eql?, so this is safe.
+      # NOTE: This hash is mutated under mutex in .build; it cannot actually be frozen.
+      @cache = {} # rubocop:disable ThreadSafety/MutableClassInstanceVariable
+      @cache_mutex = Mutex.new
+
+      class << self
+        # Build (or retrieve from cache) a Parslet::Parser subclass for the given Config.
+        #
+        # @param config [Config] Token configuration
+        # @return [Class] A Parslet::Parser subclass
+        def build(config)
+          @cache_mutex.synchronize do
+            @cache[config] ||= build_parser_class(config)
+          end
+        end
+
+        # Clear the grammar cache. Mostly useful for testing.
+        #
+        # @return [void]
+        def clear_cache!
+          @cache_mutex.synchronize do
+            @cache.clear
+          end
+        end
+
+        private
+
+        def build_parser_class(config)
+          pre_str = config.pre
+          post_str = config.post
+          separators = config.separators
+          min_segs = config.min_segments
+          max_segs = config.max_segments
+
+          Class.new(Parslet::Parser) do
+            # A segment is one or more characters that are not a separator or post delimiter.
+            # We need to exclude ALL separators and the post delimiter.
+            define_method(:_config) { config }
+
+            # Build the set of strings that terminate a segment
+            terminators = ([post_str] + separators).uniq
+
+            # segment: one or more chars that aren't any terminator
+            rule(:segment) {
+              terminator_absent = terminators.map { |t| str(t).absent? }.reduce(:>>)
+              (terminator_absent >> any).repeat(1)
+            }
+
+            # token: pre + segment + (sep + segment).repeat + post
+            # with min/max segment constraints
+            rule(:token) {
+              # Build the repeating part: (separator + segment)
+              # For sequential separators, we'd need to handle them specially.
+              # However, parslet rules are declarative, so we handle sequential seps
+              # by building a chain: first_sep >> segment >> second_sep >> segment >> ...
+              # For the general case with repeating last separator, we use a dynamic approach.
+
+              # Simple case: build "pre segment (sep segment)* post" and validate segment count after
+              # We use the first separator for the first boundary, second for the second, etc.
+              # Last separator repeats.
+
+              # For parslet, we need to build this statically. The simplest approach:
+              # Match pre + segment + (any_sep + segment)* + post, capture all segments,
+              # then validate count in the Transform step.
+
+              # Build alternation of all separators
+              sep_match = if separators.length == 1
+                str(separators[0])
+              else
+                separators.map { |s| str(s) }.reduce(:|)
+              end
+
+              base = str(pre_str) >>
+                segment.as(:seg) >>
+                (sep_match >> segment.as(:seg)).repeat(min_segs - 1, max_segs ? max_segs - 1 : nil) >>
+                str(post_str)
+
+              base
+            }
+
+            # text_char: any single character that doesn't start a valid token
+            rule(:text_char) {
+              # If we see pre_str, try to match a token. If it fails, consume pre_str as text.
+              # Parslet's ordered choice handles this: token is tried first in document.
+              # Here we just need to match any single character.
+              any
+            }
+
+            # document: sequence of tokens and text characters
+            rule(:document) {
+              (token.as(:token) | text_char.as(:text)).repeat
+            }
+
+            root(:document)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/token/resolver/node/text.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+module Token
+  module Resolver
+    module Node
+      # Represents plain text content (not a token).
+      #
+      # Text nodes hold the literal string content between (or outside of) tokens.
+      # They are frozen after creation.
+      #
+      # @example
+      #   text = Token::Resolver::Node::Text.new("Hello ")
+      #   text.to_s     # => "Hello "
+      #   text.content  # => "Hello "
+      #
+      class Text
+        # @return [String] The text content
+        attr_reader :content
+
+        # @param content [String] The text content
+        def initialize(content)
+          @content = content.frozen? ? content : content.dup.freeze
+          freeze
+        end
+
+        # @return [String] The text content
+        def to_s
+          @content
+        end
+
+        # @return [Boolean]
+        def token?
+          false
+        end
+
+        # @return [Boolean]
+        def text?
+          true
+        end
+
+        # Equality based on content.
+        #
+        # @param other [Object]
+        # @return [Boolean]
+        def eql?(other)
+          other.is_a?(Text) && content == other.content
+        end
+
+        alias_method :==, :eql?
+
+        # @return [Integer]
+        def hash
+          [self.class, content].hash
+        end
+
+        # @return [String]
+        def inspect
+          "#<#{self.class} #{content.inspect}>"
+        end
+      end
+    end
+  end
+end

data/lib/token/resolver/node/token.rb

@@ -0,0 +1,107 @@
+# frozen_string_literal: true
+
+module Token
+  module Resolver
+    module Node
+      # Represents a structured token found in the input.
+      #
+      # A token consists of segments separated by configured separators,
+      # wrapped in pre/post delimiters. For example, with default config,
+      # `{KJ|GEM_NAME}` has segments `["KJ", "GEM_NAME"]`.
+      #
+      # Token nodes are frozen after creation.
+      #
+      # @example Single separator
+      #   token = Token::Resolver::Node::Token.new(["KJ", "GEM_NAME"], config)
+      #   token.key       # => "KJ|GEM_NAME"
+      #   token.prefix    # => "KJ"
+      #   token.segments  # => ["KJ", "GEM_NAME"]
+      #   token.to_s      # => "{KJ|GEM_NAME}"
+      #
+      # @example Sequential separators
+      #   config = Config.new(separators: ["|", ":"])
+      #   token = Token::Resolver::Node::Token.new(["KJ", "SECTION", "NAME"], config)
+      #   token.key       # => "KJ|SECTION:NAME"
+      #   token.to_s      # => "{KJ|SECTION:NAME}"
+      #
+      class Token
+        # @return [Array<String>] The token segments
+        attr_reader :segments
+
+        # @return [Config] The config used to parse this token
+        attr_reader :config
+
+        # @param segments [Array<String>] The token segments
+        # @param config [Config] The config that defined this token's structure
+        def initialize(segments, config)
+          @segments = segments.map { |s| s.frozen? ? s : s.dup.freeze }.freeze
+          @config = config
+          freeze
+        end
+
+        # The canonical key for this token, suitable for use as a replacement map key.
+        #
+        # Joins segments using the actual separators in order. For `separators: ["|", ":"]`
+        # and segments `["KJ", "SECTION", "NAME"]`, returns `"KJ|SECTION:NAME"`.
+        #
+        # @return [String]
+        def key
+          return @segments[0] if @segments.length == 1
+
+          result = +""
+          @segments.each_with_index do |seg, i|
+            if i > 0
+              result << @config.separator_at(i - 1)
+            end
+            result << seg
+          end
+          result.freeze
+        end
+
+        # The first segment (typically a prefix/namespace like "KJ").
+        #
+        # @return [String]
+        def prefix
+          @segments[0]
+        end
+
+        # Reconstruct the original token string with delimiters.
+        #
+        # @return [String]
+        def to_s
+          "#{@config.pre}#{key}#{@config.post}"
+        end
+
+        # @return [Boolean]
+        def token?
+          true
+        end
+
+        # @return [Boolean]
+        def text?
+          false
+        end
+
+        # Equality based on segments and config.
+        #
+        # @param other [Object]
+        # @return [Boolean]
+        def eql?(other)
+          other.is_a?(Token) && segments == other.segments && config == other.config
+        end
+
+        alias_method :==, :eql?
+
+        # @return [Integer]
+        def hash
+          [self.class, segments, config].hash
+        end
+
+        # @return [String]
+        def inspect
+          "#<#{self.class} #{to_s.inspect} segments=#{segments.inspect}>"
+        end
+      end
+    end
+  end
+end

data/lib/token/resolver/node.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module Token
+  module Resolver
+    # Namespace for node types produced by parsing.
+    module Node
+      autoload :Text, "token/resolver/node/text"
+      autoload :Token, "token/resolver/node/token"
+    end
+  end
+end