lex-transformer 0.2.1 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d7f4f19cb531c6f7cb50302bb3f2aab973b55976bb05d1b870ac2ad4f7bdb9a0
-  data.tar.gz: a445f592b23aae48a5510a7c6f17ee3cf0ea0a29dd3c9d67bc405b5b208a0c72
+  metadata.gz: 111582b1dfcbe44b2cf91d3b316bd6394eb09e4d750a7b19be8844e91257e2c5
+  data.tar.gz: 6efebcb5ea769c9426e17e6be93aa1cc0ca1bf5093e02d9f5ea0fe5254809760
 SHA512:
-  metadata.gz: 21096cee7c407f8df2aa5307e3fe71e35008103a798bc7f9c9cd8dfd1724728af7d65fbeeada8b5df657c412bdcfd44c60fd100efb2a3f63815c6615e63f9917
-  data.tar.gz: 4e6348dc31ae9217b34040010ecf8b613b8b0e97ae1995854a3f30c821bce8a5b6d190bb283ad9a06e305fe128b68bd8a3533772451d608c55912659c1970ec9
+  metadata.gz: 8af3daa3467d9e37e571319c876a17ff772e3558c0510fcca75a633e970ed32b0497c1e31f63faa047e0eb2f6e8488d8a55820d0d5e46394450957228f4ec3af
+  data.tar.gz: a8dc1caf0ce16261c8eecf6d6ed37b2af4ff1cf4736465047a67cfd96eeb431a1454d4a31f2c34861219fea481b8d79b42d13b860a3935d240f81b89669c870d

data/.gitignore

@@ -11,3 +11,4 @@
 .rspec_status
 
 Gemfile.lock
+Gemfile.lock

data/CHANGELOG.md

@@ -1,5 +1,24 @@
 # Changelog
 
+## [0.3.0] - 2026-03-19
+
+### Added
+- LLM engine error handling with categorized retry (timeout/network retry, auth errors raise, provider errors return failure)
+- LLM engine model/provider/temperature/system_prompt kwargs via `engine_options`
+- LLM engine structured output support (`structured: true` + `schema:`)
+- LLM engine JSON response validation with correction prompt on retry
+- Settings-based LLM defaults (`lex-transformer.llm.*`)
+- Named transform definitions via Settings (`lex-transformer.definitions.*`)
+- `Definitions` class for Settings-based definition lookup
+- `name:` parameter on `Client#transform` for named definition execution
+- `engine_options:` parameter on `Client#transform` and `Client#transform_chain`
+- Conditioner integration for named definitions with conditions
+- LLM failure hash passthrough (no dispatch on engine failure)
+- Integration specs for LLM engine with client, chain, and named definitions
+
+### Changed
+- All engine `render` signatures accept `**opts` (backward-compatible, non-LLM engines ignore)
+
 ## [0.2.1] - 2026-03-17
 
 ### Added

data/CLAUDE.md

@@ -6,7 +6,7 @@
 
 ## Purpose
 
-Legion Extension that transforms task payloads between services in a relationship chain. Uses ERB template-based transformation (via the `tilt` gem) to map data from one task's output into the format expected by the next task's input. Supports single-hash output (1:1 dispatch) and array output (fan-out/multiply). Requires `legion-data` (`data_required? true`).
+Legion Extension that transforms task payloads between services in a relationship chain. Uses pluggable template engines (ERB, Static, Liquid, JSONPath, LLM) to map data from one task's output into the format expected by the next task's input. Supports single-hash output (1:1 dispatch), array output (fan-out/multiply), schema validation, and sequential transform chains. Requires `legion-data` (`data_required? true`).
 
 **GitHub**: https://github.com/LegionIO/lex-transformer
 **License**: MIT
@@ -18,18 +18,30 @@ Legion Extension that transforms task payloads between services in a relationshi
 Legion::Extensions::Transformer
 ├── Actors/
 │   └── Transform          # Subscription actor consuming transform requests
+├── Engines/
+│   ├── Base               # Abstract engine interface
+│   ├── Registry           # Maps engine name symbols to engine classes
+│   ├── Erb                # ERB rendering via tilt
+│   ├── Static             # JSON passthrough (no templating)
+│   ├── Liquid             # Liquid template rendering
+│   ├── Jsonpath           # Dot-notation value extraction from payload
+│   └── Llm                # Natural language transformation via Legion::LLM
+├── Helpers/
+│   └── SchemaValidator    # Validates transform output against required_keys/types schema
 ├── Runners/
 │   └── Transform          # Executes template-based payload transformation
-│       ├── transform             # Entry point: render + dispatch
-│       ├── render_transformation # ERB rendering via tilt, or plain JSON if no ERB tags
-│       ├── build_template_variables # Inject crypt/settings/cache/task into ERB scope
+│       ├── transform             # Entry point: render + dispatch or validate
+│       ├── transform_chain       # Sequential pipeline: N steps, output feeds next
+│       ├── render_transformation # Engine dispatch (ERB/Static/Liquid/JSONPath/LLM)
+│       ├── build_template_variables # Inject crypt/settings/cache/task into scope
 │       ├── dispatch_transformed  # Route Hash (single) or Array (fan-out) results
 │       ├── dispatch_multiplied   # Fan-out: create a new task per array element
 │       └── send_task             # Publish transformed payload to next runner
-└── Transport/
-    ├── Exchanges/Task     # Publishes to the task exchange
-    ├── Queues/Transform   # Subscribes to transformation queue
-    └── Messages/Message   # Transform request message format
+├── Transport/
+│   ├── Exchanges/Task     # Publishes to the task exchange
+│   ├── Queues/Transform   # Subscribes to transformation queue
+│   └── Messages/Message   # Transform request message format
+└── Client                 # Standalone client: transform + transform_chain (uses Engines::Registry directly)
 ```
 
 ## Key Files
@@ -39,8 +51,25 @@ Legion::Extensions::Transformer
 | `lib/legion/extensions/transformer.rb` | Entry point (`data_required? true`) |
 | `lib/legion/extensions/transformer/runners/transform.rb` | Core transformation logic |
 | `lib/legion/extensions/transformer/actors/transform.rb` | AMQP subscription actor |
+| `lib/legion/extensions/transformer/client.rb` | Standalone client (transform, transform_chain) |
+| `lib/legion/extensions/transformer/engines/registry.rb` | Engine name -> class lookup + auto-detection |
+| `lib/legion/extensions/transformer/helpers/schema_validator.rb` | Output schema validation |
 | `lib/legion/extensions/transformer/transport.rb` | Transport setup |
 
+## Template Engines
+
+| Engine | Name | Detection | Description |
+|--------|------|-----------|-------------|
+| ERB | `:erb` | `<%` or `%>` in template | Full ERB template rendering via `tilt` |
+| Static | `:static` | Default (no ERB markers) | Plain JSON passthrough |
+| Liquid | `:liquid` | Explicit only | Liquid template rendering (`{{ var }}`) |
+| JSONPath | `:jsonpath` | Explicit only | Dot-notation value extraction from payload |
+| LLM | `:llm` | Explicit only | Natural language transformation via `Legion::LLM` |
+
+Auto-detection: ERB when template contains `<%` or `%>`, otherwise Static. Pass `engine:` to force a specific engine.
+
+The LLM engine requires `legion-llm` to be started; it is provider-agnostic (Ollama, Bedrock, Anthropic, OpenAI, Gemini).
+
 ## Template Variables Available in ERB
 
 | Variable | Available when |
@@ -51,28 +80,55 @@ Legion::Extensions::Transformer
 | `cache` | Template string contains `'cache'` |
 | `task` | Template string contains `'task'` and payload has `task_id` |
 
+## Schema Validation
+
+```ruby
+schema = {
+  required_keys: [:name, :email],
+  types: { name: String, email: String, age: Integer }
+}
+```
+
+On failure: `{ success: false, status: 'transformer.validation_failed', errors: [...] }` — no dispatch.
+
 ## Dispatch Behavior
 
 - **Hash result**: Updates task to `transformer.succeeded`, dispatches single task, updates to `task.queued`
-- **Array result**: Fan-out via `dispatch_multiplied` - creates a new task record per element, dispatches each, marks original as `task.multiplied`
-- **Plain JSON** (no ERB tags): Parsed directly without template rendering
+- **Array result**: Fan-out via `dispatch_multiplied` — creates a new task record per element, dispatches each, marks original as `task.multiplied`
+- **Schema failure**: `transformer.validation_failed`, no dispatch
+
+## Transform Chains
+
+`transform_chain(steps:, **payload)` runs steps sequentially. Each step specifies `:transformation`, optional `:engine`, and optional `:schema`. Output of step N is merged into the running payload for step N+1. Stops on first schema failure.
+
+## Standalone Client
+
+`Legion::Extensions::Transformer::Client` includes the Transform runner:
+
+```ruby
+require 'legion/extensions/transformer/client'
+client = Legion::Extensions::Transformer::Client.new
+result = client.transform(transformation: '{"x":"<%= y %>"}', payload: { y: 'hello' })
+result[:success] # => true
+result[:result]  # => { x: "hello" }
+```
 
 ## Dependencies
 
 | Gem | Purpose |
 |-----|---------|
 | `tilt` (>= 2.3) | Template engine abstraction for ERB rendering |
-| `legion-data` | Required - task record creation for fan-out |
+| `legion-data` | Required — task record creation for fan-out |
 
 ## Testing
 
 ```bash
 bundle install
-bundle exec rspec
-bundle exec rubocop
+bundle exec rspec     # 86 examples, 0 failures
+bundle exec rubocop   # 0 offenses
 ```
 
-Spec files: `spec/legion/extensions/tranformer_spec.rb` (note: typo in filename), `spec/legion/extensions/transform_runner_spec.rb`
+Spec files: `spec/legion/extensions/tranformer_spec.rb` (note: typo in filename is intentional in repo), `spec/legion/extensions/transform_runner_spec.rb`
 
 ---
 

data/lib/legion/extensions/transformer/client.rb

@@ -2,16 +2,21 @@
 
 require_relative 'engines/registry'
 require_relative 'helpers/schema_validator'
+require_relative 'definitions'
 
 module Legion
   module Extensions
     module Transformer
       class Client
-        def transform(transformation:, payload:, engine: nil, schema: nil)
+        def transform(payload:, transformation: nil, engine: nil, schema: nil, engine_options: {}, name: nil)
+          return transform_by_name(name: name, payload: payload, engine_options: engine_options) if transformation.nil? && name
+
           eng = resolve_engine(engine, transformation)
-          rendered = eng.render(transformation, payload)
+          rendered = eng.render(transformation, payload, **engine_options)
           rendered = parse_rendered(rendered)
 
+          return rendered if rendered.is_a?(Hash) && rendered[:success] == false
+
           if schema
             validation = Helpers::SchemaValidator.validate(schema: schema, data: rendered)
             return { success: false, status: 'transformer.validation_failed', errors: validation[:errors] } unless validation[:valid]
@@ -24,9 +29,12 @@ module Legion
           result = payload.dup
           steps.each do |step|
             eng = resolve_engine(step[:engine], step[:transformation])
-            rendered = eng.render(step[:transformation], result)
+            step_opts = step[:engine_options] || {}
+            rendered = eng.render(step[:transformation], result, **step_opts)
             rendered = parse_rendered(rendered)
 
+            return rendered if rendered.is_a?(Hash) && rendered[:success] == false
+
             if step[:schema]
               validation = Helpers::SchemaValidator.validate(schema: step[:schema], data: rendered)
               return { success: false, status: 'transformer.validation_failed', errors: validation[:errors] } unless validation[:valid]
@@ -43,6 +51,38 @@ module Legion
 
         private
 
+        def transform_by_name(name:, payload:, engine_options: {})
+          definition = Definitions.fetch(name)
+          return { success: false, error: 'definition_not_found' } unless definition
+
+          if definition[:conditions] && conditioner_available?
+            cond_result = evaluate_conditions(definition[:conditions], payload)
+            return { success: false, reason: 'conditions_not_met' } unless cond_result
+          end
+
+          merged_opts = Definitions.merge_options(definition, **engine_options)
+
+          transform(
+            transformation: definition[:transformation],
+            payload:        payload,
+            engine:         definition[:engine],
+            schema:         definition[:schema],
+            engine_options: merged_opts
+          )
+        end
+
+        def conditioner_available?
+          defined?(Legion::Extensions::Conditioner::Client)
+        end
+
+        def evaluate_conditions(conditions, payload)
+          client = Legion::Extensions::Conditioner::Client.new
+          result = client.evaluate(conditions: conditions, values: payload)
+          result[:passed]
+        rescue StandardError
+          true
+        end
+
         def resolve_engine(engine_name, transformation)
           if engine_name
             Engines::Registry.fetch(engine_name)

data/lib/legion/extensions/transformer/definitions.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+
+module Legion
+  module Extensions
+    module Transformer
+      class Definitions
+        class << self
+          def fetch(name)
+            defns = load_definitions
+            return nil unless defns&.key?(name)
+
+            symbolize_definition(defns[name])
+          end
+
+          def names
+            defns = load_definitions
+            return [] unless defns
+
+            defns.keys
+          end
+
+          def merge_options(definition, **overrides)
+            base = definition[:engine_options] || {}
+            base.merge(overrides)
+          end
+
+          private
+
+          def load_definitions
+            return nil unless defined?(Legion::Settings)
+
+            Legion::Settings.dig('lex-transformer', 'definitions')
+          rescue StandardError
+            nil
+          end
+
+          def symbolize_definition(raw)
+            defn = {}
+            defn[:transformation] = raw['transformation'] || raw[:transformation]
+            defn[:engine] = (raw['engine'] || raw[:engine])&.to_sym
+            defn[:engine_options] = symbolize_hash(raw['engine_options'] || raw[:engine_options] || {})
+            defn[:schema] = raw['schema'] || raw[:schema]
+            defn[:conditions] = raw['conditions'] || raw[:conditions]
+            defn
+          end
+
+          def symbolize_hash(hash)
+            return {} unless hash.is_a?(Hash)
+
+            hash.transform_keys(&:to_sym)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/legion/extensions/transformer/engines/base.rb

@@ -5,7 +5,7 @@ module Legion
     module Transformer
       module Engines
         class Base
-          def render(template, payload)
+          def render(template, payload, **_opts)
             raise NotImplementedError, "#{self.class}#render must be implemented"
           end
 

data/lib/legion/extensions/transformer/engines/erb.rb

@@ -12,7 +12,7 @@ module Legion
             :erb
           end
 
-          def render(template, payload)
+          def render(template, payload, **_opts)
             tilt_template = Tilt['erb'].new { template }
             variables = build_variables(template, payload)
             tilt_template.render(Object.new, variables)

data/lib/legion/extensions/transformer/engines/jsonpath.rb

@@ -11,7 +11,7 @@ module Legion
             :jsonpath
           end
 
-          def render(expression, payload)
+          def render(expression, payload, **_opts)
             extract(expression, payload)
           end
 

data/lib/legion/extensions/transformer/engines/liquid.rb

@@ -11,7 +11,7 @@ module Legion
             :liquid
           end
 
-          def render(template, payload)
+          def render(template, payload, **_opts)
             require 'liquid'
             liquid_template = ::Liquid::Template.parse(template)
             liquid_template.render(stringify_keys(payload))

data/lib/legion/extensions/transformer/engines/llm.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require 'timeout'
 require_relative 'base'
 
 module Legion
@@ -7,28 +8,144 @@ module Legion
     module Transformer
       module Engines
         class Llm < Base
+          AUTH_ERRORS = /auth|credentials|forbidden|api.key|unauthorized/i
+          RETRY_SLEEP = 1
+
           def name
             :llm
           end
 
-          def render(prompt, payload)
-            raise 'Legion::LLM is not available or not started' unless defined?(Legion::LLM) && Legion::LLM.respond_to?(:started?) && Legion::LLM.started?
+          def render(prompt, payload, **opts)
+            raise 'Legion::LLM is not available or not started' unless llm_available?
 
-            context = Legion::JSON.dump(payload)
-            full_prompt = "#{prompt}\n\nPayload:\n```json\n#{context}\n```\n\nRespond with valid JSON only. No explanation, no markdown fences."
+            @last_raw = nil
+            resolved = resolve_options(opts)
+            max_retries = resolved.delete(:max_retries)
+            attempts = 0
+
+            loop do
+              result = attempt_render(prompt, payload, resolved, attempts)
+              return result unless result == :retry
 
-            chat = Legion::LLM.chat(message: full_prompt)
-            extract_response(chat)
+              attempts += 1
+              return build_failure if attempts > max_retries
+
+              sleep(RETRY_SLEEP)
+            end
           end
 
           private
 
+          def llm_available?
+            defined?(Legion::LLM) && Legion::LLM.respond_to?(:started?) && Legion::LLM.started?
+          end
+
+          def resolve_options(opts)
+            defaults = settings_defaults
+            resolved = defaults.merge(opts)
+            resolved[:max_retries] = (resolved[:max_retries] || 1).to_i
+            resolved
+          end
+
+          def settings_defaults
+            return {} unless defined?(Legion::Settings)
+
+            settings = begin
+              Legion::Settings.dig('lex-transformer', 'llm')
+            rescue StandardError
+              nil
+            end
+            return {} unless settings.is_a?(Hash)
+
+            settings.transform_keys(&:to_sym)
+          end
+
+          def attempt_render(prompt, payload, opts, attempt)
+            chat = call_llm(prompt, payload, opts, attempt)
+            content = extract_response(chat)
+            validate_json(content)
+            content
+          rescue Timeout::Error, IOError, Errno::ECONNREFUSED, Errno::ECONNRESET
+            :retry
+          rescue ::JSON::ParserError
+            @last_raw = content
+            :retry
+          rescue RuntimeError => e
+            raise if auth_error?(e)
+
+            { success: false, error: e.class.to_s, message: e.message }
+          rescue StandardError => e
+            { success: false, error: e.class.to_s, message: e.message }
+          end
+
+          def call_llm(prompt, payload, opts, attempt)
+            context = Legion::JSON.dump(payload)
+
+            return call_structured(prompt, context, opts) if opts[:structured] && opts[:schema] && structured_available?
+
+            full_prompt = build_prompt(prompt, context, attempt)
+            full_prompt = inject_schema_into_prompt(full_prompt, opts[:schema]) if opts[:structured] && opts[:schema]
+
+            llm_opts = build_llm_opts(opts)
+            Legion::LLM.chat(message: full_prompt, **llm_opts)
+          end
+
+          def structured_available?
+            Legion::LLM.respond_to?(:structured)
+          end
+
+          def call_structured(prompt, context, opts)
+            llm_opts = build_llm_opts(opts)
+            Legion::LLM.structured(
+              message: "#{prompt}\n\nPayload:\n```json\n#{context}\n```",
+              schema:  opts[:schema],
+              **llm_opts
+            )
+          end
+
+          def inject_schema_into_prompt(prompt, schema)
+            schema_json = Legion::JSON.dump(schema)
+            "#{prompt}\n\nYour response MUST conform to this JSON schema:\n```json\n#{schema_json}\n```"
+          end
+
+          def build_prompt(prompt, context, attempt)
+            base = "#{prompt}\n\nPayload:\n```json\n#{context}\n```\n\nRespond with valid JSON only. No explanation, no markdown fences."
+            return base unless attempt.positive?
+
+            "#{base}\n\nIMPORTANT: Your previous response was not valid JSON. Return ONLY a valid JSON object or array, nothing else."
+          end
+
           def extract_response(chat)
             content = chat.respond_to?(:content) ? chat.content : chat.to_s
             content = content.strip
             content = content.sub(/\A```(?:json)?\n?/, '').sub(/\n?```\z/, '') if content.start_with?('```')
             content
           end
+
+          def validate_json(content)
+            ::JSON.parse(content)
+          end
+
+          def auth_error?(error)
+            AUTH_ERRORS.match?(error.message)
+          end
+
+          def build_llm_opts(opts)
+            llm_opts = {}
+            llm_opts[:model] = opts[:model] if opts[:model]
+            llm_opts[:provider] = opts[:provider] if opts[:provider]
+            llm_opts[:temperature] = opts[:temperature] if opts[:temperature]
+            llm_opts[:system_prompt] = opts[:system_prompt] if opts[:system_prompt]
+            llm_opts
+          end
+
+          def build_failure
+            if @last_raw
+              { success: false, error: 'invalid_json', message: 'max retries exhausted', raw: @last_raw }
+            else
+              { success: false, error: 'timeout', message: 'max retries exhausted' }
+            end
+          end
         end
       end
     end

data/lib/legion/extensions/transformer/engines/static.rb

@@ -11,7 +11,7 @@ module Legion
             :static
           end
 
-          def render(template, _payload)
+          def render(template, _payload, **_opts)
             template
           end
         end

data/lib/legion/extensions/transformer/runners/transform.rb

@@ -8,8 +8,11 @@ module Legion
     module Transformer
       module Runners
         module Transform
-          def transform(transformation:, engine: nil, schema: nil, **payload)
-            payload[:args] = render_transformation(transformation, payload, engine: engine)
+          def transform(transformation:, engine: nil, schema: nil, engine_options: {}, **payload)
+            payload[:args] = render_transformation(transformation, payload, engine: engine, engine_options: engine_options)
+
+            return payload[:args] if payload[:args].is_a?(Hash) && payload[:args][:success] == false
+
             if schema
               validation = Helpers::SchemaValidator.validate(schema: schema, data: payload[:args])
               return { success: false, status: 'transformer.validation_failed', errors: validation[:errors] } unless validation[:valid]
@@ -22,8 +25,12 @@ module Legion
           def transform_chain(steps:, **payload)
             result = payload
             steps.each do |step|
-              rendered = render_transformation(step[:transformation], result, engine: step[:engine])
+              step_opts = step[:engine_options] || {}
+              rendered = render_transformation(step[:transformation], result, engine: step[:engine], engine_options: step_opts)
               rendered = from_json(rendered) if rendered.is_a?(String)
+
+              return rendered if rendered.is_a?(Hash) && rendered[:success] == false
+
               if step[:schema]
                 validation = Helpers::SchemaValidator.validate(schema: step[:schema], data: rendered)
                 return { success: false, status: 'transformer.validation_failed', errors: validation[:errors] } unless validation[:valid]
@@ -33,14 +40,16 @@ module Legion
             { success: true, **result }
           end
 
-          def render_transformation(transformation, payload, engine: nil)
+          def render_transformation(transformation, payload, engine: nil, engine_options: {})
             eng = if engine
                     Engines::Registry.fetch(engine)
                   else
                     Engines::Registry.detect(transformation)
                   end
 
-            rendered = eng.render(transformation, payload)
+            rendered = eng.render(transformation, payload, **engine_options)
+            return rendered if rendered.is_a?(Hash) && rendered[:success] == false
+
             rendered.is_a?(String) ? from_json(rendered) : rendered
           end
 

data/lib/legion/extensions/transformer/version.rb

@@ -3,7 +3,7 @@
 module Legion
   module Extensions
     module Transformer
-      VERSION = '0.2.1'
+      VERSION = '0.3.0'
     end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: lex-transformer
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.3.0
 platform: ruby
 authors:
 - Esity
@@ -47,6 +47,7 @@ files:
 - lib/legion/extensions/transformer.rb
 - lib/legion/extensions/transformer/actors/transform.rb
 - lib/legion/extensions/transformer/client.rb
+- lib/legion/extensions/transformer/definitions.rb
 - lib/legion/extensions/transformer/engines/base.rb
 - lib/legion/extensions/transformer/engines/erb.rb
 - lib/legion/extensions/transformer/engines/jsonpath.rb