language-operator 0.1.61 → 0.1.62

data/lib/language_operator/learning/task_synthesizer.rb

@@ -1,288 +0,0 @@
-# frozen_string_literal: true
-
-require 'json'
-
-module LanguageOperator
-  module Learning
-    # Synthesizes deterministic code for neural tasks using LLM analysis
-    #
-    # TaskSynthesizer uses an LLM to analyze task definitions and execution traces,
-    # then generates optimized Ruby code if the task can be made deterministic.
-    # This approach can handle inconsistent traces better than pure pattern matching
-    # because the LLM can understand intent and unify variations.
-    #
-    # @example Basic usage
-    #   synthesizer = TaskSynthesizer.new(
-    #     llm_client: my_llm_client,
-    #     validator: ASTValidator.new
-    #   )
-    #
-    #   result = synthesizer.synthesize(
-    #     task_definition: task_def,
-    #     traces: execution_traces,
-    #     available_tools: tool_list
-    #   )
-    #
-    #   if result[:is_deterministic]
-    #     puts result[:code]
-    #   end
-    class TaskSynthesizer
-      # Template file name
-      TEMPLATE_FILE = 'task_synthesis.tmpl'
-
-      # Initialize synthesizer
-      #
-      # @param llm_client [Object] Client for LLM API calls (must respond to #chat)
-      # @param validator [Agent::Safety::ASTValidator] Code validator
-      # @param logger [Logger, nil] Logger instance
-      def initialize(llm_client:, validator:, logger: nil)
-        @llm_client = llm_client
-        @validator = validator
-        @logger = logger || ::Logger.new($stdout, level: ::Logger::INFO)
-      end
-
-      # Synthesize deterministic code for a task
-      #
-      # @param task_definition [Dsl::TaskDefinition] Task to optimize
-      # @param traces [Array<Hash>] Execution traces from TraceAnalyzer
-      # @param available_tools [Array<String>] List of available tool names
-      # @param consistency_score [Float] Current consistency score
-      # @param common_pattern [String, nil] Most common tool pattern
-      # @return [Hash] Synthesis result with :is_deterministic, :code, :explanation, :confidence
-      def synthesize(task_definition:, traces:, available_tools: [], consistency_score: 0.0, common_pattern: nil)
-        # Build prompt from template
-        prompt = build_prompt(
-          task_definition: task_definition,
-          traces: traces,
-          available_tools: available_tools,
-          consistency_score: consistency_score,
-          common_pattern: common_pattern
-        )
-
-        @logger.info("Task synthesis prompt:\n#{prompt}")
-
-        # Call LLM
-        response = call_llm(prompt)
-
-        # Parse JSON response
-        result = parse_response(response)
-
-        # Validate generated code if present
-        if result[:is_deterministic] && result[:code]
-          validation = validate_code(result[:code])
-          unless validation[:valid]
-            @logger.warn("Generated code failed validation: #{validation[:errors].join(', ')}")
-            result[:validation_errors] = validation[:errors]
-            result[:is_deterministic] = false
-            result[:explanation] = "Generated code failed safety validation: #{validation[:errors].first}"
-          end
-        end
-
-        result
-      rescue StandardError => e
-        @logger.error("Task synthesis failed: #{e.message}")
-        @logger.error(e.backtrace&.first(10)&.join("\n"))
-        {
-          is_deterministic: false,
-          confidence: 0.0,
-          explanation: "Synthesis error: #{e.message}",
-          code: nil
-        }
-      end
-
-      private
-
-      # Build synthesis prompt from template
-      #
-      # @return [String] Rendered prompt
-      def build_prompt(task_definition:, traces:, available_tools:, consistency_score:, common_pattern:)
-        template = load_template
-
-        # Pre-render traces since our template engine doesn't support loops
-        traces_text = format_traces(traces)
-        inputs_text = format_schema(task_definition.inputs)
-        outputs_text = format_schema(task_definition.outputs)
-        tools_text = available_tools.map { |t| "- #{t}" }.join("\n")
-
-        # Count unique patterns
-        unique_patterns = traces.map { |t| (t[:tool_calls] || []).map { |tc| tc[:tool_name] }.join(' → ') }.uniq.size
-
-        # Build data hash for template substitution
-        data = {
-          'TaskName' => task_definition.name.to_s,
-          'Instructions' => task_definition.instructions || '(none)',
-          'Inputs' => inputs_text,
-          'Outputs' => outputs_text,
-          'TaskCode' => format_task_code(task_definition),
-          'TraceCount' => traces.size.to_s,
-          'Traces' => traces_text,
-          'CommonPattern' => common_pattern || '(none detected)',
-          'ConsistencyScore' => (consistency_score * 100).round(1).to_s,
-          'UniquePatternCount' => unique_patterns.to_s,
-          'ToolsList' => tools_text.empty? ? '(none available)' : tools_text
-        }
-
-        render_template(template, data)
-      end
-
-      # Load template file
-      #
-      # @return [String] Template content
-      def load_template
-        template_path = File.join(__dir__, '..', 'templates', TEMPLATE_FILE)
-        File.read(template_path)
-      end
-
-      # Render Go-style template with data
-      #
-      # @param template [String] Template content
-      # @param data [Hash] Data to substitute
-      # @return [String] Rendered content
-      def render_template(template, data)
-        result = template.dup
-
-        # Remove range blocks (we pre-render these)
-        result.gsub!(/{{range.*?}}.*?{{end}}/m, '')
-
-        # Remove if blocks for empty values
-        result.gsub!(/{{if not \.Inputs}}.*?{{end}}/m, '')
-        result.gsub!(/{{if \.InputSummary}}.*?{{end}}/m, '')
-
-        # Replace simple variables {{.Variable}}
-        data.each do |key, value|
-          result.gsub!("{{.#{key}}}", value.to_s)
-        end
-
-        result
-      end
-
-      # Format traces for template
-      #
-      # @param traces [Array<Hash>] Execution traces
-      # @return [String] Formatted traces text
-      def format_traces(traces)
-        return '(no traces available)' if traces.empty?
-
-        traces.first(10).each_with_index.map do |trace, idx|
-          format_single_trace(trace, idx)
-        end.join("\n")
-      end
-
-      # Format a single trace execution
-      #
-      # @param trace [Hash] Single trace data
-      # @param idx [Integer] Trace index
-      # @return [String] Formatted trace
-      def format_single_trace(trace, idx)
-        tool_sequence = trace[:tool_calls]&.map { |tc| tc[:tool_name] }&.join(' → ') || '(no tools)'
-        duration = trace[:duration_ms]&.round(1) || 'unknown'
-        inputs_summary = trace[:inputs]&.keys&.join(', ') || 'none'
-        tool_details = format_tool_calls(trace[:tool_calls])
-
-        <<~TRACE
-          ### Execution #{idx + 1}
-          - **Tool Sequence:** #{tool_sequence}
-          - **Duration:** #{duration}ms
-          - **Inputs:** #{inputs_summary}
-          - **Tool Calls:**
-          #{tool_details}
-        TRACE
-      end
-
-      # Format tool call details
-      #
-      # @param tool_calls [Array<Hash>, nil] Tool call data
-      # @return [String] Formatted tool calls
-      def format_tool_calls(tool_calls)
-        return '  (no tool calls)' unless tool_calls&.any?
-
-        tool_calls.map do |tc|
-          details = "  - #{tc[:tool_name]}"
-          details += "\n    Args: #{tc[:arguments]}" if tc[:arguments]
-          details += "\n    Result: #{tc[:result]}" if tc[:result]
-          details
-        end.join("\n")
-      end
-
-      # Format schema hash as text
-      #
-      # @param schema [Hash] Input/output schema
-      # @return [String] Formatted text
-      def format_schema(schema)
-        return '(none)' if schema.nil? || schema.empty?
-
-        schema.map { |k, v| "- #{k}: #{v}" }.join("\n")
-      end
-
-      # Format task definition as Ruby code
-      #
-      # @param task_def [Dsl::TaskDefinition] Task definition
-      # @return [String] Ruby code representation
-      def format_task_code(task_def)
-        inputs_str = (task_def.inputs || {}).map { |k, v| "#{k}: '#{v}'" }.join(', ')
-        outputs_str = (task_def.outputs || {}).map { |k, v| "#{k}: '#{v}'" }.join(', ')
-
-        <<~RUBY
-          task :#{task_def.name},
-            instructions: "#{task_def.instructions}",
-            inputs: { #{inputs_str} },
-            outputs: { #{outputs_str} }
-        RUBY
-      end
-
-      # Call LLM with prompt
-      #
-      # @param prompt [String] Synthesis prompt
-      # @return [String] LLM response
-      def call_llm(prompt)
-        @llm_client.chat(prompt)
-      end
-
-      # Parse JSON response from LLM
-      #
-      # @param response [String] LLM response text
-      # @return [Hash] Parsed result
-      def parse_response(response)
-        # Extract JSON from response (may be wrapped in markdown code block)
-        json_match = response.match(/```json\s*(.*?)\s*```/m) ||
-                     response.match(/\{.*\}/m)
-
-        json_str = json_match ? json_match[1] || json_match[0] : response
-
-        parsed = JSON.parse(json_str, symbolize_names: true)
-
-        {
-          is_deterministic: parsed[:is_deterministic] == true,
-          confidence: parsed[:confidence].to_f,
-          explanation: parsed[:explanation] || 'No explanation provided',
-          code: parsed[:code]
-        }
-      rescue JSON::ParserError => e
-        @logger.warn("Failed to parse LLM response as JSON: #{e.message}")
-        {
-          is_deterministic: false,
-          confidence: 0.0,
-          explanation: "Failed to parse synthesis response: #{e.message}",
-          code: nil
-        }
-      end
-
-      # Validate generated code
-      #
-      # @param code [String] Ruby code to validate
-      # @return [Hash] Validation result with :valid and :errors
-      def validate_code(code)
-        errors = @validator.validate(code)
-        {
-          valid: errors.empty?,
-          errors: errors
-        }
-      rescue StandardError => e
-        {
-          valid: false,
-          errors: ["Validation error: #{e.message}"]
-        }
-      end
-    end
-  end
-end

data/lib/language_operator/learning/trace_analyzer.rb

@@ -1,285 +0,0 @@
-# frozen_string_literal: true
-
-require 'logger'
-require_relative 'adapters/base_adapter'
-
-module LanguageOperator
-  module Learning
-    # Analyzes OpenTelemetry traces to detect patterns in task execution
-    #
-    # The TraceAnalyzer queries OTLP backends (SigNoz, Jaeger, Tempo) to retrieve
-    # execution traces for neural tasks, then analyzes them to determine if they
-    # exhibit consistent patterns that can be codified into symbolic implementations.
-    #
-    # Auto-detects available backends in order: SigNoz → Jaeger → Tempo
-    # Falls back gracefully if no backend is available (learning disabled).
-    #
-    # @example Basic usage
-    #   analyzer = TraceAnalyzer.new(
-    #     endpoint: ENV['OTEL_QUERY_ENDPOINT'],
-    #     api_key: ENV['OTEL_QUERY_API_KEY']
-    #   )
-    #
-    #   analysis = analyzer.analyze_patterns(task_name: 'fetch_user_data')
-    #   if analysis && analysis[:consistency] >= 0.85
-    #     puts "Task is ready for learning!"
-    #     puts "Tool sequence: #{analysis[:common_pattern]}"
-    #   end
-    #
-    # @example Explicit backend selection
-    #   ENV['OTEL_QUERY_BACKEND'] = 'signoz'
-    #   analyzer = TraceAnalyzer.new(endpoint: 'https://signoz.example.com')
-    class TraceAnalyzer
-      # Minimum pattern consistency required for learning (configurable)
-      DEFAULT_CONSISTENCY_THRESHOLD = 0.85
-
-      # Default time range for queries (24 hours)
-      DEFAULT_TIME_RANGE = 24 * 60 * 60
-
-      # Initialize trace analyzer with backend connection
-      #
-      # @param endpoint [String, nil] OTLP backend endpoint (auto-detected from ENV if nil)
-      # @param api_key [String, nil] API key for authentication (if required)
-      # @param backend [String, nil] Explicit backend type ('signoz', 'jaeger', 'tempo')
-      # @param logger [Logger, nil] Logger instance (creates default if nil)
-      def initialize(endpoint: nil, api_key: nil, backend: nil, logger: nil)
-        @endpoint = endpoint || ENV.fetch('OTEL_QUERY_ENDPOINT', nil)
-        @api_key = api_key || ENV.fetch('OTEL_QUERY_API_KEY', nil)
-        @backend_type = backend || ENV.fetch('OTEL_QUERY_BACKEND', nil)
-        @logger = logger || ::Logger.new($stdout, level: ::Logger::WARN)
-        @adapter = detect_backend_adapter
-      end
-
-      # Check if learning is available (backend connected)
-      #
-      # @return [Boolean] True if a backend adapter is available
-      def available?
-        !@adapter.nil?
-      end
-
-      # Query task execution traces from backend
-      #
-      # @param task_name [String] Name of task to query
-      # @param agent_name [String, nil] Optional agent name to filter by
-      # @param limit [Integer] Maximum number of traces to return
-      # @param time_range [Integer, Range<Time>] Time range in seconds or explicit range
-      # @return [Array<Hash>] Task execution data
-      def query_task_traces(task_name:, agent_name: nil, limit: 100, time_range: DEFAULT_TIME_RANGE)
-        unless available?
-          @logger.warn('No OTLP backend available, learning disabled')
-          return []
-        end
-
-        range = normalize_time_range(time_range)
-
-        filter = { task_name: task_name }
-        filter[:agent_name] = agent_name if agent_name
-
-        spans = @adapter.query_spans(
-          filter: filter,
-          time_range: range,
-          limit: limit
-        )
-
-        @adapter.extract_task_data(spans)
-      rescue StandardError => e
-        @logger.error("Failed to query task traces: #{e.message}")
-        @logger.debug(e.backtrace.join("\n"))
-        []
-      end
-
-      # Analyze task execution patterns for consistency
-      #
-      # Determines if a neural task exhibits consistent behavior that can be
-      # learned and converted to a symbolic implementation.
-      #
-      # @param task_name [String] Name of task to analyze
-      # @param agent_name [String, nil] Optional agent name to filter by
-      # @param min_executions [Integer] Minimum executions required for analysis
-      # @param consistency_threshold [Float] Required consistency (0.0-1.0)
-      # @param time_range [Integer, Range<Time>, nil] Time range for query (seconds or explicit range)
-      # @return [Hash, nil] Analysis results or nil if insufficient data
-      def analyze_patterns(task_name:, agent_name: nil, min_executions: 10, consistency_threshold: DEFAULT_CONSISTENCY_THRESHOLD,
-                           time_range: nil)
-        executions = query_task_traces(task_name: task_name, agent_name: agent_name, limit: 1000, time_range: time_range || DEFAULT_TIME_RANGE)
-
-        if executions.empty?
-          @logger.info("No executions found for task '#{task_name}'")
-          return nil
-        end
-
-        if executions.size < min_executions
-          @logger.info("Insufficient executions for task '#{task_name}': #{executions.size}/#{min_executions}")
-          return {
-            task_name: task_name,
-            execution_count: executions.size,
-            required_count: min_executions,
-            ready_for_learning: false,
-            reason: "Need #{min_executions - executions.size} more executions"
-          }
-        end
-
-        consistency_data = calculate_consistency(executions)
-
-        # Task is ready for learning only if:
-        # 1. Consistency meets threshold
-        # 2. There's an actual tool pattern to learn (not empty/pure LLM)
-        has_pattern = !consistency_data[:common_pattern].nil? && !consistency_data[:common_pattern].empty?
-        ready = consistency_data[:score] >= consistency_threshold && has_pattern
-
-        {
-          task_name: task_name,
-          execution_count: executions.size,
-          consistency_score: consistency_data[:score],
-          consistency_threshold: consistency_threshold,
-          ready_for_learning: ready,
-          reason: has_pattern ? nil : 'No tool calls to learn (pure LLM task)',
-          common_pattern: consistency_data[:common_pattern],
-          input_signatures: consistency_data[:input_signatures],
-          analysis_timestamp: Time.now.iso8601
-        }
-      end
-
-      # Calculate pattern consistency across executions
-      #
-      # Groups executions by input signature and analyzes tool call sequences
-      # to determine how often the same pattern is used for the same inputs.
-      #
-      # @param executions [Array<Hash>] Task execution data
-      # @return [Hash] Consistency analysis with score and common pattern
-      def calculate_consistency(executions)
-        # Group by input signature
-        by_inputs = executions.group_by { |ex| normalize_inputs(ex[:inputs]) }
-
-        # For each input signature, find the most common tool call pattern
-        signature_patterns = by_inputs.map do |input_sig, execs|
-          patterns = execs.map { |ex| normalize_tool_calls(ex[:tool_calls]) }
-          pattern_counts = patterns.tally
-          most_common = pattern_counts.max_by { |_, count| count }
-
-          {
-            input_signature: input_sig,
-            total_executions: execs.size,
-            most_common_pattern: most_common[0],
-            pattern_count: most_common[1],
-            consistency: most_common[1].to_f / execs.size
-          }
-        end
-
-        # Overall consistency is weighted average across input signatures
-        total_execs = executions.size
-        weighted_consistency = signature_patterns.sum do |sig_data|
-          weight = sig_data[:total_executions].to_f / total_execs
-          weight * sig_data[:consistency]
-        end
-
-        # Find the globally most common pattern
-        all_patterns = signature_patterns.map { |s| s[:most_common_pattern] }
-        common_pattern = all_patterns.max_by { |p| all_patterns.count(p) }
-
-        {
-          score: weighted_consistency.round(3),
-          common_pattern: common_pattern,
-          input_signatures: signature_patterns.size
-        }
-      end
-
-      private
-
-      # Detect and initialize the appropriate backend adapter
-      #
-      # Auto-detection order: SigNoz → Jaeger → Tempo
-      # Falls back to nil if no backend is available
-      #
-      # @return [BaseAdapter, nil] Initialized adapter or nil
-      def detect_backend_adapter
-        return nil unless @endpoint
-
-        # Explicit backend selection
-        if @backend_type
-          adapter = create_adapter(@backend_type)
-          return adapter if adapter
-
-          @logger.warn("Requested backend '#{@backend_type}' not available, trying auto-detection")
-        end
-
-        # Auto-detect with fallback chain
-        %w[signoz jaeger tempo].each do |backend|
-          adapter = create_adapter(backend)
-          if adapter
-            @logger.info("Detected OTLP backend: #{backend} at #{@endpoint}")
-            return adapter
-          end
-        end
-
-        @logger.warn("No OTLP backend available at #{@endpoint}, learning disabled")
-        nil
-      end
-
-      # Create adapter instance for specified backend
-      #
-      # @param backend_type [String] Backend type
-      # @return [BaseAdapter, nil] Adapter instance or nil if unavailable
-      def create_adapter(backend_type)
-        require_relative "adapters/#{backend_type}_adapter"
-
-        adapter_class = case backend_type.downcase
-                        when 'signoz'
-                          Adapters::SignozAdapter
-                        when 'jaeger'
-                          Adapters::JaegerAdapter
-                        when 'tempo'
-                          Adapters::TempoAdapter
-                        else
-                          @logger.error("Unknown backend type: #{backend_type}")
-                          return nil
-                        end
-
-        return nil unless adapter_class.available?(@endpoint, @api_key)
-
-        adapter_class.new(@endpoint, @api_key, logger: @logger)
-      rescue LoadError => e
-        @logger.debug("Adapter #{backend_type} not available: #{e.message}")
-        nil
-      rescue StandardError => e
-        @logger.error("Failed to create #{backend_type} adapter: #{e.message}")
-        nil
-      end
-
-      # Normalize time range to Range<Time>
-      #
-      # @param time_range [Integer, Range<Time>] Time range
-      # @return [Range<Time>] Normalized time range
-      def normalize_time_range(time_range)
-        case time_range
-        when Range
-          time_range
-        when Integer
-          (Time.now - time_range)..Time.now
-        else
-          (Time.now - DEFAULT_TIME_RANGE)..Time.now
-        end
-      end
-
-      # Normalize inputs for comparison
-      #
-      # @param inputs [Hash] Task inputs
-      # @return [String] Normalized input signature
-      def normalize_inputs(inputs)
-        return '' unless inputs.is_a?(Hash)
-
-        inputs.sort.to_h.to_s
-      end
-
-      # Normalize tool calls for pattern matching
-      #
-      # @param tool_calls [Array<Hash>] Tool call sequence
-      # @return [String] Normalized pattern signature
-      def normalize_tool_calls(tool_calls)
-        return '' unless tool_calls.is_a?(Array)
-
-        tool_calls.map { |tc| tc[:tool_name] }.join(' → ')
-      end
-    end
-  end
-end

data/lib/language_operator/templates/task_synthesis.tmpl

@@ -1,98 +0,0 @@
-You are analyzing whether an agentic task can be converted to symbolic Ruby code.
-
-## What "Symbolic" Means
-
-A task is **symbolic** (can be optimized) if it follows a predictable algorithm:
-- Reading files, calling APIs, transforming data = SYMBOLIC (even though outputs vary based on data)
-- The same code logic applies regardless of the actual data values
-- Conditional branches based on data (if file exists, if value > threshold) are fine
-
-A task is **neural** (cannot be optimized) only if it requires:
-- Creative text generation (writing stories, poems, marketing copy)
-- Subjective reasoning or judgment calls
-- Understanding nuanced human intent that varies per request
-
-**Key insight:** File I/O, API calls, and data transformation are deterministic CODE even if their outputs depend on external state. "Read a file and count lines" is symbolic - the algorithm is fixed.
-
-## Task Definition
-
-**Name:** {{.TaskName}}
-**Instructions:** {{.Instructions}}
-
-**Inputs:**
-{{.Inputs}}
-
-**Outputs:**
-{{.Outputs}}
-
-## Current Task Code
-
-```ruby
-{{.TaskCode}}
-```
-
-## Execution Traces ({{.TraceCount}} samples)
-
-{{.Traces}}
-
-## Pattern Analysis
-
-- **Most Common Pattern:** {{.CommonPattern}}
-- **Pattern Consistency:** {{.ConsistencyScore}}%
-- **Unique Patterns Observed:** {{.UniquePatternCount}}
-
-## Available Tools
-
-{{.ToolsList}}
-
----
-
-## Your Task
-
-Analyze whether this neural task can be converted to symbolic Ruby code.
-
-Questions to ask:
-1. Is there a clear algorithm implied by the instructions?
-2. Do the tool call patterns show a logical sequence?
-3. Can conditional logic handle the variations seen in traces?
-
-Tasks that ARE symbolic (optimize these):
-- "Read file X and return its contents" → read_file, return content
-- "Check if file exists, create if not" → get_file_info, conditional write_file
-- "Fetch data from API and transform it" → api_call, data transformation
-
-Tasks that are NOT symbolic (don't optimize):
-- "Write a creative story continuation"
-- "Decide what the user probably meant"
-- "Generate marketing copy for this product"
-
-## Output Format
-
-Respond with valid JSON:
-
-```json
-{
-  "is_deterministic": true/false,
-  "confidence": 0.0-1.0,
-  "explanation": "Brief explanation",
-  "code": "Ruby code if symbolic, null otherwise"
-}
-```
-
-**Code Requirements (if symbolic):**
-- Use the DSL task format with a do block:
-  ```ruby
-  task :task_name,
-       instructions: "Keep the original instructions for documentation",
-       inputs: { ... },
-       outputs: { ... } do |inputs|
-    # Helper methods available: execute_tool, execute_task, execute_llm
-    result = execute_tool(:tool_name, { arg: value })
-    { output_key: result }
-  end
-  ```
-- Use `execute_tool(:tool_name, { arg: value })` for MCP tool calls
-- Use `execute_task(:task_name, inputs: { ... })` to call other tasks
-- Access inputs via the `inputs` hash parameter
-- Return a hash matching the output schema
-- Do NOT use system(), eval(), or other unsafe methods