active_genie 0.26.5 → 0.27.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ad223ae84ac62d9375af07ba18dca9152296cac5e7c432a6fd07118107c4ab6e
-  data.tar.gz: 5faa2d7ad89cf2200296f94892470248a544d86cd37880191777226f121d9cf4
+  metadata.gz: 752b254545d62f5e618ca56935dca6e89dce29e0353af995f96f276ac2852662
+  data.tar.gz: 7df0c3373970c5da1cf12cea004839a17e8489bf52b2e3d14dedcd61c04f91cc
 SHA512:
-  metadata.gz: 2a6bd702967d5948e37fed48a9b92d0c2061b4893579e67d526cb3bf9b71b44889b7a5a457d5c78a768f74d6cc650beb17a52c1ce00bf1a5c7d2dc553d88b866
-  data.tar.gz: 845fecdb4f036ce4a97b99ea54d0f5600544d6ce0b052da99e4533e4555d774ce09bff1dc037d3752f30aabe69acc5b9362ab77eaf17c62b8ebb5244e02b5df8
+  metadata.gz: dd4d86d0bd8a8be3c44355d997883db01b424a070ab51b555e1f1b8a10c3c0fbc46f8afc4c5b8764b4462b9e1e5e711be38ad25a77a21e26b3ea55668fd61ccf
+  data.tar.gz: bb69781f7f05a339833e553ab7cb1dc805044cda7fb2615712c7fea3f5ce9f71dce82b38edd1b70aa6fca36d6ef6538c573b34058e80498a0a5a65bf6fd58821

data/README.md

@@ -1,13 +1,16 @@
 # ActiveGenie 🧞‍♂️
-> The lodash for GenAI, stop reinventing the wheel
+> The Lodash for GenAI: Real Value + Consistent + Model-Agnostic
 
 [![Gem Version](https://badge.fury.io/rb/active_genie.svg?icon=si%3Arubygems)](https://badge.fury.io/rb/active_genie)
 [![Ruby](https://github.com/roriz/active_genie/actions/workflows/benchmark.yml/badge.svg)](https://github.com/roriz/active_genie/actions/workflows/benchmark.yml)
 
-ActiveGenie is a Ruby gem that provides valuable solutions powered by Generative AI (GenAI) models. Just like Lodash or ActiveStorage, ActiveGenie brings a set of Modules to reach real value fast and reliably.
-ActiveGenie is backed by a custom benchmarking system that ensures consistent quality and performance across different models and providers in every release.
+ActiveGenie is a developer-first library for GenAI workflows, designed to help you compare, rank, and score LLM outputs with consistency and model-agnostic flexibility. Think of it as the Lodash for GenAI: built for real value, consistent results, and freedom from vendor lock-in. It solves the biggest pain in GenAI today: getting predictable, trustworthy answers across use cases, models, and providers.
 
-## Installation
+Behind the scenes, a custom benchmarking system keeps everything consistent across LLM vendors and versions, release after release.
+
+For full documentation, visit [activegenie.ai](https://activegenie.ai).
+
+# Installation
 
 1. Add to your Gemfile:
 ```ruby
@@ -32,9 +35,7 @@ ActiveGenie.configure do |config|
 end
 ```
 
-## Quick Start
-
-### Data Extractor
+## Quick start example
 
 Extract structured data from text using AI-powered analysis, handling informal language and complex expressions.
 
@@ -71,193 +72,9 @@ result = ActiveGenie::DataExtractor.call(
 #    }
 ```
 
-*Recommended model*: `gpt-4o-mini`
-
-Features:
-- Structured data extraction with type validation
-- Schema-based extraction with custom constraints
-- Informal text analysis (litotes, hedging)
-- Detailed explanations for extracted values
-
-See the [Data Extractor README](lib/active_genie/data_extractor/README.md) for informal text processing, advanced schemas, and detailed interface documentation.
-
-### Scoring
-Text evaluation system that provides detailed scoring and feedback using multiple expert reviewers. Get balanced scoring through AI-powered expert reviewers that automatically adapt to your content.
-
-```ruby
-text = "The code implements a binary search algorithm with O(log n) complexity"
-criteria = "Evaluate technical accuracy and clarity"
-
-result = ActiveGenie::Scoring.call(
-  text,
-  criteria,
-  config: { provider: :anthropic, model: 'claude-3-5-haiku-20241022' } # optional
-)
-# => {
-#      algorithm_expert_score: 95,
-#      algorithm_expert_reasoning: "Accurately describes binary search and its complexity",
-#      technical_writer_score: 90,
-#      technical_writer_reasoning: "Clear and concise explanation of the algorithm",
-#      final_score: 92.5
-#    }
-```
-
-*Recommended model*: `claude-3-5-haiku-20241022`
-
-Features:
-- Multi-reviewer evaluation with automatic expert selection
-- Detailed feedback with scoring reasoning
-- Customizable reviewer weights
-- Flexible evaluation criteria
-
-See the [Scoring README](lib/active_genie/scoring/README.md) for advanced usage, custom reviewers, and detailed interface documentation.
-
-### Battle
-AI-powered battle evaluation system that determines winners between two players based on specified criteria.
-
-```ruby
-require 'active_genie'
-
-player_a = "Implementation uses dependency injection for better testability"
-player_b = "Code has high test coverage but tightly coupled components"
-criteria = "Evaluate code quality and maintainability"
-
-result = ActiveGenie::Battle.call(
-  player_a,
-  player_b,
-  criteria,
-  config: { provider: :google, model: 'gemini-2.0-flash-lite' } # optional
-)
-# => {
-#      winner_player: "Implementation uses dependency injection for better testability",
-#      reasoning: "Player 1 implementation demonstrates better maintainability through dependency injection,
-#                 which allows for easier testing and component replacement. While Player 2 has good test coverage,
-#                 the tight coupling makes the code harder to maintain and modify.",
-#      what_could_be_changed_to_avoid_draw: "Focus on specific architectural patterns and design principles"
-#    }
-```
-
-*Recommended model*: `claude-3-5-haiku`
-
-Features:
-- Multi-reviewer evaluation with automatic expert selection
-- Detailed feedback with scoring reasoning
-- Customizable reviewer weights
-- Flexible evaluation criteria
-
-See the [Battle README](lib/active_genie/battle/README.md) for advanced usage, custom reviewers, and detailed interface documentation.
-
-### Ranking
-The Ranking module provides competitive ranking through multi-stage evaluation:
-
-```ruby
-require 'active_genie'
-
-players = ['REST API', 'GraphQL API', 'SOAP API', 'gRPC API', 'Websocket API']
-criteria = "Best one to be used into a high changing environment"
-
-result = ActiveGenie::Ranking.call(
-  players,
-  criteria,
-  config: { provider: :google, model: 'gemini-2.0-flash-lite' } # optional
-)
-# => {
-#      winner_player: "gRPC API",
-#      reasoning: "gRPC API is the best one to be used into a high changing environment",
-#    }
-```
-
-*Recommended model*: `gemini-2.0-flash-lite`
-
-- **Multi-phase ranking system** combining expert scoring and ELO algorithms
-- **Automatic elimination** of inconsistent performers using statistical analysis
-- **Dynamic ranking adjustments** based on simulated pairwise battles, from bottom to top
-
-See the [Ranking README](lib/active_genie/ranking/README.md) for implementation details, configuration, and advanced ranking strategies.
-
-### Text Summarizer (Future)
-### Categorizer (Future)
-### Language detector (Future)
-### Translator (Future)
-### Sentiment analyzer (Future)
-
-## Benchmarking 🧪
-
-ActiveGenie includes a comprehensive benchmarking system to ensure consistent, high-quality outputs across different LLM models and providers.
-
-```ruby
-# Run all benchmarks
-bundle exec rake active_genie:benchmark
-
-# Run benchmarks for a specific module
-bundle exec rake active_genie:benchmark[data_extractor]
-```
-
-### Latest Results
-
-| Model | Overall Precision |
-|-------|-------------------|
-| claude-3-5-haiku-20241022 | 92.25% |
-| gemini-2.0-flash-lite | 84.25% |
-| gpt-4o-mini | 62.75% |
-| deepseek-chat | 57.25% |
-
-See the [Benchmark README](benchmark/README.md) for detailed results, methodology, and how to contribute to our test suite.
-
-## Configuration
-
-| Config | Description | Default |
-|--------|-------------|---------|
-| `provider` | LLM provider (openai, anthropic, etc) | `nil` |
-| `model` | Model to use | `nil` |
-| `api_key` | Provider API key | `nil` |
-| `timeout` | Request timeout in seconds | `5` |
-| `max_retries` | Maximum retry attempts | `3` |
+## Documentation
 
-> **Note:** Each module can append its own set of configuration, see the individual module documentation for details.
-
-## How to create a new provider
-
-ActiveGenie supports adding custom providers to integrate with different LLM services. To create a new provider:
-
-1. Create a configuration class for your provider in `lib/active_genie/configuration/providers/`:
-2. Register your client
-
-```ruby
-class InternalCompanyApi
-  # @param messages [Array<Hash>] A list of messages representing the conversation history.
-  #   Each hash should have :role ('user', 'assistant', or 'system') and :content (String).
-  # @param function [Hash] A JSON schema definition describing the desired output format.
-  # @return [Hash, nil] The parsed JSON object matching the schema, or nil if parsing fails or content is empty.
-  def function_calling(messages, function)
-    # ...
-  end
-end
-
-ActiveGenie.configure do |config|
-  config.llm.client = InternalCompanyApi
-end
-# or
-ActiveGenie::Battle.call('player_a', 'player_b', 'criteria', { client: InternalCompanyApi })
-```
-
-## Observability
-Fundamental to managing any production system, observability is crucial for GenAI features. At a minimum, track these key metrics:
-
-- Usage Rate (e.g., uses_per_minute): Detect anomalies like sudden traffic spikes (potential DDoS) or drops (feature outage or declining usage).
-- Failure/Retry Rate (e.g., retry_count, fail_count): Monitor the frequency of errors. Exceeding a defined threshold should trigger downtime or degradation alerts.
-- Token Consumption (e.g., tokens_used): Track usage to monitor costs. Set alerts if tokens_used * price_per_token exceeds budget thresholds.
-
-```ruby
-ActiveGenie.configure do |config|
-  config.log.add_observer(scope: { code: :llm_usage }) do |log|
-    puts "LLM Usage: #{log[:model]} - #{log[:total_tokens]} tokens"
-  end
-  config.log.add_observer(scope: { code: :retry_attempt }) do |log|
-    puts "Retry Attempt: #{log[:attempt]} of #{log[:max_retries]}"
-  end
-end
-```
+For full documentation, visit [activegenie.ai](https://activegenie.ai).
 
 ## Contributing
 

data/VERSION

@@ -1 +1 @@
-0.26.5
+0.27.1

data/lib/active_genie/battle/generalist.rb

@@ -48,19 +48,19 @@ module ActiveGenie
           config: @config
         )
 
-        ActiveGenie::Logger.call({
-                                   code: :battle,
-                                   player_a: @player_a[0..30],
-                                   player_b: @player_b[0..30],
-                                   criteria: @criteria[0..30],
-                                   winner: response['impartial_judge_winner'],
-                                   reasoning: response['impartial_judge_winner_reasoning']
-                                 })
+        @config.logger.call({
+                              code: :battle,
+                              player_a: @player_a[0..30],
+                              player_b: @player_b[0..30],
+                              criteria: @criteria[0..30],
+                              winner: response['impartial_judge_winner'],
+                              reasoning: response['impartial_judge_winner_reasoning']
+                            })
 
         response_formatted(response)
       end
 
-      PROMPT = File.read(File.join(__dir__, 'generalist.md'))
+      PROMPT = File.read(File.join(__dir__, 'generalist.prompt.md'))
       FUNCTION = JSON.parse(File.read(File.join(__dir__, 'generalist.json')), symbolize_names: true)
 
       private

data/lib/active_genie/clients/providers/anthropic_client.rb

@@ -55,7 +55,7 @@ module ActiveGenie
       def request(payload)
         response = post(url, payload, headers:)
 
-        ActiveGenie::Logger.call(
+        @config.logger.call(
           {
             code: :llm_usage,
             input_tokens: response.dig('usage', 'input_tokens'),
@@ -67,7 +67,7 @@ module ActiveGenie
             usage: response['usage']
           }
         )
-        ActiveGenie::Logger.call(
+        @config.logger.call(
           {
             code: :function_calling,
             fine_tune: true,

data/lib/active_genie/clients/providers/base_client.rb

@@ -151,7 +151,7 @@ module ActiveGenie
       #
       # @param details [Hash] Request and response details
       def log_request_details(uri:, request:, response:, start_time:, parsed_response:)
-        ActiveGenie::Logger.call(
+        @config.logger.call(
           {
             code: :http_request,
             uri: uri.to_s,
@@ -176,7 +176,7 @@ module ActiveGenie
           sleep_time = retry_delay * (2**retries)
           retries += 1
 
-          ActiveGenie::Logger.call(
+          @config.logger.call(
             code: :retry_attempt,
             attempt: retries,
             max_retries: max_retries,

data/lib/active_genie/clients/providers/deepseek_client.rb

@@ -32,7 +32,7 @@ module ActiveGenie
                 "Invalid response: #{response}"
         end
 
-        ActiveGenie::Logger.call({ code: :function_calling, fine_tune: true, payload:, response: })
+        @config.logger.call({ code: :function_calling, fine_tune: true, payload:, response: })
 
         response
       end
@@ -44,7 +44,7 @@ module ActiveGenie
 
         return nil if response.nil?
 
-        ActiveGenie::Logger.call(
+        @config.logger.call(
           {
             code: :llm_usage,
             input_tokens: response.dig('usage', 'prompt_tokens'),

data/lib/active_genie/clients/providers/google_client.rb

@@ -34,6 +34,8 @@ module ActiveGenie
         json_string = response&.dig('candidates', 0, 'content', 'parts', 0, 'text')
         return nil if json_string.nil? || json_string.empty?
 
+        @config.logger.call({ code: :function_calling, fine_tune: true, payload:, parsed_response: json_string })
+
         normalize_response(json_string)
       end
 
@@ -48,7 +50,7 @@ module ActiveGenie
       def request(payload, params)
         response = post(url, payload, headers: DEFAULT_HEADERS, params:)
 
-        ActiveGenie::Logger.call(
+        @config.logger.call(
           {
             code: :llm_usage,
             input_tokens: response['usageMetadata']['promptTokenCount'] || 0,
@@ -59,15 +61,6 @@ module ActiveGenie
           }
         )
 
-        ActiveGenie::Logger.call(
-          {
-            code: :function_calling,
-            fine_tune: true,
-            payload:,
-            parsed_response: response&.dig('candidates', 0, 'content', 'parts', 0, 'text')
-          }
-        )
-
         response
       end
 

data/lib/active_genie/clients/providers/openai_client.rb

@@ -29,7 +29,7 @@ module ActiveGenie
 
         raise InvalidResponseError, "Invalid response: #{response}" if response.nil? || response.keys.empty?
 
-        ActiveGenie::Logger.call({ code: :function_calling, fine_tune: true, payload:, response: })
+        @config.logger.call({ code: :function_calling, fine_tune: true, payload:, response: })
 
         response
       end
@@ -41,7 +41,7 @@ module ActiveGenie
 
         return nil if response.nil?
 
-        ActiveGenie::Logger.call(
+        @config.logger.call(
           {
             code: :llm_usage,
             input_tokens: response.dig('usage', 'prompt_tokens'),

data/lib/active_genie/config/log_config.rb

@@ -4,6 +4,7 @@ module ActiveGenie
   module Config
     class LogConfig
       attr_writer :file_path, :fine_tune_file_path
+      attr_reader :output, :observers
 
       def file_path
         @file_path || 'log/active_genie.log'
@@ -13,8 +14,12 @@ module ActiveGenie
         @fine_tune_file_path || 'log/active_genie_fine_tune.log'
       end
 
-      def output
-        @output || ->(log) { $stdout.puts log }
+      def additional_context
+        @additional_context || {}
+      end
+
+      def additional_context=(context)
+        @additional_context = additional_context.merge(context).compact
       end
 
       def output=(output)
@@ -23,29 +28,17 @@ module ActiveGenie
         @output = output
       end
 
-      def output_call(log)
-        output&.call(log)
-
-        Array(@observers).each do |obs|
-          next unless obs[:scope].all? { |key, value| log[key.to_sym] == value }
-
-          obs[:observer]&.call(log)
-        rescue StandardError => e
-          ActiveGenie::Logger.call(code: :observer_error, **obs, error: e.message)
-        end
-      end
-
       def add_observer(observers: [], scope: {}, &block)
         @observers ||= []
 
         raise ArgumentError, 'Scope must be a hash' if scope && !scope.is_a?(Hash)
 
-        @observers << { observer: block, scope: } if block_given?
         Array(observers).each do |observer|
           next unless observer.respond_to?(:call)
 
           @observers << { observer:, scope: }
         end
+        @observers << { observer: block, scope: } if block_given?
       end
 
       def remove_observer(observers)
@@ -60,6 +53,10 @@ module ActiveGenie
 
       def merge(config_params = {})
         dup.tap do |config|
+          config_params.compact.each do |key, value|
+            config.send("#{key}=", value) if config.respond_to?("#{key}=")
+          end
+
           config.add_observer(config_params[:observers]) if config_params[:observers]
         end
       end

data/lib/active_genie/config/providers/anthropic_config.rb

@@ -48,8 +48,8 @@ module ActiveGenie
         # Retrieves the model name designated for the upper tier (e.g., most capable).
         # Defaults to 'claude-3-opus'.
         # @return [String] The upper tier model name.
-        def upper_tier_model
-          @upper_tier_model || 'claude-3-opus-20240229'
+        def higher_tier_model
+          @higher_tier_model || 'claude-3-opus-20240229'
         end
       end
     end

data/lib/active_genie/config/providers/deepseek_config.rb

@@ -41,8 +41,8 @@ module ActiveGenie
         # Retrieves the model name designated for the upper tier (e.g., most capable).
         # Defaults to 'deepseek-reasoner'.
         # @return [String] The upper tier model name.
-        def upper_tier_model
-          @upper_tier_model || 'deepseek-reasoner'
+        def higher_tier_model
+          @higher_tier_model || 'deepseek-reasoner'
         end
       end
     end

data/lib/active_genie/config/providers/google_config.rb

@@ -43,8 +43,8 @@ module ActiveGenie
         # Retrieves the model name designated for the upper tier (e.g., most capable).
         # Defaults to 'gemini-2.5-pro-experimental'.
         # @return [String] The upper tier model name.
-        def upper_tier_model
-          @upper_tier_model || 'gemini-2.5-pro-experimental'
+        def higher_tier_model
+          @higher_tier_model || 'gemini-2.5-pro-experimental'
         end
       end
     end

data/lib/active_genie/config/providers/openai_config.rb

@@ -41,8 +41,8 @@ module ActiveGenie
         # Retrieves the model name designated for the upper tier (e.g., most capable).
         # Defaults to 'o1-preview'.
         # @return [String] The upper tier model name.
-        def upper_tier_model
-          @upper_tier_model || 'o3-mini'
+        def higher_tier_model
+          @higher_tier_model || 'o3-mini'
         end
       end
     end

data/lib/active_genie/config/providers/provider_base.rb

@@ -7,7 +7,7 @@ module ActiveGenie
         NAME = :unknown
 
         attr_writer :api_key, :organization, :api_url, :client,
-                    :lower_tier_model, :middle_tier_model, :upper_tier_model
+                    :lower_tier_model, :middle_tier_model, :higher_tier_model
 
         # Maps a symbolic tier (:lower_tier, :middle_tier, :upper_tier) to a specific model name.
         # Falls back to the lower_tier_model if the tier is nil or unrecognized.
@@ -17,7 +17,7 @@ module ActiveGenie
           {
             lower_tier: lower_tier_model,
             middle_tier: middle_tier_model,
-            upper_tier: upper_tier_model
+            upper_tier: higher_tier_model
           }[tier&.to_sym] || lower_tier_model
         end
 
@@ -31,7 +31,7 @@ module ActiveGenie
             api_url:,
             lower_tier_model:,
             middle_tier_model:,
-            upper_tier_model:,
+            higher_tier_model:,
             **config
           }
         end
@@ -80,7 +80,7 @@ module ActiveGenie
         # Retrieves the model name designated for the upper tier (e.g., most capable).
         # Defaults to 'o1-preview'.
         # @return [String] The upper tier model name.
-        def upper_tier_model
+        def higher_tier_model
           raise NotImplementedError, 'Subclasses must implement this method'
         end
       end

data/lib/active_genie/configuration.rb

@@ -38,6 +38,10 @@ module ActiveGenie
       @llm ||= Config::LlmConfig.new
     end
 
+    def logger
+      @logger ||= ActiveGenie::Logger.new(log_config: log)
+    end
+
     SUB_CONFIGS = %w[log providers llm ranking scoring data_extractor battle].freeze
 
     def merge(config_params = {})
@@ -54,20 +58,28 @@ module ActiveGenie
 
         new_configuration.send("#{key}=", new_config)
       end
+      @logger = nil
 
       new_configuration
     end
 
+    # Merges a sub config with the config_params.
+    # Examples:
+    #   config.merge({ 'log' => { file_path: 'log/active_genie.log' } })
+    #   config.merge({ log: { file_path: 'log/active_genie.log' } })
+    #   config.merge({ file_path: 'log/active_genie.log' })
+    # these are all the same
+
     def sub_config_merge(config, key, config_params)
-      if config_params.key?(key.to_s)
+      if config_params&.key?(key.to_s)
         config.merge(config_params[key.to_s])
-      elsif config_params.key?(key.to_sym)
+      elsif config_params&.key?(key.to_sym)
         config.merge(config_params[key.to_sym])
       else
-        config.merge(config_params)
+        config.merge(config_params || {})
       end
     end
 
-    attr_writer :log, :providers, :ranking, :scoring, :data_extractor, :battle, :llm
+    attr_writer :log, :providers, :ranking, :scoring, :data_extractor, :battle, :llm, :logger
   end
 end

data/lib/active_genie/data_extractor/generalist.rb

@@ -87,7 +87,7 @@ module ActiveGenie
           config: @config
         )
 
-        ActiveGenie::Logger.call(
+        @config.logger.call(
           {
             code: :data_extractor,
             text: @text[0..30],
@@ -119,7 +119,7 @@ module ActiveGenie
       end
 
       def prompt
-        File.read(File.join(__dir__, 'generalist.md'))
+        File.read(File.join(__dir__, 'generalist.prompt.md'))
       end
     end
   end