active_genie 0.0.10 → 0.0.12

data/lib/active_genie/{league → ranking}/README.md

@@ -1,22 +1,22 @@
-# league
+# Ranking
 
-The `ActiveGenie::League` module organizes players based on scores derived from textual evaluations and then ranks them using a multi-stage process. It leverages the scoring system from the `ActiveGenie::Scoring` module to assign initial scores, and then applies advanced ranking methods to produce a competitive league.
+The `ActiveGenie::Ranking` module organizes players based on scores derived from textual evaluations and then ranks them using a multi-stage process. It leverages the scoring system from the `ActiveGenie::Scoring` module to assign initial scores, and then applies advanced ranking methods to produce a competitive ranking.
 
 ## Overview
 
-The league system performs the following steps:
+The ranking system performs the following steps:
 
 1. **Initial Scoring**: Each player’s textual content is evaluated using `ActiveGenie::Scoring`. This produces a `score` based on multiple expert reviews.
 
 2. **Elimination of Poor Performers**: Players whose scores show a high coefficient of variation (indicating inconsistency) are progressively eliminated. This ensures that only competitive candidates continue in the ranking process.
 
-3. **ELO Ranking**: If there are more than 10 eligible players, an ELO-based ranking is applied. Battles between players are simulated via `ActiveGenie::Battle`, and scores are updated using an ELO algorithm tailored to the league.
+3. **ELO Ranking**: If there are more than 10 eligible players, an ELO-based ranking is applied. Battles between players are simulated via `ActiveGenie::Battle`, and scores are updated using an ELO algorithm tailored to the ranking.
 
 4. **Free for all Matches**: Finally, the remaining players engage in head-to-head matches where each unique pair competes. Match outcomes (wins, losses, draws) are recorded to finalize the rankings.
 
 ## Components
 
-- **league**: Orchestrates the entire ranking process. Initializes player scores, eliminates outliers, and coordinates ranking rounds.
+- **ranking**: Orchestrates the entire ranking process. Initializes player scores, eliminates outliers, and coordinates ranking rounds.
 
 - **EloRanking**: Applies an ELO-based system to rank players through simulated battles. It updates players’ ELO scores based on match outcomes and predefined rules (including penalties for losses).
 
@@ -24,10 +24,10 @@ The league system performs the following steps:
 
 ## Usage
 
-Call the league using:
+Call the ranking using:
 
 ```ruby
-result = ActiveGenie::League::league.call(players, criteria, config: {})
+result = ActiveGenie::Ranking.call(players, criteria, config: {})
 ```
 
 - `players`: A collection of player instances, each containing textual content to be scored.

data/lib/active_genie/ranking/elo_round.rb

@@ -0,0 +1,113 @@
+require_relative '../battle/basic'
+
+module ActiveGenie::Ranking
+  class EloRound
+    def self.call(...)
+      new(...).call
+    end
+
+    def initialize(players, criteria, config: {})
+      @players = players
+      @relegation_tier = players.calc_relegation_tier
+      @defender_tier = players.calc_defender_tier
+      @criteria = criteria
+      @config = config
+      @tmp_defenders = []
+    end
+
+    def call
+      ActiveGenie::Logger.with_context(log_context) do
+        matches.each do |player_a, player_b|
+          # TODO: battle can take a while, can be parallelized
+          winner, loser = battle(player_a, player_b)
+
+          next if winner.nil? || loser.nil?
+
+          new_winner_elo, new_loser_elo = calculate_new_elo(winner.elo, loser.elo)
+
+          winner.elo = new_winner_elo
+          loser.elo = new_loser_elo
+        end
+
+        # TODO: add a round report. Duration, Elo changes, etc.
+      end
+    end
+
+    private
+
+    BATTLE_PER_PLAYER = 3
+    LOSE_PENALTY = 15
+    K = 32
+
+    def matches
+      @relegation_tier.reduce([]) do |matches, attack_player|
+        BATTLE_PER_PLAYER.times do
+          matches << [attack_player, next_defense_player].shuffle
+        end
+        matches
+      end
+    end
+
+    def next_defense_player
+      @tmp_defenders = @defender_tier if @tmp_defenders.size.zero?
+
+      @tmp_defenders.shuffle.pop!
+    end
+
+    def battle(player_a, player_b)
+      result = ActiveGenie::Battle.basic(
+        player_a,
+        player_b,
+        @criteria,
+        config: @config
+      )
+      
+      winner, loser = case result['winner']
+        when 'player_a' then [player_a, player_b]
+        when 'player_b' then [player_b, player_a]
+        when 'draw' then [nil, nil]
+      end
+
+      ActiveGenie::Logger.debug({
+        step: :elo_round_battle,
+        player_ids: [player_a.id, player_b.id],
+        winner_id: winner&.id,
+        loser_id: loser&.id,
+        reasoning: result['reasoning']
+      })
+
+      [winner, loser]
+    end
+
+    # INFO: Read more about the Elo rating system on https://en.wikipedia.org/wiki/Elo_rating_system
+    def calculate_new_elo(winner_elo, loser_elo)
+      expected_score_a = 1 / (1 + 10**((loser_elo - winner_elo) / 400))
+      expected_score_b = 1 - expected_score_a
+
+      new_winner_elo = [winner_elo + K * (1 - expected_score_a), max_defense_elo].min
+      new_loser_elo = [loser_elo + K * (1 - expected_score_b) - LOSE_PENALTY, min_relegation_elo].max
+
+      [new_winner_elo, new_loser_elo]
+    end
+
+    def max_defense_elo
+      @defender_tier.max_by(&:elo).elo
+    end
+
+    def min_relegation_elo
+      @relegation_tier.min_by(&:elo).elo
+    end
+
+    def log_context
+      { elo_round_id: }
+    end
+
+    def elo_round_id
+      relegation_tier_ids = @relegation_tier.map(&:id).join(',')
+      defender_tier_ids = @defender_tier.map(&:id).join(',')
+
+      ranking_unique_key = [relegation_tier_ids, defender_tier_ids, @criteria, @config.to_json].join('-')
+      Digest::MD5.hexdigest(ranking_unique_key)
+    end
+  end
+end

data/lib/active_genie/ranking/free_for_all.rb

@@ -0,0 +1,76 @@
+require_relative '../battle/basic'
+
+module ActiveGenie::Ranking
+  class FreeForAll
+    def self.call(...)
+      new(...).call
+    end
+
+    def initialize(players, criteria, config: {})
+      @players = players
+      @criteria = criteria
+      @config = config
+    end
+
+    def call
+      ActiveGenie::Logger.with_context(log_context) do
+        matches.each do |player_a, player_b|
+          winner, loser = battle(player_a, player_b)
+          
+          if winner.nil? || loser.nil?
+            player_a.draw!
+            player_b.draw!
+          else
+            winner.win!
+            loser.lose!
+          end
+        end
+
+        # TODO: add a freeForAll report. Duration, Elo changes, etc.
+      end
+    end
+
+    private
+
+    # TODO: reduce the number of matches based on transitivity.
+    #       For example, if A is better than B, and B is better than C, then battle between A and C should be auto win A
+    def matches
+      @players.eligible.combination(2).to_a
+    end
+
+    def battle(player_a, player_b)
+      result = ActiveGenie::Battle.basic(
+        player_a,
+        player_b,
+        @criteria,
+        config: @config
+      )
+
+      winner, loser = case result['winner']
+        when 'player_a' then [player_a, player_b, result['reasoning']]
+        when 'player_b' then [player_b, player_a, result['reasoning']]
+        when 'draw' then [nil, nil, result['reasoning']]
+      end
+
+      ActiveGenie::Logger.debug({
+        step: :free_for_all_battle,
+        player_ids: [player_a.id, player_b.id],
+        winner_id: winner&.id,
+        loser_id: loser&.id,
+        reasoning: result['reasoning']
+      })
+
+      [winner, loser]
+    end
+
+    def log_context
+      { free_for_all_id: }
+    end
+
+    def free_for_all_id
+      eligible_ids = @players.eligible.map(&:id).join(',')
+      ranking_unique_key = [eligible_ids, @criteria, @config.to_json].join('-')
+      Digest::MD5.hexdigest(ranking_unique_key)
+    end
+  end
+end

data/lib/active_genie/ranking/player.rb

@@ -0,0 +1,97 @@
+require 'digest'
+
+module ActiveGenie::Ranking
+  class Player
+    def initialize(params)
+      params = { content: params } if params.is_a?(String)
+
+      @content = params.dig(:content) || params
+      @id = params.dig(:id) || Digest::MD5.hexdigest(@content)
+      @score = params.dig(:score) || nil
+      @elo = params.dig(:elo) || nil
+      @ffa_win_count = params.dig(:ffa_win_count) || 0
+      @ffa_lose_count = params.dig(:ffa_lose_count) || 0
+      @ffa_draw_count = params.dig(:ffa_draw_count) || 0
+      @eliminated = params.dig(:eliminated) || nil
+    end
+
+    attr_reader :id, :content, :score, :elo, 
+      :ffa_win_count, :ffa_lose_count, :ffa_draw_count,
+      :eliminated
+    attr_accessor :rank
+
+    def score=(value)
+      @score = value
+      ActiveGenie::Logger.debug({ step: :new_score, player_id: id, score: value })
+    end
+
+    def elo
+      generate_elo_by_score if @elo.nil?
+
+      @elo
+    end
+
+    def elo=(value)
+      @elo = value
+      ActiveGenie::Logger.debug({ step: :new_elo, player_id: id, elo: value })
+    end
+
+    def eliminated=(value)
+      @eliminated = value
+      ActiveGenie::Logger.debug({ step: :new_eliminated, player_id: id, eliminated: value })
+    end
+
+    def draw!
+      @ffa_draw_count += 1
+      ActiveGenie::Logger.debug({ step: :new_ffa_score, player_id: id, result: 'draw', ffa_score: })
+    end
+
+    def win!
+      @ffa_win_count += 1
+      ActiveGenie::Logger.debug({ step: :new_ffa_score, player_id: id, result: 'win', ffa_score: })
+    end
+
+    def lose!
+      @ffa_lose_count += 1
+      ActiveGenie::Logger.debug({ step: :new_ffa_score, player_id: id, result: 'lose', ffa_score: })
+    end
+
+    def ffa_score
+      @ffa_win_count * 3 + @ffa_draw_count
+    end
+
+    def to_h
+      {
+        id:, content:, score:, elo:,
+        ffa_win_count:, ffa_lose_count:, ffa_draw_count:,
+        eliminated:, ffa_score:
+      }
+    end
+
+    def method_missing(method_name, *args, &block)
+      if method_name == :[] && args.size == 1
+        attr_name = args.first.to_sym
+
+        if respond_to?(attr_name) 
+          return send(attr_name)
+        else
+          return nil
+        end
+      end
+
+      super
+    end
+
+    def respond_to_missing?(method_name, include_private = false)
+      method_name == :[] || super
+    end
+
+    private
+    
+    BASE_ELO = 1000
+
+    def generate_elo_by_score
+      @elo = BASE_ELO + ((@score || 0) - 50)
+    end
+  end
+end

data/lib/active_genie/{league → ranking}/players_collection.rb

@@ -1,7 +1,6 @@
-require_relative '../utils/math'
 require_relative './player'
 
-module ActiveGenie::Leaderboard
+module ActiveGenie::Ranking
   class PlayersCollection
     def initialize(param_players)
       @players = build(param_players)
@@ -9,9 +8,11 @@ module ActiveGenie::Leaderboard
     attr_reader :players
 
     def coefficient_of_variation
-      score_list = eligible.map(&:score)
+      score_list = eligible.map(&:score).compact
+      return nil if score_list.empty?
+
       mean = score_list.sum.to_f / score_list.size
-      return nil if mean == 0  # To avoid division by zero
+      return nil if mean == 0
 
       variance = score_list.map { |num| (num - mean) ** 2 }.sum / score_list.size
       standard_deviation = Math.sqrt(variance)
@@ -19,11 +20,11 @@ module ActiveGenie::Leaderboard
       (standard_deviation / mean) * 100
     end
 
-    def tier_relegation
+    def calc_relegation_tier
       eligible[(tier_size*-1)..-1]
     end
 
-    def tier_defense
+    def calc_defender_tier
       eligible[(tier_size*-2)...(tier_size*-1)]
     end
 
@@ -35,18 +36,24 @@ module ActiveGenie::Leaderboard
       @players.reject(&:eliminated).size
     end
 
+    def elo_eligible?
+      eligible.size > 15
+    end
+
+    def sorted
+      @players.sort_by { |p| [-p.ffa_score, -(p.elo || 0), -(p.score || 0)] }
+      @players.each_with_index { |p, i| p.rank = i + 1 }
+      @players
+    end
+
     def to_h
-      sorted.map(&:to_h)
+      sorted.map { |p| p.to_h }
     end
 
     def method_missing(...)
       @players.send(...)
     end
 
-    def sorted
-      @players.sort_by { |p| [-p.league_score, -(p.elo || 0), -p.score] }
-    end
-
     private
 
     def build(param_players)

data/lib/active_genie/ranking/ranking.rb

@@ -0,0 +1,98 @@
+require_relative './players_collection'
+require_relative './free_for_all'
+require_relative './elo_round'
+require_relative './ranking_scoring'
+
+# This class orchestrates player ranking through multiple evaluation stages
+# using Elo ranking and free-for-all match simulations.
+# 1. Sets initial scores
+# 2. Eliminates low performers
+# 3. Runs Elo ranking (for large groups)
+# 4. Conducts free-for-all matches
+#
+# @example Basic usage
+#   Ranking.call(players, criteria)
+#
+# @param param_players [Array<Hash|String>] Collection of player objects to evaluate
+#   Example: ["Circle", "Triangle", "Square"]
+#            or
+#   [
+#     { content: "Circle", score: 10 },
+#     { content: "Triangle", score: 7 },
+#     { content: "Square", score: 5 }
+#   ]
+# @param criteria [String] Evaluation criteria configuration
+#   Example: "What is more similar to the letter 'O'?"
+# @param config [Hash] Additional configuration config
+#   Example: { model: "gpt-4o", api_key: ENV['OPENAI_API_KEY'] }
+# @return [Hash] Final ranked player results
+module ActiveGenie::Ranking
+  class Ranking
+    def self.call(...)
+      new(...).call
+    end
+
+    def initialize(param_players, criteria, reviewers: [], config: {})
+      @param_players = param_players
+      @criteria = criteria
+      @reviewers = Array(reviewers).compact.uniq
+      @config = ActiveGenie::Configuration.to_h(config)
+      @players = nil
+    end
+
+    def call
+      @players = PlayersCollection.new(@param_players)
+
+      ActiveGenie::Logger.with_context(log_context) do
+        set_initial_player_scores!
+        eliminate_obvious_bad_players!
+
+        while @players.elo_eligible?
+          run_elo_round!
+          eliminate_relegation_players!
+        end
+  
+        run_free_for_all!
+      end
+
+      @players.sorted
+    end
+
+    private
+
+    SCORE_VARIATION_THRESHOLD = 10
+
+    def set_initial_player_scores!
+      RankingScoring.call(@players, @criteria, reviewers: @reviewers, config: @config)
+    end
+
+    def eliminate_obvious_bad_players!
+      while @players.coefficient_of_variation >= SCORE_VARIATION_THRESHOLD
+        @players.eligible.last.eliminated = 'variation_too_high'
+      end
+    end
+
+    def run_elo_round!
+      EloRound.call(@players, @criteria, config: @config)
+    end
+
+    def eliminate_relegation_players!
+      @players.calc_relegation_tier.each { |player| player.eliminated = 'relegation_tier' }
+    end
+
+    def run_free_for_all!
+      FreeForAll.call(@players, @criteria, config: @config)
+    end
+
+    def log_context
+      { config: @config[:log], ranking_id: }
+    end
+
+    def ranking_id
+      player_ids = @players.map(&:id).join(',')
+      ranking_unique_key = [player_ids, @criteria, @config.to_json].join('-')
+
+      Digest::MD5.hexdigest(ranking_unique_key)
+    end
+  end
+end

data/lib/active_genie/ranking/ranking_scoring.rb

@@ -0,0 +1,71 @@
+require_relative '../scoring/recommended_reviewers'
+
+module ActiveGenie::Ranking
+  class RankingScoring
+    def self.call(...)
+      new(...).call
+    end
+
+    def initialize(players, criteria, reviewers: [], config: {})
+      @players = players
+      @criteria = criteria
+      @config = ActiveGenie::Configuration.to_h(config)
+      @reviewers = Array(reviewers).compact.uniq
+    end
+
+    def call
+      ActiveGenie::Logger.with_context(log_context) do
+        @reviewers = generate_reviewers
+
+        players_without_score.each do |player|
+          # TODO: This can take a while, can be parallelized
+          player.score = generate_score(player)
+        end
+      end
+    end
+
+    private
+
+    def players_without_score
+      @players_without_score ||= @players.select { |player| player.score.nil? }
+    end
+
+    def generate_score(player)
+      score, reasoning = ActiveGenie::Scoring::Basic.call(
+        player.content,
+        @criteria,
+        @reviewers,
+        config: @config
+      ).values_at('final_score', 'final_reasoning')
+
+      ActiveGenie::Logger.debug({step: :new_score, player_id: player.id, score:, reasoning: })
+
+      score
+    end
+
+    def generate_reviewers
+      return @reviewers if @reviewers.size > 0
+
+      reviewer1, reviewer2, reviewer3 = ActiveGenie::Scoring::RecommendedReviewers.call(
+        [@players.sample.content, @players.sample.content].join("\n\n"),
+        @criteria,
+        config: @config
+      ).values_at('reviewer1', 'reviewer2', 'reviewer3')
+      
+      ActiveGenie::Logger.debug({step: :new_reviewers, reviewers: [reviewer1, reviewer2, reviewer3] })
+
+      [reviewer1, reviewer2, reviewer3]
+    end
+
+    def log_context
+      { ranking_scoring_id: }
+    end
+
+    def ranking_scoring_id
+      player_ids = players_without_score.map(&:id).join(',')
+      ranking_unique_key = [player_ids, @criteria, @config.to_json].join('-')
+
+      Digest::MD5.hexdigest(ranking_unique_key)
+    end
+  end
+end

data/lib/active_genie/ranking.rb

@@ -0,0 +1,12 @@
+require_relative 'ranking/ranking'
+
+module ActiveGenie
+  # See the [ranking README](lib/active_genie/ranking/README.md) for more information.
+  module Ranking
+    module_function
+
+    def call(...)
+      Ranking.call(...)
+    end
+  end
+end

data/lib/active_genie/scoring/README.md

@@ -58,7 +58,7 @@ Main interface for scoring text content.
 - `reviewers` [Array<String>] - Optional list of specific reviewers
 - `config` [Hash] - Additional configuration config
 
-### `RecommendedReviews.call(text, criteria, config: {})`
+### `RecommendedReviewers.call(text, criteria, config: {})`
 Recommends appropriate reviewers based on content and criteria.
 
 #### Parameters