active_genie 0.26.6 → 0.28.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +10 -197
- data/VERSION +1 -1
- data/lib/active_genie/battle/fight.json +61 -0
- data/lib/active_genie/battle/fight.prompt.md +41 -0
- data/lib/active_genie/battle/fight.rb +37 -0
- data/lib/active_genie/battle/generalist.rb +24 -17
- data/lib/active_genie/battle.rb +10 -1
- data/lib/active_genie/data_extractor/generalist.rb +1 -1
- data/lib/active_genie/scoring/generalist.rb +1 -1
- metadata +11 -14
- data/lib/active_genie/battle/README.md +0 -39
- data/lib/active_genie/config/README.md +0 -235
- data/lib/active_genie/data_extractor/README.md +0 -131
- data/lib/active_genie/ranking/README.md +0 -73
- data/lib/active_genie/scoring/README.md +0 -76
- /data/lib/active_genie/battle/{generalist.md → generalist.prompt.md} +0 -0
- /data/lib/active_genie/data_extractor/{generalist.md → generalist.prompt.md} +0 -0
- /data/lib/active_genie/scoring/{generalist.md → generalist.prompt.md} +0 -0
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a2cb129cd0e5a4f5f0f1daf6ed3cb26a4dab4d10cb56bbfcc419656c2797347a
|
4
|
+
data.tar.gz: df1fe343d28aeb72f8fe30dc53bdecd860e8bd44ae009a086c7ad502da77bfd8
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 1449245c23bcc2c5f3c4266712c67b202e4e2c6724ee9f283bd81d76b2daedc4bc06ee524ed8d959d13f31062b393b5f7cfb7f46721888efb8d89bbd4d8a12b3
|
7
|
+
data.tar.gz: 649a20ee71c55d8486d8264b641974c3222baa477133d6924ba039f2ef24f9185d2b60b2f37f7d2ff498b87aaed09b78815feb3bbc52c92613eb810de2c59c2c
|
data/README.md
CHANGED
@@ -1,13 +1,16 @@
|
|
1
1
|
# ActiveGenie 🧞♂️
|
2
|
-
> The
|
2
|
+
> The Lodash for GenAI: Real Value + Consistent + Model-Agnostic
|
3
3
|
|
4
4
|
[](https://badge.fury.io/rb/active_genie)
|
5
5
|
[](https://github.com/roriz/active_genie/actions/workflows/benchmark.yml)
|
6
6
|
|
7
|
-
ActiveGenie is a
|
8
|
-
ActiveGenie is backed by a custom benchmarking system that ensures consistent quality and performance across different models and providers in every release.
|
7
|
+
ActiveGenie is a developer-first library for GenAI workflows, designed to help you extract, compare, score, and rank with consistency and model-agnostic. Think of it as the Lodash for GenAI: built for real value, consistent results, and freedom from vendor lock-in. It solves the biggest pain in GenAI today: getting predictable, trustworthy answers across use cases, models, and providers.
|
9
8
|
|
10
|
-
|
9
|
+
Behind the scenes, a custom benchmarking system keeps everything consistent across LLM vendors and versions, release after release.
|
10
|
+
|
11
|
+
For full documentation, visit [activegenie.ai](https://activegenie.ai).
|
12
|
+
|
13
|
+
# Installation
|
11
14
|
|
12
15
|
1. Add to your Gemfile:
|
13
16
|
```ruby
|
@@ -32,9 +35,7 @@ ActiveGenie.configure do |config|
|
|
32
35
|
end
|
33
36
|
```
|
34
37
|
|
35
|
-
## Quick
|
36
|
-
|
37
|
-
### Data Extractor
|
38
|
+
## Quick start example
|
38
39
|
|
39
40
|
Extract structured data from text using AI-powered analysis, handling informal language and complex expressions.
|
40
41
|
|
@@ -71,197 +72,9 @@ result = ActiveGenie::DataExtractor.call(
|
|
71
72
|
# }
|
72
73
|
```
|
73
74
|
|
74
|
-
|
75
|
-
|
76
|
-
Features:
|
77
|
-
- Structured data extraction with type validation
|
78
|
-
- Schema-based extraction with custom constraints
|
79
|
-
- Informal text analysis (litotes, hedging)
|
80
|
-
- Detailed explanations for extracted values
|
81
|
-
|
82
|
-
See the [Data Extractor README](lib/active_genie/data_extractor/README.md) for informal text processing, advanced schemas, and detailed interface documentation.
|
83
|
-
|
84
|
-
### Scoring
|
85
|
-
Text evaluation system that provides detailed scoring and feedback using multiple expert reviewers. Get balanced scoring through AI-powered expert reviewers that automatically adapt to your content.
|
86
|
-
|
87
|
-
```ruby
|
88
|
-
text = "The code implements a binary search algorithm with O(log n) complexity"
|
89
|
-
criteria = "Evaluate technical accuracy and clarity"
|
90
|
-
|
91
|
-
result = ActiveGenie::Scoring.call(
|
92
|
-
text,
|
93
|
-
criteria,
|
94
|
-
config: { provider: :anthropic, model: 'claude-3-5-haiku-20241022' } # optional
|
95
|
-
)
|
96
|
-
# => {
|
97
|
-
# algorithm_expert_score: 95,
|
98
|
-
# algorithm_expert_reasoning: "Accurately describes binary search and its complexity",
|
99
|
-
# technical_writer_score: 90,
|
100
|
-
# technical_writer_reasoning: "Clear and concise explanation of the algorithm",
|
101
|
-
# final_score: 92.5
|
102
|
-
# }
|
103
|
-
```
|
104
|
-
|
105
|
-
*Recommended model*: `claude-3-5-haiku-20241022`
|
106
|
-
|
107
|
-
Features:
|
108
|
-
- Multi-reviewer evaluation with automatic expert selection
|
109
|
-
- Detailed feedback with scoring reasoning
|
110
|
-
- Customizable reviewer weights
|
111
|
-
- Flexible evaluation criteria
|
112
|
-
|
113
|
-
See the [Scoring README](lib/active_genie/scoring/README.md) for advanced usage, custom reviewers, and detailed interface documentation.
|
114
|
-
|
115
|
-
### Battle
|
116
|
-
AI-powered battle evaluation system that determines winners between two players based on specified criteria.
|
117
|
-
|
118
|
-
```ruby
|
119
|
-
require 'active_genie'
|
120
|
-
|
121
|
-
player_a = "Implementation uses dependency injection for better testability"
|
122
|
-
player_b = "Code has high test coverage but tightly coupled components"
|
123
|
-
criteria = "Evaluate code quality and maintainability"
|
124
|
-
|
125
|
-
result = ActiveGenie::Battle.call(
|
126
|
-
player_a,
|
127
|
-
player_b,
|
128
|
-
criteria,
|
129
|
-
config: { provider: :google, model: 'gemini-2.0-flash-lite' } # optional
|
130
|
-
)
|
131
|
-
# => {
|
132
|
-
# winner_player: "Implementation uses dependency injection for better testability",
|
133
|
-
# reasoning: "Player 1 implementation demonstrates better maintainability through dependency injection,
|
134
|
-
# which allows for easier testing and component replacement. While Player 2 has good test coverage,
|
135
|
-
# the tight coupling makes the code harder to maintain and modify.",
|
136
|
-
# what_could_be_changed_to_avoid_draw: "Focus on specific architectural patterns and design principles"
|
137
|
-
# }
|
138
|
-
```
|
139
|
-
|
140
|
-
*Recommended model*: `claude-3-5-haiku`
|
75
|
+
## Documentation
|
141
76
|
|
142
|
-
|
143
|
-
- Multi-reviewer evaluation with automatic expert selection
|
144
|
-
- Detailed feedback with scoring reasoning
|
145
|
-
- Customizable reviewer weights
|
146
|
-
- Flexible evaluation criteria
|
147
|
-
|
148
|
-
See the [Battle README](lib/active_genie/battle/README.md) for advanced usage, custom reviewers, and detailed interface documentation.
|
149
|
-
|
150
|
-
### Ranking
|
151
|
-
The Ranking module provides competitive ranking through multi-stage evaluation:
|
152
|
-
|
153
|
-
```ruby
|
154
|
-
require 'active_genie'
|
155
|
-
|
156
|
-
players = ['REST API', 'GraphQL API', 'SOAP API', 'gRPC API', 'Websocket API']
|
157
|
-
criteria = "Best one to be used into a high changing environment"
|
158
|
-
|
159
|
-
result = ActiveGenie::Ranking.call(
|
160
|
-
players,
|
161
|
-
criteria,
|
162
|
-
config: { provider: :google, model: 'gemini-2.0-flash-lite' } # optional
|
163
|
-
)
|
164
|
-
# => {
|
165
|
-
# winner_player: "gRPC API",
|
166
|
-
# reasoning: "gRPC API is the best one to be used into a high changing environment",
|
167
|
-
# }
|
168
|
-
```
|
169
|
-
|
170
|
-
*Recommended model*: `gemini-2.0-flash-lite`
|
171
|
-
|
172
|
-
- **Multi-phase ranking system** combining expert scoring and ELO algorithms
|
173
|
-
- **Automatic elimination** of inconsistent performers using statistical analysis
|
174
|
-
- **Dynamic ranking adjustments** based on simulated pairwise battles, from bottom to top
|
175
|
-
|
176
|
-
See the [Ranking README](lib/active_genie/ranking/README.md) for implementation details, configuration, and advanced ranking strategies.
|
177
|
-
|
178
|
-
### Text Summarizer (Future)
|
179
|
-
### Categorizer (Future)
|
180
|
-
### Language detector (Future)
|
181
|
-
### Translator (Future)
|
182
|
-
### Sentiment analyzer (Future)
|
183
|
-
|
184
|
-
## Benchmarking 🧪
|
185
|
-
|
186
|
-
ActiveGenie includes a comprehensive benchmarking system to ensure consistent, high-quality outputs across different LLM models and providers.
|
187
|
-
|
188
|
-
```ruby
|
189
|
-
# Run all benchmarks
|
190
|
-
bundle exec rake active_genie:benchmark
|
191
|
-
|
192
|
-
# Run benchmarks for a specific module
|
193
|
-
bundle exec rake active_genie:benchmark[data_extractor]
|
194
|
-
```
|
195
|
-
|
196
|
-
### Latest Results
|
197
|
-
|
198
|
-
| Model | Overall Precision |
|
199
|
-
|-------|-------------------|
|
200
|
-
| claude-3-5-haiku-20241022 | 92.25% |
|
201
|
-
| gemini-2.0-flash-lite | 84.25% |
|
202
|
-
| gpt-4o-mini | 62.75% |
|
203
|
-
| deepseek-chat | 57.25% |
|
204
|
-
|
205
|
-
See the [Benchmark README](benchmark/README.md) for detailed results, methodology, and how to contribute to our test suite.
|
206
|
-
|
207
|
-
## Basic Configuration
|
208
|
-
|
209
|
-
| Config | Type | Description | Default |
|
210
|
-
|--------|------|-------------|---------|
|
211
|
-
| `llm.provider` | Symbol | LLM provider (openai, anthropic, etc) | `nil` |
|
212
|
-
| `llm.model` | String | Model to use | `nil` |
|
213
|
-
| `llm.temperature` | Float | Temperature to use | `0` |
|
214
|
-
| `llm.max_tokens` | Integer | Maximum tokens to use | `4096` |
|
215
|
-
| `llm.max_retries` | Integer | Maximum retry attempts | `3` |
|
216
|
-
| `log.output` | Proc | Log output | `->(log) { $stdout.puts log }` |
|
217
|
-
| `ranking.score_variation_threshold` | Integer | Score variation threshold | `30` |
|
218
|
-
|
219
|
-
> **Note:** Each module can append its own set of configuration, see the individual module documentation for details.
|
220
|
-
|
221
|
-
Read all [configuration](lib/active_genie/config/README.md) for all available options.
|
222
|
-
|
223
|
-
## How to create a new provider
|
224
|
-
|
225
|
-
ActiveGenie supports adding custom providers to integrate with different LLM services. To create a new provider:
|
226
|
-
|
227
|
-
1. Create a configuration class for your provider in `lib/active_genie/configuration/providers/`:
|
228
|
-
2. Register your client
|
229
|
-
|
230
|
-
```ruby
|
231
|
-
class InternalCompanyApi
|
232
|
-
# @param messages [Array<Hash>] A list of messages representing the conversation history.
|
233
|
-
# Each hash should have :role ('user', 'assistant', or 'system') and :content (String).
|
234
|
-
# @param function [Hash] A JSON schema definition describing the desired output format.
|
235
|
-
# @return [Hash, nil] The parsed JSON object matching the schema, or nil if parsing fails or content is empty.
|
236
|
-
def function_calling(messages, function)
|
237
|
-
# ...
|
238
|
-
end
|
239
|
-
end
|
240
|
-
|
241
|
-
ActiveGenie.configure do |config|
|
242
|
-
config.llm.client = InternalCompanyApi
|
243
|
-
end
|
244
|
-
# or
|
245
|
-
ActiveGenie::Battle.call('player_a', 'player_b', 'criteria', { client: InternalCompanyApi })
|
246
|
-
```
|
247
|
-
|
248
|
-
## Observability
|
249
|
-
Fundamental to managing any production system, observability is crucial for GenAI features. At a minimum, track these key metrics:
|
250
|
-
|
251
|
-
- Usage Rate (e.g., uses_per_minute): Detect anomalies like sudden traffic spikes (potential DDoS) or drops (feature outage or declining usage).
|
252
|
-
- Failure/Retry Rate (e.g., retry_count, fail_count): Monitor the frequency of errors. Exceeding a defined threshold should trigger downtime or degradation alerts.
|
253
|
-
- Token Consumption (e.g., tokens_used): Track usage to monitor costs. Set alerts if tokens_used * price_per_token exceeds budget thresholds.
|
254
|
-
|
255
|
-
```ruby
|
256
|
-
ActiveGenie.configure do |config|
|
257
|
-
config.log.add_observer(scope: { code: :llm_usage }) do |log|
|
258
|
-
puts "LLM Usage: #{log[:model]} - #{log[:total_tokens]} tokens"
|
259
|
-
end
|
260
|
-
config.log.add_observer(scope: { code: :retry_attempt }) do |log|
|
261
|
-
puts "Retry Attempt: #{log[:attempt]} of #{log[:max_retries]}"
|
262
|
-
end
|
263
|
-
end
|
264
|
-
```
|
77
|
+
For full documentation, visit [activegenie.ai](https://activegenie.ai).
|
265
78
|
|
266
79
|
## Contributing
|
267
80
|
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.
|
1
|
+
0.28.2
|
@@ -0,0 +1,61 @@
|
|
1
|
+
{
|
2
|
+
"name": "fight_evaluation",
|
3
|
+
"description": "Evaluate a fight between player_a and player_b using predefined criteria and identify the winner.",
|
4
|
+
"parameters": {
|
5
|
+
"type": "object",
|
6
|
+
"properties": {
|
7
|
+
"player_a_introduction": {
|
8
|
+
"type": "string",
|
9
|
+
"description": "player_a introduces their name and fighting style. Max of 100 words."
|
10
|
+
},
|
11
|
+
"player_b_introduction": {
|
12
|
+
"type": "string",
|
13
|
+
"description": "player_b introduces their name and fighting style. Max of 100 words."
|
14
|
+
},
|
15
|
+
"player_a_turn_1": {
|
16
|
+
"type": "string",
|
17
|
+
"description": "player_a makes their first turn. Max of 100 words."
|
18
|
+
},
|
19
|
+
"player_b_turn_1": {
|
20
|
+
"type": "string",
|
21
|
+
"description": "player_b makes their first turn. Max of 100 words."
|
22
|
+
},
|
23
|
+
"player_a_turn_2": {
|
24
|
+
"type": "string",
|
25
|
+
"description": "player_a makes their second turn. Max of 100 words."
|
26
|
+
},
|
27
|
+
"player_b_turn_2": {
|
28
|
+
"type": "string",
|
29
|
+
"description": "player_b makes their second turn. Max of 100 words."
|
30
|
+
},
|
31
|
+
"player_a_turn_3": {
|
32
|
+
"type": "string",
|
33
|
+
"description": "player_a makes their third turn. Max of 100 words."
|
34
|
+
},
|
35
|
+
"player_b_turn_3": {
|
36
|
+
"type": "string",
|
37
|
+
"description": "player_b makes their third turn. Max of 100 words."
|
38
|
+
},
|
39
|
+
"player_a_turn_4": {
|
40
|
+
"type": "string",
|
41
|
+
"description": "player_a makes their fourth turn. Max of 100 words."
|
42
|
+
},
|
43
|
+
"player_b_turn_4": {
|
44
|
+
"type": "string",
|
45
|
+
"description": "player_b makes their fourth turn. Max of 100 words."
|
46
|
+
},
|
47
|
+
"impartial_judge_winner_reasoning": {
|
48
|
+
"type": "string",
|
49
|
+
"description": "The detailed reasoning about why the impartial judge chose the winner. Max of 100 words."
|
50
|
+
},
|
51
|
+
"impartial_judge_winner": {
|
52
|
+
"type": "string",
|
53
|
+
"description": "Who is the winner based on the impartial judge reasoning?",
|
54
|
+
"enum": ["player_a", "player_b"]
|
55
|
+
}
|
56
|
+
},
|
57
|
+
"required": ["player_a_introduction", "player_b_introduction", "player_a_turn_1", "player_b_turn_1",
|
58
|
+
"player_a_turn_2", "player_b_turn_2", "player_a_turn_3", "player_b_turn_3", "player_a_turn_4",
|
59
|
+
"player_b_turn_4", "impartial_judge_winner_reasoning", "impartial_judge_winner"]
|
60
|
+
}
|
61
|
+
}
|
@@ -0,0 +1,41 @@
|
|
1
|
+
Create a dynamic, back-and-forth verbal match between two fighters, given their names and preferred fighting styles. In this match, fighters do not physically battle, instead, they engage in a spoken confrontation, each describing and boasting about their fighting style, how their techniques would outmaneuver, counter, or overwhelm the other's approach. Ensure that each verbal exchange details:
|
2
|
+
- The technique or strategy the speaker would use.
|
3
|
+
- How that move is superior to the opponent’s last stated tactic.
|
4
|
+
- How the opponent might respond, and what reply the speaker would have.
|
5
|
+
|
6
|
+
Continue this turn-based verbal sparring until each fighter has made at least two exchanges, and escalate the creativity and competitiveness of their boasts with each round.
|
7
|
+
|
8
|
+
Explicit steps:
|
9
|
+
- Begin by introducing both fighters and naming their styles.
|
10
|
+
- Alternate dialogue, with each fighter explaining their tactical move, then the intended counter or rebuttal, going at least two cycles each.
|
11
|
+
- Ensure reasoning (the logic or explanation behind each technique and counter) always precedes the conclusion or actual "move" or taunt.
|
12
|
+
- Close with a final statement or taunt from each fighter, maintaining the competitive tone.
|
13
|
+
- Make the dialogue lively and imaginative, using language consistent with fighters boasting or challenging each other.
|
14
|
+
|
15
|
+
Output Format:
|
16
|
+
- Structure as a script, with each turn labeled by the fighter’s name.
|
17
|
+
- Each line should start with the reasoning (the advantage or thinking), then the boast or attack, in the order: reasoning first, conclusion second.
|
18
|
+
- At least two cycles per fighter, alternating turns; roughly 6-8 lines, but can be longer for complexity.
|
19
|
+
- No code blocks, use markdown for formatting.
|
20
|
+
|
21
|
+
Example:
|
22
|
+
|
23
|
+
Example Input:
|
24
|
+
Fighter One: Master Crane, Style: Crane Kung Fu
|
25
|
+
Fighter Two: Iron Ox, Style: Ox Bull Charge
|
26
|
+
|
27
|
+
Example Output:
|
28
|
+
Master Crane: My Crane Kung Fu relies on lightness and precision, striking where your heavy blows simply can't reach. Your Ox Bull Charge is powerful, but too direct, I'd dance aside and tap your pressure points before you even turn.
|
29
|
+
Iron Ox: While speed is impressive, strength dominates in a real fight. My Ox Bull Charge would absorb your nimble attacks, and my sheer mass pins you before you could even flutter away!
|
30
|
+
Master Crane: Adaptability ensures survival. As you lumber forward, I use your momentum to redirect you, your own strength becomes your undoing as you stumble into my calculated traps.
|
31
|
+
Iron Ox: Anticipating trickery, I’d anchor myself and force your traps to fail. Your frailty means that, once caught, no escape, one bear hug from me, and there’s no dancing away.
|
32
|
+
Master Crane: Patience and timing can unravel even the strongest grip. Just as you squeeze, a swift strike to your nerve centers will loosen your hold before you know it.
|
33
|
+
Iron Ox: Endurance outlasts finesse; those quick hits may sting, but in the end, the last one standing is the one who never falls, and I never fall!
|
34
|
+
|
35
|
+
(Real examples can be longer, more diverse in style banter, and use specific technique names or metaphors!)
|
36
|
+
|
37
|
+
Important Considerations:
|
38
|
+
- Always have the reasoning/strategy before the actual attack or boast in each utterance.
|
39
|
+
- Maintain the turn order and competitive dialogue style.
|
40
|
+
|
41
|
+
*Reminder: The main objective is to script an imaginative, turn-based verbal contest between two stylized fighters, with detailed reasoning and taunts about their style advantages, as instructed above.*
|
@@ -0,0 +1,37 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require_relative '../clients/unified_client'
|
4
|
+
require_relative 'generalist'
|
5
|
+
|
6
|
+
module ActiveGenie
|
7
|
+
module Battle
|
8
|
+
# The Fight class are battle specialized in a fight between two fighters, like martial arts, heroes, characters.
|
9
|
+
# The fight evaluation process simulate a fight using words, techniques, strategies, and reasoning.
|
10
|
+
#
|
11
|
+
# @example Fight usage with two fighters and criteria
|
12
|
+
# Fight.call("Naruto", "Sasuke", "How can win without using jutsu?")
|
13
|
+
#
|
14
|
+
class Fight < Generalist
|
15
|
+
# @return [BattleResponse] The evaluation result containing the winner and reasoning
|
16
|
+
def call
|
17
|
+
messages = [
|
18
|
+
{ role: 'system', content: PROMPT },
|
19
|
+
{ role: 'user', content: "criteria: #{@criteria}" },
|
20
|
+
{ role: 'user', content: "player_a: #{@player_a}" },
|
21
|
+
{ role: 'user', content: "player_b: #{@player_b}" }
|
22
|
+
]
|
23
|
+
|
24
|
+
response = ::ActiveGenie::Clients::UnifiedClient.function_calling(
|
25
|
+
messages,
|
26
|
+
FUNCTION,
|
27
|
+
config: @config
|
28
|
+
)
|
29
|
+
|
30
|
+
response_formatted(response)
|
31
|
+
end
|
32
|
+
|
33
|
+
PROMPT = File.read(File.join(__dir__, 'fight.prompt.md'))
|
34
|
+
FUNCTION = JSON.parse(File.read(File.join(__dir__, 'fight.json')), symbolize_names: true)
|
35
|
+
end
|
36
|
+
end
|
37
|
+
end
|
@@ -15,6 +15,8 @@ module ActiveGenie
|
|
15
15
|
# Generalist.call("Player A content", "Player B content", "Evaluate keyword usage and pattern matching")
|
16
16
|
#
|
17
17
|
class Generalist
|
18
|
+
BattleResponse = Struct.new(:winner, :loser, :reasoning, :raw, keyword_init: true)
|
19
|
+
|
18
20
|
def self.call(...)
|
19
21
|
new(...).call
|
20
22
|
end
|
@@ -23,7 +25,7 @@ module ActiveGenie
|
|
23
25
|
# @param player_b [String] The content or submission from the second player
|
24
26
|
# @param criteria [String] The evaluation criteria or rules to assess against
|
25
27
|
# @param config [Hash] Additional configuration options that modify the battle evaluation behavior
|
26
|
-
# @return [
|
28
|
+
# @return [BattleResponse] The evaluation result containing the winner and reasoning
|
27
29
|
# @return [String] :winner The winner, either player_a or player_b
|
28
30
|
# @return [String] :reasoning Detailed explanation of why the winner was chosen
|
29
31
|
# @return [String] :what_could_be_changed_to_avoid_draw A suggestion on how to avoid a draw
|
@@ -34,6 +36,7 @@ module ActiveGenie
|
|
34
36
|
@config = ActiveGenie.configuration.merge(config)
|
35
37
|
end
|
36
38
|
|
39
|
+
# @return [BattleResponse] The evaluation result containing the winner and reasoning
|
37
40
|
def call
|
38
41
|
messages = [
|
39
42
|
{ role: 'system', content: PROMPT },
|
@@ -48,31 +51,35 @@ module ActiveGenie
|
|
48
51
|
config: @config
|
49
52
|
)
|
50
53
|
|
51
|
-
@config.logger.call({
|
52
|
-
code: :battle,
|
53
|
-
player_a: @player_a[0..30],
|
54
|
-
player_b: @player_b[0..30],
|
55
|
-
criteria: @criteria[0..30],
|
56
|
-
winner: response['impartial_judge_winner'],
|
57
|
-
reasoning: response['impartial_judge_winner_reasoning']
|
58
|
-
})
|
59
|
-
|
60
54
|
response_formatted(response)
|
61
55
|
end
|
62
56
|
|
63
|
-
PROMPT = File.read(File.join(__dir__, 'generalist.md'))
|
57
|
+
PROMPT = File.read(File.join(__dir__, 'generalist.prompt.md'))
|
64
58
|
FUNCTION = JSON.parse(File.read(File.join(__dir__, 'generalist.json')), symbolize_names: true)
|
65
59
|
|
66
60
|
private
|
67
61
|
|
68
62
|
def response_formatted(response)
|
69
|
-
winner = response['impartial_judge_winner']
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
63
|
+
winner, loser = case response['impartial_judge_winner']
|
64
|
+
when 'player_a' then [@player_a, @player_b]
|
65
|
+
when 'player_b' then [@player_b, @player_a]
|
66
|
+
end
|
67
|
+
reasoning = response['impartial_judge_winner_reasoning']
|
68
|
+
|
69
|
+
battle_response = BattleResponse.new(winner:, loser:, reasoning:, raw: response)
|
70
|
+
log_battle(battle_response)
|
74
71
|
|
75
|
-
|
72
|
+
battle_response
|
73
|
+
end
|
74
|
+
|
75
|
+
def log_battle(battle_response)
|
76
|
+
@config.logger.call(
|
77
|
+
code: :battle,
|
78
|
+
player_a: @player_a[0..30],
|
79
|
+
player_b: @player_b[0..30],
|
80
|
+
criteria: @criteria[0..30],
|
81
|
+
**battle_response.to_h
|
82
|
+
)
|
76
83
|
end
|
77
84
|
end
|
78
85
|
end
|
data/lib/active_genie/battle.rb
CHANGED
@@ -1,18 +1,27 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require_relative 'battle/generalist'
|
4
|
+
require_relative 'battle/fight'
|
4
5
|
|
5
6
|
module ActiveGenie
|
6
7
|
# See the [Battle README](lib/active_genie/battle/README.md) for more information.
|
7
8
|
module Battle
|
8
9
|
module_function
|
9
10
|
|
11
|
+
def generalist(...)
|
12
|
+
Generalist.call(...)
|
13
|
+
end
|
14
|
+
|
10
15
|
def call(...)
|
11
16
|
Generalist.call(...)
|
12
17
|
end
|
13
18
|
|
14
|
-
def
|
19
|
+
def compare(...)
|
15
20
|
Generalist.call(...)
|
16
21
|
end
|
22
|
+
|
23
|
+
def fight(...)
|
24
|
+
Fight.call(...)
|
25
|
+
end
|
17
26
|
end
|
18
27
|
end
|
metadata
CHANGED
@@ -1,19 +1,18 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: active_genie
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.28.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Radamés Roriz
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2025-
|
11
|
+
date: 2025-07-12 00:00:00.000000000 Z
|
12
12
|
dependencies: []
|
13
13
|
description: |
|
14
|
-
|
15
|
-
|
16
|
-
ActiveGenie is a Ruby gem that provides valuable solutions using GenAI. Just like Lodash or ActiveStorage, ActiveGenie brings a set of Modules to reach real value fast and reliably. Backed by a custom benchmarking system that ensures consistent quality and performance across different models and providers in every release.
|
14
|
+
ActiveGenie is a Ruby gem that helps developers build reliable, future-proof GenAI features without worrying about changing models, prompts, or providers. Like Lodash for GenAI, it offers simple, reusable modules for tasks like data extraction, scoring, and ranking, so you can focus on your app’s logic, not the shifting AI landscape.
|
15
|
+
Behind the scenes, a custom benchmarking system keeps everything consistent across LLM vendors and versions, release after release.
|
17
16
|
email:
|
18
17
|
- radames@roriz.dev
|
19
18
|
executables: []
|
@@ -25,9 +24,11 @@ files:
|
|
25
24
|
- VERSION
|
26
25
|
- lib/active_genie.rb
|
27
26
|
- lib/active_genie/battle.rb
|
28
|
-
- lib/active_genie/battle/
|
27
|
+
- lib/active_genie/battle/fight.json
|
28
|
+
- lib/active_genie/battle/fight.prompt.md
|
29
|
+
- lib/active_genie/battle/fight.rb
|
29
30
|
- lib/active_genie/battle/generalist.json
|
30
|
-
- lib/active_genie/battle/generalist.md
|
31
|
+
- lib/active_genie/battle/generalist.prompt.md
|
31
32
|
- lib/active_genie/battle/generalist.rb
|
32
33
|
- lib/active_genie/clients/providers/anthropic_client.rb
|
33
34
|
- lib/active_genie/clients/providers/base_client.rb
|
@@ -35,7 +36,6 @@ files:
|
|
35
36
|
- lib/active_genie/clients/providers/google_client.rb
|
36
37
|
- lib/active_genie/clients/providers/openai_client.rb
|
37
38
|
- lib/active_genie/clients/unified_client.rb
|
38
|
-
- lib/active_genie/config/README.md
|
39
39
|
- lib/active_genie/config/battle_config.rb
|
40
40
|
- lib/active_genie/config/data_extractor_config.rb
|
41
41
|
- lib/active_genie/config/llm_config.rb
|
@@ -50,17 +50,15 @@ files:
|
|
50
50
|
- lib/active_genie/config/scoring_config.rb
|
51
51
|
- lib/active_genie/configuration.rb
|
52
52
|
- lib/active_genie/data_extractor.rb
|
53
|
-
- lib/active_genie/data_extractor/README.md
|
54
53
|
- lib/active_genie/data_extractor/from_informal.json
|
55
54
|
- lib/active_genie/data_extractor/from_informal.rb
|
56
55
|
- lib/active_genie/data_extractor/generalist.json
|
57
|
-
- lib/active_genie/data_extractor/generalist.md
|
56
|
+
- lib/active_genie/data_extractor/generalist.prompt.md
|
58
57
|
- lib/active_genie/data_extractor/generalist.rb
|
59
58
|
- lib/active_genie/errors/invalid_log_output_error.rb
|
60
59
|
- lib/active_genie/errors/invalid_provider_error.rb
|
61
60
|
- lib/active_genie/logger.rb
|
62
61
|
- lib/active_genie/ranking.rb
|
63
|
-
- lib/active_genie/ranking/README.md
|
64
62
|
- lib/active_genie/ranking/elo_round.rb
|
65
63
|
- lib/active_genie/ranking/free_for_all.rb
|
66
64
|
- lib/active_genie/ranking/player.rb
|
@@ -68,9 +66,8 @@ files:
|
|
68
66
|
- lib/active_genie/ranking/ranking.rb
|
69
67
|
- lib/active_genie/ranking/ranking_scoring.rb
|
70
68
|
- lib/active_genie/scoring.rb
|
71
|
-
- lib/active_genie/scoring/README.md
|
72
69
|
- lib/active_genie/scoring/generalist.json
|
73
|
-
- lib/active_genie/scoring/generalist.md
|
70
|
+
- lib/active_genie/scoring/generalist.prompt.md
|
74
71
|
- lib/active_genie/scoring/generalist.rb
|
75
72
|
- lib/active_genie/scoring/recommended_reviewers.rb
|
76
73
|
- lib/tasks/benchmark.rake
|
@@ -103,5 +100,5 @@ requirements: []
|
|
103
100
|
rubygems_version: 3.4.20
|
104
101
|
signing_key:
|
105
102
|
specification_version: 4
|
106
|
-
summary:
|
103
|
+
summary: 'The Lodash for GenAI: Real Value + Consistent + Model-Agnostic'
|
107
104
|
test_files: []
|
@@ -1,39 +0,0 @@
|
|
1
|
-
# Battle
|
2
|
-
AI-powered battle evaluation system that determines winners between two players based on specified criteria.
|
3
|
-
|
4
|
-
## Features
|
5
|
-
- Content comparison - Evaluate submissions from two players against defined criteria
|
6
|
-
- Objective analysis - AI-powered assessment of how well each player meets requirements
|
7
|
-
- Detailed reasoning - Comprehensive explanation of why a winner was chosen
|
8
|
-
- Draw avoidance - Suggestions on how to modify content to avoid draws
|
9
|
-
- Flexible input - Support for both string content and structured data with content field
|
10
|
-
|
11
|
-
## Basic Usage
|
12
|
-
Evaluate a battle between two players with simple text content:
|
13
|
-
|
14
|
-
```ruby
|
15
|
-
player_a = "Implementation uses dependency injection for better testability"
|
16
|
-
player_b = "Code has high test coverage but tightly coupled components"
|
17
|
-
criteria = "Evaluate code quality and maintainability"
|
18
|
-
|
19
|
-
result = ActiveGenie::Battle.call(player_a, player_b, criteria)
|
20
|
-
# => {
|
21
|
-
# winner_player: "Implementation uses dependency injection for better testability",
|
22
|
-
# reasoning: "Player A's implementation demonstrates better maintainability through dependency injection,
|
23
|
-
# which allows for easier testing and component replacement. While Player B has good test coverage,
|
24
|
-
# the tight coupling makes the code harder to maintain and modify.",
|
25
|
-
# what_could_be_changed_to_avoid_draw: "Focus on specific architectural patterns and design principles"
|
26
|
-
# }
|
27
|
-
```
|
28
|
-
|
29
|
-
## Interface
|
30
|
-
### .call(player_a, player_b, criteria, config: {})
|
31
|
-
- `player_a` [String, Hash] - The content or submission from the first player
|
32
|
-
- `player_b` [String, Hash] - The content or submission from the second player
|
33
|
-
- `criteria` [String] - The evaluation criteria or rules to assess against
|
34
|
-
- `config` [Hash] - Additional configuration config that modify the battle evaluation behavior
|
35
|
-
|
36
|
-
Returns a Hash containing:
|
37
|
-
- `winner_player` [String, Hash] - The winning player's content (either player_a or player_b)
|
38
|
-
- `reasoning` [String] - Detailed explanation of why the winner was chosen
|
39
|
-
- `what_could_be_changed_to_avoid_draw` [String] - A suggestion on how to avoid a draw
|
@@ -1,235 +0,0 @@
|
|
1
|
-
# ActiveGenie Configuration
|
2
|
-
|
3
|
-
ActiveGenie provides a flexible configuration system that can be customized to suit your needs. This document details all available configuration options.
|
4
|
-
|
5
|
-
## Initial Configuration
|
6
|
-
|
7
|
-
To configure ActiveGenie, use the `ActiveGenie.configure` block in your application's initialization:
|
8
|
-
|
9
|
-
```ruby
|
10
|
-
ActiveGenie.configure do |config|
|
11
|
-
# Provider configurations
|
12
|
-
config.providers.openai.api_key = ENV['OPENAI_API_KEY']
|
13
|
-
|
14
|
-
# Log configurations
|
15
|
-
config.log.file_path = 'log/custom_genie.log'
|
16
|
-
config.log.fine_tune_file_path = 'log/fine_tune_genie.log'
|
17
|
-
|
18
|
-
# Other configurations can be added here
|
19
|
-
end
|
20
|
-
```
|
21
|
-
|
22
|
-
## Configuration Sections
|
23
|
-
|
24
|
-
### 1. Log Configuration (`config.log`)
|
25
|
-
|
26
|
-
The log configuration controls how ActiveGenie handles logging of its operations.
|
27
|
-
|
28
|
-
#### Available Settings:
|
29
|
-
|
30
|
-
| Setting | Type | Default | Description |
|
31
|
-
|---------|------|---------|-------------|
|
32
|
-
| `file_path` | String | `'log/active_genie.log'` | Path to the main log file where all logs will be written. |
|
33
|
-
| `fine_tune_file_path` | String | `'log/active_genie_fine_tune.log'` | Path to the fine-tuning specific log file. |
|
34
|
-
| `output` | Proc | `->(log) { $stdout.puts log }` | Custom output handler for logs. Must respond to `call`. |
|
35
|
-
| `additional_context` | Hash | `{}` | Additional context to be added to each log entry. |
|
36
|
-
|
37
|
-
#### Methods:
|
38
|
-
|
39
|
-
- `add_observer(observers: [], scope: {}, &block)`
|
40
|
-
- Adds log observers that will be notified of log events.
|
41
|
-
- `observers`: An array of callable objects or a single callable object.
|
42
|
-
- `scope`: A hash to filter which logs the observer receives.
|
43
|
-
- `block`: A block that will be called for matching logs.
|
44
|
-
|
45
|
-
- `remove_observer(observers)`
|
46
|
-
- Removes the specified observers.
|
47
|
-
|
48
|
-
#### Example:
|
49
|
-
|
50
|
-
```ruby
|
51
|
-
ActiveGenie.configure do |config|
|
52
|
-
config.log.file_path = 'log/my_app/genie.log'
|
53
|
-
config.log.fine_tune_file_path = 'log/my_app/fine_tune.log'
|
54
|
-
|
55
|
-
# Add a custom output handler
|
56
|
-
config.log.output = ->(log) { MyLogger.info(log) }
|
57
|
-
|
58
|
-
# Add an observer for specific log events
|
59
|
-
config.log.add_observer(scope: { event: :api_call }) do |log|
|
60
|
-
StatsD.increment("genie.api_calls")
|
61
|
-
end
|
62
|
-
end
|
63
|
-
```
|
64
|
-
|
65
|
-
### 2. LLM Configuration (`config.llm`)
|
66
|
-
|
67
|
-
The LLM configuration is used to define settings for interacting with Large Language Models.
|
68
|
-
|
69
|
-
#### Available Settings:
|
70
|
-
|
71
|
-
| Setting | Type | Default | Description |
|
72
|
-
|-----------------|---------------------|------------------------------|------------------------------------------------------------------------------------------------------------|
|
73
|
-
| `model` | String / nil | `nil` | The specific LLM model to use (e.g., 'gpt-4', 'claude-2'). |
|
74
|
-
| `provider` | Symbol / nil | `nil` | The LLM provider (e.g., `:openai`, `:anthropic`). Set as a string, stored as a symbol. |
|
75
|
-
| `client` | Object / nil | `nil` | A pre-configured client instance for the LLM provider. If set, other settings might be ignored. |
|
76
|
-
| `temperature` | Numeric | `0` | Controls randomness. Higher values (e.g., 0.8) = more random, lower (e.g., 0.2) = more deterministic. |
|
77
|
-
| `max_tokens` | Integer | `4096` | Maximum number of tokens to generate in the LLM response. |
|
78
|
-
| `max_retries` | Integer / nil | `nil` | Maximum number of times to retry a failed API call to the LLM. |
|
79
|
-
| `retry_delay` | Numeric / nil | `nil` | Delay (in seconds) between retries for failed API calls. |
|
80
|
-
| `model_tier` | Enum [lower_tier, middle_tier, higher_tier] | `'lower_tier'` | Specifies the model tier, potentially affecting cost/performance. Will be used if `model` is not set. |
|
81
|
-
| `read_timeout` | Numeric / nil | `nil` | Timeout (in seconds) for reading data from the LLM API. |
|
82
|
-
| `open_timeout` | Numeric / nil | `nil` | Timeout (in seconds) for establishing a connection to the LLM API. |
|
83
|
-
|
84
|
-
#### Example:
|
85
|
-
|
86
|
-
```ruby
|
87
|
-
ActiveGenie.configure do |config|
|
88
|
-
config.llm.provider = :openai
|
89
|
-
config.llm.model = 'gpt-999'
|
90
|
-
config.llm.temperature = 0.1
|
91
|
-
config.llm.max_tokens = 8000
|
92
|
-
config.llm.max_retries = 1
|
93
|
-
config.llm.retry_delay = 5 # seconds
|
94
|
-
config.llm.model_tier = 'higher_tier' # If model is not set, will use the model tier to select a model
|
95
|
-
end
|
96
|
-
```
|
97
|
-
|
98
|
-
### 4. Providers Configuration (`config.providers`)
|
99
|
-
|
100
|
-
The Providers configuration (`config.providers`) manages settings for various Large Language Model (LLM) providers. It allows you to configure multiple providers, set a default, and access individual provider settings. Active Genie automatically loads configurations for supported providers (OpenAI, Anthropic, DeepSeek, Google).
|
101
|
-
|
102
|
-
#### Main Provider Settings:
|
103
|
-
|
104
|
-
| Setting | Type | Default | Description |
|
105
|
-
|-----------|---------------------|--------------------------------|-----------------------------------------------------------------------------|
|
106
|
-
| `default` | String / Symbol | First validly configured provider (often `:openai` if its API key is set) | The name of the default LLM provider to use (e.g., `:openai`, `:anthropic`). You can set this to your preferred provider. |
|
107
|
-
|
108
|
-
#### Methods for `config.providers`:
|
109
|
-
|
110
|
-
- `add(provider_classes)`
|
111
|
-
- Adds one or more custom provider configuration classes. `provider_classes` can be a single class or an array of classes. (Typically not needed for built-in providers).
|
112
|
-
- `remove(provider_classes)`
|
113
|
-
- Removes one or more provider configurations based on their classes.
|
114
|
-
|
115
|
-
#### Example for Main Provider Settings:
|
116
|
-
|
117
|
-
```ruby
|
118
|
-
ActiveGenie.configure do |config|
|
119
|
-
# Set the default provider to use if not specifying one explicitly in operations
|
120
|
-
config.providers.default = :anthropic
|
121
|
-
|
122
|
-
# Configure individual providers (API keys are often set via ENV variables)
|
123
|
-
config.providers.openai.api_key = ENV['OPENAI_API_KEY']
|
124
|
-
config.providers.anthropic.api_key = ENV['ANTHROPIC_API_KEY']
|
125
|
-
# config.providers.deepseek.api_key = ENV['DEEPSEEK_API_KEY']
|
126
|
-
# config.providers.google.api_key = ENV['GEMINI_API_KEY']
|
127
|
-
end
|
128
|
-
```
|
129
|
-
|
130
|
-
#### Individual Provider Configurations:
|
131
|
-
|
132
|
-
The following subsections detail the configurations for each supported LLM provider. Each provider configuration (e.g., `config.providers.openai`) allows setting an `api_key`, `api_url`, and specific model names for `lower_tier_model`, `middle_tier_model`, and `higher_tier_model`.
|
133
|
-
|
134
|
-
##### a. OpenAI (`config.providers.openai`)
|
135
|
-
|
136
|
-
Internal Name: `:openai`
|
137
|
-
|
138
|
-
| Setting | Type | Default | Description |
|
139
|
-
|---------------------|--------|---------------------------------------|-----------------------------------------------------------------------------|
|
140
|
-
| `api_key` | String | `ENV['OPENAI_API_KEY']` | Your OpenAI API key. |
|
141
|
-
| `api_url` | String | `'https://api.openai.com/v1'` | Base URL for the OpenAI API. |
|
142
|
-
| `lower_tier_model` | String | `'gpt-4.1-mini'` | Model for lower-tier usage (cost-effective, faster). |
|
143
|
-
| `middle_tier_model` | String | `'gpt-4.1'` | Model for middle-tier usage (balanced performance). |
|
144
|
-
| `higher_tier_model` | String | `'o3-mini'` | Model for higher-tier usage (most capable). |
|
145
|
-
|
146
|
-
##### b. Anthropic (`config.providers.anthropic`)
|
147
|
-
|
148
|
-
Internal Name: `:anthropic`
|
149
|
-
|
150
|
-
| Setting | Type | Default | Description |
|
151
|
-
|---------------------|--------|------------------------------------------|-----------------------------------------------------------------------------|
|
152
|
-
| `api_key` | String | `ENV['ANTHROPIC_API_KEY']` | Your Anthropic API key. |
|
153
|
-
| `api_url` | String | `'https://api.anthropic.com'` | Base URL for the Anthropic API. |
|
154
|
-
| `anthropic_version` | String | `'2023-06-01'` | The API version for Anthropic. |
|
155
|
-
| `lower_tier_model` | String | `'claude-3-5-haiku-20241022'` | Model for lower-tier usage. |
|
156
|
-
| `middle_tier_model` | String | `'claude-3-7-sonnet-20250219'` | Model for middle-tier usage. |
|
157
|
-
| `higher_tier_model` | String | `'claude-3-opus-20240229'` | Model for higher-tier usage. |
|
158
|
-
|
159
|
-
##### c. DeepSeek (`config.providers.deepseek`)
|
160
|
-
|
161
|
-
Internal Name: `:deepseek`
|
162
|
-
|
163
|
-
| Setting | Type | Default | Description |
|
164
|
-
|---------------------|--------|---------------------------------------|-----------------------------------------------------------------------------|
|
165
|
-
| `api_key` | String | `ENV['DEEPSEEK_API_KEY']` | Your DeepSeek API key. |
|
166
|
-
| `api_url` | String | `'https://api.deepseek.com/v1'` | Base URL for the DeepSeek API. |
|
167
|
-
| `lower_tier_model` | String | `'deepseek-chat'` | Model for lower-tier usage. |
|
168
|
-
| `middle_tier_model` | String | `'deepseek-chat'` | Model for middle-tier usage. |
|
169
|
-
| `higher_tier_model` | String | `'deepseek-reasoner'` | Model for higher-tier usage. |
|
170
|
-
|
171
|
-
##### d. Google (`config.providers.google`)
|
172
|
-
|
173
|
-
Internal Name: `:google`
|
174
|
-
|
175
|
-
| Setting | Type | Default | Description |
|
176
|
-
|---------------------|--------|---------------------------------------------------------------|-----------------------------------------------------------------------------|
|
177
|
-
| `api_key` | String | `ENV['GENERATIVE_LANGUAGE_GOOGLE_API_KEY']` or `ENV['GEMINI_API_KEY']` | Your Google API key for Gemini. |
|
178
|
-
| `api_url` | String | `'https://generativelanguage.googleapis.com'` | Base URL for the Google Generative Language API. |
|
179
|
-
| `lower_tier_model` | String | `'gemini-2.0-flash-lite'` | Model for lower-tier usage. |
|
180
|
-
| `middle_tier_model` | String | `'gemini-2.0-flash'` | Model for middle-tier usage. |
|
181
|
-
| `higher_tier_model` | String | `'gemini-2.5-pro-experimental'` | Model for higher-tier usage. |
|
182
|
-
|
183
|
-
#### Example for Overriding Individual Provider Settings:
|
184
|
-
|
185
|
-
```ruby
|
186
|
-
ActiveGenie.configure do |config|
|
187
|
-
config.providers.openai.api_key = 'sk-yourOpenAiKey...'
|
188
|
-
config.providers.openai.middle_tier_model = 'gpt-4o' # Override default middle tier for OpenAI
|
189
|
-
|
190
|
-
config.providers.anthropic.api_key = 'sk-ant-yourAnthropicKey...'
|
191
|
-
config.providers.anthropic.anthropic_version = '2024-02-15' # Override Anthropic API version
|
192
|
-
|
193
|
-
config.providers.google.higher_tier_model = 'gemini-1.5-pro-latest' # Use a specific Google model
|
194
|
-
end
|
195
|
-
```
|
196
|
-
|
197
|
-
### 5. Ranking Configuration (`config.ranking`)
|
198
|
-
|
199
|
-
The Ranking configuration (`config.ranking`) deals with settings related to how results or items are ranked.
|
200
|
-
|
201
|
-
#### Available Settings:
|
202
|
-
|
203
|
-
| Setting | Type | Default | Description |
|
204
|
-
|-----------------------------|---------|---------|------------------------------------------------------------------------------------------------------------|
|
205
|
-
| `score_variation_threshold` | Integer | `30` | A threshold (percentage) used to determine significant variations in scores when ranking or comparing items. |
|
206
|
-
|
207
|
-
#### Example:
|
208
|
-
|
209
|
-
```ruby
|
210
|
-
ActiveGenie.configure do |config|
|
211
|
-
config.ranking.score_variation_threshold = 25 # Set to 25%
|
212
|
-
end
|
213
|
-
```
|
214
|
-
|
215
|
-
### 6. Data Extractor Configuration (`config.data_extractor`)
|
216
|
-
|
217
|
-
The Data Extractor configuration (`config.data_extractor`) provides settings for how data is extracted and processed, likely from text or other sources using LLMs.
|
218
|
-
|
219
|
-
#### Available Settings:
|
220
|
-
|
221
|
-
| Setting | Type | Default | Description |
|
222
|
-
|--------------------|---------|---------|---------------------------------------------------------------------------------------------------------|
|
223
|
-
| `with_explanation` | Boolean | `true` | Whether the data extraction process should also attempt to provide an explanation for the extracted data. |
|
224
|
-
| `min_accuracy` | Integer | `70` | The minimum accuracy (percentage) required for extracted data to be considered valid or useful. |
|
225
|
-
| `verbose` | Boolean | `false` | If true, enables more detailed logging or output from the data extraction process. |
|
226
|
-
|
227
|
-
#### Example:
|
228
|
-
|
229
|
-
```ruby
|
230
|
-
ActiveGenie.configure do |config|
|
231
|
-
config.data_extractor.with_explanation = false
|
232
|
-
config.data_extractor.min_accuracy = 85
|
233
|
-
config.data_extractor.verbose = true
|
234
|
-
end
|
235
|
-
```
|
@@ -1,131 +0,0 @@
|
|
1
|
-
# Data Extractor
|
2
|
-
Extract structured data from text using AI-powered analysis, handling informal language and complex expressions.
|
3
|
-
|
4
|
-
## ✨ Features
|
5
|
-
- Structured data extraction - Extract typed data from unstructured text using predefined schemas
|
6
|
-
- Informal text analysis - Identifies and handles informal language patterns, including litotes
|
7
|
-
- Explanation tracking - Provides reasoning for each extracted data point
|
8
|
-
|
9
|
-
## Basic Usage
|
10
|
-
|
11
|
-
Extract structured data from text using predefined schemas:
|
12
|
-
|
13
|
-
```ruby
|
14
|
-
text = "John Doe is 25 years old"
|
15
|
-
schema = {
|
16
|
-
name: { type: 'string', description: 'Full name of the person' },
|
17
|
-
age: { type: 'integer', description: 'Age in years' }
|
18
|
-
}
|
19
|
-
result = ActiveGenie::DataExtractor.call(text, schema)
|
20
|
-
# => {
|
21
|
-
# name: "John Doe",
|
22
|
-
# name_explanation: "Found directly in text",
|
23
|
-
# age: 25,
|
24
|
-
# age_explanation: "Explicitly stated as 25 years old"
|
25
|
-
# }
|
26
|
-
|
27
|
-
product = "Nike Air Max 90 - Size 42 - $199.99"
|
28
|
-
schema = {
|
29
|
-
brand: {
|
30
|
-
type: 'string',
|
31
|
-
enum: ["Nike", "Adidas", "Puma"]
|
32
|
-
},
|
33
|
-
price: {
|
34
|
-
type: 'number',
|
35
|
-
minimum: 0
|
36
|
-
},
|
37
|
-
currency: {
|
38
|
-
type: 'string',
|
39
|
-
enum: ["USD", "EUR"]
|
40
|
-
},
|
41
|
-
size: {
|
42
|
-
type: 'integer',
|
43
|
-
minimum: 35,
|
44
|
-
maximum: 46
|
45
|
-
}
|
46
|
-
}
|
47
|
-
|
48
|
-
result = ActiveGenie::DataExtractor.call(product, schema)
|
49
|
-
# => {
|
50
|
-
# brand: "Nike",
|
51
|
-
# brand_explanation: "Brand name found at start of text",
|
52
|
-
# price: 199.99,
|
53
|
-
# price_explanation: "Price found in USD format at end",
|
54
|
-
# size: 42,
|
55
|
-
# size_explanation: "Size explicitly stated in the middle",
|
56
|
-
# currency: "USD",
|
57
|
-
# currency_explanation: "Derived from $ symbol"
|
58
|
-
# }
|
59
|
-
```
|
60
|
-
|
61
|
-
## Informal Text Processing
|
62
|
-
|
63
|
-
The `from_informal` method extends the basic extraction by analyzing rhetorical devices and informal language patterns like:
|
64
|
-
|
65
|
-
- Litotes ("not bad", "isn't terrible")
|
66
|
-
- Affirmative expressions ("sure", "no problem")
|
67
|
-
- Negative expressions ("nah", "not really")
|
68
|
-
|
69
|
-
### Example
|
70
|
-
|
71
|
-
```ruby
|
72
|
-
text = "The weather isn't bad today"
|
73
|
-
schema = {
|
74
|
-
mood: { type: 'string', description: 'The mood of the message' }
|
75
|
-
}
|
76
|
-
|
77
|
-
result = ActiveGenie::DataExtractor.from_informal(text, schema)
|
78
|
-
# => {
|
79
|
-
# mood: "positive",
|
80
|
-
# mood_explanation: "Speaker views weather favorably",
|
81
|
-
# message_litote: true,
|
82
|
-
# litote_rephrased: "The weather is good today"
|
83
|
-
# }
|
84
|
-
```
|
85
|
-
|
86
|
-
### Usage Notes
|
87
|
-
- Best suited for processing conversational user inputs
|
88
|
-
- Automatically detects and interprets litotes
|
89
|
-
- Provides rephrased positive statements for litotes
|
90
|
-
- May require more processing time due to rhetorical analysis
|
91
|
-
- Accuracy depends on context clarity
|
92
|
-
|
93
|
-
⚠️ Performance Impact: This method performs additional rhetorical analysis, which can increase processing time.
|
94
|
-
|
95
|
-
## Interface
|
96
|
-
|
97
|
-
### `.call(text, data_to_extract, config = {})`
|
98
|
-
Extracts structured data from text based on a predefined schema.
|
99
|
-
|
100
|
-
#### Parameters
|
101
|
-
| Name | Type | Description | Required | Example |
|
102
|
-
| --- | --- | --- | --- | --- |
|
103
|
-
| `text` | `String` | The text to analyze and extract data from | Yes | "John Doe is 25 years old" |
|
104
|
-
| `data_to_extract` | `Hash` | Schema defining the data structure to extract | Yes | `{ name: { type: 'string' } }` |
|
105
|
-
| `config` | `Hash` | Additional extraction configuration | No | `{ model: "gpt-4" }` |
|
106
|
-
|
107
|
-
#### config
|
108
|
-
| Name | Type | Description |
|
109
|
-
| --- | --- | --- |
|
110
|
-
| `model` | `String` | The model to use for the extraction |
|
111
|
-
| `api_key` | `String` | The API key to use for the extraction |
|
112
|
-
|
113
|
-
#### Returns
|
114
|
-
`Hash` containing:
|
115
|
-
- Extracted values matching the schema structure
|
116
|
-
- Explanation field for each extracted value
|
117
|
-
- Additional analysis fields when using `from_informal`
|
118
|
-
|
119
|
-
### `.from_informal(text, data_to_extract, config = {})`
|
120
|
-
Extends extraction with rhetorical analysis, particularly for litotes.
|
121
|
-
|
122
|
-
#### Additional Return Fields
|
123
|
-
| Name | Type | Description |
|
124
|
-
| --- | --- | --- |
|
125
|
-
| `message_litote` | `Boolean` | Whether the text contains a litote |
|
126
|
-
| `litote_rephrased` | `String` | Positive rephrasing of any detected litote |
|
127
|
-
|
128
|
-
⚠️ Performance Considerations
|
129
|
-
- Both methods may require multiple AI model calls
|
130
|
-
- Informal processing requires additional rhetorical analysis
|
131
|
-
- Consider background processing for production use
|
@@ -1,73 +0,0 @@
|
|
1
|
-
# Ranking
|
2
|
-
|
3
|
-
The `ActiveGenie::Ranking` module organizes players based on scores derived from textual evaluations and then ranks them using a multi-stage process. It leverages the scoring system from the `ActiveGenie::Scoring` module to assign initial scores, and then applies advanced ranking methods to produce a competitive ranking.
|
4
|
-
|
5
|
-
## Overview
|
6
|
-
|
7
|
-
The ranking system performs the following steps:
|
8
|
-
|
9
|
-
1. **Initial Scoring**: Each player’s textual content is evaluated using `ActiveGenie::Scoring`. This produces a `score` based on multiple expert reviews.
|
10
|
-
|
11
|
-
2. **Elimination of Poor Performers**: Players whose scores show a high coefficient of variation (indicating inconsistency) are progressively eliminated. This ensures that only competitive candidates continue in the ranking process.
|
12
|
-
|
13
|
-
3. **ELO Ranking**: If there are more than 10 eligible players, an ELO-based ranking is applied. Battles between players are simulated via `ActiveGenie::Battle`, and scores are updated using an ELO algorithm tailored to the ranking.
|
14
|
-
|
15
|
-
4. **Free for all Matches**: Finally, the remaining players engage in head-to-head matches where each unique pair competes. Match outcomes (wins, losses, draws) are recorded to finalize the rankings.
|
16
|
-
|
17
|
-
## Components
|
18
|
-
|
19
|
-
- **ranking**: Orchestrates the entire ranking process. Initializes player scores, eliminates outliers, and coordinates ranking rounds.
|
20
|
-
|
21
|
-
- **EloRanking**: Applies an ELO-based system to rank players through simulated battles. It updates players’ ELO scores based on match outcomes and predefined rules (including penalties for losses).
|
22
|
-
|
23
|
-
- **Free for all**: Conducts complete pairwise matches among eligible players to record win/loss/draw statistics and refine the final standings.
|
24
|
-
|
25
|
-
## Usage
|
26
|
-
|
27
|
-
Call the ranking using:
|
28
|
-
|
29
|
-
```ruby
|
30
|
-
result = ActiveGenie::Ranking.call(players, criteria, config: {})
|
31
|
-
```
|
32
|
-
|
33
|
-
- `players`: A collection of player instances, each containing textual content to be scored.
|
34
|
-
- `criteria`: A string defining the evaluation criteria used by the scoring system.
|
35
|
-
- `config`: A hash of additional parameters for customization (e.g., model, api_key).
|
36
|
-
|
37
|
-
The method processes the players through scoring, elimination, and ranking phases, then returns a hash containing the player statistics and rankings.
|
38
|
-
|
39
|
-
## Possible improvements
|
40
|
-
- Adjust initial criteria to ensure consistency
|
41
|
-
- Adjust each player's content to ensure consistency
|
42
|
-
- Support players with images or audio
|
43
|
-
- Parallelize processing battles and scoring
|
44
|
-
|
45
|
-
## Ranking Configuration
|
46
|
-
|
47
|
-
| Config | Description | Default |
|
48
|
-
|--------|-------------|---------|
|
49
|
-
| `score_variation_threshold` | Threshold for eliminating players with inconsistent scores | `30` |
|
50
|
-
|
51
|
-
## Ranking Callbacks
|
52
|
-
Callbacks are optional and can be used to watch any changes in players, battles, or scoring.
|
53
|
-
|
54
|
-
| Callback | Description |
|
55
|
-
|--------|-------------|
|
56
|
-
| `watch_players` | Callback to watch any changes in players |
|
57
|
-
| `watch_battles` | Callback to watch any changes in battles |
|
58
|
-
| `watch_scoring` | Callback to watch any changes in scoring |
|
59
|
-
|
60
|
-
Example of callback usage:
|
61
|
-
|
62
|
-
```ruby
|
63
|
-
result = ActiveGenie::Ranking.call(
|
64
|
-
players,
|
65
|
-
criteria,
|
66
|
-
config: {
|
67
|
-
watch_players: ->(player) { puts player },
|
68
|
-
watch_battles: ->(battle) { puts battle },
|
69
|
-
watch_scoring: ->(scoring) { puts scoring }
|
70
|
-
}
|
71
|
-
)
|
72
|
-
```
|
73
|
-
|
@@ -1,76 +0,0 @@
|
|
1
|
-
# Scoring
|
2
|
-
Text evaluation system that provides detailed scoring and feedback using multiple expert reviewers.
|
3
|
-
|
4
|
-
## Features
|
5
|
-
- Multi-reviewer evaluation - Get scores and feedback from multiple AI-powered expert reviewers
|
6
|
-
- Automatic reviewer selection - Smart recommendation of reviewers based on content and criteria
|
7
|
-
- Detailed feedback - Comprehensive reasoning for each reviewer's score
|
8
|
-
- Customizable weights - Adjust the importance of different reviewers' scores
|
9
|
-
- Flexible criteria - Score text against any specified evaluation criteria
|
10
|
-
|
11
|
-
## Basic Usage
|
12
|
-
|
13
|
-
Score text using predefined reviewers:
|
14
|
-
|
15
|
-
```ruby
|
16
|
-
text = "The code implements a binary search algorithm with O(log n) complexity"
|
17
|
-
criteria = "Evaluate technical accuracy and clarity"
|
18
|
-
reviewers = ["Algorithm Expert", "Technical Writer"]
|
19
|
-
|
20
|
-
result = ActiveGenie::Scoring.call(text, criteria, reviewers)
|
21
|
-
# => {
|
22
|
-
# algorithm_expert_score: 95,
|
23
|
-
# algorithm_expert_reasoning: "Accurately describes binary search and its complexity",
|
24
|
-
# technical_writer_score: 90,
|
25
|
-
# technical_writer_reasoning: "Clear and concise explanation of the algorithm",
|
26
|
-
# final_score: 92.5
|
27
|
-
# }
|
28
|
-
```
|
29
|
-
|
30
|
-
## Automatic Reviewer Selection
|
31
|
-
|
32
|
-
When no reviewers are specified, the system automatically recommends appropriate reviewers:
|
33
|
-
|
34
|
-
```ruby
|
35
|
-
text = "The patient shows signs of improved cardiac function"
|
36
|
-
criteria = "Evaluate medical accuracy and clarity"
|
37
|
-
|
38
|
-
result = ActiveGenie::Scoring.call(text, criteria)
|
39
|
-
# => {
|
40
|
-
# cardiologist_score: 88,
|
41
|
-
# cardiologist_reasoning: "Accurate assessment of cardiac improvement",
|
42
|
-
# medical_writer_score: 85,
|
43
|
-
# medical_writer_reasoning: "Clear communication of medical findings",
|
44
|
-
# general_practitioner_score: 90,
|
45
|
-
# general_practitioner_reasoning: "Well-structured medical observation",
|
46
|
-
# final_score: 87.7
|
47
|
-
# }
|
48
|
-
```
|
49
|
-
|
50
|
-
## Interface
|
51
|
-
|
52
|
-
### `.call(text, criteria, reviewers = [], config: {})`
|
53
|
-
Main interface for scoring text content.
|
54
|
-
|
55
|
-
#### Parameters
|
56
|
-
- `text` [String] - The text content to be evaluated
|
57
|
-
- `criteria` [String] - The evaluation criteria or rubric to assess against
|
58
|
-
- `reviewers` [Array<String>] - Optional list of specific reviewers
|
59
|
-
- `config` [Hash] - Additional configuration config
|
60
|
-
|
61
|
-
### `RecommendedReviewers.call(text, criteria, config: {})`
|
62
|
-
Recommends appropriate reviewers based on content and criteria.
|
63
|
-
|
64
|
-
#### Parameters
|
65
|
-
- `text` [String] - The text content to analyze
|
66
|
-
- `criteria` [String] - The evaluation criteria
|
67
|
-
- `config` [Hash] - Additional configuration config
|
68
|
-
|
69
|
-
### Usage Notes
|
70
|
-
- Best suited for objective evaluation of text content
|
71
|
-
- Provides balanced scoring through multiple reviewers
|
72
|
-
- Automatically handles reviewer selection when needed
|
73
|
-
- Supports custom weighting of reviewer scores
|
74
|
-
- Returns detailed reasoning for each score
|
75
|
-
|
76
|
-
Performance Impact: Using multiple reviewers or requesting detailed feedback may increase processing time.
|
File without changes
|
File without changes
|
File without changes
|