ask-agent 0.12.1 → 0.15.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1c3eb4f6e24d5acd650f8e11614dcd3257b9f57b252ba6df2c8c48ca3dffdd1e
-  data.tar.gz: efc969c3abc319814c4afecc9103750a37962cbd664d1944d84ec3d2d89adb71
+  metadata.gz: 816f21ed178b31227a58afeeed65f2428dbe9cce7d4a6ddab2c6fda7a6a60a89
+  data.tar.gz: 4dec09ce3cc94c6abc0e94bd5dee9f6ac9bde083dc7995af601a6650e692fd37
 SHA512:
-  metadata.gz: c9a78d1dd7e990eb11bd56997f9da131e0bb11f88affca0b4e01d9504711aa17787e98bde447bcd397045dcb97e4930cd04437638b3bebc83d166e8f10c34d37
-  data.tar.gz: 4d17c16b767ff92973d32acff24232c5a5d86739ead407f1e92f92707a8d685559e146c4c5430861e397f6bd5cf5c9d2fcdbf503015e6adeae3821f1567041ca
+  metadata.gz: 8de0c033834105f7c147115bd84b97306fc91b787690fc9e82563835f53b8fad69ff90dfabeeada12be9a8651e9ecfffd960d8e2fa6e84ffff4709f30519a5de
+  data.tar.gz: b5d46db9e66b242ea35fafaf86c67401ad27ee8ef0099ad4811a3d8442e63bd54aa2b16be5e4a7b2aa0f7305603cf39d1b5497e66648f59a9d701489bbdc6c8c

data/CHANGELOG.md

@@ -1,3 +1,94 @@
+## [0.15.0] — 2026-07-24
+
+### Added
+
+- **Independent Evaluator — `Ask::Agent::Evaluator`** — Generator/evaluator separation.
+  A separate model (configured independently from the session's model) judges the
+  agent's output against a structured rubric before delivery. This prevents the
+  anti-pattern of a model grading its own work.
+
+  ```ruby
+  # Evaluate with a different model — the recommended approach
+  session = Ask::Agent::Session.new(
+    model: "gpt-4o",
+    evaluator: { model: "claude-sonnet-4", goal: "Write an email validator" }
+  )
+  session.run("Write email validation")
+  ```
+
+  Three verdicts:
+  - **`:accept`** — output meets the goal, passes through to reflection
+  - **`:revise`** — evaluator provides actionable feedback; session runs another
+    turn with the feedback injected into system context
+  - **`:block`** — output is fundamentally wrong; session returns blocked message
+    and emits `Events::EvaluationBlocked`
+
+  Rubric dimensions (each scored 0-2):
+  - correctness (3× weight), completeness (2×), verification (2×), scope (1×), clarity (1×)
+
+  Custom rubrics supported:
+  ```ruby
+  evaluator = Ask::Agent::Evaluator.new(
+    model: "claude-sonnet-4",
+    rubric: [
+      Ask::Agent::Evaluator::Dimension.new(name: "performance", description: "Is it fast?", weight: 2)
+    ]
+  )
+  ```
+
+- **`evaluator:` option on `Session`** — accepts `true`, `false`/`nil`, or a Hash:
+  - `evaluator: true` — uses `config.default_evaluator_model` (falls back to the
+    session's model, though using a different model is strongly recommended)
+  - `evaluator: { model: "claude-sonnet-4", goal: "Custom goal" }` — explicit config
+  - `evaluator: false` (default) — no evaluation, backward compatible
+
+- **`default_evaluator_model` config option** — set a global default:
+  ```ruby
+  Ask::Agent.configure do |c|
+    c.default_evaluator_model = "claude-sonnet-4"
+  end
+  ```
+
+- **New event types** for streaming evaluation:
+  - `Events::EvaluationStart` — emitted when evaluation begins (includes dimension list)
+  - `Events::EvaluationDelta` — streamed evaluation text from the evaluator model
+  - `Events::EvaluationEnd` — emitted with decision, feedback, scores, and evidence
+  - `Events::EvaluationBlocked` — emitted when evaluator returns `:block`
+
+- **17 unit tests** for Evaluator — construction, rubric, all three verdicts, event
+  emission, custom rubrics, JSON parsing, and malformed response fallback.
+
+- **7 integration tests** for Session with evaluator — config (true/hash),
+  revise triggers improvement, revise skips reflector, block returns blocked
+  message, block emits event, evaluator-not-configured skips evaluation.
+
+## [0.14.0] — 2026-07-23
+
+### Added
+
+- **`state:` keyword on Session** — Accepts any `Ask::State::Adapter` directly. Sessions persist conversation state, tool results, and metadata. Replaces `persistence:` keyword (still supported for backward compatibility).
+- **Per-turn persistence** — Session now persists after every LLM turn, not just at the end of `run()`. Mid-session crashes no longer lose progress.
+- **`Session.load` restores `@messages`** — Previously `session.messages` returned `nil` after loading. Now it's populated from the restored chat messages.
+- **`Ask::State::Adapter#clear`** — Abstract method added to the adapter contract. Memory adapter implements it.
+
+### Changed
+
+- **Session behind the scenes now uses `@state.set`/`@state.get`/`@state.delete`** instead of the old `@persistence.save`/`@persistence.load`/`@persistence.delete`. Custom adapters must respond to `set`/`get`/`delete`.
+
+## [0.13.0] — 2026-07-23
+
+### Added
+
+- **ModelFallback middleware** — Switches to a fallback model+provider when the primary LLM call fails with a rate limit, server error, or service unavailable. Supports static and dynamic (lambda-based) fallback lists. Credentials resolve automatically via `Ask::Auth`.
+  - Static fallbacks: ordered list of `{ model:, provider: }` hashes
+  - Dynamic fallbacks: lambda receiving `(error, request)` returning the list
+  - Custom eligible errors: configure which errors trigger fallback
+  - Each fallback builds its own provider instance with resolved credentials
+
+### Changed
+
+- `Pipeline::KNOWN_MIDDLEWARES` now includes `:model_fallback`.
+
 ## [0.12.0] — 2026-07-22
 
 ### Added

data/README.md

@@ -37,7 +37,88 @@ puts response
 | `Ask::Agent::Telemetry` | telemetry.rb | File-backed telemetry for error tracking |
 | `Ask::Agent::Reflector` | reflector.rb | Assistant response self-evaluation |
 | `Ask::Agent::MetaAgent` | meta_agent.rb | LLM-powered self-improvement from telemetry |
-| `Ask::Agent::Configuration` | configuration.rb | Global config: model, turns, concurrency |
+| `Ask::Agent::Evaluator` | evaluator.rb | Independent response evaluation with structured rubric — different model, isolated context |
+| `Ask::Agent::Configuration` | configuration.rb | Global config: model, turns, concurrency, evaluator |
+
+## Evaluator
+
+Independent response evaluation with generator/evaluator separation. The
+evaluator uses a **separate model** (different from the session's model) and an
+**isolated context** to judge the agent's output — preventing the anti-pattern
+of a model grading its own work.
+
+### Quick start
+
+```ruby
+session = Ask::Agent::Session.new(
+  model: "gpt-4o",
+  evaluator: { model: "claude-sonnet-4", goal: "Write an email validator" }
+)
+session.run("Write email validation")
+```
+
+### Verdicts
+
+| Verdict | Behavior |
+|---------|----------|
+| `:accept` | Output passes — falls through to reflection |
+| `:revise` | Evaluator provides feedback; session runs another turn with it injected |
+| `:block` | Output is fundamentally wrong — returns blocked message, emits `EvaluationBlocked` |
+
+### Configuration
+
+```ruby
+# Set a global default evaluator model
+Ask::Agent.configure do |c|
+  c.default_evaluator_model = "claude-sonnet-4"
+end
+
+# Then use evaluator: true to enable with the default
+session = Ask::Agent::Session.new(model: "gpt-4o", evaluator: true)
+```
+
+### Custom rubric
+
+```ruby
+evaluator = Ask::Agent::Evaluator.new(
+  model: "claude-sonnet-4",
+  rubric: [
+    Ask::Agent::Evaluator::Dimension.new(
+      name: "performance",
+      description: "Is the implementation efficient?",
+      weight: 2
+    )
+  ]
+)
+
+result = evaluator.evaluate(
+  goal: "Write an email validator",
+  response: agent_output
+)
+result.accept?  # => true/false
+result.scores   # => { performance: 2 }
+result.feedback # => "Add edge case for unicode characters"
+```
+
+### Events
+
+The evaluator emits its own events during evaluation:
+
+```ruby
+session.on_event do |event|
+  case event
+  when Ask::Agent::Events::EvaluationStart
+    puts "Evaluating against: #{event.dimensions.join(', ')}"
+  when Ask::Agent::Events::EvaluationDelta
+    print event.content
+  when Ask::Agent::Events::EvaluationEnd
+    puts "Decision: #{event.decision}"
+    puts "Scores: #{event.scores}"
+  when Ask::Agent::Events::EvaluationBlocked
+    puts "Blocked: #{event.feedback}"
+  end
+end
+```
 
 ## Events
 
@@ -60,13 +141,13 @@ end
 
 Opt-in safety modules:
 
-- **PermissionGate** — Require approval for destructive tools (write, edit, bash, destroy)
+- **Permissions** — Access control for tools. Supports named access modes (`:full_access`, `:read_only`, `:ask_before_changes`) or custom blocked-tool lists.
 - **RateLimiter** — Prevent runaway tool calls (configurable per-minute and per-turn limits)
 - **AuditLog** — Immutable, append-only log of every tool call
 
 ```ruby
 extensions = [
-  Ask::Agent::Extensions::PermissionGate.new,
+  Ask::Agent::Extensions::Permissions.new(mode: :read_only),
   Ask::Agent::Extensions::RateLimiter.new(max_calls_per_minute: 30),
   Ask::Agent::Extensions::AuditLog.new(path: "agent.log")
 ]
@@ -81,6 +162,57 @@ session = Ask::Agent::Session.new(
 )
 ```
 
+## Middleware
+
+Wrapping LLM provider calls with cross-cutting behavior:
+
+- **RetryOnFailure** — Retry on rate limits and server errors with exponential backoff
+- **ModelFallback** — Switch to a fallback model+provider on transient errors
+- **LogCalls** — Log every LLM provider call
+- **DefaultSettings** — Inject default generation parameters
+
+```ruby
+Ask::Agent.configure do |c|
+  c.middleware.use :retry_on_failure, max_retries: 3
+  c.middleware.use :model_fallback, fallbacks: [
+    { model: "claude-sonnet-4",  provider: :anthropic },
+    { model: "gemini-2.0-flash", provider: :google }
+  ]
+  c.middleware.use :log_calls, logger: Rails.logger
+  c.middleware.use :default_settings, temperature: 0.7
+end
+```
+
+### ModelFallback
+
+When the primary LLM is overloaded or down, `ModelFallback` transparently switches to a backup model+provider. Credentials for each provider are resolved automatically.
+
+**Static fallbacks** — ordered list tried in sequence:
+```ruby
+c.middleware.use :model_fallback, fallbacks: [
+  { model: "claude-sonnet-4",  provider: :anthropic },
+  { model: "gemini-2.0-flash", provider: :google }
+]
+```
+
+**Dynamic fallbacks** — lambda that receives the error and request:
+```ruby
+c.middleware.use :model_fallback, fallbacks: ->(error, request) {
+  if request[:messages].sum { |m| m[:content].to_s.length } > 100_000
+    [{ model: "claude-sonnet-4", provider: :anthropic }]  # long-context
+  else
+    [{ model: "gpt-4o-mini", provider: :openai }]           # cheaper
+  end
+}
+```
+
+**Custom eligible errors** — by default rate limits, server errors, and service unavailable:
+```ruby
+c.middleware.use :model_fallback,
+  fallbacks: [{ model: "claude-sonnet-4", provider: :anthropic }],
+  eligible_errors: [Ask::RateLimitError, Ask::ServerError]
+```
+
 ## Configuration
 
 ```ruby

data/lib/ask/agent/configuration.rb

@@ -5,7 +5,7 @@ module Ask
     class Configuration
       attr_accessor :default_model, :default_max_turns, :compactor_enabled,
                     :compactor_threshold, :parallel_tool_execution, :max_tool_retries,
-                    :prompt_caching
+                    :prompt_caching, :default_evaluator_model
 
       # @return [Middleware::Pipeline] the middleware pipeline for provider calls
       attr_reader :middleware

data/lib/ask/agent/evaluator.rb

@@ -0,0 +1,164 @@
+# frozen_string_literal: true
+
+module Ask
+  module Agent
+    class Evaluator
+      # Structured result from an evaluation.
+      # - decision :accept — response meets the goal
+      #           :revise — response needs improvement (feedback provided)
+      #           :block  — response is fundamentally wrong (hard stop)
+      # - feedback actionable text the generator can use to improve
+      # - scores   hash of dimension name => score (0, 1, or 2)
+      # - evidence array of specific evidence strings
+      Result = Data.define(:decision, :feedback, :scores, :evidence) do
+        def accept? = decision == :accept
+        def revise? = decision == :revise
+        def block?  = decision == :block
+      end
+
+      # A single dimension in the evaluation rubric.
+      Dimension = Data.define(:name, :description, :weight) do
+        def initialize(name:, description:, weight: 1)
+          super(name: name, description: description, weight: weight)
+        end
+      end
+
+      # Default rubric borrowed from the course's evaluator-rubric template.
+      DEFAULT_DIMENSIONS = [
+        Dimension.new(name: "correctness",  description: "Does the output match the requested goal?",                               weight: 3),
+        Dimension.new(name: "completeness", description: "Are all aspects of the goal addressed?",                                  weight: 2),
+        Dimension.new(name: "verification", description: "Is there evidence that the output actually works?",                        weight: 2),
+        Dimension.new(name: "scope",        description: "Did it stay within the defined boundaries without overreaching?",          weight: 1),
+        Dimension.new(name: "clarity",      description: "Is the output clear, well-structured, and maintainable?",                  weight: 1),
+      ].freeze
+
+      # How many times the evaluator may retry on a malformed response.
+      MAX_EVAL_RETRIES = 2
+
+      attr_reader :model, :rubric
+
+      # @param model [String] the model id to use for evaluation (should differ from the generator's model)
+      # @param rubric [Array<Dimension>] the rubric dimensions to evaluate against
+      def initialize(model:, rubric: DEFAULT_DIMENSIONS)
+        @model = model
+        @rubric = rubric
+      end
+
+      # Evaluate a response against a goal.
+      #
+      # @param goal [String] what the generator was asked to do
+      # @param response [String] what the generator produced
+      # @param event_emitter [#emit, nil] optional event emitter for streaming evaluation
+      # @return [Result] structured evaluation result
+      def evaluate(goal:, response:, event_emitter: nil)
+        event_emitter&.emit(Events::EvaluationStart.new(dimensions: @rubric.map(&:name)))
+
+        chat = build_chat
+        chat.with_instructions(evaluation_prompt(goal))
+
+        accumulated = +""
+        chat.ask(response.to_s) do |chunk|
+          if chunk.content.to_s.strip.length > 0
+            accumulated << chunk.content.to_s
+            event_emitter&.emit(Events::EvaluationDelta.new(content: chunk.content.to_s))
+          end
+        end
+
+        result = parse_result(accumulated)
+        event_emitter&.emit(Events::EvaluationEnd.new(
+          decision: result.decision,
+          feedback: result.feedback,
+          scores: result.scores,
+          evidence: result.evidence
+        ))
+
+        result
+      end
+
+      private
+
+      def build_chat
+        Chat.new(model: @model)
+      end
+
+      def evaluation_prompt(goal)
+        dimensions_text = @rubric.each_with_index.map { |d, i|
+          weight_label = d.weight > 1 ? " (weight: #{d.weight}x)" : ""
+          "#{i + 1}. **#{d.name}**#{weight_label} — #{d.description}"
+        }.join("\n")
+
+        <<~PROMPT
+          You are an independent evaluator. Your job is to assess whether a response
+          successfully achieves the given goal. You are NOT the agent that produced
+          this response — you are a neutral, objective judge.
+
+          ## Goal
+
+          #{goal}
+
+          ## Rubric
+
+          Evaluate the response against these dimensions:
+
+          #{dimensions_text}
+
+          For each dimension, assign a score:
+          - **0** = fails completely
+          - **1** = partially meets
+          - **2** = fully meets
+
+          Then provide:
+          - A final **decision**: "accept" (response meets the goal), "revise" (needs specific improvements), or "block" (fundamentally wrong — cannot be fixed with revisions)
+          - **Actionable feedback** the generator can use to improve (if decision is revise or block)
+          - **Concrete evidence** for your scores
+
+          Return valid JSON only — no other text:
+          {
+            "scores": { "correctness": 2, "completeness": 1, ... },
+            "decision": "accept",
+            "feedback": "Specific feedback here (or empty string if accepted)",
+            "evidence": ["Evidence point 1", "Evidence point 2"]
+          }
+        PROMPT
+      end
+
+      def parse_result(text)
+        json = extract_json(text)
+        return default_fallback unless json
+
+        scores = json["scores"] || {}
+
+        decision = case json["decision"].to_s.strip.downcase
+                   when "revise" then :revise
+                   when "block"  then :block
+                   else :accept
+                   end
+
+        Result.new(
+          decision: decision,
+          feedback: json["feedback"].to_s.strip,
+          scores: scores.transform_keys(&:to_sym),
+          evidence: Array(json["evidence"])
+        )
+      end
+
+      def extract_json(text)
+        # Try direct parse first
+        JSON.parse(text.strip)
+      rescue JSON::ParserError
+        # Fall back to extracting the first JSON object
+        match = text.match(/\{.*\}/m)
+        match ? JSON.parse(match[0]) : nil
+      end
+
+      def default_fallback
+        Result.new(
+          decision: :accept,
+          feedback: "",
+          scores: {},
+          evidence: []
+        )
+      end
+    end
+  end
+end

data/lib/ask/agent/events.rb

@@ -28,6 +28,11 @@ module Ask
       ReflectionDelta = Data.define(:content)
       ReflectionEnd = Data.define(:decision, :feedback)
 
+      EvaluationStart = Data.define(:dimensions)
+      EvaluationDelta = Data.define(:content)
+      EvaluationEnd = Data.define(:decision, :feedback, :scores, :evidence)
+      EvaluationBlocked = Data.define(:feedback, :scores, :evidence)
+
       MetaAgentAnalysis = Data.define(:results, :count)
 
       Error = Data.define(:error, :recoverable)

data/lib/ask/agent/extensions/{permission_gate.rb → permissions.rb}

@@ -3,14 +3,30 @@
 module Ask
   module Agent
     module Extensions
-      class PermissionGate
+      class Permissions
         DEFAULT_TOOLS = %i[write edit bash destroy].freeze
 
-        def initialize(blocked_tools: DEFAULT_TOOLS, timeout: nil)
-          @blocked_tools = Array(blocked_tools).map(&:to_sym)
+        ACCESS_MODES = {
+          full_access:       { blocked_tools: [] }.freeze,
+          ask_before_changes: { blocked_tools: DEFAULT_TOOLS }.freeze,
+          read_only:         { blocked_tools: %i[write edit bash destroy] }.freeze
+        }.freeze
+
+        def initialize(mode: nil, blocked_tools: nil, timeout: nil)
+          @mode = mode
           @timeout = timeout
           @pending = {}
           @mutex = Mutex.new
+
+          @blocked_tools = if mode
+            config = ACCESS_MODES[mode]
+            raise ArgumentError, "Unknown access mode: #{mode.inspect}. Valid: #{ACCESS_MODES.keys.join(', ')}" unless config
+            config[:blocked_tools].dup
+          elsif blocked_tools
+            Array(blocked_tools).map(&:to_sym)
+          else
+            DEFAULT_TOOLS.dup
+          end
         end
 
         def before_tool_call(tool_call, _context)
@@ -61,7 +77,7 @@ module Ask
             }
           end
 
-          warn "[PermissionGate] Tool '#{tool_call.name}' requires approval. Call approve('#{tool_call.id}') to allow."
+          warn "[Permissions] Tool '#{tool_call.name}' requires approval. Call approve('#{tool_call.id}') to allow."
           { action: :block, reason: "Tool '#{tool_call.name}' requires approval" }
         end
       end

data/lib/ask/agent/loop.rb

@@ -17,7 +17,7 @@ module Ask
       @max_consecutive_tool_turns = max_consecutive_tool_turns
       end
 
-      def run_turn(chat:, message:, tools:, tool_executor:, compactor:, hooks:, event_emitter:, session_id: nil)
+      def run_turn(chat:, message:, tools:, tool_executor:, compactor:, hooks:, event_emitter:, session_id: nil, persist: nil)
         raise MaxTurnsExceeded if @turn_count >= @max_turns
 
         event_emitter.emit(Events::TurnStart.new)
@@ -103,6 +103,9 @@ module Ask
           compactor.run(event_emitter: event_emitter)
         end
 
+        # Persist after each turn so mid-session crashes don't lose progress
+        persist&.call
+
         raise MaxTurnsExceeded if @turn_count >= @max_turns
 
         # Recursive call — LLM processes tool results
@@ -114,7 +117,8 @@ module Ask
           compactor: compactor,
           hooks: hooks,
           event_emitter: event_emitter,
-          session_id: session_id
+          session_id: session_id,
+          persist: persist
         )
       end
 

data/lib/ask/agent/middleware/model_fallback.rb

@@ -0,0 +1,110 @@
+# frozen_string_literal: true
+
+module Ask
+  module Agent
+    module Middleware
+      # Switches to a fallback model+provider when the primary LLM call fails
+      # with a transient error (rate limit, server error, service unavailable).
+      #
+      # Each fallback specifies both a model and a provider slug, so the
+      # middleware can switch from e.g. OpenAI to Anthropic transparently.
+      # Credentials for each provider are resolved automatically via
+      # {Ask::Auth.resolve}.
+      #
+      # @example Basic usage — fallback to Anthropic when OpenAI is overloaded
+      #   pipeline.use :model_fallback, fallbacks: [
+      #     { model: "claude-sonnet-4",     provider: :anthropic },
+      #     { model: "gemini-2.0-flash",    provider: :google }
+      #   ]
+      #
+      # @example With failure-trigger customization
+      #   pipeline.use :model_fallback, fallbacks: [
+      #     { model: "claude-sonnet-4", provider: :anthropic, on_error: [Ask::RateLimitError, Ask::ServerError] }
+      #   ]
+      #
+      # @example Using the block form to choose fallbacks dynamically
+      #   pipeline.use :model_fallback, fallbacks: ->(error, request) {
+      #     if request[:messages].sum { |m| m[:content].to_s.length } > 100_000
+      #       [{ model: "claude-sonnet-4", provider: :anthropic }]  # use long-context model
+      #     else
+      #       [{ model: "gpt-4o-mini", provider: :openai }]          # use cheaper model
+      #     end
+      #   }
+      class ModelFallback < Base
+        DEFAULT_ELIGIBLE_ERRORS = [
+          Ask::RateLimitError, Ask::ServerError, Ask::ServiceUnavailable
+        ].freeze
+
+        def initialize(fallbacks:, eligible_errors: nil)
+          @fallbacks = fallbacks.respond_to?(:call) ? fallbacks : Array(fallbacks)
+          @eligible_errors = Array(eligible_errors || DEFAULT_ELIGIBLE_ERRORS)
+          raise ArgumentError, "At least one fallback is required" if Array(@fallbacks).empty?
+        end
+
+        def around_request(provider, request)
+          # Try primary provider
+          begin
+            return yield
+          rescue *@eligible_errors => e
+            result = try_fallbacks(request, error: e)
+            return result if result
+            raise
+          end
+        end
+
+        private
+
+        def try_fallbacks(request, error:)
+          fallback_list = resolve_fallback_list(error, request)
+
+          fallback_list.each do |fb|
+            begin
+              new_provider = build_fallback_provider(fb[:provider])
+              request[:model] = fb[:model]
+              return invoke_fallback(new_provider, request)
+            rescue *@eligible_errors
+              next  # Try next fallback
+            end
+          end
+
+          nil  # All fallbacks exhausted
+        end
+
+        def resolve_fallback_list(error, request)
+          list = if @fallbacks.respond_to?(:call)
+            @fallbacks.call(error, request)
+          else
+            @fallbacks
+          end
+          raise "Fallback list must be an array of hashes, got #{list.class}" unless list.is_a?(Array)
+          list
+        end
+
+        def build_fallback_provider(provider_slug)
+          slug = provider_slug.to_s
+          klass = Ask::Provider.resolve(slug)
+          klass.new(fallback_config(slug))
+        end
+
+        def invoke_fallback(provider, request)
+          provider.chat(
+            request[:messages],
+            model: request[:model],
+            tools: request[:tools],
+            temperature: request[:temperature],
+            stream: request[:stream],
+            schema: request[:schema],
+            **request[:extra_params]
+          )
+        end
+
+        def fallback_config(slug)
+          key = Ask::Auth.resolve(:"#{slug}_api_key") rescue nil
+          config = { api_key: key }
+          config[:"#{slug}_api_key"] = key
+          Ask::LLM::Config.new(config)
+        end
+      end
+    end
+  end
+end

data/lib/ask/agent/middleware/pipeline.rb

@@ -19,7 +19,8 @@ module Ask
         KNOWN_MIDDLEWARES = {
           retry_on_failure: "Ask::Agent::Middleware::RetryOnFailure",
           log_calls: "Ask::Agent::Middleware::LogCalls",
-          default_settings: "Ask::Agent::Middleware::DefaultSettings"
+          default_settings: "Ask::Agent::Middleware::DefaultSettings",
+          model_fallback: "Ask::Agent::Middleware::ModelFallback"
         }.freeze
 
         def initialize

data/lib/ask/agent/session.rb

@@ -18,10 +18,10 @@ module Ask
       attr_reader :skills_registry
 
       def initialize(model:, tools: [], max_turns: 25, max_tool_retries: 3,
-                     compactor: nil, hooks: {}, persistence: nil,
+                     compactor: nil, hooks: {}, state: nil, persistence: nil,
                      id: nil, system_prompt: nil, parallel_tools: true,
                      reflector: nil, telemetry: true, meta_agent: nil,
-                     agent_dir: nil, **chat_options)
+                     agent_dir: nil, evaluator: nil, **chat_options)
         @id = id || SecureRandom.uuid
         @agent_dir = agent_dir
         @max_turns = max_turns
@@ -48,11 +48,10 @@ module Ask
         @compactor = compactor ? build_compactor(compactor) : nil
         @hooks = Hooks.new(hooks)
 
-        # Build system context from typed sources
         @system_context = build_system_context(system_prompt)
         apply_system_context
 
-        @persistence = persistence
+        @state = state || persistence
 
         reflector_opts = reflector.is_a?(Hash) ? reflector : {}
         @reflector = if reflector
@@ -66,6 +65,21 @@ module Ask
         @meta_agent_results = nil
 
         @compactor&.chat = @chat
+
+        # Parse evaluator configuration
+        @evaluator = nil
+        @evaluator_config = {}
+
+        if evaluator
+          eval_model = if evaluator.is_a?(Hash)
+                         @evaluator_config = evaluator
+                         evaluator[:model] || Ask::Agent.configuration.default_evaluator_model || model_id_from(@chat)
+                       else
+                         Ask::Agent.configuration.default_evaluator_model || model_id_from(@chat)
+                       end
+
+          @evaluator = Evaluator.new(model: eval_model)
+        end
       end
 
       def run(message, tools: nil)
@@ -89,16 +103,17 @@ module Ask
         begin
           @tool_executor.telemetry = @telemetry
 
-          response = @loop.run_turn(
-            chat: @chat,
-            message: message,
-            tools: active_tools,
-            tool_executor: @tool_executor,
-            compactor: @compactor,
-            hooks: @hooks,
-            event_emitter: self,
-            session_id: @id
-          )
+	          response = @loop.run_turn(
+	            chat: @chat,
+	            message: message,
+	            tools: active_tools,
+	            tool_executor: @tool_executor,
+	            compactor: @compactor,
+	            hooks: @hooks,
+	            event_emitter: self,
+	            session_id: @id,
+	            persist: @state ? method(:persist!) : nil
+	          )
 
           @total_input_tokens += @loop.last_input_tokens.to_i
           @total_output_tokens += @loop.last_output_tokens.to_i
@@ -122,12 +137,67 @@ module Ask
           raise
         ensure
           @running = false
-          persist! if @persistence
+          persist! if @state
         end
 
         @tool_calls_made = @tool_executor.total_executions
 
-        if @reflector && @reflector.reflect?(@tool_calls_made) && !@abort_requested
+        # Independent evaluator step (generator/evaluator separation).
+        # Runs BEFORE self-reflection so the evaluator gets a fresh, unbiased look
+        # at the generator's output using a separate model and isolated context.
+        @skip_reflector = false
+
+        if @evaluator && !@abort_requested
+          goal = @evaluator_config[:goal] || message
+
+          eval_result = @evaluator.evaluate(
+            goal: goal.to_s,
+            response: response,
+            event_emitter: self
+          )
+
+          @telemetry.log(:evaluation_end, session_id: @id,
+                         decision: eval_result.decision,
+                         feedback: eval_result.feedback,
+                         scores: eval_result.scores)
+
+          case eval_result.decision
+          when :revise
+            @chat.add_message(
+              role: :system,
+              content: "An independent evaluator has requested revisions:\n\n#{eval_result.feedback}"
+            )
+
+            response = @loop.run_turn(
+              chat: @chat,
+              message: "",
+              tools: active_tools,
+              tool_executor: @tool_executor,
+              compactor: @compactor,
+              hooks: @hooks,
+              event_emitter: self,
+              session_id: @id
+            )
+
+            @total_input_tokens += @loop.last_input_tokens.to_i
+            @total_output_tokens += @loop.last_output_tokens.to_i
+            @total_cost += @loop.last_cost.to_f
+
+            # Skip reflector — we already iterated based on evaluator feedback
+            @skip_reflector = true
+          when :block
+            emit(Events::EvaluationBlocked.new(
+              feedback: eval_result.feedback,
+              scores: eval_result.scores,
+              evidence: eval_result.evidence
+            ))
+            response = "This response was blocked by the evaluator: #{eval_result.feedback}"
+          when :accept
+            # Fall through to reflector for backward compatibility
+          end
+        end
+
+        if @reflector && !@skip_reflector && @reflector.reflect?(@tool_calls_made) && !@abort_requested
           eval_result = @reflector.evaluate(response: response, event_emitter: self)
           @telemetry.log(:reflection_end, session_id: @id, decision: eval_result[:decision], feedback: eval_result[:feedback])
 
@@ -193,34 +263,37 @@ module Ask
       def deleted? = @deleted
 
       def save
-        persist! if @persistence
+        persist! if @state
       end
 
       def self.load(id, adapter:)
-        data = adapter.load(id)
+        data = adapter.get(id)
         return nil unless data
 
+        data = deep_symbolize_keys(data)
+
         session = new(
           id: data[:id],
           model: data.dig(:metadata, :model),
           tools: data.dig(:metadata, :tools)&.map(&:constantize) || [],
-          persistence: adapter
+          state: adapter
         )
 
-        data[:messages].each do |msg|
-          session.chat.add_message(
-            role: msg[:role].to_sym,
-            content: msg[:content],
-            tool_call_id: msg[:tool_call_id]
-          )
-        end
-
-        session
+	        data[:messages].each do |msg|
+	          session.chat.add_message(
+	            role: msg[:role].to_sym,
+	            content: msg[:content],
+	            tool_call_id: msg[:tool_call_id]
+	          )
+	        end
+	
+	        session.instance_variable_set(:@messages, session.chat.messages.dup)
+	        session
       end
 
       def delete
         @deleted = true
-        @persistence&.delete(@id)
+        @state&.delete(@id)
       end
 
       def abort
@@ -290,7 +363,7 @@ module Ask
       end
 
       def persist!
-        @persistence.save(@id, {
+        @state.set(@id, {
           id: @id,
           messages: @chat.messages.map { |m|
             {
@@ -358,6 +431,19 @@ module Ask
         SystemContext.new(sources)
       end
 
+      # Recursively convert string keys to symbol keys in hashes.
+      # Needed when loading session data that was serialized through JSON.
+      def self.deep_symbolize_keys(obj)
+        case obj
+        when Hash
+          obj.each_with_object({}) { |(k, v), h| h[k.to_sym] = deep_symbolize_keys(v) }
+        when Array
+          obj.map { |e| deep_symbolize_keys(e) }
+        else
+          obj
+        end
+      end
+
       # Render the system context and apply it to the chat.
       def apply_system_context
         rendered = @system_context.render

data/lib/ask/agent/version.rb

@@ -2,6 +2,6 @@
 
 module Ask
   module Agent
-    VERSION = "0.12.1"
+    VERSION = "0.15.0"
   end
 end

data/lib/ask/agent.rb

@@ -21,7 +21,7 @@ module Ask
     class UnknownAgent < Error; end
 
     module Extensions
-      autoload :PermissionGate, "ask/agent/extensions/permission_gate"
+      autoload :Permissions, "ask/agent/extensions/permissions"
       autoload :RateLimiter, "ask/agent/extensions/rate_limiter"
       autoload :AuditLog, "ask/agent/extensions/audit_log"
     end
@@ -32,6 +32,7 @@ module Ask
       autoload :RetryOnFailure, "ask/agent/middleware/retry_on_failure"
       autoload :LogCalls, "ask/agent/middleware/log_calls"
       autoload :DefaultSettings, "ask/agent/middleware/default_settings"
+      autoload :ModelFallback, "ask/agent/middleware/model_fallback"
     end
 
     module StreamTransforms
@@ -231,6 +232,7 @@ require_relative "agent/tool_abort_controller"
 require_relative "agent/session"
 require_relative "agent/loop"
 require_relative "agent/reflector"
+require_relative "agent/evaluator"
 require_relative "agent/tool_executor"
 require_relative "agent/compactor"
 require_relative "agent/hooks"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ask-agent
 version: !ruby/object:Gem::Version
-  version: 0.12.1
+  version: 0.15.0
 platform: ruby
 authors:
 - Kaka Ruto
@@ -23,6 +23,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0.1'
+- !ruby/object:Gem::Dependency
+  name: ask-state-providers
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0.1'
 - !ruby/object:Gem::Dependency
   name: ask-llm-providers
   requirement: !ruby/object:Gem::Requirement
@@ -157,9 +171,10 @@ files:
 - lib/ask/agent/context_source.rb
 - lib/ask/agent/context_sources.rb
 - lib/ask/agent/definition.rb
+- lib/ask/agent/evaluator.rb
 - lib/ask/agent/events.rb
 - lib/ask/agent/extensions/audit_log.rb
-- lib/ask/agent/extensions/permission_gate.rb
+- lib/ask/agent/extensions/permissions.rb
 - lib/ask/agent/extensions/rate_limiter.rb
 - lib/ask/agent/hooks.rb
 - lib/ask/agent/loop.rb
@@ -167,6 +182,7 @@ files:
 - lib/ask/agent/middleware/base.rb
 - lib/ask/agent/middleware/default_settings.rb
 - lib/ask/agent/middleware/log_calls.rb
+- lib/ask/agent/middleware/model_fallback.rb
 - lib/ask/agent/middleware/pipeline.rb
 - lib/ask/agent/middleware/retry_on_failure.rb
 - lib/ask/agent/persistence/base.rb