ask-decisions 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 0eecbf122542bb526917a22123bd02d15ed9aaee01db6c3283024858cbb5a867
+  data.tar.gz: 6192d3b7bab381fe65d3f43119ee836598aa68b2639eae0fb87379b432bd838a
+SHA512:
+  metadata.gz: f90ced1aac865272ac6bdbc55f734ba55be112d0178b3fdc07214b3758b12399606bc6ccb3df48f2ca20f2968288759fa02bf975e20add4e1faeb8f8b52fc7a3
+  data.tar.gz: efad93c3e452c09bc1b0e188c6445066dd6787a13ed9d6d9698e2198f7a90a17faff13b7aa5c26f92bd87c7c0fcefd1edda23dd84f489ae15ef7ae1497216ef4

data/CHANGELOG.md

@@ -0,0 +1,67 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+
+## [0.1.0] - 2026-09-18
+
+The first release, so everything here is new. It is a decision layer for the
+ask-rb ecosystem: ask Jev (or any System One model) typed questions and get
+back answers with calibrated probabilities. LLMs generate text; deciders
+decide.
+
+### Added
+
+- `Ask::Decisions::Reader` — asks a described set of options as one Choice,
+  with anything else the caller needs riding along in the same request.
+  `ToolRouter` and `Triage` are façades over it.
+- `Ask::Decisions::Triage` — reads a message into a caller-defined lane and
+  asks the mood and whether the person wants a human, all in one request.
+  Measured against a 19-tool roster: lane-level routing was right 19/20 where
+  tool-level routing was right 10/16 — the tools overlapped, and a lane is the
+  part of the decision the message actually carries.
+- `Ask::Decisions::AgentAdapter` — wires `Gate`, `OutputJudge`,
+  `FailureClassifier`, `LoopDetector`, `QualityJudge`, `ReflectionJudge`,
+  `ToolRepairer` and `ConfidencePolicy` into ask-agent's `before_tool` /
+  `after_tool` hooks, so one config line activates the guard half:
+  `Ask::Agent.configure { |c| c.decision_provider = :typesafe }`.
+- `Ask::Decision::Choice`, `Score`, `Noul` primitives in ask-core
+- `Ask::DecisionResult::ChoiceAnswer`, `ScoreAnswer`, `NoulAnswer`, `Batch` in ask-core
+- `Ask::DecisionProvider` base class + registry in ask-core
+- `Ask::Decisions::Typesafe` — HTTP client for TypeSafe/System One API
+- `Ask::Decisions::Batcher` — one-call batching for parallel questions
+- `Ask::Decisions::Cache` — MD5-keyed result caching (120s TTL)
+- `Ask::Decisions::Static` — canned answers for tests
+- `Ask::Decisions::Lint` — anti-pattern detection (8 rules, 2 measured from pi-jev)
+- `Ask::Decisions::Gate` — pre-tool-call intent judge (4 calibrated questions)
+- `Ask::Decisions::OutputJudge` — post-tool-call leak detection + failure classification (6 classes)
+- `Ask::Decisions::FailureClassifier` — retry logic with attempt tracking
+- `Ask::Decisions::ToolRouter` — Choice over tool roster + non-tool outcomes
+- `Ask::Decisions::ArgumentResolver` — enum→Choice, bool→Noul, free text→generator
+- `Ask::Decisions::DecisionState` — compact state projection with token budget
+- `Ask::Decisions::ConfidencePolicy` — risk-tiered act/review/escalate thresholds
+- `Ask::Decisions::LoopDetector` — Jev-based stuck/repeating detection
+- `Ask::Decisions::QualityJudge` — composite scoring over rubric dimensions
+- `Ask::Decisions::ReflectionJudge` — Noul-based self-critique
+- `Ask::Decisions::ToolRepairer` — Choice over valid tool names + enum values
+- `Ask::Decisions::Reranker` — Score per query-passage pair for ask-rag
+- `Ask::Decisions::StructuredStateLoop` — OCR/DOM→Jev→action for ask-computer
+- `Ask::Decisions::CalibrationReport` — reliability curves per decision id
+- `Ask::Decisions::CalibrationHarness` — run test cases and measure calibration
+- `Ask::Tools::Decide` — bridge tool for LLM agents (requires ask-tools)
+- `Ask::Decisions::MCPHelper` — one-liner to add decide to MCP servers
+- `Ask.decide` and `Ask::Decisions.batch` convenience methods
+- Provider registry: `Ask::DecisionProvider.register`
+- `ask-tools` `param :enum` support (backward compatible)
+
+### Changed
+
+- `Ask::Decisions::ToolRouter` takes `criteria:` — routing-grade descriptions,
+  tool name to when to choose it — and a `limit:`. A tool's own description is
+  written for the model that already holds the tool, so two accurate
+  descriptions can still fail to separate their tools from the outside.
+- `Triage::Verdict#wants_human?` requires the probability to be *above* the
+  threshold. A noul at exactly 0.5 is the model saying it has no idea, which
+  is the one answer that must not read as consent.
+- Requires `ask-core >= 0.12.0` for the decision vocabulary.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kaka Ruto
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,238 @@
+# ask-decisions
+
+Decision primitives and providers for the [ask-rb](https://github.com/ask-rb) ecosystem. Ask Jev (or any System One model) structured questions — classify, route, score, guard — and get typed answers with calibrated probabilities. LLMs generate text. Deciders decide.
+
+```ruby
+result = Ask.decide(
+  state: "Help! My payouts have been failing for 3 days.",
+  decisions: {
+    "route" => Ask::Decision::Choice.new(
+      instructions: "Which team should handle this?",
+      criteria: { billing: "Payment issues", technical: "Bugs", sales: "Pricing" }
+    ),
+    "urgent" => Ask::Decision::Noul.new(
+      instructions: "Does this message convey urgency?"
+    )
+  }
+)
+# result["route"].choice       # => "billing"
+# result["route"].confidence  # => 0.89
+# result["urgent"].noul       # => 0.96
+```
+
+## Installation
+
+```ruby
+gem "ask-decisions"
+```
+
+## Setup
+
+```ruby
+require "ask-decisions"
+
+Ask::Decisions.configure do |c|
+  c.default_provider = :typesafe  # or :static for tests
+  c.default_model = "jev-latest"
+end
+```
+
+## The four verbs
+
+Everything Jev does with LLMs falls into one of these patterns:
+
+| Verb | Shape | Example |
+|---|---|---|
+| **Route** | decide → generate | Jev picks the tool; LLM writes the answer |
+| **Filter** | generate → decide | LLM proposes facts; Jev keeps the good ones |
+| **Replace** | decide, no generate | Triage, scoring, relevance — no text needed |
+| **Guard** | decide ⟶ gate generate | Screen input/output; gate actions on confidence |
+
+Anything expressible as a closed set becomes a decision. Anything that must produce new text stays a generation.
+
+## Core primitives
+
+Three question types, three answer types:
+
+```ruby
+# Choice — pick one option from a set
+Ask::Decision::Choice.new(
+  instructions: "Which team?",
+  criteria: { billing: "Payments", technical: "Bugs", none: "None of these" }
+)
+
+# Score — rate on a spectrum
+Ask::Decision::Score.new(
+  instructions: "How urgent?",
+  criteria: ["Not urgent", "Somewhat urgent", "Very urgent"]
+)
+
+# Noul — yes/no with probability
+Ask::Decision::Noul.new(
+  instructions: "Does this convey urgency?"
+)
+```
+
+## Batching
+
+Jev evaluates all questions in one request in parallel. Send every question you might need:
+
+```ruby
+result = Ask::Decisions.batch(state: "...") do |b|
+  b.ask("route", Ask::Decision::Choice.new(...))
+  b.ask("urgent", Ask::Decision::Noul.new(...))
+  b.ask("quality", Ask::Decision::Score.new(...))
+end
+# One API call, three answers.
+```
+
+## Confidence
+
+Every choice/score answer carries calibrated confidence. Noul answers carry a value (0–1) with no separate confidence — use the distance from 0.5 as strength.
+
+```ruby
+result["route"].confident?(0.7)  # => true
+result["urgent"].strength        # => 0.46 (distance from 0.5)
+result["urgent"].yes?            # => true (noul >= 0.5)
+```
+
+## Swapping providers
+
+The registry makes providers swappable:
+
+```ruby
+# Use Jev in production
+Ask::Decisions.configure { |c| c.default_provider = :typesafe }
+
+# Use canned answers in tests
+Ask::Decisions.configure { |c| c.default_provider = :static }
+
+# Register a future provider
+Ask::DecisionProvider.register(:openjev, MyProvider)
+Ask::Decisions.configure { |c| c.default_provider = :openjev }
+```
+
+## Gate (pre-tool-call)
+
+Calibrated intent judge — 4 questions in one request:
+
+```ruby
+gate = Ask::Decisions::Gate.new(provider)
+verdict = gate.judge(tool: "bash", args: { command: "rm -rf src" })
+verdict.passed?    # => false
+verdict.flagged    # => [:destructive, :beyond_scope]
+```
+
+## OutputJudge (post-tool-call)
+
+Screens tool output for leaks and classifies failures:
+
+```ruby
+judge = Ask::Decisions::OutputJudge.new(provider)
+result = judge.judge(tool: "bash", output: "npm ERR! code ECONNRESET")
+result.leak?           # => false
+result.failure_class   # => "transient"
+result.advice          # => "Retry unchanged."
+```
+
+## Triage
+
+Read a message once and answer everything a turn needs about it — which lane
+it belongs to, how the person sounds, whether they want a human. All in one
+request, so the extra questions cost no extra latency:
+
+```ruby
+triage = Ask::Decisions::Triage.new(provider, lanes: {
+  "knowledge" => "Asks about the business, its services, prices, or hours",
+  "booking"   => "Wants to book an appointment or asks what times are free",
+  "human"     => "Wants to speak to a person, or describes an emergency",
+  "close"     => "Says goodbye or is done",
+  "chat"      => "Small talk or a greeting needing no action",
+  "unclear"   => "None of these is clear; a clarifying question is needed first"
+})
+
+verdict = triage.read(message: "What time do you close on Saturdays?")
+verdict.lane        # => "knowledge"
+verdict.confidence  # => 1.0
+verdict.certain?(0.7)   # => true
+verdict.sentiment       # => 1.0 (0 = upset, 1 = neutral, 2 = warm)
+verdict.wants_human?    # => false
+```
+
+### Lanes, not tools
+
+Route to a **lane**, then let code map the lane to its tools. Measured on a
+19-tool roster, same model and same messages:
+
+| Routed to | Correct |
+|---|---|
+| one of the 19 tools | 10/16 |
+| one of 7 lanes | **19/20** |
+
+The tools overlapped — seven of them answered questions about the business,
+and which one holds the answer is found by calling them, not by reading the
+message. Routing straight to a tool asks for a distinction the message does
+not carry. A lane is the part that *is* decidable from the message alone.
+
+A reading should narrow, never grant: let the lane take tools away from a
+turn, and let the agent's own definition stay the ceiling.
+
+## ToolRouter
+
+Routes user turns to the right tool:
+
+```ruby
+router = Ask::Decisions::ToolRouter.new(provider, tools: tool_roster)
+result = router.route(user_turn: "run the tests")
+result.tool         # => "bash"
+result.confidence   # => 0.92
+```
+
+## ConfidencePolicy
+
+Risk-tiered action gating:
+
+```ruby
+policy = Ask::Decisions::ConfidencePolicy.new
+policy.add_rule("bash", risk: :low)
+policy.add_rule("rm", risk: :high)
+
+policy.evaluate(tool: "bash", confidence: 0.6).action   # => :act
+policy.evaluate(tool: "rm", confidence: 0.8).action      # => :review
+```
+
+## Calibration
+
+Measure whether confidence is trustworthy on YOUR decisions:
+
+```ruby
+harness = Ask::Decisions::CalibrationHarness.new(provider)
+harness.add_case(
+  id: "urgent_ticket",
+  state: "Help! Server down!",
+  decisions: { "urgent" => Ask::Decision::Noul.new(instructions: "Is this urgent?") },
+  expected: { "urgent" => { noul_above: 0.7 } },
+  runs: 10
+)
+report = harness.run
+puts report  # reliability curve per decision id, accuracy by confidence band
+```
+
+## Lint
+
+Catch anti-patterns before they hit the API:
+
+```ruby
+warnings = Ask::Decisions::Lint.check({
+  "good" => Ask::Decision::Noul.new(instructions: "Is this urgent?"),
+  "bad" => Ask::Decision::Choice.new(
+    instructions: "Which team? Since this cannot be recovered...",
+    criteria: { a: "A", b: "B" }
+  )
+})
+# => ["bad: instructions contain a reasoning path...", "bad: Choice without a none/other option..."]
+```
+
+## License
+
+MIT

data/lib/ask/decisions/agent_adapter.rb

@@ -0,0 +1,178 @@
+# frozen_string_literal: true
+
+module Ask
+  module Decisions
+    # Bridges ask-decisions into ask-agent. Configures an agent session
+    # to use decision providers for the gate, output judge, evaluator,
+    # reflector, failure classification, loop detection, and confidence
+    # policy — while leaving the LLM in place for text generation.
+    #
+    # This is the "seamlessly usable" integration: one config line
+    # activates the full decision layer.
+    #
+    #   Ask::Agent.configure do |c|
+    #     c.default_model = "gpt-4o"
+    #     c.decision_provider = :typesafe
+    #   end
+    #
+    # Or per-session:
+    #
+    #   session = Ask::Agent::Session.new(
+    #     model: "gpt-4o",
+    #     decision_provider: :typesafe
+    #   )
+    #
+    class AgentAdapter
+      attr_reader :provider, :gate, :output_judge, :failure_classifier,
+                  :loop_detector, :confidence_policy, :quality_judge,
+                  :reflection_judge, :tool_repairer
+
+      # @param provider_name [Symbol, String] the decision provider to use
+      # @param config [Hash] optional overrides
+      def initialize(provider_name, config = {})
+        @provider = resolve_provider(provider_name)
+        @config = config
+
+        # Build all decision components with the same provider.
+        @gate = Gate.new(@provider, **gate_config)
+        @output_judge = OutputJudge.new(@provider, **output_judge_config)
+        @failure_classifier = FailureClassifier.new(@provider, **failure_classifier_config)
+        @loop_detector = LoopDetector.new(@provider)
+        @confidence_policy = build_confidence_policy
+        @quality_judge = QualityJudge.new(@provider)
+        @reflection_judge = ReflectionJudge.new(@provider, **reflection_judge_config)
+        @tool_repairer = ToolRepairer.new(@provider)
+      end
+
+      # Generate before_tool hooks for ask-agent.
+      # Returns an array of callables that match the Hooks interface.
+      def before_tool_hooks
+        [build_gate_hook]
+      end
+
+      # Generate after_tool hooks for ask-agent.
+      def after_tool_hooks
+        [build_output_judge_hook]
+      end
+
+      # Build a tool-call repair function that uses the ToolRepairer.
+      def repair_tool_call(tool_call, available_tools)
+        @tool_repairer.repair(
+          attempted_tool: tool_call.name,
+          attempted_args: tool_call.arguments || {},
+          available_tools: available_tools
+        )
+      end
+
+      # Classify a tool result for retry decisions.
+      def classify_failure(tool:, output:, args: {}, attempt: 1)
+        @failure_classifier.classify(
+          tool: tool, output: output, args: args, attempt: attempt
+        )
+      end
+
+      # Check for agent loops.
+      def check_loop(recent_turns:, current_tool: nil, current_args: nil, turn_count: 0)
+        @loop_detector.check(
+          recent_turns: recent_turns,
+          current_tool: current_tool,
+          current_args: current_args,
+          turn_count: turn_count
+        )
+      end
+
+      # Evaluate answer quality.
+      def evaluate_quality(request:, response:, rubric: nil)
+        opts = { request: request, response: response }
+        opts[:rubric] = rubric if rubric
+        @quality_judge.evaluate(**opts)
+      end
+
+      # Run reflection.
+      def reflect(request:, response:, attempt: 1)
+        @reflection_judge.reflect(
+          request: request, response: response, attempt: attempt
+        )
+      end
+
+      # Evaluate confidence for an action.
+      def evaluate_confidence(tool:, confidence:)
+        @confidence_policy.evaluate(tool: tool, confidence: confidence)
+      end
+
+      private
+
+      def resolve_provider(name)
+        Ask::Decisions.resolve_provider(name)
+      end
+
+      def gate_config
+        { tools: @config[:gate_tools], thresholds: @config[:gate_thresholds] || {} }.compact
+      end
+
+      def output_judge_config
+        { tools: @config[:output_tools], output_limit: @config[:output_limit] }.compact
+      end
+
+      def failure_classifier_config
+        { max_retries: @config[:max_retries] }.compact
+      end
+
+      def reflection_judge_config
+        { max_reflections: @config[:max_reflections] }.compact
+      end
+
+      def build_confidence_policy
+        policy = ConfidencePolicy.new
+        @config[:risk_rules]&.each do |tool, risk|
+          policy.add_rule(tool, risk: risk)
+        end
+        policy
+      end
+
+      # Build a before_tool hook that runs the Gate.
+      def build_gate_hook
+        gate = @gate
+        lambda do |tool_call, context|
+          args = tool_call.respond_to?(:arguments) ? tool_call.arguments : {}
+          working_dir = context[:working_dir] || context["working_dir"]
+          user_message = context[:user_message] || context["user_message"]
+
+          verdict = gate.judge(
+            tool: tool_call.name,
+            args: args || {},
+            working_dir: working_dir,
+            user_message: user_message
+          )
+
+          if verdict.flagged?
+            { action: :block, reason: verdict.to_s }
+          else
+            { action: :proceed }
+          end
+        end
+      end
+
+      # Build an after_tool hook that runs the OutputJudge.
+      def build_output_judge_hook
+        judge = @output_judge
+        lambda do |tool_call, result|
+          output = result.respond_to?(:output) ? result.output : result.to_s
+          args = tool_call.respond_to?(:arguments) ? tool_call.arguments : {}
+
+          output_result = judge.judge(
+            tool: tool_call.name,
+            output: output.to_s,
+            args: args || {}
+          )
+
+          if output_result.leak?
+            { action: :block, reason: "Secret detected in output" }
+          else
+            { action: :proceed, failure_class: output_result.failure_class }
+          end
+        end
+      end
+    end
+  end
+end

data/lib/ask/decisions/argument_resolver.rb

@@ -0,0 +1,136 @@
+# frozen_string_literal: true
+
+module Ask
+  module Decisions
+    # Resolves tool arguments by asking Jev to fill enum/bool params from the
+    # user's turn, while routing free-text params to the generator.
+    #
+    # This is the hybrid: Jev decides closed-set args, the LLM writes free-text.
+    #
+    #   resolver = Ask::Decisions::ArgumentResolver.new(provider)
+    #   result = resolver.resolve(
+    #     tool_name: "linear.create_issue",
+    #     params_schema: tool.params_schema,
+    #     user_turn: "create a bug in the eng team about login failures",
+    #     model: "jev-latest"
+    #   )
+    #   result.resolved_args   # => {"team" => "eng", "priority" => "high"}
+    #   result.needs_generation # => ["title", "description"]
+    #   result.confidence      # => 0.85
+    #
+    class ArgumentResolver
+      # @param provider [Ask::DecisionProvider]
+      def initialize(provider)
+        @provider = provider
+      end
+
+      # Resolve as many arguments as possible via Jev decisions.
+      #
+      # @param tool_name [String] the tool's name (for context in instructions)
+      # @param params_schema [Hash] the tool's JSON Schema
+      # @param user_turn [String] the user's message
+      # @param model [String, nil] model override
+      # @return [Resolution]
+      def resolve(tool_name:, params_schema:, user_turn:, model: nil)
+        properties = params_schema["properties"] || {}
+        required = Array(params_schema["required"] || [])
+
+        resolved = {}
+        needs_generation = []
+        questions = {}
+
+        properties.each do |param_name, prop_schema|
+          enum = prop_schema["enum"]
+          type = prop_schema["type"]
+          desc = prop_schema["description"] || param_name
+
+          if enum && !enum.empty?
+            # Closed set → Choice question
+            criteria = {}
+            enum.each { |v| criteria[v.to_s] = "#{v}" }
+            questions[param_name] = Ask::Decision::Choice.new(
+              instructions: "What value should the `#{param_name}` parameter be for #{tool_name}?",
+              criteria: criteria
+            )
+          elsif type == "boolean"
+            # Boolean → Noul
+            questions[param_name] = Ask::Decision::Noul.new(
+              instructions: "Should the `#{param_name}` parameter be true?"
+            )
+          else
+            # Free text → generator
+            needs_generation << param_name
+          end
+        end
+
+        if questions.empty?
+          return Resolution.new(
+            resolved: {},
+            needs_generation: required + (properties.keys - required),
+            confidence: nil,
+            result: nil
+          )
+        end
+
+        result = @provider.evaluate(
+          state: { tool_name: tool_name, user_turn: user_turn },
+          decisions: questions,
+          model: model
+        )
+
+        # Extract resolved values from answers.
+        questions.each_key do |param_name|
+          answer = result[param_name]
+          next unless answer
+
+          case answer
+          when DecisionResult::ChoiceAnswer
+            resolved[param_name] = answer.choice
+          when DecisionResult::NoulAnswer
+            resolved[param_name] = answer.yes?
+          end
+        end
+
+        Resolution.new(
+          resolved: resolved,
+          needs_generation: needs_generation,
+          confidence: result.min_confidence,
+          result: result
+        )
+      end
+
+      # Resolution result.
+      class Resolution
+        attr_reader :resolved, :needs_generation, :confidence, :result
+
+        def initialize(resolved:, needs_generation:, confidence:, result:)
+          @resolved = resolved
+          @needs_generation = needs_generation
+          @confidence = confidence
+          @result = result
+        end
+
+        # All args that can be filled right now (resolved + defaults for
+        # unmentioned optional params).
+        def auto_fill_args
+          resolved.dup
+        end
+
+        # Whether the LLM needs to generate any arguments.
+        def needs_generation? = !needs_generation.empty?
+
+        def confident?(threshold = 0.7)
+          return false if confidence.nil?
+          confidence >= threshold
+        end
+
+        def to_s
+          parts = []
+          resolved.each { |k, v| parts << "#{k}=#{v.inspect}" }
+          parts << "generate: #{needs_generation.join(', ')}" if needs_generation.any?
+          parts.join(", ")
+        end
+      end
+    end
+  end
+end