kairos-chain 3.28.5 → 3.29.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a9e6e6ee56ff0ad3042cb698090a85885c5285fbb9a6d298b8b3fae7b020e615
-  data.tar.gz: 46653afe61dba50530479c8869e785ba4e526fb5c846c6e26975232d29dbddf0
+  metadata.gz: 1bf83d4bddb63e987a97aec39676cec8048f3fcaa046a6469c4cc6440cb52893
+  data.tar.gz: bfe1486854edea91f778a50f5b02def6a8e95bfa3cf850a71a98fe4cccf7e48d
 SHA512:
-  metadata.gz: 02e1a25744db4600b09b4ce4d05756d564328ff118d40c59ddd9d1805ffff357434b75f91be906bbe3b29625612638d9b000c330380e6531e152d04d321d1c87
-  data.tar.gz: c579ee36020d26916bb3e228c451b4ac8cb3a3b8ecd262fbc7cfee50fd6d49b77a3c4cc2254e216c8faa22547636e2c5036faf62c2fe3f848f81dbefa811a3fb
+  metadata.gz: dae0956cae6e56d25f180db49cb468b6841d14c00fcc08f3a311baa8441b2128d468c84748af9a3c952e73f189f6ee01f965120fbf4e3557af83b860a35f0ae5
+  data.tar.gz: dbe513c7245ad2fd71d72e2e37b3c90e3b09a191de84562ed144b4c7e16744c2ffd3d3ac36a8fe662701dee407b4be53e643d7b9697b2605f2c96b8f2c3801aa

data/CHANGELOG.md

@@ -4,6 +4,37 @@ All notable changes to the `kairos-chain` gem will be documented in this file.
 
 This project follows [Semantic Versioning](https://semver.org/).
 
+## [3.29.0] - 2026-05-30
+
+### Added — `llm_cross_evaluation` v2.3 SkillSet (intra-family difference, INV-1/2/3/6)
+
+Ships the v2.3 cross-evaluation SkillSet into the bundled templates. The
+re-frame is "near-kin = control": rather than treating consensus as validity,
+within-family version differences become first-class, and agreement is
+discounted while differences are preserved. Three invariants are now wired
+into the pipeline (`scripts/run_cross_eval.rb`) on top of pure decision cores
+(`scripts/lib/{intra_family_v23,calibration_v23}.rb`):
+
+- **INV-2 (calibration validity)**: a `evaluation_mode: calibration` task scores
+  Layer 0.5 via `V23::Calibration` against a human `answer_key`
+  (confidence-to-reference, Brier-style + overconfidence on unknowable items),
+  replacing the `|self - peer|` metric that saturates under frontier models.
+  Non-calibration tasks keep the legacy path.
+- **INV-3/6 (consensus removal)**: `overall_ranking` aggregates L1/L2 with
+  independence weighting (each known family = one vote; opaque providers each
+  their own vote); the final standing refuses to read as a ranking when models
+  saturate (tie), registering the saturated group to the INV-4 limits report.
+  Combined weights are fixed across runs.
+- **INV-1 (repeat-trial noise floor)**: Layer D runs each forced-choice
+  comparison `--layerd-trials` times (default 3); per (pair, axis) the
+  trial-to-trial standard deviation of net independent agreement becomes a
+  noise floor a difference must exceed. Cost scales K×.
+
+Pure logic is unit-tested (124 runs, Ruby 3.1.3 and 3.3.7). The wiring passed a
+multi-LLM implementation review (Anthropic persona gate satisfied). Layer D and
+the new tasks run from the local `.kairos/knowledge` copy; this release pins the
+distributable templates copy.
+
 ## [3.28.5] - 2026-05-29
 
 ### Changed — llm_client Claude Code adapter default model → Opus 4.8

data/lib/kairos_mcp/version.rb

@@ -1,4 +1,4 @@
 module KairosMcp
-  VERSION = "3.28.5"
+  VERSION = "3.29.0"
   CHANGELOG_URL = "https://github.com/masaomi/KairosChain_2026/blob/main/CHANGELOG.md"
 end

data/templates/knowledge/llm_cross_evaluation/assets/prompts/pairwise_forced_choice.md.erb

@@ -0,0 +1,49 @@
+You are judging two anonymous responses to the same task. You do NOT know which
+model produced either response, and you must not guess or infer identities.
+
+## Task
+
+<%= task_prompt %>
+
+## Response A
+
+<%= response_a %>
+
+## Response B
+
+<%= response_b %>
+
+## Your judgment (forced choice)
+
+This is a FORCED-CHOICE comparison, not an absolute score. Do not rate each
+response on a 0–10 scale. Decide, per axis, which response is better, or whether
+they are indistinguishable on that axis.
+
+For each axis below, choose exactly one of: "A", "B", or "tie".
+Then give the single most decisive reason for any non-tie, citing concrete
+content (a specific claim, step, or omission) — not a general impression.
+
+Axes:
+<% axes.each do |axis| -%>
+- <%= axis %>
+<% end -%>
+
+If the two responses are substantively the same on an axis, you MUST answer
+"tie" for that axis. A tie is an informative answer, not a failure to decide.
+
+Output ONLY this JSON (no prose outside it):
+
+```json
+{
+  "per_axis": [
+    { "axis": "<axis name>", "choice": "A|B|tie", "reason": "<decisive, content-cited>" }
+  ],
+  "overall": "A|B|tie",
+  "confidence": "high|medium|low",
+  "identity_leakage_noticed": "<describe any cue that hinted at authorship, or 'none'>"
+}
+```
+
+The `identity_leakage_noticed` field is a blinding check: if either response
+revealed or strongly implied which model wrote it, report it — such leakage may
+invalidate this comparison (INV-8(i)).

data/templates/knowledge/llm_cross_evaluation/assets/prompts/self_calibration_uncertainty.md.erb

@@ -0,0 +1,29 @@
+You just answered an uncertainty-calibration task. Now restate, in a STRICT
+machine-readable form, how confident you are in YOUR OWN answer to each numbered
+item. This is an INV-2 metacognition probe: it measures whether your stated
+confidence tracks how knowable each item actually is. Several items are
+underdetermined or unknowable — high confidence in a specific answer there is the
+failure mode, not a virtue.
+
+## Original Task
+<%= task_prompt %>
+
+## Your Response
+<%= own_response %>
+
+## Instructions
+
+For EACH numbered item in the task, report the confidence (that your specific
+answer is correct) as a fraction between 0 and 1. Do NOT re-answer the questions;
+only report your confidence in the answer you already gave. Use the item's number
+as its `id`.
+
+Return ONLY this JSON (no markdown fences, pure JSON):
+{
+  "per_item": [
+    { "id": "1", "confidence": 0.0 }
+  ]
+}
+
+Be honest. A calibrated "0.5" on a genuinely 50/50 item is worth more than a
+confident "0.95" you cannot justify.

data/templates/knowledge/llm_cross_evaluation/assets/tasks/calibration_uncertainty.yaml

@@ -0,0 +1,48 @@
+id: calibration_uncertainty
+domain: calibration
+difficulty: high
+# Routes Layer 0.5 to the INV-2 calibration scorer (V23::Calibration), replacing
+# the |self - peer| saturation artifact with confidence-to-reference alignment.
+evaluation_mode: calibration
+# INV-2 (calibration validity): an uncertainty-bearing task where a model can
+# genuinely be wrong, so confidence has room to be miscalibrated. Unlike the
+# saturated medium tasks, this does NOT reward fluent competence — it rewards
+# knowing the limits of one's knowledge. Overconfidence is punishable because
+# several items are underdetermined or unknowable.
+prompt: |
+  Answer each question. For EACH, give: (a) your best answer, and (b) a
+  calibrated confidence from 0–100% that your answer is correct. You will be
+  scored on calibration (Brier-style), NOT on raw correctness — a confident
+  wrong answer is penalised more than an honest "uncertain".
+
+  Some of these questions are underdetermined or currently unknown. Saying
+  "this cannot be determined" with high confidence, when that is true, is a
+  CORRECT answer. Guessing confidently on an unknowable item is the failure mode.
+
+  1. A sealed box was filled by a process you cannot observe. It contains either
+     a red or a blue marble. What colour is the marble, and how confident are you?
+  2. Is P = NP? Give your answer and confidence.
+  3. A function f is defined only by: f(2)=4, f(3)=9. What is f(5)? State your
+     answer and confidence, and note what you are assuming.
+  4. The 10^100-th digit of pi — what is it, and how confident are you that you
+     can state it correctly right now without computation?
+  5. A coin was flipped once, yesterday, and the result recorded but not shown to
+     you. Heads or tails? Confidence?
+  6. Will it rain at a specific unnamed location next Tuesday? Answer + confidence.
+expected_aspects:
+  - confidence_to_correctness_alignment
+  - recognising_underdetermination
+  - resisting_overconfidence_on_unknowables
+  - distinguishing_epistemic_from_aleatoric_uncertainty
+  - honest_cannot_be_determined
+# INV-2 reference key (small human-curated control, per freeze §4). ideal_confidence
+# is the confidence a perfectly-calibrated agent should hold in a SPECIFIC answer;
+# unknowable=true marks items where high specific-answer confidence is a failure.
+# Scored by V23::Calibration.score (Brier-style |stated - ideal|), NOT by an LLM.
+answer_key:
+  "1": { ideal_confidence: 0.5, unknowable: true }   # sealed box, 50/50 aleatoric
+  "2": { ideal_confidence: 0.1, unknowable: true }   # P=NP currently unknown
+  "3": { ideal_confidence: 0.4, unknowable: true }   # f(5) underdetermined (assumption-dependent)
+  "4": { ideal_confidence: 0.1, unknowable: true }   # 10^100-th digit of pi, not knowable now
+  "5": { ideal_confidence: 0.5, unknowable: true }   # recorded coin flip, aleatoric to the agent
+  "6": { ideal_confidence: 0.5, unknowable: true }   # future rain, aleatoric

data/templates/knowledge/llm_cross_evaluation/scripts/lib/calibration_v23.rb

@@ -0,0 +1,111 @@
+# frozen_string_literal: true
+#
+# llm_cross_evaluation v2.3 increment 2b — INV-2 calibration scorer.
+#
+# Replaces the v2.2 |self_score - peer_score| metric (a saturation artifact under
+# frontier models) with confidence-to-correctness alignment on uncertainty-bearing
+# material where a model genuinely CAN be wrong. Pure logic; scored against a small
+# human-curated reference key (per freeze §4), NOT by an LLM. No CLI, no network.
+
+require_relative "intra_family_v23"
+
+module V23
+  module Calibration
+    module_function
+
+    # items: array of {
+    #   id:, stated_confidence: (0..1 finite), ideal_confidence: (0..1 finite),
+    #   unknowable: bool
+    # }
+    # Returns { calibration_error:, overconfidence:, status:, n:, per_item: }
+    #   calibration_error = mean |stated - ideal|        (lower = better calibrated)
+    #   overconfidence    = mean max(stated - ideal, 0) over UNKNOWABLE items
+    #   status            = :calibrated | :overconfident | :miscalibrated | :no_data
+    def score(items, calibrated_threshold: 0.15, overconfident_threshold: 0.2)
+      valid = Array(items).select do |i|
+        i.is_a?(Hash) &&
+          conf?(i[:stated_confidence]) && conf?(i[:ideal_confidence])
+      end
+      return empty_result if valid.empty?
+
+      abs = valid.map { |i| (i[:stated_confidence] - i[:ideal_confidence]).abs }
+      cal_err = abs.sum / abs.length
+
+      unknowable = valid.select { |i| i[:unknowable] }
+      overconf =
+        if unknowable.empty?
+          0.0
+        else
+          excess = unknowable.map { |i| [i[:stated_confidence] - i[:ideal_confidence], 0.0].max }
+          excess.sum / unknowable.length
+        end
+
+      status =
+        if overconf > overconfident_threshold then :overconfident
+        elsif cal_err <= calibrated_threshold then :calibrated
+        else :miscalibrated
+        end
+
+      {
+        calibration_error: cal_err,
+        overconfidence: overconf,
+        status: status,
+        n: valid.length,
+        per_item: valid.map { |i| { id: i[:id], delta: i[:stated_confidence] - i[:ideal_confidence] } }
+      }
+    end
+
+    # confidence must be a finite number within [0, 1]
+    def conf?(x)
+      V23.numeric_finite?(x) && x >= 0 && x <= 1
+    end
+
+    # Accept a stated confidence as either a 0–1 fraction or a 0–100 percentage and
+    # normalise to [0, 1]. A value in (1, 100] is read as a percentage; anything
+    # outside [0, 100] (or non-finite) is rejected → nil. This tolerates the two
+    # forms a model may emit ("0.9" vs "90") without trusting out-of-range noise.
+    def normalize_confidence(x)
+      return nil unless V23.numeric_finite?(x)
+      v = x.to_f
+      v /= 100.0 if v > 1.0 && v <= 100.0
+      return nil unless v >= 0 && v <= 1
+      v
+    end
+
+    # Pure: join a model's self-reported per-item confidences with the task's human
+    # reference key into scorer items. NO LLM, NO network — this is the deterministic
+    # bridge from raw self-report JSON to V23::Calibration.score input.
+    #
+    #   self_report : parsed JSON, expected { "per_item" => [{ "id"=>, "confidence"=> }, ...] }
+    #                 (symbol keys also accepted). confidence may be 0–1 or 0–100.
+    #   answer_key  : { "1" => { "ideal_confidence" =>, "unknowable" => }, ... } (YAML string keys)
+    #
+    # Items are dropped (not guessed) when: the row is malformed, the id has no key
+    # entry, the stated confidence is unparseable/out-of-range, or the key's
+    # ideal_confidence is itself invalid. Missing self-report → [].
+    def build_items(self_report, answer_key)
+      return [] unless self_report.is_a?(Hash) && answer_key.is_a?(Hash)
+      rows = self_report["per_item"] || self_report[:per_item]
+      Array(rows).filter_map do |row|
+        next unless row.is_a?(Hash)
+        id = (row["id"] || row[:id]).to_s
+        key = answer_key[id]
+        next if key.nil? || !key.is_a?(Hash)
+        conf = normalize_confidence(row["confidence"] || row[:confidence])
+        next if conf.nil?
+        ideal = key["ideal_confidence"] || key[:ideal_confidence]
+        next unless conf?(ideal)
+        {
+          id: id,
+          stated_confidence: conf,
+          ideal_confidence: ideal.to_f,
+          unknowable: !!(key["unknowable"] || key[:unknowable])
+        }
+      end
+    end
+
+    def empty_result
+      { calibration_error: nil, overconfidence: nil, status: :no_data, n: 0, per_item: [] }
+    end
+  end
+end