stilltrue 0.6.0 → 0.7.0

package/CHANGELOG.md

@@ -0,0 +1,57 @@
+# Changelog
+
+## 0.7.0
+
+**A different value now counts as a contradiction.**
+
+The judge used to answer `cant_tell` when a source stated its own figure for
+the thing a claim quantified. Asked whether `apple.com/apple-card` supports
+"Apple Card gives 10% Daily Cash back on every purchase", it reasoned:
+
+> The source states that Apple Card offers 2% Daily Cash for most purchases,
+> 3% at Apple and select merchants, and 1% on the titanium card, but it does
+> not mention a 10% Daily Cash rate.
+
+It had read every rate on the page and still treated the absence of the
+claimed number as the claim being unaddressed. Nothing on that page denies
+10% in words, so the old wording — `cant_tell: the source does not address
+the claim` — left it nowhere else to land.
+
+Explicit denial is the rare shape of a false claim. The common one is a wrong
+number, and a checker that goes quiet on wrong numbers is quiet exactly when
+it is most useful. It also collapsed the distinction the verdict set exists
+for: a claim the source contradicts and a claim the source never mentions
+came back identical.
+
+- `not_supported` now explicitly covers a source stating a DIFFERENT VALUE for
+  what the claim quantifies — a rate, price, date, count, duration or name.
+- `cant_tell` explicitly excludes that case.
+- The contradiction-quote rule gains the matching line: where the conflict is
+  a differing value, the sentence to cite is the one stating the source's own
+  value.
+- "Compare like with like" ships in the same change, because the obvious
+  over-correction is ruling `not_supported` against a page that simply lists
+  several values for different cases.
+
+**Minor, not patch.** The public API is unchanged, but verdicts change for the
+same inputs — and for a library whose product IS the verdict, that should not
+reach a consumer pinned to `^0.6.0` silently.
+
+Verified against a local gpt-oss-20b (the weaker of two available engines),
+five claims against one live page:
+
+| claim | before | after |
+|---|---|---|
+| 3% Daily Cash at Apple | supported | supported |
+| **10% on every purchase** | **cant_tell** | **not_supported** |
+| works with Windows PCs | cant_tell | cant_tell |
+| 2% with Apple Pay | — | supported |
+| no annual fee | — | supported |
+
+The last two are over-correction checks. The contradiction guard did not
+downgrade the 10% verdict, so it arrived with a verbatim quote that passed the
+contiguous-run check — refused with evidence, not merely refused.
+
+## 0.6.0 and earlier
+
+Not recorded here; see the git history.

package/dist/support.d.ts

@@ -87,6 +87,20 @@ export interface SupportStageOptions {
 /**
  * The shipped judge prompt.
  *
+ * The "different value" clause was added 2026-09-11 after a measured failure:
+ * asked whether apple.com/apple-card supports "Apple Card gives 10% Daily Cash",
+ * the judge answered cant_tell, reasoning "the source states 2%... 3%... and 1%,
+ * but it does not mention a 10% Daily Cash rate". It had read every rate on the
+ * page and still treated the absence of the claimed number as the claim being
+ * unaddressed. Nothing on that page denies 10% in words, so the original
+ * wording gave it nowhere else to land.
+ *
+ * That is the common shape of a false claim in the wild — a wrong number, not
+ * an explicit denial — so a checker that goes quiet on it is quiet exactly when
+ * it is most needed. The compare-like-with-like line ships alongside because
+ * the obvious over-correction is ruling not_supported on a page that simply
+ * lists several values for different cases.
+ *
  * The line about qualification is load-bearing rather than decorative.
  * Measured 2026-09-04 without it: adding two lines to a source that never
  * mention the claim flipped a verdict from supported to not_supported — three
@@ -94,7 +108,7 @@ export interface SupportStageOptions {
  * model stayed stable. A package promising engine-agnostic verification has to
  * survive the weaker engine, so the instruction ships by default.
  */
-export declare const SUPPORT_SYSTEM = "You are a strict fact-check judge. Decide whether the SOURCE supports the CLAIM using ONLY the source text \u2014 no outside knowledge, no charitable guessing.\nVerdicts:\n- \"supported\": the source affirmatively supports the claim.\n- \"not_supported\": the source contradicts the claim or asserts something incompatible with it.\n- \"cant_tell\": the source does not address the claim, or addresses it only partially. Silence is NOT support.\nA qualification is not a contradiction. If the source substantially supports the claim but adds an exception, a carve-out or a condition, it has NOT said otherwise \u2014 do not rule not_supported. Reserve not_supported for a source that asserts something the claim cannot be true alongside.\nTo rule \"not_supported\" you MUST fill \"contradiction\" with the exact sentence from the source that says otherwise, copied verbatim. If no such sentence exists, you may not use that verdict.\nReturn STRICT JSON only:\n{\"verdict\":\"supported\"|\"not_supported\"|\"cant_tell\",\"reasoning\":\"<one or two plain sentences>\",\"evidence\":\"<short verbatim quote from the source that decided it, or null>\",\"contradiction\":\"<verbatim sentence that contradicts the claim, or null>\"}";
+export declare const SUPPORT_SYSTEM = "You are a strict fact-check judge. Decide whether the SOURCE supports the CLAIM using ONLY the source text \u2014 no outside knowledge, no charitable guessing.\nVerdicts:\n- \"supported\": the source affirmatively supports the claim.\n- \"not_supported\": the source contradicts the claim or asserts something incompatible with it. This INCLUDES the source stating a DIFFERENT VALUE for the same thing the claim quantifies \u2014 a rate, price, date, count, duration or name. If the claim says 10% and the source says the rate is 3%, the source has contradicted it; it does not have to mention 10% to do so.\n- \"cant_tell\": the source never addresses the subject of the claim at all, or addresses it only partially. A source that states its own value for the very thing the claim asserts HAS addressed it \u2014 that is not_supported, not cant_tell. Silence is NOT support.\nA qualification is not a contradiction. If the source substantially supports the claim but adds an exception, a carve-out or a condition, it has NOT said otherwise \u2014 do not rule not_supported. Reserve not_supported for a source that asserts something the claim cannot be true alongside.\nTo rule \"not_supported\" you MUST fill \"contradiction\" with the exact sentence from the source that says otherwise, copied verbatim. Where the conflict is a differing value, that sentence is the one stating the SOURCE's own value. If no such sentence exists, you may not use that verdict.\nCompare like with like: a different value only contradicts when it describes the same thing under the same conditions. A source listing several values for different cases has not contradicted a claim that matches one of them.\nReturn STRICT JSON only:\n{\"verdict\":\"supported\"|\"not_supported\"|\"cant_tell\",\"reasoning\":\"<one or two plain sentences>\",\"evidence\":\"<short verbatim quote from the source that decided it, or null>\",\"contradiction\":\"<verbatim sentence that contradicts the claim, or null>\"}";
 /** JSON Schema for the judgement, for providers that constrain decoding. */
 export declare const SUPPORT_SCHEMA: {
     readonly type: "object";

package/dist/support.js

@@ -18,6 +18,20 @@
 /**
  * The shipped judge prompt.
  *
+ * The "different value" clause was added 2026-09-11 after a measured failure:
+ * asked whether apple.com/apple-card supports "Apple Card gives 10% Daily Cash",
+ * the judge answered cant_tell, reasoning "the source states 2%... 3%... and 1%,
+ * but it does not mention a 10% Daily Cash rate". It had read every rate on the
+ * page and still treated the absence of the claimed number as the claim being
+ * unaddressed. Nothing on that page denies 10% in words, so the original
+ * wording gave it nowhere else to land.
+ *
+ * That is the common shape of a false claim in the wild — a wrong number, not
+ * an explicit denial — so a checker that goes quiet on it is quiet exactly when
+ * it is most needed. The compare-like-with-like line ships alongside because
+ * the obvious over-correction is ruling not_supported on a page that simply
+ * lists several values for different cases.
+ *
  * The line about qualification is load-bearing rather than decorative.
  * Measured 2026-09-04 without it: adding two lines to a source that never
  * mention the claim flipped a verdict from supported to not_supported — three
@@ -28,10 +42,11 @@
 export const SUPPORT_SYSTEM = `You are a strict fact-check judge. Decide whether the SOURCE supports the CLAIM using ONLY the source text — no outside knowledge, no charitable guessing.
 Verdicts:
 - "supported": the source affirmatively supports the claim.
-- "not_supported": the source contradicts the claim or asserts something incompatible with it.
-- "cant_tell": the source does not address the claim, or addresses it only partially. Silence is NOT support.
+- "not_supported": the source contradicts the claim or asserts something incompatible with it. This INCLUDES the source stating a DIFFERENT VALUE for the same thing the claim quantifies — a rate, price, date, count, duration or name. If the claim says 10% and the source says the rate is 3%, the source has contradicted it; it does not have to mention 10% to do so.
+- "cant_tell": the source never addresses the subject of the claim at all, or addresses it only partially. A source that states its own value for the very thing the claim asserts HAS addressed it — that is not_supported, not cant_tell. Silence is NOT support.
 A qualification is not a contradiction. If the source substantially supports the claim but adds an exception, a carve-out or a condition, it has NOT said otherwise — do not rule not_supported. Reserve not_supported for a source that asserts something the claim cannot be true alongside.
-To rule "not_supported" you MUST fill "contradiction" with the exact sentence from the source that says otherwise, copied verbatim. If no such sentence exists, you may not use that verdict.
+To rule "not_supported" you MUST fill "contradiction" with the exact sentence from the source that says otherwise, copied verbatim. Where the conflict is a differing value, that sentence is the one stating the SOURCE's own value. If no such sentence exists, you may not use that verdict.
+Compare like with like: a different value only contradicts when it describes the same thing under the same conditions. A source listing several values for different cases has not contradicted a claim that matches one of them.
 Return STRICT JSON only:
 {"verdict":"supported"|"not_supported"|"cant_tell","reasoning":"<one or two plain sentences>","evidence":"<short verbatim quote from the source that decided it, or null>","contradiction":"<verbatim sentence that contradicts the claim, or null>"}`;
 /** JSON Schema for the judgement, for providers that constrain decoding. */

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "stilltrue",
-  "version": "0.6.0",
+  "version": "0.7.0",
   "description": "Your tests check your code. stilltrue checks your facts. CI checks for AI-in-the-loop apps: drift (curated facts vs live sources), golden (regression evals), verify (judge answers against sources).",
   "keywords": [
     "drift",
@@ -20,7 +20,8 @@
     "stilltrue": "dist/cli.js"
   },
   "files": [
-    "dist"
+    "dist",
+    "CHANGELOG.md"
   ],
   "scripts": {
     "build": "tsc",