stilltrue 0.7.1 → 0.8.0

package/CHANGELOG.md

@@ -1,5 +1,46 @@
 # Changelog
 
+## 0.8.0 — 2026-10-28
+
+**A list is not a denial.**
+
+Asked whether `apple.com/apple-card` supports "Apple Card works with Windows
+PCs", the judge ruled `not_supported`, reasoning:
+
+> by listing only Apple devices as having the technology, it implies
+> exclusion of other platforms like Windows.
+
+The page never mentions Windows, so the right answer is `cant_tell`. The
+contradiction guard could not catch it, because the quoted sentence was real.
+It was about Apple Pay, though, not Apple Card. Two lines were added to the
+shipped judge prompt:
+
+- A source that names the cases something applies to has not ruled out the
+  ones it leaves unnamed, unless it says so in words ("only", "not available
+  on").
+- A contradicting sentence must be about the same subject as the claim.
+
+Measured on qwen3.6-35b-a3b, the weakest engine behind the demo:
+
+| Set | 0.7.1 | 0.8.0 |
+|---|---|---|
+| Apple: Windows PCs, 3 runs | 0/3 | 3/3 |
+| Apple: 10% Daily Cash, 3 runs | 3/3 | 3/3 |
+| Controls | 4/4 | 4/4 |
+| 21-case comparison set | 21/21 | 21/21 |
+| False supported, all sets | 0 | 0 |
+
+The box is not deterministic at temperature 0. On a second sample 0.8.0
+scored 20/21 on the comparison set, missing "Visitors must create an
+account". Rerun five times on each prompt, that case scored 4/5 on 0.7.1
+and 5/5 on 0.8.0, so the miss is run-to-run variance rather than a
+regression.
+
+Known gap: two synthetic list cases still come back `not_supported` on that
+model, which reads "issued at Duluth, Norcross and Snellville" as "issued
+only at" them. Generating reasoning before the verdict fixed those two but
+broke the real Apple case 3/3, so that change did not ship.
+
 ## 0.7.1 — 2026-09-13
 
 Docs restructure, no behavior changes except one message:

package/dist/support.d.ts

@@ -101,6 +101,19 @@ export interface SupportStageOptions {
  * the obvious over-correction is ruling not_supported on a page that simply
  * lists several values for different cases.
  *
+ * The list and same-subject lines were added 2026-10-28, the mirror image of
+ * the 10% failure on the same page. Asked whether apple.com/apple-card supports
+ * "Apple Card works with Windows PCs", the judge ruled not_supported, quoting
+ * "Apple Pay ... built into iPhone, Apple Watch, iPad, and Mac" and reasoning
+ * that listing only Apple devices "implies exclusion". That is an argument from
+ * silence, and the quote was about Apple Pay rather than Apple Card. The guard
+ * could not catch it, because the quote was real. On qwen3.6-35b-a3b the real
+ * case went from 0/3 to 3/3 correct, with no change on the 21-case comparison
+ * set, the controls, or the 10% case. Two synthetic list cases ("issued at the
+ * Duluth, Norcross and Snellville branches") still fail on that model, which
+ * reads a plain list as "only" these. Putting reasoning ahead of the verdict
+ * fixed those and broke the real Apple case 3/3, so it did not ship.
+ *
  * The line about qualification is load-bearing rather than decorative.
  * Measured 2026-09-04 without it: adding two lines to a source that never
  * mention the claim flipped a verdict from supported to not_supported — three
@@ -108,7 +121,7 @@ export interface SupportStageOptions {
  * model stayed stable. A package promising engine-agnostic verification has to
  * survive the weaker engine, so the instruction ships by default.
  */
-export declare const SUPPORT_SYSTEM = "You are a strict fact-check judge. Decide whether the SOURCE supports the CLAIM using ONLY the source text \u2014 no outside knowledge, no charitable guessing.\nVerdicts:\n- \"supported\": the source affirmatively supports the claim.\n- \"not_supported\": the source contradicts the claim or asserts something incompatible with it. This INCLUDES the source stating a DIFFERENT VALUE for the same thing the claim quantifies \u2014 a rate, price, date, count, duration or name. If the claim says 10% and the source says the rate is 3%, the source has contradicted it; it does not have to mention 10% to do so.\n- \"cant_tell\": the source never addresses the subject of the claim at all, or addresses it only partially. A source that states its own value for the very thing the claim asserts HAS addressed it \u2014 that is not_supported, not cant_tell. Silence is NOT support.\nA qualification is not a contradiction. If the source substantially supports the claim but adds an exception, a carve-out or a condition, it has NOT said otherwise \u2014 do not rule not_supported. Reserve not_supported for a source that asserts something the claim cannot be true alongside.\nTo rule \"not_supported\" you MUST fill \"contradiction\" with the exact sentence from the source that says otherwise, copied verbatim. Where the conflict is a differing value, that sentence is the one stating the SOURCE's own value. If no such sentence exists, you may not use that verdict.\nCompare like with like: a different value only contradicts when it describes the same thing under the same conditions. A source listing several values for different cases has not contradicted a claim that matches one of them.\nReturn STRICT JSON only:\n{\"verdict\":\"supported\"|\"not_supported\"|\"cant_tell\",\"reasoning\":\"<one or two plain sentences>\",\"evidence\":\"<short verbatim quote from the source that decided it, or null>\",\"contradiction\":\"<verbatim sentence that contradicts the claim, or null>\"}";
+export declare const SUPPORT_SYSTEM = "You are a strict fact-check judge. Decide whether the SOURCE supports the CLAIM using ONLY the source text \u2014 no outside knowledge, no charitable guessing.\nVerdicts:\n- \"supported\": the source affirmatively supports the claim.\n- \"not_supported\": the source contradicts the claim or asserts something incompatible with it. This INCLUDES the source stating a DIFFERENT VALUE for the same thing the claim quantifies \u2014 a rate, price, date, count, duration or name. If the claim says 10% and the source says the rate is 3%, the source has contradicted it; it does not have to mention 10% to do so.\n- \"cant_tell\": the source never addresses the subject of the claim at all, or addresses it only partially. A source that states its own value for the very thing the claim asserts HAS addressed it \u2014 that is not_supported, not cant_tell. Silence is NOT support.\nA qualification is not a contradiction. If the source substantially supports the claim but adds an exception, a carve-out or a condition, it has NOT said otherwise \u2014 do not rule not_supported. Reserve not_supported for a source that asserts something the claim cannot be true alongside.\nTo rule \"not_supported\" you MUST fill \"contradiction\" with the exact sentence from the source that says otherwise, copied verbatim. Where the conflict is a differing value, that sentence is the one stating the SOURCE's own value. If no such sentence exists, you may not use that verdict.\nCompare like with like: a different value only contradicts when it describes the same thing under the same conditions. A source listing several values for different cases has not contradicted a claim that matches one of them.\nA list is not a denial. A source that names the devices, places, people or cases something applies to has NOT ruled out the ones it leaves unnamed, unless it says so in words \u2014 \"only\", \"exclusively\", \"not available on\", \"does not work with\". Absence from a list is silence, and silence is cant_tell.\nThe contradicting sentence must be about the same subject as the claim. A sentence about a different product, service or program does not contradict it.\nReturn STRICT JSON only:\n{\"verdict\":\"supported\"|\"not_supported\"|\"cant_tell\",\"reasoning\":\"<one or two plain sentences>\",\"evidence\":\"<short verbatim quote from the source that decided it, or null>\",\"contradiction\":\"<verbatim sentence that contradicts the claim, or null>\"}";
 /** JSON Schema for the judgement, for providers that constrain decoding. */
 export declare const SUPPORT_SCHEMA: {
     readonly type: "object";

package/dist/support.js

@@ -32,6 +32,19 @@
  * the obvious over-correction is ruling not_supported on a page that simply
  * lists several values for different cases.
  *
+ * The list and same-subject lines were added 2026-10-28, the mirror image of
+ * the 10% failure on the same page. Asked whether apple.com/apple-card supports
+ * "Apple Card works with Windows PCs", the judge ruled not_supported, quoting
+ * "Apple Pay ... built into iPhone, Apple Watch, iPad, and Mac" and reasoning
+ * that listing only Apple devices "implies exclusion". That is an argument from
+ * silence, and the quote was about Apple Pay rather than Apple Card. The guard
+ * could not catch it, because the quote was real. On qwen3.6-35b-a3b the real
+ * case went from 0/3 to 3/3 correct, with no change on the 21-case comparison
+ * set, the controls, or the 10% case. Two synthetic list cases ("issued at the
+ * Duluth, Norcross and Snellville branches") still fail on that model, which
+ * reads a plain list as "only" these. Putting reasoning ahead of the verdict
+ * fixed those and broke the real Apple case 3/3, so it did not ship.
+ *
  * The line about qualification is load-bearing rather than decorative.
  * Measured 2026-09-04 without it: adding two lines to a source that never
  * mention the claim flipped a verdict from supported to not_supported — three
@@ -47,6 +60,8 @@ Verdicts:
 A qualification is not a contradiction. If the source substantially supports the claim but adds an exception, a carve-out or a condition, it has NOT said otherwise — do not rule not_supported. Reserve not_supported for a source that asserts something the claim cannot be true alongside.
 To rule "not_supported" you MUST fill "contradiction" with the exact sentence from the source that says otherwise, copied verbatim. Where the conflict is a differing value, that sentence is the one stating the SOURCE's own value. If no such sentence exists, you may not use that verdict.
 Compare like with like: a different value only contradicts when it describes the same thing under the same conditions. A source listing several values for different cases has not contradicted a claim that matches one of them.
+A list is not a denial. A source that names the devices, places, people or cases something applies to has NOT ruled out the ones it leaves unnamed, unless it says so in words — "only", "exclusively", "not available on", "does not work with". Absence from a list is silence, and silence is cant_tell.
+The contradicting sentence must be about the same subject as the claim. A sentence about a different product, service or program does not contradict it.
 Return STRICT JSON only:
 {"verdict":"supported"|"not_supported"|"cant_tell","reasoning":"<one or two plain sentences>","evidence":"<short verbatim quote from the source that decided it, or null>","contradiction":"<verbatim sentence that contradicts the claim, or null>"}`;
 /** JSON Schema for the judgement, for providers that constrain decoding. */

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "stilltrue",
-  "version": "0.7.1",
+  "version": "0.8.0",
   "description": "Your tests check your code. stilltrue checks your facts. CI checks for AI-in-the-loop apps: drift (curated facts vs live sources), golden (regression evals), verify (judge answers against sources).",
   "keywords": [
     "drift",
@@ -31,7 +31,7 @@
   },
   "dependencies": {
     "jiti": "^2.7.0",
-    "zod": "^3.23.0"
+    "zod": "^3.23.0 || ^4.0.0"
   },
   "devDependencies": {
     "@types/node": "^22.0.0",