stilltrue 0.7.0 → 0.8.0

package/CHANGELOG.md

@@ -1,5 +1,61 @@
 # Changelog
 
+## 0.8.0 — 2026-10-28
+
+**A list is not a denial.**
+
+Asked whether `apple.com/apple-card` supports "Apple Card works with Windows
+PCs", the judge ruled `not_supported`, reasoning:
+
+> by listing only Apple devices as having the technology, it implies
+> exclusion of other platforms like Windows.
+
+The page never mentions Windows, so the right answer is `cant_tell`. The
+contradiction guard could not catch it, because the quoted sentence was real.
+It was about Apple Pay, though, not Apple Card. Two lines were added to the
+shipped judge prompt:
+
+- A source that names the cases something applies to has not ruled out the
+  ones it leaves unnamed, unless it says so in words ("only", "not available
+  on").
+- A contradicting sentence must be about the same subject as the claim.
+
+Measured on qwen3.6-35b-a3b, the weakest engine behind the demo:
+
+| Set | 0.7.1 | 0.8.0 |
+|---|---|---|
+| Apple: Windows PCs, 3 runs | 0/3 | 3/3 |
+| Apple: 10% Daily Cash, 3 runs | 3/3 | 3/3 |
+| Controls | 4/4 | 4/4 |
+| 21-case comparison set | 21/21 | 21/21 |
+| False supported, all sets | 0 | 0 |
+
+The box is not deterministic at temperature 0. On a second sample 0.8.0
+scored 20/21 on the comparison set, missing "Visitors must create an
+account". Rerun five times on each prompt, that case scored 4/5 on 0.7.1
+and 5/5 on 0.8.0, so the miss is run-to-run variance rather than a
+regression.
+
+Known gap: two synthetic list cases still come back `not_supported` on that
+model, which reads "issued at Duluth, Norcross and Snellville" as "issued
+only at" them. Generating reasoning before the verdict fixed those two but
+broke the real Apple case 3/3, so that change did not ship.
+
+## 0.7.1 — 2026-09-13
+
+Docs restructure, no behavior changes except one message:
+
+- README cut from 408 lines to ~190. A fresh-eyes pass (install → first
+  green check in three minutes) showed setup was never the problem — the
+  document performed difficulty the tool doesn't have. The drift
+  quickstart, which tested perfect, is untouched.
+- New `docs/CONFIG.md`: the reference shelf (config fields, compare modes,
+  helpers, CLI flags), moved verbatim out of the tutorial path.
+- `docs/VERIFY.md` gained the `supportStage` section from the README, so
+  the ready-made stage lives beside the judge walkthrough it builds on.
+- `stilltrue init` now prints the full MARKERS.md URL — npm users have no
+  repo checkout for a relative path to resolve against.
+
 ## 0.7.0
 
 **A different value now counts as a contradiction.**

package/README.md

@@ -71,7 +71,7 @@ Office", and "12 locations" all still appear on it.* Swap in your own URL
 and your own facts — that's a working check, no programming required.
 
 Every other field, compare mode, and helper is in
-[Configuration](#configuration) below. Choosing *what* to assert is the
+[the configuration reference](docs/CONFIG.md). Choosing *what* to assert is the
 judgment call — surnames beat full names, `"143 schools"` beats a bare
 `"143"`. The short guide: [docs/MARKERS.md](docs/MARKERS.md).
 
@@ -115,172 +115,6 @@ You'll see one line per check:
   This only *warns*. An unreachable site doesn't mean your facts are wrong,
   and an alarm that cries wolf gets ignored — so outages never fail the build.
 
-## Configuration
-
-stilltrue loads a config file from the current directory. The first of
-these that exists wins:
-
-`stilltrue.config.ts` · `.mts` · `.js` · `.mjs` · `.cjs`
-
-(`npx stilltrue init` writes `.mjs`.) Point at a file anywhere with
-`stilltrue drift --config path/to/file`.
-
-Wrap the export with `defineStilltrue` so TypeScript can check it:
-
-```js
-import { defineStilltrue, corpus, json, surname } from 'stilltrue';
-
-export default defineStilltrue({
-  drift: [ /* one entry per fact-set */ ],
-  verify: { /* optional named pipelines — library use, see below */ },
-});
-```
-
-### Drift checks
-
-Each `drift` entry is one curated fact-set vs its live source.
-
-| Field | Required | Default | What it is |
-|---|---|---|---|
-| `name` | yes | — | Short id. Shows up in the terminal, `--only`, and the report. |
-| `source` | yes | — | `async (ctx) => actual`. Fetch/parse the live authority. **Throw** if the source is unreachable — that is outcome `error`, never `rot`. |
-| `expect` | yes | — | The curated facts. A value, or `async (ctx) => expected` (e.g. `json(...)`). A throw here is also `error` (broken config, not rot). |
-| `compare` | no | `'deep-equal'` | How to compare `expect` against what `source` returned. |
-
-`source` and function-form `expect` receive `ctx`:
-
-| | |
-|---|---|
-| `ctx.warn(message)` | Record a non-fatal note (one of several pages down, odd markup). Shows as a warning; does not fail the run. |
-| `ctx.configDir` | Absolute directory of the config file. Data paths resolve from here. |
-
-### `compare`
-
-**`'contains-all'`** — `expect` is a list of marker strings; `source` must
-return a text corpus (what `corpus()` returns). Every marker must appear
-somewhere in that text, case-insensitively. Extra text on the page is
-ignored. This is the webpage-watch check.
-
-**`'deep-equal'`** — structural equality. Extra keys, missing keys, and
-value mismatches all rot, with pathed messages
-(`$.members[2].name: expected "Smith", source has "Jones"`). Use this when
-`source()` already returns structured data that should match `expect`
-exactly.
-
-**A function** — `(expected, actual) => string[]`. Return `[]` to pass;
-each string is one mismatch (outcome `rot`). Throw if the shapes are
-unusable (outcome `error`). Use this for numeric tolerance, order-insensitive
-lists, or any comparison that isn't "markers in a page" or "these two
-objects are identical."
-
-```js
-compare: (expected, actual) =>
-  expected.rate === actual.rate
-    ? []
-    : [`tax rate ${expected.rate} → ${actual.rate}`],
-```
-
-Surnames are a *marker choice*, not a compare mode: derive them from your
-data with `surname()` and use `'contains-all'`.
-
-### Helpers
-
-These are the built-in `source` / `expect` builders. You can always write
-your own functions instead.
-
-**`corpus(urls, options?)`** — fetch pages, strip HTML to text,
-concatenate. Individual unreachable pages call `ctx.warn`; if *every* page
-fails, it throws (`error`). Pair with `'contains-all'`.
-
-| Option | Default | What it is |
-|---|---|---|
-| `timeoutMs` | `20000` | Per-page fetch timeout, milliseconds. |
-| `userAgent` | `stilltrue drift check (github.com/sg-bfoster/stilltrue)` | Sent on each request so operators can identify the check. |
-
-```js
-source: corpus(
-  ['https://example.org/about', 'https://example.org/board'],
-  { timeoutMs: 15_000, userAgent: 'my-app drift check' },
-),
-```
-
-**`json(path, derive?)`** — read a JSON file relative to the config file.
-With no `derive`, the parsed value *is* the expectation (typical with
-`'deep-equal'`). With `derive`, you map that value into whatever `compare`
-needs — usually a marker list:
-
-```js
-expect: json('./data/board.json', (data) =>
-  data.members.map((m) => surname(m.name)),
-),
-```
-
-**`surname(fullName)`** — last token of a name, with leading honorifics
-(`Dr.`, `Mr.`, `Mrs.`, `Ms.`) and trailing generational suffixes (`Jr.`,
-`Sr.`, `II`–`V`) stripped. `"Dr. Jane Smith III"` → `"Smith"`.
-
-**`htmlToText(html)`** — the same tag-stripping `corpus()` uses. For a
-custom `source` that already has HTML in hand.
-
-### Custom `source`
-
-A throw is an outage; a return value is data to compare. Warn for partial
-trouble:
-
-```js
-source: async (ctx) => {
-  const res = await fetch('https://example.org/api/rates');
-  if (!res.ok) throw new Error(`HTTP ${res.status}`);
-  const data = await res.json();
-  if (!data.updatedAt) ctx.warn('response missing updatedAt');
-  return data;
-},
-expect: json('./data/rates.json'),
-// compare defaults to 'deep-equal'
-```
-
-### `verify` pipelines
-
-`verify` is an optional map of named stage lists, for use from your app
-(`runVerify(config.verify.answers, draft)`). There is no `stilltrue verify`
-CLI command — verification runs in the request path, not in CI.
-
-Each stage:
-
-| Field | Required | Default | What it is |
-|---|---|---|---|
-| `name` | yes | — | Id of this checkpoint. The first rejection reports this name. |
-| `check` | yes | — | `(input, ctx) => { ok: true, value? } \| { ok: false, messages, revision? }`. |
-| `failPolicy` | no | `'open'` | What a **thrown** stage means. `'open'`: warn and continue (a judge outage must not block the app). `'closed'`: treat the throw as a rejection. |
-| `timeoutMs` | no | none | Abort the stage after this many ms. The timeout follows `failPolicy`. |
-| `tier` | no | `'inline'` | `'inline'` always runs; `'ci'` runs only when you call `runVerify(..., { tier: 'ci' })`. |
-
-Stages run in order and stop at the first rejection. `zodStage(name, schema)`
-builds a parse stage from a Zod schema.
-`generateVerified({ generate, stages, attempts, tier })` is the generate →
-verify → retry loop (`attempts` defaults to `2`). See
-[For developers: `verify`](#for-developers-verify) below.
-
-### CLI flags
-
-`stilltrue drift`:
-
-| Flag | What it does |
-|---|---|
-| `--config <path>` | Config file. Default: first `stilltrue.config.*` in the current directory. |
-| `--only <name,name>` | Run only these checks. |
-| `--json <path>` | Write the result array as JSON. |
-| `--record` | Append this run to history. |
-| `--history <path>` | History file for `--record`. Default: `.stilltrue/history.jsonl`. |
-
-`stilltrue report`:
-
-| Flag | What it does |
-|---|---|
-| `--history <path>` | History file to read. Default: `.stilltrue/history.jsonl`. |
-| `--out <path>` | HTML output. Default: `stilltrue-report.html`. |
-| `--title <text>` | Page `<title>` and heading. Default: `stilltrue drift report`. |
-
 ## Running it automatically every week
 
 If your project is on GitHub, add this file as
@@ -329,79 +163,29 @@ comparison stays mechanical.)
 ## For developers: `verify`
 
 `verify` is a library for teams shipping AI-generated answers: an ordered
-series of checkpoints (a schema parse, a rules check, an AI judge — your
-choice) that each answer must pass before publishing. Rejections say
-exactly which checkpoint failed and why, in a format you can feed back to
-the model for a retry. A broken checker never blocks your app — it fails
-open. See the [project docs](docs/BRIEF.md) and the typed API in
-`stilltrue`'s exports (`runVerify`, `zodStage`, `generateVerified`).
-
-**Bring your own judge — any provider, or none.** stilltrue ships no AI
-and never touches a model key: a checkpoint is just a function you supply,
-so the judge stage can call Gemini, OpenAI, Anthropic, a local model — or
-be a plain schema/rules check with no AI at all. Swapping providers means
-editing your function, never this package.
-
-### The common case, in six lines
-
-Most people arrive wanting one thing: *does this source actually support
-this claim?* `supportStage` is that check, ready-made. You bring a function
-that calls your model; it brings the prompt, the response schema, and the
-guard.
-
-```js
-import { runVerify, supportStage } from 'stilltrue';
-
-const stage = supportStage({
-  // The only required option. Return the model's reply — parsed or raw JSON.
-  judge: ({ system, user, schema }) => callMyModel({ system, user, schema }),
-});
-
-const result = await runVerify([stage], { claim, source });
-// result.ok      → true when the source supports the claim
-// result.value   → { verdict, reasoning, evidence, contradiction, guard }
-// result.messages → why it was rejected, ready for a retry
-```
-
-`verdict` is one of `supported`, `not_supported` or `cant_tell`. Silence is
-not support: a source that never addresses the claim returns `cant_tell`,
-even when the claim happens to be true.
-
-**"Not supported" has to earn itself.** That verdict tells a reader the
-source *says otherwise*, which is the strongest thing a judge can assert
-and the one most worth being wrong about. So the stage requires the judge
-to quote the contradicting sentence, and checks that quote against the
-source in code. If it cannot point at real text, the verdict is downgraded
-to `cant_tell` and a warning explains why. The check only ever downgrades —
-a missing quote is evidence of nothing except a missing quote.
-
-That guard is not theoretical. Judged by a small local model, adding two
-lines to a source that never mentioned the claim flipped a verdict from
-`supported` to `not_supported` — three runs each way, so not noise — where
-a frontier model stayed stable. A package that promises engine-agnostic
-verification has to survive the weaker engine.
-
-Useful options, all defaulted: `minRun` (how much of the contradiction
-quote must match the source — 24 characters, measured on civic notices and
-fee tables, not universal), `requireContradiction: false` to switch the
-guard off, `treatCantTellAs: 'warn'` to pass unanswerable claims instead of
-rejecting them, and `system` to replace the prompt entirely. `SUPPORT_SYSTEM`
-and `SUPPORT_SCHEMA` are exported too, so a provider with constrained
-decoding can take the schema directly.
-
-### Writing your own stages
-
-`supportStage` is one stage; a pipeline is a list of them, and any function
-of the right shape will do. **[docs/VERIFY.md](docs/VERIFY.md)** is the full
-walkthrough — the judging prompt explained, response mapping, retries, and
-the generate → verify → retry loop — for when the ready-made stage is not
-the shape you need.
-
-## What about testing prompts?
-
-That's a different job with a great existing tool:
-[promptfoo](https://promptfoo.dev) tests your prompts; stilltrue tests your
-facts. How they fit together: [docs/GOLDEN.md](docs/GOLDEN.md).
+series of checkpoints — a schema parse, a rules check, an AI judge, your
+choice — that each answer must pass before publishing. Rejections say which
+checkpoint failed and why, in a format you can feed back to the model for a
+retry; a broken checker fails open rather than blocking your app. stilltrue
+ships no AI and never touches a model key — the judge is a function you
+supply, so swapping providers means editing your function, never this
+package.
+
+**[docs/VERIFY.md](docs/VERIFY.md) is the whole story**, including a
+complete copy-paste judge (Gemini shown, OpenAI-compatible variant inline —
+which also covers LM Studio and every hosted provider) and `supportStage`,
+the ready-made "does this source support this claim?" check.
+
+## Reference
+
+- **[Configuration](docs/CONFIG.md)** — every config field, compare mode,
+  helper, and CLI flag.
+- **[Picking good markers](docs/MARKERS.md)** — the judgment calls
+  (`surnames beat full names`, `"143 schools"` beats `"143"`).
+- **[Verify](docs/VERIFY.md)** — building a judge, writing stages,
+  `generateVerified`.
+- **[Prompts vs facts](docs/GOLDEN.md)** — [promptfoo](https://promptfoo.dev)
+  tests your prompts; stilltrue tests your facts. How they fit together.
 
 ## License
 

package/dist/cli.js

@@ -106,7 +106,7 @@ try {
                     '  1. Edit stilltrue.config.mjs — point source at the real page(s) your facts came from,\n' +
                     '     and replace the example markers with your own facts (plain strings)\n' +
                     '  2. Run: npx stilltrue drift\n' +
-                    'Picking good markers: docs/MARKERS.md in the stilltrue repo');
+                    'Picking good markers: https://github.com/sg-bfoster/stilltrue/blob/main/docs/MARKERS.md');
             }
             process.exit(0);
             break;

package/dist/support.d.ts

@@ -101,6 +101,19 @@ export interface SupportStageOptions {
  * the obvious over-correction is ruling not_supported on a page that simply
  * lists several values for different cases.
  *
+ * The list and same-subject lines were added 2026-10-28, the mirror image of
+ * the 10% failure on the same page. Asked whether apple.com/apple-card supports
+ * "Apple Card works with Windows PCs", the judge ruled not_supported, quoting
+ * "Apple Pay ... built into iPhone, Apple Watch, iPad, and Mac" and reasoning
+ * that listing only Apple devices "implies exclusion". That is an argument from
+ * silence, and the quote was about Apple Pay rather than Apple Card. The guard
+ * could not catch it, because the quote was real. On qwen3.6-35b-a3b the real
+ * case went from 0/3 to 3/3 correct, with no change on the 21-case comparison
+ * set, the controls, or the 10% case. Two synthetic list cases ("issued at the
+ * Duluth, Norcross and Snellville branches") still fail on that model, which
+ * reads a plain list as "only" these. Putting reasoning ahead of the verdict
+ * fixed those and broke the real Apple case 3/3, so it did not ship.
+ *
  * The line about qualification is load-bearing rather than decorative.
  * Measured 2026-09-04 without it: adding two lines to a source that never
  * mention the claim flipped a verdict from supported to not_supported — three
@@ -108,7 +121,7 @@ export interface SupportStageOptions {
  * model stayed stable. A package promising engine-agnostic verification has to
  * survive the weaker engine, so the instruction ships by default.
  */
-export declare const SUPPORT_SYSTEM = "You are a strict fact-check judge. Decide whether the SOURCE supports the CLAIM using ONLY the source text \u2014 no outside knowledge, no charitable guessing.\nVerdicts:\n- \"supported\": the source affirmatively supports the claim.\n- \"not_supported\": the source contradicts the claim or asserts something incompatible with it. This INCLUDES the source stating a DIFFERENT VALUE for the same thing the claim quantifies \u2014 a rate, price, date, count, duration or name. If the claim says 10% and the source says the rate is 3%, the source has contradicted it; it does not have to mention 10% to do so.\n- \"cant_tell\": the source never addresses the subject of the claim at all, or addresses it only partially. A source that states its own value for the very thing the claim asserts HAS addressed it \u2014 that is not_supported, not cant_tell. Silence is NOT support.\nA qualification is not a contradiction. If the source substantially supports the claim but adds an exception, a carve-out or a condition, it has NOT said otherwise \u2014 do not rule not_supported. Reserve not_supported for a source that asserts something the claim cannot be true alongside.\nTo rule \"not_supported\" you MUST fill \"contradiction\" with the exact sentence from the source that says otherwise, copied verbatim. Where the conflict is a differing value, that sentence is the one stating the SOURCE's own value. If no such sentence exists, you may not use that verdict.\nCompare like with like: a different value only contradicts when it describes the same thing under the same conditions. A source listing several values for different cases has not contradicted a claim that matches one of them.\nReturn STRICT JSON only:\n{\"verdict\":\"supported\"|\"not_supported\"|\"cant_tell\",\"reasoning\":\"<one or two plain sentences>\",\"evidence\":\"<short verbatim quote from the source that decided it, or null>\",\"contradiction\":\"<verbatim sentence that contradicts the claim, or null>\"}";
+export declare const SUPPORT_SYSTEM = "You are a strict fact-check judge. Decide whether the SOURCE supports the CLAIM using ONLY the source text \u2014 no outside knowledge, no charitable guessing.\nVerdicts:\n- \"supported\": the source affirmatively supports the claim.\n- \"not_supported\": the source contradicts the claim or asserts something incompatible with it. This INCLUDES the source stating a DIFFERENT VALUE for the same thing the claim quantifies \u2014 a rate, price, date, count, duration or name. If the claim says 10% and the source says the rate is 3%, the source has contradicted it; it does not have to mention 10% to do so.\n- \"cant_tell\": the source never addresses the subject of the claim at all, or addresses it only partially. A source that states its own value for the very thing the claim asserts HAS addressed it \u2014 that is not_supported, not cant_tell. Silence is NOT support.\nA qualification is not a contradiction. If the source substantially supports the claim but adds an exception, a carve-out or a condition, it has NOT said otherwise \u2014 do not rule not_supported. Reserve not_supported for a source that asserts something the claim cannot be true alongside.\nTo rule \"not_supported\" you MUST fill \"contradiction\" with the exact sentence from the source that says otherwise, copied verbatim. Where the conflict is a differing value, that sentence is the one stating the SOURCE's own value. If no such sentence exists, you may not use that verdict.\nCompare like with like: a different value only contradicts when it describes the same thing under the same conditions. A source listing several values for different cases has not contradicted a claim that matches one of them.\nA list is not a denial. A source that names the devices, places, people or cases something applies to has NOT ruled out the ones it leaves unnamed, unless it says so in words \u2014 \"only\", \"exclusively\", \"not available on\", \"does not work with\". Absence from a list is silence, and silence is cant_tell.\nThe contradicting sentence must be about the same subject as the claim. A sentence about a different product, service or program does not contradict it.\nReturn STRICT JSON only:\n{\"verdict\":\"supported\"|\"not_supported\"|\"cant_tell\",\"reasoning\":\"<one or two plain sentences>\",\"evidence\":\"<short verbatim quote from the source that decided it, or null>\",\"contradiction\":\"<verbatim sentence that contradicts the claim, or null>\"}";
 /** JSON Schema for the judgement, for providers that constrain decoding. */
 export declare const SUPPORT_SCHEMA: {
     readonly type: "object";

package/dist/support.js

@@ -32,6 +32,19 @@
  * the obvious over-correction is ruling not_supported on a page that simply
  * lists several values for different cases.
  *
+ * The list and same-subject lines were added 2026-10-28, the mirror image of
+ * the 10% failure on the same page. Asked whether apple.com/apple-card supports
+ * "Apple Card works with Windows PCs", the judge ruled not_supported, quoting
+ * "Apple Pay ... built into iPhone, Apple Watch, iPad, and Mac" and reasoning
+ * that listing only Apple devices "implies exclusion". That is an argument from
+ * silence, and the quote was about Apple Pay rather than Apple Card. The guard
+ * could not catch it, because the quote was real. On qwen3.6-35b-a3b the real
+ * case went from 0/3 to 3/3 correct, with no change on the 21-case comparison
+ * set, the controls, or the 10% case. Two synthetic list cases ("issued at the
+ * Duluth, Norcross and Snellville branches") still fail on that model, which
+ * reads a plain list as "only" these. Putting reasoning ahead of the verdict
+ * fixed those and broke the real Apple case 3/3, so it did not ship.
+ *
  * The line about qualification is load-bearing rather than decorative.
  * Measured 2026-09-04 without it: adding two lines to a source that never
  * mention the claim flipped a verdict from supported to not_supported — three
@@ -47,6 +60,8 @@ Verdicts:
 A qualification is not a contradiction. If the source substantially supports the claim but adds an exception, a carve-out or a condition, it has NOT said otherwise — do not rule not_supported. Reserve not_supported for a source that asserts something the claim cannot be true alongside.
 To rule "not_supported" you MUST fill "contradiction" with the exact sentence from the source that says otherwise, copied verbatim. Where the conflict is a differing value, that sentence is the one stating the SOURCE's own value. If no such sentence exists, you may not use that verdict.
 Compare like with like: a different value only contradicts when it describes the same thing under the same conditions. A source listing several values for different cases has not contradicted a claim that matches one of them.
+A list is not a denial. A source that names the devices, places, people or cases something applies to has NOT ruled out the ones it leaves unnamed, unless it says so in words — "only", "exclusively", "not available on", "does not work with". Absence from a list is silence, and silence is cant_tell.
+The contradicting sentence must be about the same subject as the claim. A sentence about a different product, service or program does not contradict it.
 Return STRICT JSON only:
 {"verdict":"supported"|"not_supported"|"cant_tell","reasoning":"<one or two plain sentences>","evidence":"<short verbatim quote from the source that decided it, or null>","contradiction":"<verbatim sentence that contradicts the claim, or null>"}`;
 /** JSON Schema for the judgement, for providers that constrain decoding. */

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "stilltrue",
-  "version": "0.7.0",
+  "version": "0.8.0",
   "description": "Your tests check your code. stilltrue checks your facts. CI checks for AI-in-the-loop apps: drift (curated facts vs live sources), golden (regression evals), verify (judge answers against sources).",
   "keywords": [
     "drift",
@@ -31,7 +31,7 @@
   },
   "dependencies": {
     "jiti": "^2.7.0",
-    "zod": "^3.23.0"
+    "zod": "^3.23.0 || ^4.0.0"
   },
   "devDependencies": {
     "@types/node": "^22.0.0",