stilltrue 0.6.0 → 0.7.1

package/CHANGELOG.md

@@ -0,0 +1,72 @@
+# Changelog
+
+## 0.7.1 — 2026-09-13
+
+Docs restructure, no behavior changes except one message:
+
+- README cut from 408 lines to ~190. A fresh-eyes pass (install → first
+  green check in three minutes) showed setup was never the problem — the
+  document performed difficulty the tool doesn't have. The drift
+  quickstart, which tested perfect, is untouched.
+- New `docs/CONFIG.md`: the reference shelf (config fields, compare modes,
+  helpers, CLI flags), moved verbatim out of the tutorial path.
+- `docs/VERIFY.md` gained the `supportStage` section from the README, so
+  the ready-made stage lives beside the judge walkthrough it builds on.
+- `stilltrue init` now prints the full MARKERS.md URL — npm users have no
+  repo checkout for a relative path to resolve against.
+
+## 0.7.0
+
+**A different value now counts as a contradiction.**
+
+The judge used to answer `cant_tell` when a source stated its own figure for
+the thing a claim quantified. Asked whether `apple.com/apple-card` supports
+"Apple Card gives 10% Daily Cash back on every purchase", it reasoned:
+
+> The source states that Apple Card offers 2% Daily Cash for most purchases,
+> 3% at Apple and select merchants, and 1% on the titanium card, but it does
+> not mention a 10% Daily Cash rate.
+
+It had read every rate on the page and still treated the absence of the
+claimed number as the claim being unaddressed. Nothing on that page denies
+10% in words, so the old wording — `cant_tell: the source does not address
+the claim` — left it nowhere else to land.
+
+Explicit denial is the rare shape of a false claim. The common one is a wrong
+number, and a checker that goes quiet on wrong numbers is quiet exactly when
+it is most useful. It also collapsed the distinction the verdict set exists
+for: a claim the source contradicts and a claim the source never mentions
+came back identical.
+
+- `not_supported` now explicitly covers a source stating a DIFFERENT VALUE for
+  what the claim quantifies — a rate, price, date, count, duration or name.
+- `cant_tell` explicitly excludes that case.
+- The contradiction-quote rule gains the matching line: where the conflict is
+  a differing value, the sentence to cite is the one stating the source's own
+  value.
+- "Compare like with like" ships in the same change, because the obvious
+  over-correction is ruling `not_supported` against a page that simply lists
+  several values for different cases.
+
+**Minor, not patch.** The public API is unchanged, but verdicts change for the
+same inputs — and for a library whose product IS the verdict, that should not
+reach a consumer pinned to `^0.6.0` silently.
+
+Verified against a local gpt-oss-20b (the weaker of two available engines),
+five claims against one live page:
+
+| claim | before | after |
+|---|---|---|
+| 3% Daily Cash at Apple | supported | supported |
+| **10% on every purchase** | **cant_tell** | **not_supported** |
+| works with Windows PCs | cant_tell | cant_tell |
+| 2% with Apple Pay | — | supported |
+| no annual fee | — | supported |
+
+The last two are over-correction checks. The contradiction guard did not
+downgrade the 10% verdict, so it arrived with a verbatim quote that passed the
+contiguous-run check — refused with evidence, not merely refused.
+
+## 0.6.0 and earlier
+
+Not recorded here; see the git history.

package/README.md

@@ -71,7 +71,7 @@ Office", and "12 locations" all still appear on it.* Swap in your own URL
 and your own facts — that's a working check, no programming required.
 
 Every other field, compare mode, and helper is in
-[Configuration](#configuration) below. Choosing *what* to assert is the
+[the configuration reference](docs/CONFIG.md). Choosing *what* to assert is the
 judgment call — surnames beat full names, `"143 schools"` beats a bare
 `"143"`. The short guide: [docs/MARKERS.md](docs/MARKERS.md).
 
@@ -115,172 +115,6 @@ You'll see one line per check:
   This only *warns*. An unreachable site doesn't mean your facts are wrong,
   and an alarm that cries wolf gets ignored — so outages never fail the build.
 
-## Configuration
-
-stilltrue loads a config file from the current directory. The first of
-these that exists wins:
-
-`stilltrue.config.ts` · `.mts` · `.js` · `.mjs` · `.cjs`
-
-(`npx stilltrue init` writes `.mjs`.) Point at a file anywhere with
-`stilltrue drift --config path/to/file`.
-
-Wrap the export with `defineStilltrue` so TypeScript can check it:
-
-```js
-import { defineStilltrue, corpus, json, surname } from 'stilltrue';
-
-export default defineStilltrue({
-  drift: [ /* one entry per fact-set */ ],
-  verify: { /* optional named pipelines — library use, see below */ },
-});
-```
-
-### Drift checks
-
-Each `drift` entry is one curated fact-set vs its live source.
-
-| Field | Required | Default | What it is |
-|---|---|---|---|
-| `name` | yes | — | Short id. Shows up in the terminal, `--only`, and the report. |
-| `source` | yes | — | `async (ctx) => actual`. Fetch/parse the live authority. **Throw** if the source is unreachable — that is outcome `error`, never `rot`. |
-| `expect` | yes | — | The curated facts. A value, or `async (ctx) => expected` (e.g. `json(...)`). A throw here is also `error` (broken config, not rot). |
-| `compare` | no | `'deep-equal'` | How to compare `expect` against what `source` returned. |
-
-`source` and function-form `expect` receive `ctx`:
-
-| | |
-|---|---|
-| `ctx.warn(message)` | Record a non-fatal note (one of several pages down, odd markup). Shows as a warning; does not fail the run. |
-| `ctx.configDir` | Absolute directory of the config file. Data paths resolve from here. |
-
-### `compare`
-
-**`'contains-all'`** — `expect` is a list of marker strings; `source` must
-return a text corpus (what `corpus()` returns). Every marker must appear
-somewhere in that text, case-insensitively. Extra text on the page is
-ignored. This is the webpage-watch check.
-
-**`'deep-equal'`** — structural equality. Extra keys, missing keys, and
-value mismatches all rot, with pathed messages
-(`$.members[2].name: expected "Smith", source has "Jones"`). Use this when
-`source()` already returns structured data that should match `expect`
-exactly.
-
-**A function** — `(expected, actual) => string[]`. Return `[]` to pass;
-each string is one mismatch (outcome `rot`). Throw if the shapes are
-unusable (outcome `error`). Use this for numeric tolerance, order-insensitive
-lists, or any comparison that isn't "markers in a page" or "these two
-objects are identical."
-
-```js
-compare: (expected, actual) =>
-  expected.rate === actual.rate
-    ? []
-    : [`tax rate ${expected.rate} → ${actual.rate}`],
-```
-
-Surnames are a *marker choice*, not a compare mode: derive them from your
-data with `surname()` and use `'contains-all'`.
-
-### Helpers
-
-These are the built-in `source` / `expect` builders. You can always write
-your own functions instead.
-
-**`corpus(urls, options?)`** — fetch pages, strip HTML to text,
-concatenate. Individual unreachable pages call `ctx.warn`; if *every* page
-fails, it throws (`error`). Pair with `'contains-all'`.
-
-| Option | Default | What it is |
-|---|---|---|
-| `timeoutMs` | `20000` | Per-page fetch timeout, milliseconds. |
-| `userAgent` | `stilltrue drift check (github.com/sg-bfoster/stilltrue)` | Sent on each request so operators can identify the check. |
-
-```js
-source: corpus(
-  ['https://example.org/about', 'https://example.org/board'],
-  { timeoutMs: 15_000, userAgent: 'my-app drift check' },
-),
-```
-
-**`json(path, derive?)`** — read a JSON file relative to the config file.
-With no `derive`, the parsed value *is* the expectation (typical with
-`'deep-equal'`). With `derive`, you map that value into whatever `compare`
-needs — usually a marker list:
-
-```js
-expect: json('./data/board.json', (data) =>
-  data.members.map((m) => surname(m.name)),
-),
-```
-
-**`surname(fullName)`** — last token of a name, with leading honorifics
-(`Dr.`, `Mr.`, `Mrs.`, `Ms.`) and trailing generational suffixes (`Jr.`,
-`Sr.`, `II`–`V`) stripped. `"Dr. Jane Smith III"` → `"Smith"`.
-
-**`htmlToText(html)`** — the same tag-stripping `corpus()` uses. For a
-custom `source` that already has HTML in hand.
-
-### Custom `source`
-
-A throw is an outage; a return value is data to compare. Warn for partial
-trouble:
-
-```js
-source: async (ctx) => {
-  const res = await fetch('https://example.org/api/rates');
-  if (!res.ok) throw new Error(`HTTP ${res.status}`);
-  const data = await res.json();
-  if (!data.updatedAt) ctx.warn('response missing updatedAt');
-  return data;
-},
-expect: json('./data/rates.json'),
-// compare defaults to 'deep-equal'
-```
-
-### `verify` pipelines
-
-`verify` is an optional map of named stage lists, for use from your app
-(`runVerify(config.verify.answers, draft)`). There is no `stilltrue verify`
-CLI command — verification runs in the request path, not in CI.
-
-Each stage:
-
-| Field | Required | Default | What it is |
-|---|---|---|---|
-| `name` | yes | — | Id of this checkpoint. The first rejection reports this name. |
-| `check` | yes | — | `(input, ctx) => { ok: true, value? } \| { ok: false, messages, revision? }`. |
-| `failPolicy` | no | `'open'` | What a **thrown** stage means. `'open'`: warn and continue (a judge outage must not block the app). `'closed'`: treat the throw as a rejection. |
-| `timeoutMs` | no | none | Abort the stage after this many ms. The timeout follows `failPolicy`. |
-| `tier` | no | `'inline'` | `'inline'` always runs; `'ci'` runs only when you call `runVerify(..., { tier: 'ci' })`. |
-
-Stages run in order and stop at the first rejection. `zodStage(name, schema)`
-builds a parse stage from a Zod schema.
-`generateVerified({ generate, stages, attempts, tier })` is the generate →
-verify → retry loop (`attempts` defaults to `2`). See
-[For developers: `verify`](#for-developers-verify) below.
-
-### CLI flags
-
-`stilltrue drift`:
-
-| Flag | What it does |
-|---|---|
-| `--config <path>` | Config file. Default: first `stilltrue.config.*` in the current directory. |
-| `--only <name,name>` | Run only these checks. |
-| `--json <path>` | Write the result array as JSON. |
-| `--record` | Append this run to history. |
-| `--history <path>` | History file for `--record`. Default: `.stilltrue/history.jsonl`. |
-
-`stilltrue report`:
-
-| Flag | What it does |
-|---|---|
-| `--history <path>` | History file to read. Default: `.stilltrue/history.jsonl`. |
-| `--out <path>` | HTML output. Default: `stilltrue-report.html`. |
-| `--title <text>` | Page `<title>` and heading. Default: `stilltrue drift report`. |
-
 ## Running it automatically every week
 
 If your project is on GitHub, add this file as
@@ -329,79 +163,29 @@ comparison stays mechanical.)
 ## For developers: `verify`
 
 `verify` is a library for teams shipping AI-generated answers: an ordered
-series of checkpoints (a schema parse, a rules check, an AI judge — your
-choice) that each answer must pass before publishing. Rejections say
-exactly which checkpoint failed and why, in a format you can feed back to
-the model for a retry. A broken checker never blocks your app — it fails
-open. See the [project docs](docs/BRIEF.md) and the typed API in
-`stilltrue`'s exports (`runVerify`, `zodStage`, `generateVerified`).
-
-**Bring your own judge — any provider, or none.** stilltrue ships no AI
-and never touches a model key: a checkpoint is just a function you supply,
-so the judge stage can call Gemini, OpenAI, Anthropic, a local model — or
-be a plain schema/rules check with no AI at all. Swapping providers means
-editing your function, never this package.
-
-### The common case, in six lines
-
-Most people arrive wanting one thing: *does this source actually support
-this claim?* `supportStage` is that check, ready-made. You bring a function
-that calls your model; it brings the prompt, the response schema, and the
-guard.
-
-```js
-import { runVerify, supportStage } from 'stilltrue';
-
-const stage = supportStage({
-  // The only required option. Return the model's reply — parsed or raw JSON.
-  judge: ({ system, user, schema }) => callMyModel({ system, user, schema }),
-});
-
-const result = await runVerify([stage], { claim, source });
-// result.ok      → true when the source supports the claim
-// result.value   → { verdict, reasoning, evidence, contradiction, guard }
-// result.messages → why it was rejected, ready for a retry
-```
-
-`verdict` is one of `supported`, `not_supported` or `cant_tell`. Silence is
-not support: a source that never addresses the claim returns `cant_tell`,
-even when the claim happens to be true.
-
-**"Not supported" has to earn itself.** That verdict tells a reader the
-source *says otherwise*, which is the strongest thing a judge can assert
-and the one most worth being wrong about. So the stage requires the judge
-to quote the contradicting sentence, and checks that quote against the
-source in code. If it cannot point at real text, the verdict is downgraded
-to `cant_tell` and a warning explains why. The check only ever downgrades —
-a missing quote is evidence of nothing except a missing quote.
-
-That guard is not theoretical. Judged by a small local model, adding two
-lines to a source that never mentioned the claim flipped a verdict from
-`supported` to `not_supported` — three runs each way, so not noise — where
-a frontier model stayed stable. A package that promises engine-agnostic
-verification has to survive the weaker engine.
-
-Useful options, all defaulted: `minRun` (how much of the contradiction
-quote must match the source — 24 characters, measured on civic notices and
-fee tables, not universal), `requireContradiction: false` to switch the
-guard off, `treatCantTellAs: 'warn'` to pass unanswerable claims instead of
-rejecting them, and `system` to replace the prompt entirely. `SUPPORT_SYSTEM`
-and `SUPPORT_SCHEMA` are exported too, so a provider with constrained
-decoding can take the schema directly.
-
-### Writing your own stages
-
-`supportStage` is one stage; a pipeline is a list of them, and any function
-of the right shape will do. **[docs/VERIFY.md](docs/VERIFY.md)** is the full
-walkthrough — the judging prompt explained, response mapping, retries, and
-the generate → verify → retry loop — for when the ready-made stage is not
-the shape you need.
-
-## What about testing prompts?
-
-That's a different job with a great existing tool:
-[promptfoo](https://promptfoo.dev) tests your prompts; stilltrue tests your
-facts. How they fit together: [docs/GOLDEN.md](docs/GOLDEN.md).
+series of checkpoints — a schema parse, a rules check, an AI judge, your
+choice — that each answer must pass before publishing. Rejections say which
+checkpoint failed and why, in a format you can feed back to the model for a
+retry; a broken checker fails open rather than blocking your app. stilltrue
+ships no AI and never touches a model key — the judge is a function you
+supply, so swapping providers means editing your function, never this
+package.
+
+**[docs/VERIFY.md](docs/VERIFY.md) is the whole story**, including a
+complete copy-paste judge (Gemini shown, OpenAI-compatible variant inline —
+which also covers LM Studio and every hosted provider) and `supportStage`,
+the ready-made "does this source support this claim?" check.
+
+## Reference
+
+- **[Configuration](docs/CONFIG.md)** — every config field, compare mode,
+  helper, and CLI flag.
+- **[Picking good markers](docs/MARKERS.md)** — the judgment calls
+  (`surnames beat full names`, `"143 schools"` beats `"143"`).
+- **[Verify](docs/VERIFY.md)** — building a judge, writing stages,
+  `generateVerified`.
+- **[Prompts vs facts](docs/GOLDEN.md)** — [promptfoo](https://promptfoo.dev)
+  tests your prompts; stilltrue tests your facts. How they fit together.
 
 ## License
 

package/dist/cli.js

@@ -106,7 +106,7 @@ try {
                     '  1. Edit stilltrue.config.mjs — point source at the real page(s) your facts came from,\n' +
                     '     and replace the example markers with your own facts (plain strings)\n' +
                     '  2. Run: npx stilltrue drift\n' +
-                    'Picking good markers: docs/MARKERS.md in the stilltrue repo');
+                    'Picking good markers: https://github.com/sg-bfoster/stilltrue/blob/main/docs/MARKERS.md');
             }
             process.exit(0);
             break;

package/dist/support.d.ts

@@ -87,6 +87,20 @@ export interface SupportStageOptions {
 /**
  * The shipped judge prompt.
  *
+ * The "different value" clause was added 2026-09-11 after a measured failure:
+ * asked whether apple.com/apple-card supports "Apple Card gives 10% Daily Cash",
+ * the judge answered cant_tell, reasoning "the source states 2%... 3%... and 1%,
+ * but it does not mention a 10% Daily Cash rate". It had read every rate on the
+ * page and still treated the absence of the claimed number as the claim being
+ * unaddressed. Nothing on that page denies 10% in words, so the original
+ * wording gave it nowhere else to land.
+ *
+ * That is the common shape of a false claim in the wild — a wrong number, not
+ * an explicit denial — so a checker that goes quiet on it is quiet exactly when
+ * it is most needed. The compare-like-with-like line ships alongside because
+ * the obvious over-correction is ruling not_supported on a page that simply
+ * lists several values for different cases.
+ *
  * The line about qualification is load-bearing rather than decorative.
  * Measured 2026-09-04 without it: adding two lines to a source that never
  * mention the claim flipped a verdict from supported to not_supported — three
@@ -94,7 +108,7 @@ export interface SupportStageOptions {
  * model stayed stable. A package promising engine-agnostic verification has to
  * survive the weaker engine, so the instruction ships by default.
  */
-export declare const SUPPORT_SYSTEM = "You are a strict fact-check judge. Decide whether the SOURCE supports the CLAIM using ONLY the source text \u2014 no outside knowledge, no charitable guessing.\nVerdicts:\n- \"supported\": the source affirmatively supports the claim.\n- \"not_supported\": the source contradicts the claim or asserts something incompatible with it.\n- \"cant_tell\": the source does not address the claim, or addresses it only partially. Silence is NOT support.\nA qualification is not a contradiction. If the source substantially supports the claim but adds an exception, a carve-out or a condition, it has NOT said otherwise \u2014 do not rule not_supported. Reserve not_supported for a source that asserts something the claim cannot be true alongside.\nTo rule \"not_supported\" you MUST fill \"contradiction\" with the exact sentence from the source that says otherwise, copied verbatim. If no such sentence exists, you may not use that verdict.\nReturn STRICT JSON only:\n{\"verdict\":\"supported\"|\"not_supported\"|\"cant_tell\",\"reasoning\":\"<one or two plain sentences>\",\"evidence\":\"<short verbatim quote from the source that decided it, or null>\",\"contradiction\":\"<verbatim sentence that contradicts the claim, or null>\"}";
+export declare const SUPPORT_SYSTEM = "You are a strict fact-check judge. Decide whether the SOURCE supports the CLAIM using ONLY the source text \u2014 no outside knowledge, no charitable guessing.\nVerdicts:\n- \"supported\": the source affirmatively supports the claim.\n- \"not_supported\": the source contradicts the claim or asserts something incompatible with it. This INCLUDES the source stating a DIFFERENT VALUE for the same thing the claim quantifies \u2014 a rate, price, date, count, duration or name. If the claim says 10% and the source says the rate is 3%, the source has contradicted it; it does not have to mention 10% to do so.\n- \"cant_tell\": the source never addresses the subject of the claim at all, or addresses it only partially. A source that states its own value for the very thing the claim asserts HAS addressed it \u2014 that is not_supported, not cant_tell. Silence is NOT support.\nA qualification is not a contradiction. If the source substantially supports the claim but adds an exception, a carve-out or a condition, it has NOT said otherwise \u2014 do not rule not_supported. Reserve not_supported for a source that asserts something the claim cannot be true alongside.\nTo rule \"not_supported\" you MUST fill \"contradiction\" with the exact sentence from the source that says otherwise, copied verbatim. Where the conflict is a differing value, that sentence is the one stating the SOURCE's own value. If no such sentence exists, you may not use that verdict.\nCompare like with like: a different value only contradicts when it describes the same thing under the same conditions. A source listing several values for different cases has not contradicted a claim that matches one of them.\nReturn STRICT JSON only:\n{\"verdict\":\"supported\"|\"not_supported\"|\"cant_tell\",\"reasoning\":\"<one or two plain sentences>\",\"evidence\":\"<short verbatim quote from the source that decided it, or null>\",\"contradiction\":\"<verbatim sentence that contradicts the claim, or null>\"}";
 /** JSON Schema for the judgement, for providers that constrain decoding. */
 export declare const SUPPORT_SCHEMA: {
     readonly type: "object";

package/dist/support.js

@@ -18,6 +18,20 @@
 /**
  * The shipped judge prompt.
  *
+ * The "different value" clause was added 2026-09-11 after a measured failure:
+ * asked whether apple.com/apple-card supports "Apple Card gives 10% Daily Cash",
+ * the judge answered cant_tell, reasoning "the source states 2%... 3%... and 1%,
+ * but it does not mention a 10% Daily Cash rate". It had read every rate on the
+ * page and still treated the absence of the claimed number as the claim being
+ * unaddressed. Nothing on that page denies 10% in words, so the original
+ * wording gave it nowhere else to land.
+ *
+ * That is the common shape of a false claim in the wild — a wrong number, not
+ * an explicit denial — so a checker that goes quiet on it is quiet exactly when
+ * it is most needed. The compare-like-with-like line ships alongside because
+ * the obvious over-correction is ruling not_supported on a page that simply
+ * lists several values for different cases.
+ *
  * The line about qualification is load-bearing rather than decorative.
  * Measured 2026-09-04 without it: adding two lines to a source that never
  * mention the claim flipped a verdict from supported to not_supported — three
@@ -28,10 +42,11 @@
 export const SUPPORT_SYSTEM = `You are a strict fact-check judge. Decide whether the SOURCE supports the CLAIM using ONLY the source text — no outside knowledge, no charitable guessing.
 Verdicts:
 - "supported": the source affirmatively supports the claim.
-- "not_supported": the source contradicts the claim or asserts something incompatible with it.
-- "cant_tell": the source does not address the claim, or addresses it only partially. Silence is NOT support.
+- "not_supported": the source contradicts the claim or asserts something incompatible with it. This INCLUDES the source stating a DIFFERENT VALUE for the same thing the claim quantifies — a rate, price, date, count, duration or name. If the claim says 10% and the source says the rate is 3%, the source has contradicted it; it does not have to mention 10% to do so.
+- "cant_tell": the source never addresses the subject of the claim at all, or addresses it only partially. A source that states its own value for the very thing the claim asserts HAS addressed it — that is not_supported, not cant_tell. Silence is NOT support.
 A qualification is not a contradiction. If the source substantially supports the claim but adds an exception, a carve-out or a condition, it has NOT said otherwise — do not rule not_supported. Reserve not_supported for a source that asserts something the claim cannot be true alongside.
-To rule "not_supported" you MUST fill "contradiction" with the exact sentence from the source that says otherwise, copied verbatim. If no such sentence exists, you may not use that verdict.
+To rule "not_supported" you MUST fill "contradiction" with the exact sentence from the source that says otherwise, copied verbatim. Where the conflict is a differing value, that sentence is the one stating the SOURCE's own value. If no such sentence exists, you may not use that verdict.
+Compare like with like: a different value only contradicts when it describes the same thing under the same conditions. A source listing several values for different cases has not contradicted a claim that matches one of them.
 Return STRICT JSON only:
 {"verdict":"supported"|"not_supported"|"cant_tell","reasoning":"<one or two plain sentences>","evidence":"<short verbatim quote from the source that decided it, or null>","contradiction":"<verbatim sentence that contradicts the claim, or null>"}`;
 /** JSON Schema for the judgement, for providers that constrain decoding. */

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "stilltrue",
-  "version": "0.6.0",
+  "version": "0.7.1",
   "description": "Your tests check your code. stilltrue checks your facts. CI checks for AI-in-the-loop apps: drift (curated facts vs live sources), golden (regression evals), verify (judge answers against sources).",
   "keywords": [
     "drift",
@@ -20,7 +20,8 @@
     "stilltrue": "dist/cli.js"
   },
   "files": [
-    "dist"
+    "dist",
+    "CHANGELOG.md"
   ],
   "scripts": {
     "build": "tsc",