stilltrue 0.4.0 → 0.5.0

package/README.md

@@ -2,111 +2,162 @@
 
 > Your tests check your code. **stilltrue** checks your facts.
 
-AI-in-the-loop apps quietly start lying three ways: code changes break old
-answers, the world changes and curated facts rot, and the model embellishes
-beyond its sources. stilltrue is one watchdog per failure mode, run from CI —
-no server, no dashboard, no SaaS.
-
-| Command | Watchdog | Status |
-|---|---|---|
-| `npx stilltrue drift` | Curated facts vs their live authoritative sources — fails the build on rot | shipped |
-| `npx stilltrue report` | Self-contained HTML drift timeline from recorded runs | shipped |
-| `verify` (library) | Staged acceptance pipeline for AI output — judge answers against sources in the request path | shipped |
-
-For golden-style regression evals (did a code change break an answer?), use
-[promptfoo](https://promptfoo.dev) — **promptfoo tests your prompts;
-stilltrue tests your facts.** Our eval doctrine and how the tools pair:
-[docs/GOLDEN.md](docs/GOLDEN.md).
-
-**Why drift first:** "drift detection" in the market means statistical ML
-feature drift; website-change monitors alert humans, not CI. Nobody ships
-*"assert my curated facts still match their live source, as a CI check."*
-This was built bespoke three times across three domains before becoming a
-package — see [docs/BRIEF.md](docs/BRIEF.md) for the full design brief,
-positioning, and build plan.
-
-## Core principle
-
-**AI may extract, but must never decide.** The default drift path has zero AI:
-fetch → parse → compare → diff. A consumer's `source()` may use an LLM to pull
-structure out of a messy page, but the comparison and the verdict are always
-deterministic. Same inputs, same verdict — a nondeterministic alarm trains
-people to ignore it.
-
-Drift has three outcomes, never two: `pass`, `rot` (facts changed — fail the
-build), and `error` (source unreachable — warn, don't fail). Conflating the
-last two causes alarm fatigue.
-
-## Quick start
-
-```ts
-// stilltrue.config.ts (or .mjs / .js)
+## The problem
+
+Say you've built an app or chatbot that answers questions using facts you
+collected — a list of school board members, office hours, tax rates,
+whatever. You checked those facts when you wrote them down. But the world
+keeps moving: a board member resigns, a rate changes, a first day of school
+rolls over. **Your app doesn't know.** It keeps confidently repeating
+yesterday's truth, and nothing alerts you.
+
+stilltrue is the alarm. Once a week (or whenever you like), it re-reads the
+official web pages your facts came from and checks that every fact you
+recorded still appears there. If something vanished, your build fails and
+tells you exactly which fact went stale — so a human can go verify and
+update it.
+
+## What you get
+
+| Command | What it does |
+|---|---|
+| `npx stilltrue drift` | Checks your saved facts against their live sources. Fails loudly if a fact has gone stale. |
+| `npx stilltrue report` | Turns past check runs into a simple web page — green/red grid, what broke and when. |
+| `verify` | For developers: a checkpoint that inspects AI-generated answers before they're shown to users. |
+
+## Getting started
+
+You'll need [Node.js](https://nodejs.org) (version 20 or newer) and a
+project folder with your facts saved in a JSON file.
+
+**1. Install it** — in your project folder, run:
+
+```bash
+npm install --save-dev stilltrue
+```
+
+**2. Scaffold a check** — this creates a ready-to-edit config file and an
+example facts file (it never overwrites anything you already have):
+
+```bash
+npx stilltrue init
+```
+
+Open the generated `stilltrue.config.mjs`. Each check answers three
+questions: *where do I fetch the truth from* (`source`), *what do I
+believe* (`expect`), and *how do I compare the two* (`compare`):
+
+```js
 import { defineStilltrue, corpus, json, surname } from 'stilltrue';
 
 export default defineStilltrue({
   drift: [
     {
       name: 'school-board-roster',
-      source: corpus(['https://district.example.org/board']),
-      expect: json('./data/school-board.json', (d) => d.members.map((m) => surname(m.name))),
+      // Where the truth lives: the district's own pages.
+      source: corpus(['https://www.myschooldistrict.org/board']),
+      // What you believe: read your saved file, keep just the last names.
+      expect: json('./data/school-board.json', (data) =>
+        data.members.map((m) => surname(m.name))
+      ),
+      // How to compare: every name must still appear on those pages.
       compare: 'contains-all',
     },
   ],
 });
 ```
 
+This example says: *fetch the board page, and make sure every last name in
+my `school-board.json` file still appears somewhere on it.*
+
+Choosing what to assert is the judgment call — surnames beat full names,
+"143 schools" beats a bare "143". The short guide:
+[docs/MARKERS.md](docs/MARKERS.md).
+
+**3. Run it:**
+
+```bash
+npx stilltrue drift
+```
+
+You'll see one line per check:
+
+```
+✓ school-board-roster — pass (1204ms)
+
+1 pass, 0 rot, 0 error (errors warn, never fail)
+```
+
+## The three results (and why there are three, not two)
+
+- **pass** — every saved fact still appears at its source. All good.
+- **rot** — a fact is gone from the source. The world changed; your data is
+  stale. This *fails the run* so you notice and fix the data.
+- **error** — the website couldn't be reached (down, timeout, blocked).
+  This only *warns*. An unreachable site doesn't mean your facts are wrong,
+  and an alarm that cries wolf gets ignored — so outages never fail the build.
+
+## Running it automatically every week
+
+If your project is on GitHub, add this file as
+`.github/workflows/drift.yml` and GitHub will run the check every Wednesday
+and email you if anything rotted:
+
 ```yaml
-# .github/workflows/drift.yml
+name: Fact drift check
 on:
-  schedule: [{ cron: '17 14 * * 3' }]  # off-beat minute: GitHub drops :00 crons
+  schedule:
+    - cron: '17 14 * * 3'   # Wednesdays, ~10am US Eastern
+  workflow_dispatch:         # adds a manual "Run workflow" button
 jobs:
   drift:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
+      - uses: actions/checkout@v5
+      - uses: actions/setup-node@v5
         with: { node-version: '22' }
       - run: npm ci
       - run: npx stilltrue drift
 ```
 
-`source` is your code (fetch+parse, or LLM extraction — the comparison stays
-deterministic either way). `expect` is your curated file. Rot fails the run
-with one message per vanished fact; unreachable sources only warn.
+## The report
 
-## Verify — staged acceptance for AI output
+Add `--record` to save each run, then generate the web page:
 
-An ordered, short-circuiting pipeline: each stage is a pluggable judge — a
-Zod parse, a deterministic validator, or an LLM. Rejection is structured
-(`{ stage, messages[] }`, one issue per entry) and designed to be fed back to
-the generator for a retry. Stages that *throw* are infrastructure failures,
-not verdicts: `failPolicy: 'open'` (default) warns and continues — a judge
-outage must never block your app.
+```bash
+npx stilltrue drift --record
+npx stilltrue report
+```
 
-```ts
-import { runVerify, zodStage, generateVerified, formatRejection } from 'stilltrue';
+That writes `stilltrue-report.html` — open it in any browser. It shows a
+timeline grid (one row per check, one column per run, green ✓ / red ✗ /
+amber ⚠) and a log of everything that ever rotted, with details.
 
-const stages = [
-  zodStage('parse_schema', AnswerSchema),
-  { name: 'grounding', timeoutMs: 10_000, check: llmJudge },     // may return a revision
-  { name: 'chase-sim', tier: 'ci', check: expensiveSimulation }, // skipped inline
-];
+## One rule underneath everything
 
-// Request path:
-const result = await runVerify(stages, draft);
-publish(result.value); // draft, parsed draft, or the judge's rewrite
+**AI may extract, but never decides.** The drift check itself uses no AI at
+all — it fetches, reads, and compares text, the same way every time, with
+no API key and no cost. Deterministic alarms are trustworthy alarms. (If
+*your* fact-gathering step wants to use AI to pull structure out of a messy
+page, that's fine — it happens inside your `source` function, and the
+comparison stays mechanical.)
 
-// Generator loop (one automatic retry, then fall back — don't loop forever):
-const gen = await generateVerified({ generate: (feedback) => ask(model, prompt, feedback && formatRejection(feedback)), stages });
-```
+## For developers: `verify`
+
+`verify` is a library for teams shipping AI-generated answers: an ordered
+series of checkpoints (a schema parse, a rules check, an AI judge — your
+choice) that each answer must pass before publishing. Rejections say
+exactly which checkpoint failed and why, in a format you can feed back to
+the model for a retry. A broken checker never blocks your app — it fails
+open. See the [project docs](docs/BRIEF.md) and the typed API in
+`stilltrue`'s exports (`runVerify`, `zodStage`, `generateVerified`).
 
-## Status
+## What about testing prompts?
 
-v0.4 — feature-complete for the 1.0 shape: `drift` and `report` run in
-production CI (AskGwinnett), `verify` runs in the production request path
-(AskGwinnett's pre-publish check sweep). Golden is deliberately out of
-scope — settled in [docs/GOLDEN.md](docs/GOLDEN.md).
+That's a different job with a great existing tool:
+[promptfoo](https://promptfoo.dev) tests your prompts; stilltrue tests your
+facts. How they fit together: [docs/GOLDEN.md](docs/GOLDEN.md).
 
 ## License
 
-MIT
+MIT — free to use, modify, and share.

package/dist/cli.js

@@ -94,13 +94,31 @@ try {
         case 'report':
             process.exit(await report(rest));
             break;
+        case 'init': {
+            const { init } = await import("./init.js");
+            const { created, skipped } = await init(process.cwd());
+            for (const f of created)
+                console.log(`created ${f}`);
+            for (const f of skipped)
+                console.log(`kept existing ${f} (never overwritten)`);
+            if (created.length) {
+                console.log('\nNext steps:\n' +
+                    '  1. Edit stilltrue.config.mjs — point source at the real page(s) your facts came from\n' +
+                    '  2. Edit data/example-facts.json — or point the config at a facts file you already have\n' +
+                    '  3. Run: npx stilltrue drift\n' +
+                    'Picking good markers: docs/MARKERS.md in the stilltrue repo');
+            }
+            process.exit(0);
+            break;
+        }
         case 'golden':
             console.error('stilltrue deliberately has no golden runner — promptfoo tests your prompts; stilltrue tests your facts.\n' +
                 'Use https://promptfoo.dev for prompt/answer regression evals. Doctrine + pairing guide: docs/GOLDEN.md');
             process.exit(1);
             break;
         default:
-            console.log('usage: stilltrue drift [--config <path>] [--only <names>] [--json <path>] [--record] [--history <path>]\n' +
+            console.log('usage: stilltrue init\n' +
+                '       stilltrue drift [--config <path>] [--only <names>] [--json <path>] [--record] [--history <path>]\n' +
                 '       stilltrue report [--history <path>] [--out <path>] [--title <text>]');
             process.exit(command ? 1 : 0);
     }

package/dist/init.d.ts

@@ -0,0 +1,6 @@
+export interface InitResult {
+    created: string[];
+    skipped: string[];
+}
+/** Write starter config + example data file. Never overwrites. */
+export declare function init(dir: string): Promise<InitResult>;

package/dist/init.js

@@ -0,0 +1,78 @@
+import { mkdir, writeFile } from 'node:fs/promises';
+import { existsSync } from 'node:fs';
+import { join } from 'node:path';
+const CONFIG_TEMPLATE = `// stilltrue configuration — one entry per fact-set you want watched.
+// Docs: https://github.com/sg-bfoster/stilltrue#readme
+// Picking good markers: https://github.com/sg-bfoster/stilltrue/blob/main/docs/MARKERS.md
+import { defineStilltrue, corpus, json, surname } from 'stilltrue';
+
+export default defineStilltrue({
+  drift: [
+    {
+      // A short id for this check — shows up in output and reports.
+      name: 'example-board-roster',
+
+      // WHERE THE TRUTH LIVES: the official page(s) your facts came from.
+      // corpus() fetches each page and strips it down to readable text.
+      // If one page is unreachable it warns; only if ALL are down does the
+      // check report an error (which never fails your build).
+      source: corpus([
+        'https://example.org/board',
+        // 'https://example.org/about',   // add more pages if facts span several
+      ]),
+
+      // WHAT YOU BELIEVE: read your saved facts file and pick out the
+      // "markers" — short, stable strings that must still appear on the
+      // page(s) above. Here: each member's last name, plus one number.
+      expect: json('./data/example-facts.json', (data) => [
+        ...data.members.map((m) => surname(m.name)),
+        \`\${data.totalSchools} schools\`,
+      ]),
+
+      // HOW TO COMPARE: 'contains-all' = every marker must appear in the
+      // fetched text (case-insensitive). Any missing marker = rot = the
+      // build fails and tells you which fact went stale.
+      compare: 'contains-all',
+    },
+  ],
+});
+`;
+const DATA_TEMPLATE = `{
+  "_comment": "Example facts file - replace with your own data, or point the config at a file you already have.",
+  "members": [
+    { "name": "Dr. Jane Smith" },
+    { "name": "Robert Jones Jr." }
+  ],
+  "totalSchools": 12
+}
+`;
+/** Write starter config + example data file. Never overwrites. */
+export async function init(dir) {
+    const created = [];
+    const skipped = [];
+    const configPath = join(dir, 'stilltrue.config.mjs');
+    const existingConfig = [
+        'stilltrue.config.mjs',
+        'stilltrue.config.ts',
+        'stilltrue.config.mts',
+        'stilltrue.config.js',
+        'stilltrue.config.cjs',
+    ].find((name) => existsSync(join(dir, name)));
+    if (existingConfig) {
+        skipped.push(existingConfig);
+    }
+    else {
+        await writeFile(configPath, CONFIG_TEMPLATE);
+        created.push('stilltrue.config.mjs');
+    }
+    const dataPath = join(dir, 'data', 'example-facts.json');
+    if (existsSync(dataPath)) {
+        skipped.push('data/example-facts.json');
+    }
+    else {
+        await mkdir(join(dir, 'data'), { recursive: true });
+        await writeFile(dataPath, DATA_TEMPLATE);
+        created.push('data/example-facts.json');
+    }
+    return { created, skipped };
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "stilltrue",
-  "version": "0.4.0",
+  "version": "0.5.0",
   "description": "Your tests check your code. stilltrue checks your facts. CI checks for AI-in-the-loop apps: drift (curated facts vs live sources), golden (regression evals), verify (judge answers against sources).",
   "keywords": [
     "drift",
@@ -37,5 +37,13 @@
   },
   "engines": {
     "node": ">=20"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/sg-bfoster/stilltrue.git"
+  },
+  "homepage": "https://github.com/sg-bfoster/stilltrue#readme",
+  "bugs": {
+    "url": "https://github.com/sg-bfoster/stilltrue/issues"
   }
 }