stilltrue 0.3.0 → 0.4.1

package/README.md

@@ -2,105 +2,152 @@
 
 > Your tests check your code. **stilltrue** checks your facts.
 
-AI-in-the-loop apps quietly start lying three ways: code changes break old
-answers, the world changes and curated facts rot, and the model embellishes
-beyond its sources. stilltrue is one watchdog per failure mode, run from CI —
-no server, no dashboard, no SaaS.
-
-| Command | Watchdog | Status |
-|---|---|---|
-| `npx stilltrue drift` | Curated facts vs their live authoritative sources — fails the build on rot | **v0.1 focus** |
-| `npx stilltrue golden` | Deterministic regression evals for answers | later (may wrap promptfoo) |
-| `verify` (library) | Staged acceptance pipeline for AI output — judge answers against sources in the request path | **shipped 0.3.0** |
-
-**Why drift first:** "drift detection" in the market means statistical ML
-feature drift; website-change monitors alert humans, not CI. Nobody ships
-*"assert my curated facts still match their live source, as a CI check."*
-This was built bespoke three times across three domains before becoming a
-package — see [docs/BRIEF.md](docs/BRIEF.md) for the full design brief,
-positioning, and build plan.
-
-## Core principle
-
-**AI may extract, but must never decide.** The default drift path has zero AI:
-fetch → parse → compare → diff. A consumer's `source()` may use an LLM to pull
-structure out of a messy page, but the comparison and the verdict are always
-deterministic. Same inputs, same verdict — a nondeterministic alarm trains
-people to ignore it.
-
-Drift has three outcomes, never two: `pass`, `rot` (facts changed — fail the
-build), and `error` (source unreachable — warn, don't fail). Conflating the
-last two causes alarm fatigue.
-
-## Quick start
-
-```ts
-// stilltrue.config.ts (or .mjs / .js)
+## The problem
+
+Say you've built an app or chatbot that answers questions using facts you
+collected — a list of school board members, office hours, tax rates,
+whatever. You checked those facts when you wrote them down. But the world
+keeps moving: a board member resigns, a rate changes, a first day of school
+rolls over. **Your app doesn't know.** It keeps confidently repeating
+yesterday's truth, and nothing alerts you.
+
+stilltrue is the alarm. Once a week (or whenever you like), it re-reads the
+official web pages your facts came from and checks that every fact you
+recorded still appears there. If something vanished, your build fails and
+tells you exactly which fact went stale — so a human can go verify and
+update it.
+
+## What you get
+
+| Command | What it does |
+|---|---|
+| `npx stilltrue drift` | Checks your saved facts against their live sources. Fails loudly if a fact has gone stale. |
+| `npx stilltrue report` | Turns past check runs into a simple web page — green/red grid, what broke and when. |
+| `verify` | For developers: a checkpoint that inspects AI-generated answers before they're shown to users. |
+
+## Getting started
+
+You'll need [Node.js](https://nodejs.org) (version 20 or newer) and a
+project folder with your facts saved in a JSON file.
+
+**1. Install it** — in your project folder, run:
+
+```bash
+npm install --save-dev stilltrue
+```
+
+**2. Describe a check** — create a file called `stilltrue.config.mjs` in
+the same folder. Each check answers three questions: *where do I fetch the
+truth from* (`source`), *what do I believe* (`expect`), and *how do I
+compare the two* (`compare`):
+
+```js
 import { defineStilltrue, corpus, json, surname } from 'stilltrue';
 
 export default defineStilltrue({
   drift: [
     {
       name: 'school-board-roster',
-      source: corpus(['https://district.example.org/board']),
-      expect: json('./data/school-board.json', (d) => d.members.map((m) => surname(m.name))),
+      // Where the truth lives: the district's own pages.
+      source: corpus(['https://www.myschooldistrict.org/board']),
+      // What you believe: read your saved file, keep just the last names.
+      expect: json('./data/school-board.json', (data) =>
+        data.members.map((m) => surname(m.name))
+      ),
+      // How to compare: every name must still appear on those pages.
       compare: 'contains-all',
     },
   ],
 });
 ```
 
+This example says: *fetch the board page, and make sure every last name in
+my `school-board.json` file still appears somewhere on it.*
+
+**3. Run it:**
+
+```bash
+npx stilltrue drift
+```
+
+You'll see one line per check:
+
+```
+✓ school-board-roster — pass (1204ms)
+
+1 pass, 0 rot, 0 error (errors warn, never fail)
+```
+
+## The three results (and why there are three, not two)
+
+- **pass** — every saved fact still appears at its source. All good.
+- **rot** — a fact is gone from the source. The world changed; your data is
+  stale. This *fails the run* so you notice and fix the data.
+- **error** — the website couldn't be reached (down, timeout, blocked).
+  This only *warns*. An unreachable site doesn't mean your facts are wrong,
+  and an alarm that cries wolf gets ignored — so outages never fail the build.
+
+## Running it automatically every week
+
+If your project is on GitHub, add this file as
+`.github/workflows/drift.yml` and GitHub will run the check every Wednesday
+and email you if anything rotted:
+
 ```yaml
-# .github/workflows/drift.yml
+name: Fact drift check
 on:
-  schedule: [{ cron: '17 14 * * 3' }]  # off-beat minute: GitHub drops :00 crons
+  schedule:
+    - cron: '17 14 * * 3'   # Wednesdays, ~10am US Eastern
+  workflow_dispatch:         # adds a manual "Run workflow" button
 jobs:
   drift:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
+      - uses: actions/checkout@v5
+      - uses: actions/setup-node@v5
         with: { node-version: '22' }
       - run: npm ci
       - run: npx stilltrue drift
 ```
 
-`source` is your code (fetch+parse, or LLM extraction — the comparison stays
-deterministic either way). `expect` is your curated file. Rot fails the run
-with one message per vanished fact; unreachable sources only warn.
+## The report
 
-## Verify — staged acceptance for AI output
+Add `--record` to save each run, then generate the web page:
 
-An ordered, short-circuiting pipeline: each stage is a pluggable judge — a
-Zod parse, a deterministic validator, or an LLM. Rejection is structured
-(`{ stage, messages[] }`, one issue per entry) and designed to be fed back to
-the generator for a retry. Stages that *throw* are infrastructure failures,
-not verdicts: `failPolicy: 'open'` (default) warns and continues — a judge
-outage must never block your app.
+```bash
+npx stilltrue drift --record
+npx stilltrue report
+```
 
-```ts
-import { runVerify, zodStage, generateVerified, formatRejection } from 'stilltrue';
+That writes `stilltrue-report.html` — open it in any browser. It shows a
+timeline grid (one row per check, one column per run, green ✓ / red ✗ /
+amber ⚠) and a log of everything that ever rotted, with details.
 
-const stages = [
-  zodStage('parse_schema', AnswerSchema),
-  { name: 'grounding', timeoutMs: 10_000, check: llmJudge },     // may return a revision
-  { name: 'chase-sim', tier: 'ci', check: expensiveSimulation }, // skipped inline
-];
+## One rule underneath everything
 
-// Request path:
-const result = await runVerify(stages, draft);
-publish(result.value); // draft, parsed draft, or the judge's rewrite
+**AI may extract, but never decides.** The drift check itself uses no AI at
+all — it fetches, reads, and compares text, the same way every time, with
+no API key and no cost. Deterministic alarms are trustworthy alarms. (If
+*your* fact-gathering step wants to use AI to pull structure out of a messy
+page, that's fine — it happens inside your `source` function, and the
+comparison stays mechanical.)
 
-// Generator loop (one automatic retry, then fall back — don't loop forever):
-const gen = await generateVerified({ generate: (feedback) => ask(model, prompt, feedback && formatRejection(feedback)), stages });
-```
+## For developers: `verify`
+
+`verify` is a library for teams shipping AI-generated answers: an ordered
+series of checkpoints (a schema parse, a rules check, an AI judge — your
+choice) that each answer must pass before publishing. Rejections say
+exactly which checkpoint failed and why, in a format you can feed back to
+the model for a retry. A broken checker never blocks your app — it fails
+open. See the [project docs](docs/BRIEF.md) and the typed API in
+`stilltrue`'s exports (`runVerify`, `zodStage`, `generateVerified`).
 
-## Status
+## What about testing prompts?
 
-v0.3 — `drift` and `report` running in production CI (AskGwinnett); `verify`
-shipped as a library. `golden` remains open: wrap promptfoo or defer to it
-(see the brief's competitive research).
+That's a different job with a great existing tool:
+[promptfoo](https://promptfoo.dev) tests your prompts; stilltrue tests your
+facts. How they fit together: [docs/GOLDEN.md](docs/GOLDEN.md).
 
 ## License
 
-MIT
+MIT — free to use, modify, and share.

package/dist/cli.js

@@ -95,7 +95,8 @@ try {
             process.exit(await report(rest));
             break;
         case 'golden':
-            console.error('stilltrue golden: not implemented yet — drift shipped first (docs/BRIEF.md)');
+            console.error('stilltrue deliberately has no golden runner — promptfoo tests your prompts; stilltrue tests your facts.\n' +
+                'Use https://promptfoo.dev for prompt/answer regression evals. Doctrine + pairing guide: docs/GOLDEN.md');
             process.exit(1);
             break;
         default:

package/dist/config.d.ts

@@ -58,8 +58,6 @@ export interface StilltrueConfig {
     drift?: DriftCheck<any, any>[];
     /** Named verify pipelines (see src/verify.ts) — usable from library code. */
     verify?: Record<string, import('./verify.ts').VerifyStage<any>[]>;
-    /** golden is a later phase — see docs/BRIEF.md build plan. */
-    golden?: unknown;
 }
 /** Identity function for typed configs: `export default defineStilltrue({...})`. */
 export declare function defineStilltrue(config: StilltrueConfig): StilltrueConfig;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "stilltrue",
-  "version": "0.3.0",
+  "version": "0.4.1",
   "description": "Your tests check your code. stilltrue checks your facts. CI checks for AI-in-the-loop apps: drift (curated facts vs live sources), golden (regression evals), verify (judge answers against sources).",
   "keywords": [
     "drift",
@@ -37,5 +37,13 @@
   },
   "engines": {
     "node": ">=20"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/sg-bfoster/stilltrue.git"
+  },
+  "homepage": "https://github.com/sg-bfoster/stilltrue#readme",
+  "bugs": {
+    "url": "https://github.com/sg-bfoster/stilltrue/issues"
   }
 }