tewip-ai 0.54.0 → 0.56.0

package/README.md

@@ -15,13 +15,42 @@ It works for every stack: Node.js (node:test, Jest, Vitest, Mocha), Python (pyte
 | **Terminal** | `npm install -g tewip-ai`, then `tewip explain report.xml`, `tewip ci owner/repo --pr 12`, `tewip mcp` |
 | **VS Code, Cursor, Windsurf, VSCodium** | the Tewip extension ([download](https://github.com/hmamut39/tewip/releases/tag/vscode-0.5.0)): registers Tewip with the editor's AI assistant and adds six commands - explain a report, why CI failed on this branch, the risk review of your change, which flaky tests to quarantine, the security check, and what your corrections add up to |
 | **JetBrains, Claude, other AI assistants** | add the MCP server `tewip mcp`: fifteen tools, including *has this happened here before?*, *is this requirement covered and passing?* and *what should be done next about this goal?* (see [editor setup](#use-it-from-your-editor-any-ide-that-speaks-mcp)) |
-| **Any model** | OpenAI, Claude, Gemini, Azure, or a model on your own machine with Ollama (no account, nothing leaves the computer): `TEWIP_PROVIDER` |
+| **Any model, including none** | OpenAI, Claude, Gemini, Azure, or a model on your own machine with Ollama - no account, no key, and no source code leaving the building: `TEWIP_PROVIDER=ollama`. Three quarters of what Tewip answers calls no model at all. [How](#running-with-no-data-leaving-the-building) |
 | **GitHub CI** | the Tewip Action: a comment on every pull request, an optional merge gate, verified fix PRs ([QUICKSTART](QUICKSTART.md)) |
 
 Formerly *triage-agent*: the repository moved to `hmamut39/tewip`, and GitHub redirects the old address. See [ROADMAP.md](ROADMAP.md) for the plan.
 
 **Status: phases 1 to 14 are built; the two auto-fix phases (5 and 8) are the ones still in progress.** Tewip collects failures from any framework, diagnoses them with rules first and a model only for what rules cannot settle, and turns that into work for each role: a CI comment and merge gate, test plans and verified tests, a flake dossier, a PR-risk review, security scanning, tickets in Jira, Rally, GitHub Issues, Azure Boards or Linear, checks generated from a Figma design, specifications written for people, and - opt in, and only when proved - fixes to test or application code. It runs live in GitHub CI and answers questions from AI assistants through MCP. On fresh real Gutenberg CI failures, with the pipeline frozen before labelling, accuracy is 93.4%; across six real datasets it ranges from 86.1% to 99.7%, and the caveats are in [eval/README.md](eval/README.md).
 
+
+## Running with no data leaving the building
+
+A hospital, a bank or a government department usually cannot send source code to a hosted API, and
+that is most of who this is built for. Tewip runs against a model on the same machine:
+
+```bash
+# once
+curl -fsSL https://ollama.com/install.sh | sh     # or ollama.com for Windows and macOS
+ollama pull qwen3:8b
+
+# then
+export TEWIP_PROVIDER=ollama
+tewip done --diff main...HEAD
+```
+
+No key, no account, no network call off the machine. `TEWIP_BASE_URL` points the same path at
+LM Studio, vLLM, llama.cpp's server, or anything else that speaks the OpenAI shape.
+
+**What changes and what does not.** The deterministic layers - `vouch`, `schema`, `access`, `perf`,
+`standards` - call no model at any time, with any provider, and are unaffected. Only the parts that
+need judgement use one: explaining a failure, writing a test, drafting a plan. A smaller local model
+is less accurate at those than a frontier one, and Tewip says which layer decided every answer, so
+the difference is visible rather than hidden.
+
+A local model is also less likely to be *constrained* to the schema it was asked for: some backends
+accept the request and quietly do not enforce it. Tewip reads a reply that arrives inside a markdown
+fence or after a sentence of preamble, and still refuses one with no JSON in it at all.
+
 ## What it costs
 
 Most failures never reach a model. On 3,556 real CI failures from six public projects, the
@@ -1067,9 +1096,18 @@ that never run, and tests that compare no value at all. No model, no cost.
 It is also an MCP tool, so a coding agent can ask about the tests it just wrote before claiming a
 change is covered.
 
-**Measured on 5,448 tests in 8 public projects across 5 languages** (express, axios, chalk, requests,
-click, gorilla/mux, cobra, gson): 19 tests were called unable to fail, and **all 19 were correct** when
-read against the files - one unconditional skip in psf/requests, eighteen `@Ignore` methods in gson.
+**Measured on 41,523 tests in six public projects** - Keycloak, OpenMRS, bitwarden/server, Saleor,
+GOV.UK Frontend and Firefly III - in Java, C#, Python, PHP and TypeScript. Every accusation was read
+against its source by hand, and the counts are kept in [`corpus/expected.json`](corpus/expected.json)
+against a pinned commit, re-derived by `npm run corpus` on every release.
+
+**It has been badly wrong once, and that is worth saying plainly.** An earlier version of this
+paragraph reported 19 of 19 correct on 5,448 tests. Then Tewip was pointed at Keycloak and **called
+166 correct tests worthless** - every one of them the standard Java way of writing "I expect this to
+throw", which `Assert.fail()` makes fail properly because `AssertionError` is not an `Exception`.
+The rule had never read the caught type. 918 unit tests passed before that and after it, which is
+why the corpus exists: the thing that matters is what Tewip says about other people's code, and
+nothing was measuring it.
 
 Two things it refuses to do, which is why the list is worth reading:
 

package/dist/engine/llm/client.js

@@ -50,7 +50,7 @@ export function loadDotEnv() {
         return;
     }
 }
-export function openAiClient({ apiKey, model, reasoningEffort, baseUrl = 'https://api.openai.com/v1' }) {
+export function openAiClient({ apiKey, model, reasoningEffort, baseUrl = 'https://api.openai.com/v1', provider = 'OpenAI', hint = '' }) {
     return {
         model: reasoningEffort ? `${model} (reasoning ${reasoningEffort})` : model,
         async complete({ system, user, schemaName, schema }) {
@@ -75,8 +75,16 @@ export function openAiClient({ apiKey, model, reasoningEffort, baseUrl = 'https:
                     });
                 }
                 catch (error) {
-                    if (attempt >= MAX_ATTEMPTS)
-                        throw new Error(`OpenAI request failed: ${String(error)}`);
+                    if (attempt >= MAX_ATTEMPTS) {
+                        // "OpenAI request failed: TypeError: fetch failed" is what somebody running a model on
+                        // their own machine used to see when they had not started it - the wrong provider named,
+                        // and no mention of the one thing wrong. A local endpoint that refuses the connection is
+                        // not an outage, it is a server nobody started yet, and the hint says how.
+                        const local = /^https?:\/\/(127\.0\.0\.1|localhost|\[::1\])/.test(baseUrl);
+                        throw new Error(local
+                            ? `no model is answering at ${baseUrl}. Nothing is listening there, so ${provider} is probably not running${hint ? ` - ${hint}` : ''}`
+                            : `${provider} request failed: ${String(error)}`);
+                    }
                     await sleep(2000 * attempt);
                     continue;
                 }
@@ -93,7 +101,7 @@ export function openAiClient({ apiKey, model, reasoningEffort, baseUrl = 'https:
                 }
                 const detail = truncate(await res.text(), 300);
                 if ((res.status !== 429 && res.status < 500) || attempt >= MAX_ATTEMPTS)
-                    throw new Error(`OpenAI API ${res.status}: ${detail}`);
+                    throw new Error(`${provider} API ${res.status}: ${detail}`);
                 await sleep(2000 * attempt);
             }
         },

package/dist/engine/llm/json-from-model.js

@@ -0,0 +1,56 @@
+// The JSON a model meant to send, out of whatever it actually sent.
+//
+// OpenAI's `response_format: json_schema` with `strict: true` constrains generation, so the content
+// is a bare JSON object and `JSON.parse` is enough. That is the only provider this was ever run
+// against, and it made the parsing look simpler than it is.
+//
+// Phase 47 put the real client in front of a local OpenAI-compatible server - the path a hospital or
+// a bank has to use, because they cannot send source code to a hosted API - and two of the three
+// commonest local-model replies crashed it:
+//
+//     ```json\n{ ... }\n```          SyntaxError: Unexpected token '`'
+//     Here is the analysis:\n\n{...} SyntaxError: Unexpected token 'H'
+//
+// Self-hosted Ollama does constrain generation through llama.cpp's grammar decoder, but its own
+// documentation records backends that accept `json_schema` and quietly do not enforce it, and every
+// other local runner - LM Studio, vLLM, llama.cpp's own server - has its own answer. A fence and a
+// sentence of preamble are not a model misbehaving; they are what an unconstrained model does.
+//
+// This is extraction, not repair. Nothing is guessed and nothing is rewritten: the fence comes off,
+// the object between the outermost braces is taken, and anything that still will not parse throws
+// with what actually arrived. A model that replies with prose and no JSON is still an error.
+const FENCED = /^\s*```(?:json|jsonc|JSON)?\s*\n([\s\S]*?)\n?\s*```\s*$/;
+/** Parses a model's reply as JSON, tolerating a fence or a sentence in front of it. */
+export function jsonFromModel(raw) {
+    const attempts = [raw, ...candidates(raw)];
+    for (const text of attempts) {
+        try {
+            return JSON.parse(text);
+        }
+        catch {
+            continue;
+        }
+    }
+    throw new Error(`the model did not return JSON: ${raw.trim().slice(0, 200)}${raw.trim().length > 200 ? '…' : ''}`);
+}
+/** The shapes worth trying, in order of how sure we are about them. */
+function candidates(raw) {
+    const out = [];
+    const fenced = FENCED.exec(raw.trim());
+    if (fenced)
+        out.push(fenced[1]);
+    // The outermost braces, which is what a sentence of preamble leaves behind. Taken from the first
+    // `{` to the last `}` rather than by counting: a brace inside a string would defeat the counter,
+    // and if the slice is unbalanced `JSON.parse` says so and the next candidate is tried.
+    const source = fenced ? fenced[1] : raw;
+    const open = source.indexOf('{');
+    const close = source.lastIndexOf('}');
+    if (open !== -1 && close > open)
+        out.push(source.slice(open, close + 1));
+    // The same for an array, which `write` and `plan` can both legitimately return.
+    const openArray = source.indexOf('[');
+    const closeArray = source.lastIndexOf(']');
+    if (openArray !== -1 && closeArray > openArray && (open === -1 || openArray < open))
+        out.push(source.slice(openArray, closeArray + 1));
+    return out;
+}

package/dist/engine/llm/llm-diagnose.js

@@ -4,9 +4,10 @@ import { recordFailure } from './usage.js';
 import { ROOT_CAUSES } from '../../reporters/types.js';
 import { truncate } from '../signature.js';
 import { OUTPUT_SCHEMA, OUTPUT_SCHEMA_NAME, PROMPT_VERSION, SYSTEM_PROMPT } from './prompt.js';
+import { jsonFromModel } from './json-from-model.js';
 /** Validates the model's JSON. Throws on anything outside the contract. */
 export function parseLlmOutput(raw) {
-    const value = JSON.parse(raw);
+    const value = jsonFromModel(raw);
     if (!ROOT_CAUSES.includes(value.category))
         throw new Error(`category outside the taxonomy: ${String(value.category)}`);
     if (typeof value.confidence !== 'number' || !Number.isFinite(value.confidence))

package/dist/engine/llm/providers.js

@@ -45,7 +45,14 @@ export function modelChoiceFromEnv(purpose) {
 export function clientFor(choice) {
     if (choice.provider === 'anthropic')
         return anthropicClient({ apiKey: choice.apiKey, model: choice.model, baseUrl: choice.baseUrl });
-    return openAiClient({ apiKey: choice.apiKey || 'local', model: choice.model, reasoningEffort: choice.reasoningEffort, baseUrl: choice.baseUrl });
+    return openAiClient({
+        apiKey: choice.apiKey || 'local',
+        model: choice.model,
+        reasoningEffort: choice.reasoningEffort,
+        baseUrl: choice.baseUrl,
+        provider: choice.provider === 'custom' ? 'the configured endpoint' : choice.provider,
+        hint: PROVIDERS[choice.provider].hint,
+    });
 }
 /** What to tell someone who has no model configured, naming every way to get one. */
 export function noModelHelp() {

package/dist/engine/quality/plan.js

@@ -7,6 +7,7 @@ import { existsSync, readdirSync } from 'node:fs';
 import { join } from 'node:path';
 import { readJsonl } from '../../reporters/store.js';
 import { changeSummary, existingTests, readPages, relatedCodeSources, relatedTestSources } from './grounding.js';
+import { jsonFromModel } from '../llm/json-from-model.js';
 export const PLAN_PROMPT_VERSION = 'plan-4';
 /** Failures Tewip already diagnosed in this project (from `tewip explain` runs kept in .tewip). */
 function failureHistory(root, cap = 60) {
@@ -115,5 +116,5 @@ export function verifyCoverage(plan, facts) {
 }
 export async function writePlan(facts, model) {
     const raw = await model.complete({ system: SYSTEM, user: factsForModel(facts), schemaName: 'tewip_test_plan', schema: SCHEMA });
-    return verifyCoverage(JSON.parse(raw), facts);
+    return verifyCoverage(jsonFromModel(raw), facts);
 }

package/dist/engine/quality/write.js

@@ -4,6 +4,7 @@
 // When a draft fails, the model repairs it from the real error - or says the app looks wrong,
 // because weakening a test until it passes would hide exactly the bug it exists to catch.
 import { readFileSync } from 'node:fs';
+import { jsonFromModel } from '../llm/json-from-model.js';
 export const WRITE_PROMPT_VERSION = 'write-6';
 const RULES = `Rules:
 - Write in the given framework and language, in the style of the example test: the same imports, helpers, and way of opening pages (if it uses a base URL with relative paths, do the same).
@@ -48,7 +49,7 @@ function contextFor(ctx, scenario) {
 }
 export async function draftTest(model, ctx, scenario) {
     const raw = await model.complete({ system: DRAFT_SYSTEM, user: JSON.stringify(contextFor(ctx, scenario)), schemaName: 'tewip_test_draft', schema: DRAFT_SCHEMA });
-    return JSON.parse(raw);
+    return jsonFromModel(raw);
 }
 /** The first lines of a failure: which check failed and how, without the page or the call log. */
 export function failureHead(error) {
@@ -61,7 +62,7 @@ export async function repairTest(model, ctx, scenario, draft, error, earlier = [
     const repeated = earlier.length > 0 ? { earlierFailures: earlier, note: 'Your earlier repairs failed too. If the same check keeps failing because the page does something other than the scenario expects, the fault is "app".' } : {};
     const user = JSON.stringify({ ...contextFor(ctx, scenario), yourTest: draft.code, realError: error.slice(-10000), ...repeated });
     const raw = await model.complete({ system: REPAIR_SYSTEM, user, schemaName: 'tewip_test_repair', schema: REPAIR_SCHEMA });
-    return JSON.parse(raw);
+    return jsonFromModel(raw);
 }
 /**
  * The broken-on-purpose version of a test, or why none could be made. The piece to change must

package/dist/engine/verdict/verdict.js

@@ -25,11 +25,33 @@ export function decide(evidence) {
     if (evidence.changedSource.length === 0) {
         checks.push({ question: 'Is the changed code tested?', standing: 'not-applicable', answer: 'this change touches no source file', decidedBy: 'the diff' });
     }
+    else if (evidence.covering.length === 0 && evidence.changedTests.length > 0) {
+        // Nothing *imports* the changed file, and the same change ships tests. Coverage here is followed
+        // through imports, which is a fact rather than a name match - and which cannot see a test that
+        // reaches the code through a framework's entry point. A Saleor test posts a GraphQL query and
+        // asserts the error it gets back; the resolver chain in between is never imported by name.
+        //
+        // Found in Phase 46 by running this verdict over six real merged Saleor pull requests. Four came
+        // back **Not ready**, and all four shipped a test for exactly the change they were making. A gate
+        // that says "not ready" to a correct pull request is uninstalled within a week, and it is the
+        // same overclaim as a false accusation, pointed at the change instead of at a test.
+        //
+        // So: cannot say. Which is this project's whole point, and this check had no way to say it.
+        checks.push({
+            question: 'Is the changed code tested?',
+            standing: 'not-checked',
+            answer: `no test imports or reaches ${evidence.changedSource.slice(0, 3).join(', ')}, but this change also ` +
+                `edits ${evidence.changedTests.length} test file(s). Coverage is followed through imports, which cannot see a test ` +
+                'that reaches the code through a GraphQL resolver, an HTTP route or any other framework entry point - ' +
+                'so whether those tests cover this is a question the imports cannot answer',
+            decidedBy: 'imports followed through the source, which is a fact rather than a name match',
+        });
+    }
     else if (evidence.covering.length === 0) {
         checks.push({
             question: 'Is the changed code tested?',
             standing: 'not-met',
-            answer: `no test imports or reaches ${evidence.changedSource.slice(0, 3).join(', ')}, so nothing here could notice a mistake in it`,
+            answer: `no test imports or reaches ${evidence.changedSource.slice(0, 3).join(', ')}, and this change adds no test either, so nothing here could notice a mistake in it`,
             decidedBy: 'imports followed through the source, which is a fact rather than a name match',
         });
     }
@@ -50,7 +72,7 @@ export function decide(evidence) {
             question: 'Can the tests in this change fail?',
             standing: 'not-met',
             answer: `${evidence.cannotFail.length} of them cannot: ${evidence.cannotFail.slice(0, 2).map((t) => `${t.name} (${t.file}:${t.line})`).join(', ')}`,
-            decidedBy: 'tewip vouch, 19 of 19 correct on 5,448 tests in 8 public projects',
+            decidedBy: 'tewip vouch, measured on 41,523 real tests in the six public projects in corpus/expected.json',
         });
     }
     else {
@@ -120,7 +142,13 @@ export function decide(evidence) {
         headline: broken.length > 0
             ? `Not ready: ${broken[0].answer}`
             : verdict === 'cannot-say'
-                ? `Cannot say: ${essential.filter((c) => c.standing === 'not-checked').map((c) => c.question.toLowerCase().replace(/\?$/, '')).join(' and ')} could not be checked, so this is not an approval`
+                ? // Quoted, because the questions are questions: lowercasing them and joining with "and"
+                    // produced "is anything failing and is the changed code tested could not be checked",
+                    // which is the headline of the verdict this whole tool is named for.
+                    `Cannot say: ${essential.filter((c) => c.standing === 'not-checked').length === 1 ? 'one question that matters' : 'two questions that matter'} here went unanswered - ${essential
+                        .filter((c) => c.standing === 'not-checked')
+                        .map((c) => `"${c.question}"`)
+                        .join(' and ')} - so this is not an approval`
                 : `Ready as far as ${checks.filter((c) => c.standing === 'met').length} check(s) can show${unknown.length > 0 ? `, with ${unknown.length} not checked` : ''}`,
         checks,
         notChecked: [

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tewip-ai",
-  "version": "0.54.0",
+  "version": "0.56.0",
   "mcpName": "io.github.hmamut39/tewip",
   "description": "Tewip finds why each failing test failed - real bug, flaky test, environment, test data or changed selector - for any stack, from the command line, CI, or your editor.",
   "keywords": [