tewip-ai 0.55.0 → 0.57.0

package/README.md

@@ -15,13 +15,42 @@ It works for every stack: Node.js (node:test, Jest, Vitest, Mocha), Python (pyte
 | **Terminal** | `npm install -g tewip-ai`, then `tewip explain report.xml`, `tewip ci owner/repo --pr 12`, `tewip mcp` |
 | **VS Code, Cursor, Windsurf, VSCodium** | the Tewip extension ([download](https://github.com/hmamut39/tewip/releases/tag/vscode-0.5.0)): registers Tewip with the editor's AI assistant and adds six commands - explain a report, why CI failed on this branch, the risk review of your change, which flaky tests to quarantine, the security check, and what your corrections add up to |
 | **JetBrains, Claude, other AI assistants** | add the MCP server `tewip mcp`: fifteen tools, including *has this happened here before?*, *is this requirement covered and passing?* and *what should be done next about this goal?* (see [editor setup](#use-it-from-your-editor-any-ide-that-speaks-mcp)) |
-| **Any model** | OpenAI, Claude, Gemini, Azure, or a model on your own machine with Ollama (no account, nothing leaves the computer): `TEWIP_PROVIDER` |
+| **Any model, including none** | OpenAI, Claude, Gemini, Azure, or a model on your own machine with Ollama - no account, no key, and no source code leaving the building: `TEWIP_PROVIDER=ollama`. Three quarters of what Tewip answers calls no model at all. [How](#running-with-no-data-leaving-the-building) |
 | **GitHub CI** | the Tewip Action: a comment on every pull request, an optional merge gate, verified fix PRs ([QUICKSTART](QUICKSTART.md)) |
 
 Formerly *triage-agent*: the repository moved to `hmamut39/tewip`, and GitHub redirects the old address. See [ROADMAP.md](ROADMAP.md) for the plan.
 
 **Status: phases 1 to 14 are built; the two auto-fix phases (5 and 8) are the ones still in progress.** Tewip collects failures from any framework, diagnoses them with rules first and a model only for what rules cannot settle, and turns that into work for each role: a CI comment and merge gate, test plans and verified tests, a flake dossier, a PR-risk review, security scanning, tickets in Jira, Rally, GitHub Issues, Azure Boards or Linear, checks generated from a Figma design, specifications written for people, and - opt in, and only when proved - fixes to test or application code. It runs live in GitHub CI and answers questions from AI assistants through MCP. On fresh real Gutenberg CI failures, with the pipeline frozen before labelling, accuracy is 93.4%; across six real datasets it ranges from 86.1% to 99.7%, and the caveats are in [eval/README.md](eval/README.md).
 
+
+## Running with no data leaving the building
+
+A hospital, a bank or a government department usually cannot send source code to a hosted API, and
+that is most of who this is built for. Tewip runs against a model on the same machine:
+
+```bash
+# once
+curl -fsSL https://ollama.com/install.sh | sh     # or ollama.com for Windows and macOS
+ollama pull qwen3:8b
+
+# then
+export TEWIP_PROVIDER=ollama
+tewip done --diff main...HEAD
+```
+
+No key, no account, no network call off the machine. `TEWIP_BASE_URL` points the same path at
+LM Studio, vLLM, llama.cpp's server, or anything else that speaks the OpenAI shape.
+
+**What changes and what does not.** The deterministic layers - `vouch`, `schema`, `access`, `perf`,
+`standards` - call no model at any time, with any provider, and are unaffected. Only the parts that
+need judgement use one: explaining a failure, writing a test, drafting a plan. A smaller local model
+is less accurate at those than a frontier one, and Tewip says which layer decided every answer, so
+the difference is visible rather than hidden.
+
+A local model is also less likely to be *constrained* to the schema it was asked for: some backends
+accept the request and quietly do not enforce it. Tewip reads a reply that arrives inside a markdown
+fence or after a sentence of preamble, and still refuses one with no JSON in it at all.
+
 ## What it costs
 
 Most failures never reach a model. On 3,556 real CI failures from six public projects, the

package/dist/engine/llm/client.js

@@ -50,7 +50,7 @@ export function loadDotEnv() {
         return;
     }
 }
-export function openAiClient({ apiKey, model, reasoningEffort, baseUrl = 'https://api.openai.com/v1' }) {
+export function openAiClient({ apiKey, model, reasoningEffort, baseUrl = 'https://api.openai.com/v1', provider = 'OpenAI', hint = '' }) {
     return {
         model: reasoningEffort ? `${model} (reasoning ${reasoningEffort})` : model,
         async complete({ system, user, schemaName, schema }) {
@@ -75,8 +75,16 @@ export function openAiClient({ apiKey, model, reasoningEffort, baseUrl = 'https:
                     });
                 }
                 catch (error) {
-                    if (attempt >= MAX_ATTEMPTS)
-                        throw new Error(`OpenAI request failed: ${String(error)}`);
+                    if (attempt >= MAX_ATTEMPTS) {
+                        // "OpenAI request failed: TypeError: fetch failed" is what somebody running a model on
+                        // their own machine used to see when they had not started it - the wrong provider named,
+                        // and no mention of the one thing wrong. A local endpoint that refuses the connection is
+                        // not an outage, it is a server nobody started yet, and the hint says how.
+                        const local = /^https?:\/\/(127\.0\.0\.1|localhost|\[::1\])/.test(baseUrl);
+                        throw new Error(local
+                            ? `no model is answering at ${baseUrl}. Nothing is listening there, so ${provider} is probably not running${hint ? ` - ${hint}` : ''}`
+                            : `${provider} request failed: ${String(error)}`);
+                    }
                     await sleep(2000 * attempt);
                     continue;
                 }
@@ -93,7 +101,7 @@ export function openAiClient({ apiKey, model, reasoningEffort, baseUrl = 'https:
                 }
                 const detail = truncate(await res.text(), 300);
                 if ((res.status !== 429 && res.status < 500) || attempt >= MAX_ATTEMPTS)
-                    throw new Error(`OpenAI API ${res.status}: ${detail}`);
+                    throw new Error(`${provider} API ${res.status}: ${detail}`);
                 await sleep(2000 * attempt);
             }
         },

package/dist/engine/llm/json-from-model.js

@@ -0,0 +1,56 @@
+// The JSON a model meant to send, out of whatever it actually sent.
+//
+// OpenAI's `response_format: json_schema` with `strict: true` constrains generation, so the content
+// is a bare JSON object and `JSON.parse` is enough. That is the only provider this was ever run
+// against, and it made the parsing look simpler than it is.
+//
+// Phase 47 put the real client in front of a local OpenAI-compatible server - the path a hospital or
+// a bank has to use, because they cannot send source code to a hosted API - and two of the three
+// commonest local-model replies crashed it:
+//
+//     ```json\n{ ... }\n```          SyntaxError: Unexpected token '`'
+//     Here is the analysis:\n\n{...} SyntaxError: Unexpected token 'H'
+//
+// Self-hosted Ollama does constrain generation through llama.cpp's grammar decoder, but its own
+// documentation records backends that accept `json_schema` and quietly do not enforce it, and every
+// other local runner - LM Studio, vLLM, llama.cpp's own server - has its own answer. A fence and a
+// sentence of preamble are not a model misbehaving; they are what an unconstrained model does.
+//
+// This is extraction, not repair. Nothing is guessed and nothing is rewritten: the fence comes off,
+// the object between the outermost braces is taken, and anything that still will not parse throws
+// with what actually arrived. A model that replies with prose and no JSON is still an error.
+const FENCED = /^\s*```(?:json|jsonc|JSON)?\s*\n([\s\S]*?)\n?\s*```\s*$/;
+/** Parses a model's reply as JSON, tolerating a fence or a sentence in front of it. */
+export function jsonFromModel(raw) {
+    const attempts = [raw, ...candidates(raw)];
+    for (const text of attempts) {
+        try {
+            return JSON.parse(text);
+        }
+        catch {
+            continue;
+        }
+    }
+    throw new Error(`the model did not return JSON: ${raw.trim().slice(0, 200)}${raw.trim().length > 200 ? '…' : ''}`);
+}
+/** The shapes worth trying, in order of how sure we are about them. */
+function candidates(raw) {
+    const out = [];
+    const fenced = FENCED.exec(raw.trim());
+    if (fenced)
+        out.push(fenced[1]);
+    // The outermost braces, which is what a sentence of preamble leaves behind. Taken from the first
+    // `{` to the last `}` rather than by counting: a brace inside a string would defeat the counter,
+    // and if the slice is unbalanced `JSON.parse` says so and the next candidate is tried.
+    const source = fenced ? fenced[1] : raw;
+    const open = source.indexOf('{');
+    const close = source.lastIndexOf('}');
+    if (open !== -1 && close > open)
+        out.push(source.slice(open, close + 1));
+    // The same for an array, which `write` and `plan` can both legitimately return.
+    const openArray = source.indexOf('[');
+    const closeArray = source.lastIndexOf(']');
+    if (openArray !== -1 && closeArray > openArray && (open === -1 || openArray < open))
+        out.push(source.slice(openArray, closeArray + 1));
+    return out;
+}

package/dist/engine/llm/llm-diagnose.js

@@ -4,9 +4,10 @@ import { recordFailure } from './usage.js';
 import { ROOT_CAUSES } from '../../reporters/types.js';
 import { truncate } from '../signature.js';
 import { OUTPUT_SCHEMA, OUTPUT_SCHEMA_NAME, PROMPT_VERSION, SYSTEM_PROMPT } from './prompt.js';
+import { jsonFromModel } from './json-from-model.js';
 /** Validates the model's JSON. Throws on anything outside the contract. */
 export function parseLlmOutput(raw) {
-    const value = JSON.parse(raw);
+    const value = jsonFromModel(raw);
     if (!ROOT_CAUSES.includes(value.category))
         throw new Error(`category outside the taxonomy: ${String(value.category)}`);
     if (typeof value.confidence !== 'number' || !Number.isFinite(value.confidence))

package/dist/engine/llm/providers.js

@@ -45,7 +45,14 @@ export function modelChoiceFromEnv(purpose) {
 export function clientFor(choice) {
     if (choice.provider === 'anthropic')
         return anthropicClient({ apiKey: choice.apiKey, model: choice.model, baseUrl: choice.baseUrl });
-    return openAiClient({ apiKey: choice.apiKey || 'local', model: choice.model, reasoningEffort: choice.reasoningEffort, baseUrl: choice.baseUrl });
+    return openAiClient({
+        apiKey: choice.apiKey || 'local',
+        model: choice.model,
+        reasoningEffort: choice.reasoningEffort,
+        baseUrl: choice.baseUrl,
+        provider: choice.provider === 'custom' ? 'the configured endpoint' : choice.provider,
+        hint: PROVIDERS[choice.provider].hint,
+    });
 }
 /** What to tell someone who has no model configured, naming every way to get one. */
 export function noModelHelp() {

package/dist/engine/quality/plan.js

@@ -7,6 +7,7 @@ import { existsSync, readdirSync } from 'node:fs';
 import { join } from 'node:path';
 import { readJsonl } from '../../reporters/store.js';
 import { changeSummary, existingTests, readPages, relatedCodeSources, relatedTestSources } from './grounding.js';
+import { jsonFromModel } from '../llm/json-from-model.js';
 export const PLAN_PROMPT_VERSION = 'plan-4';
 /** Failures Tewip already diagnosed in this project (from `tewip explain` runs kept in .tewip). */
 function failureHistory(root, cap = 60) {
@@ -115,5 +116,5 @@ export function verifyCoverage(plan, facts) {
 }
 export async function writePlan(facts, model) {
     const raw = await model.complete({ system: SYSTEM, user: factsForModel(facts), schemaName: 'tewip_test_plan', schema: SCHEMA });
-    return verifyCoverage(JSON.parse(raw), facts);
+    return verifyCoverage(jsonFromModel(raw), facts);
 }

package/dist/engine/quality/write.js

@@ -4,6 +4,7 @@
 // When a draft fails, the model repairs it from the real error - or says the app looks wrong,
 // because weakening a test until it passes would hide exactly the bug it exists to catch.
 import { readFileSync } from 'node:fs';
+import { jsonFromModel } from '../llm/json-from-model.js';
 export const WRITE_PROMPT_VERSION = 'write-6';
 const RULES = `Rules:
 - Write in the given framework and language, in the style of the example test: the same imports, helpers, and way of opening pages (if it uses a base URL with relative paths, do the same).
@@ -48,7 +49,7 @@ function contextFor(ctx, scenario) {
 }
 export async function draftTest(model, ctx, scenario) {
     const raw = await model.complete({ system: DRAFT_SYSTEM, user: JSON.stringify(contextFor(ctx, scenario)), schemaName: 'tewip_test_draft', schema: DRAFT_SCHEMA });
-    return JSON.parse(raw);
+    return jsonFromModel(raw);
 }
 /** The first lines of a failure: which check failed and how, without the page or the call log. */
 export function failureHead(error) {
@@ -61,7 +62,7 @@ export async function repairTest(model, ctx, scenario, draft, error, earlier = [
     const repeated = earlier.length > 0 ? { earlierFailures: earlier, note: 'Your earlier repairs failed too. If the same check keeps failing because the page does something other than the scenario expects, the fault is "app".' } : {};
     const user = JSON.stringify({ ...contextFor(ctx, scenario), yourTest: draft.code, realError: error.slice(-10000), ...repeated });
     const raw = await model.complete({ system: REPAIR_SYSTEM, user, schemaName: 'tewip_test_repair', schema: REPAIR_SCHEMA });
-    return JSON.parse(raw);
+    return jsonFromModel(raw);
 }
 /**
  * The broken-on-purpose version of a test, or why none could be made. The piece to change must

package/dist/engine/vouch/assertions.js

@@ -68,6 +68,21 @@ const CHECKS = [
     // in laravel/framework's SupportStrTest came back as "Tewip would not judge" because the
     // `assertSame(...)` in them was being read as an unknown helper that might assert.
     /\$this\s*->\s*assert/,
+    // Laravel's fluent assertions, which is what a Laravel HTTP test actually asserts with:
+    //
+    //     $response = $this->get(route('api.v1.autocomplete.accounts'));
+    //     $response->assertUnauthorized();
+    //     $response->assertHeader('Content-Type', 'application/json');
+    //
+    // The pattern above only reads `$this->assert`, so every one of those was an unknown helper that
+    // might assert - and firefly-iii, a real finance application, came back **72% "Tewip would not
+    // judge"**. A capital after `assert` is what keeps this from matching an ordinary method: it is
+    // assertOk, assertStatus, assertJson, assertUnauthorized, never assertion() or asserts().
+    // `$response->`, not any `->`: Java's lambda arrow is also `->`, so the first version of this
+    // quietly moved thirteen Keycloak verdicts - `coseKey -> assertEcKey(coseKey, ...)` matched. It
+    // happened to be right about them, and that is not the point: a PHP fix had changed Java answers
+    // nobody had looked at. The corpus caught it on the next run, which is what it is for.
+    /\$\w+\s*->\s*assert[A-Z]\w*\s*\(/,
     /\b(?:self|static|Assert)\s*::\s*assert/,
     /\bexpect\s*\(.*\)\s*->\s*\w+/,
     // Rust. A #[should_panic] test asserts by panicking, so it can fail without an assert in it.
@@ -90,7 +105,8 @@ export function checksSomething(text) {
  */
 export function unfinishedExpectations(body) {
     const found = [];
-    for (const line of body.split(String.fromCharCode(10))) {
+    const lines = body.split(String.fromCharCode(10));
+    for (const [i, line] of lines.entries()) {
         const text = line.trim();
         if (!/^(await\s+)?expect(\.soft)?\s*\(/.test(text))
             continue;
@@ -98,6 +114,27 @@ export function unfinishedExpectations(body) {
         if (after === null)
             continue; // the call spans more lines; judged as part of the whole body
         const rest = after.trim();
+        // An expectation kept in a variable and finished on a later line. RSpec allows it, and
+        // discourse writes it:
+        //
+        //     preview =
+        //       expect(Onebox.preview(url).to_s)
+        //     preview.to match(/start=120/)
+        //
+        // `expect(...)` returns the target and `.to` is a method on it, so this is a finished
+        // expectation that happens to span three lines. Reading only the line the `expect` is on called
+        // it unfinished - "an expectation was written but no matcher was called on it" - about a test
+        // that checks two things.
+        if (rest === '' || rest === ';') {
+            // Only the continuation form, because that is the only one that reaches here: a line reading
+            // `preview = expect(x)` does not start with `expect(` and the loop above never looks at it.
+            const named = /([A-Za-z_$][\w$]*)\s*=\s*$/.exec(lines[i - 1] ?? '');
+            // Built by concatenation on purpose. Written as a template literal, `\b` is a backspace
+            // character and `\s` is the letter s, so the pattern silently matched nothing.
+            const finished = new RegExp('(^|[^\\w$.])' + named?.[1] + '\\s*\\.\\s*(to|not_to|to_not)\\b');
+            if (named && finished.test(body))
+                continue;
+        }
         if (rest === '' || rest === ';')
             found.push(text);
         // A matcher named but not called: `.toBeVisible;` or `.toBeVisible` at the end of a statement.

package/dist/engine/vouch/find-tests.js

@@ -193,7 +193,9 @@ const DECLARES = {
     rust: /#\s*\[\s*(?:test|tokio::test|async_std::test|rstest)\s*\]/g,
     // PHPUnit accepts three markers and real suites use all of them: the testFoo naming convention
     // (laravel/framework does this), the #[Test] attribute of PHPUnit 10, and the @test docblock.
-    php: /(?:#\s*\[\s*Test[^\]]*\]|\*\s*@test\b|(?:public|protected|private)?\s*function\s+test\w*\s*\()/g,
+    // `private`/`protected` dropped: PHPUnit collects public methods only, and a private helper whose
+    // name happens to start with "test" is not one. See tests-in.ts for the production file this cost.
+    php: /(?:#\s*\[\s*Test[^\]]*\]|\*\s*@test\b|(?:^|[^\w$>])(?:public\s+)?(?:static\s+)?function\s+test\w*\s*\()/gm,
 };
 /** JUnit, Kotlin, Scala, Go, C#, PHP and Rust: an annotation or a naming convention, then a braced body. */
 function bracedTests(language, text) {
@@ -281,6 +283,15 @@ function skipsAtTopLevel(body) {
     if (lines.length === 0)
         return false;
     const base = Math.min(...lines.map((l) => l.length - l.trimStart().length));
-    return lines.some((l) => l.length - l.trimStart().length === base && /^\s*(?:skip|pending)\b/.test(l));
+    // Not a `skip` carrying a trailing condition. Ruby writes a guard as a modifier -
+    //
+    //     skip "suggested topic not found in results" if suggested_topic.nil?
+    //
+    // - and that test runs every time the condition is false, which is most of the time. The Go side
+    // has refused conditional skips since it was written; Ruby's was added in Phase 36 without it, and
+    // discourse has four tests that run perfectly well being called "never runs".
+    return lines.some((l) => l.length - l.trimStart().length === base &&
+        /^\s*(?:skip|pending)\b/.test(l) &&
+        !/\s(?:if|unless)\s/.test(l));
 }
 export { jsTests, pythonTests, rubyTests, bracedTests };

package/dist/engine/vouch/tests-in.js

@@ -30,14 +30,47 @@ export function languageOf(file) {
     return null;
 }
 /** Does this file look like tests at all? Vouching for a source file is a category error. */
+/**
+ * Minified or vendored: a file nobody in this project wrote, and nobody is going to fix.
+ *
+ * firefly-iii ships `public/v1/lib/jquery-4.0.0.min.js`, and minified JavaScript is full of `test(`
+ * - it is how a regular expression is applied - so it read as a test file and was judged as one.
+ * Nothing useful can come of that: the author cannot act on it, and it is not their code.
+ *
+ * Length is the signal rather than the name, because plenty of vendored bundles are not called
+ * `.min.js`. A minified file is one enormous line; source is not.
+ */
+function minifiedOrVendored(file, text) {
+    if (/[.-]min\.(js|css|mjs)$/i.test(file))
+        return true;
+    if (/(^|[\/])(vendor|third[_-]?party|bundles?|dist)[\/]/i.test(file))
+        return true;
+    const lines = text.split(String.fromCharCode(10));
+    const longest = lines.reduce((most, line) => Math.max(most, line.length), 0);
+    return longest > 2000 && text.length / Math.max(lines.length, 1) > 400;
+}
 export function looksLikeTests(file, text) {
+    if (minifiedOrVendored(file, text))
+        return false;
     const named = /(^|[\\/])(tests?|spec|specs|__tests__|e2e|it|integration)[\\/]|[._-](test|tests|spec|specs|it)\.[a-z]+$|(^|[\\/])test_[^\\/]+\.py$|_test\.(go|py|rb)$|Tests?\.(java|kt|cs|scala)$/i.test(file);
     const declares = /\b(it|test|describe|Scenario)\s*\(|^\s*def\s+test_|@Test\b|\[(Test|Fact|Theory)\]|\bfunc\s+Test[A-Z]/m.test(text) ||
         // Ruby, PHP and Rust, added in Phase 36. PHPUnit names a test `testSomething`, so the `test\s*\(`
         // above never matched it, and Rust marks one with an attribute no pattern here knew. Both came
         // back as "0 test(s) read in 0 file(s)", which reads like a clean answer and is not one.
         /#\s*\[\s*(?:test|tokio::test|rstest)\s*\]/.test(text) ||
-        /\bfunction\s+test\w*\s*\(|#\s*\[\s*Test\b|\*\s*@test\b/.test(text) ||
+        // `public`, or no modifier at all: PHPUnit runs public methods and nothing else. Without that,
+        // firefly-iii's `private function testing()` - a middleware helper asking whether the app is in
+        // the test environment - made `app/Http/Middleware/InterestingMessage.php` a test file, and the
+        // whole of that production class was judged as tests. Vouching for source code is a different
+        // question, and this file says so three lines further down.
+        // A PHP method called `test*` is not a PHPUnit test. firefly-iii has eight public ones in
+        // production controllers - `testRule`, `testNotification` - which are endpoints a user hits to
+        // try a rule or send a test notification, and every one of them made its controller a test file.
+        // What makes a PHPUnit test is the class it is in, so that is what is asked for: a TestCase to
+        // extend, or the attribute and annotation forms that need no base class.
+        (/#\s*\[\s*Test\b|\*\s*@test\b/.test(text) ||
+            (/(?:^|[^\w$>])(?:public\s+)?(?:static\s+)?function\s+test\w*\s*\(/m.test(text) &&
+                /\bextends\s+\w*TestCase\b|\buse\s+PHPUnit\{2}Framework/.test(text))) ||
         /^\s*(?:RSpec\.)?(?:describe|context)\s+|^\s*(?:x|f)?it\s+["'].*\bdo\b/m.test(text);
     return named || declares;
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tewip-ai",
-  "version": "0.55.0",
+  "version": "0.57.0",
   "mcpName": "io.github.hmamut39/tewip",
   "description": "Tewip finds why each failing test failed - real bug, flaky test, environment, test data or changed selector - for any stack, from the command line, CI, or your editor.",
   "keywords": [