@sifurai/cli 0.1.1 → 0.1.4

package/README.md

@@ -6,7 +6,7 @@ send the results back to Sifur.
 Use it when your app only runs on your team's servers, behind a firewall or VPN that Sifur can't
 reach. Sifur never deploys to your server, never holds VPN or server credentials, and never connects
 into your network: you deploy the ticket's branch, this tool runs the cases on your machine, and only
-the results and screenshots go to Sifur.
+the results, screenshots and a failed step's evidence (below) go to Sifur.
 
 ## Requirements
 
@@ -103,10 +103,37 @@ sifur accept AZA-12 --url https://test.example.internal
    for the result.
 5. For a case with no script, shows its steps and expected result and asks you for the result:
    passed, failed or blocked, with a note.
-6. Uploads the screenshots and results, prints a summary, and links to the ticket.
+6. After a case that failed or was blocked, asks whether the app or the test is wrong (below).
+7. Uploads the screenshots and results, prints a summary, and links to the ticket.
 
 Every case passed: the ticket moves on to approval. Any failed or blocked: it goes back to be fixed.
 
+### When a case fails
+
+When a UI step fails, the CLI also records what the screen showed at that moment, so Sifur can
+tell a wrong app from a wrong test:
+
+- the page's address, with secrets and the values of query keys like `token`, `password`, `key`,
+  `secret` or `session` replaced;
+- the visible text of the page and every frame it can read, at most 20,000 characters;
+- up to 40 buttons, links and fields on the screen, nearest the step's target first, each with its
+  role, its label and a selector for it;
+- how many frames couldn't be read, if any.
+
+Secrets are replaced with `[secret]` in all of it. It is kept with the results until they're
+uploaded, and nothing else from the page leaves your machine. If the screen can't be read, the
+result goes without it.
+
+After a case that failed or was blocked, UI or automated, the CLI asks:
+
+```
+Is the app wrong, or the test? [a]pp / [t]est / [s]kip:
+```
+
+`a` or `t` asks for an optional reason; `s` or Enter gives no answer. The answer goes to Sifur
+with the result, and Sifur takes it as given: `test` means the case is wrong and is rewritten,
+and it's never sent to the coding agent to change the code; `app` sends it to be fixed.
+
 `--url` is the root of your test server: the cases' paths are opened under it. The CLI warns when
 it has a path or a query string, since that's rarely what's wanted.
 
@@ -153,6 +180,53 @@ afterwards it removes them again (also on Ctrl-C), along with any folder it made
 `git status` is just as it was. A file of yours at the same path is set aside and put back; one
 that's already identical is left alone. A path outside the checkout is refused.
 
+## Record a case
+
+For a screen Sifur's agents can't see, record the case yourself in a browser:
+
+```sh
+sifur record AZA-12 TC-3                                  # record TC-3's steps again
+sifur record AZA-12 --new "The filter keeps the choice"   # add a case
+sifur record AZA-12 --new "…" --repository web            # when the project has more than one repository
+```
+
+Like `accept`, it uses the project's test server unless `--url` names another. The CLI:
+
+1. Downloads the ticket's cases and signs in to your test server with Sifur's sign-in (the steps
+   every UI case starts with), in a browser you don't see (`--headed` shows it). The sign-in's
+   secrets come from `SIFUR_SECRET_<NAME>`, or are asked for, as for `accept`.
+2. Opens Playwright's recorder in Chrome (or Edge), already signed in, on the page the sign-in
+   left it at. Do the case's steps, then check the result with the toolbar's **Assert text** or
+   **Assert visibility** on what the change shows. Close the window when you're done.
+3. Turns the recording into Sifur's steps and shows them. The sign-in isn't part of them: Sifur
+   puts it in front of every UI case.
+4. Asks `Upload these N steps? [y/N]`, and uploads them as the case's new version (or as a new
+   case). Sifur refuses a recording made against an older version of the case, or after the
+   ticket has moved on; record it again.
+
+A recording needs at least one check: what proves the case is yours to choose, so the CLI never
+adds one. Sifur replays opening a page, clicks (a double-click, check or uncheck counts as a
+click), typing into a field, key presses and choosing an option, on elements found by role, text,
+label, placeholder, test id or CSS, in the page or one of its frames. Anything else (hovering,
+uploading a file, dragging, the keyboard or mouse on their own, dialogs, popups and new tabs, a
+value or snapshot check) is refused, with each line and the reason, and nothing is uploaded.
+
+**Credentials.** What you type is never uploaded as typed where it's a credential. A value that
+matches a secret the CLI knows (the sign-in's, or any `SIFUR_SECRET_<NAME>`) becomes its
+placeholder, `{{NAME}}`. Anything else typed into a field whose selector or label looks like a
+password, email or username field, the CLI asks you to name a placeholder for (like
+`APP_PASSWORD`); when the case runs, it comes from `SIFUR_SECRET_<NAME>` or is asked for.
+The recorder only sees the field's selector, not its type, so check the steps it shows you before
+you upload them.
+
+The signed-in session and the recorder's raw output (which holds everything you typed) are kept in
+a private temporary folder (the session file with owner-only permissions, 0600) and deleted as
+soon as the recorder closes, whatever happens, Ctrl-C included.
+
+The session carries into the recorder through its cookies and storage (`localStorage` and
+IndexedDB). An app that keeps it only in `sessionStorage` opens signed out there, and can't be
+recorded this way yet.
+
 ## VPN: split tunnel or full tunnel
 
 With a **split-tunnel** VPN (only company traffic goes through it), one command does everything:
@@ -207,6 +281,8 @@ SIFUR_SECRET_ADMIN_PASSWORD='…' sifur accept AZA-12 --url https://test.example
 | `--download` / `--run` / `--upload` | The three steps for a full-tunnel VPN |
 | `--apply` | With `accept`: put Sifur's changes in your checkouts first, as `sifur apply` does |
 | `--revert` | With `apply`: take the changes back out |
+| `--new <title>` | With `record`: record a new case with this title |
+| `--repository <name>` | With `record --new`: the case's repository, when the project has more than one |
 | `--api <url>` | Sifur's API, for `login` (default `https://api.sifur.ai`, or `SIFUR_API_URL`). Not your test server: that's `--url` |
 
 ## Troubleshooting
@@ -224,6 +300,6 @@ certificate with `NODE_EXTRA_CA_CERTS=<root cert>`.
 
 | Code | Meaning |
 |---|---|
-| 0 | Every case passed (or `--download` finished) |
-| 1 | At least one case failed or was blocked |
+| 0 | Every case passed (or `--download` finished, or `record` uploaded the steps) |
+| 1 | At least one case failed or was blocked (or you chose not to upload a recording) |
 | 2 | An error: not signed in, the ticket isn't in acceptance testing, Sifur unreachable, no Chrome or Edge, … |

package/dist/accept.js

@@ -70,6 +70,20 @@ function judgeWith(io) {
         return { ok: false, note: note || undefined };
     };
 }
+export const VERDICT_EXPLAINED = "Your answer goes to Sifur with the result: 'test' means the case is wrong and is rewritten, never sent to the coding agent.";
+/** Whose fault a failed or blocked case is, in the engineer's view; nothing when they skip it. */
+async function askVerdict(io) {
+    const choices = { a: 'app', app: 'app', t: 'test', test: 'test', s: null, skip: null, '': null };
+    for (;;) {
+        const verdict = choices[(await io.ask('  Is the app wrong, or the test? [a]pp / [t]est / [s]kip: ')).trim().toLowerCase()];
+        if (verdict === null)
+            return undefined;
+        if (verdict) {
+            const note = await askNote(io, '  Why? (optional): ');
+            return note ? { verdict, note } : { verdict };
+        }
+    }
+}
 function filesFor(testCase, files, workdirs) {
     return files.filter((f) => f.repository === testCase.repository || (!testCase.repository && !workdirs[f.repository]));
 }
@@ -144,6 +158,7 @@ export async function runCases(acceptance, options, deps) {
     const runToken = newRunToken();
     let browser = null;
     const results = [];
+    let verdictExplained = false;
     try {
         for (const testCase of acceptance.cases) {
             io.out(`${testCase.id}  ${testCase.title}`);
@@ -151,7 +166,17 @@ export async function runCases(acceptance, options, deps) {
             const notes = [];
             let steps = [];
             let output;
-            if (testCase.check === 'ui' || testCase.check === 'both') {
+            let evidence;
+            const withoutLogin = secretUses([testCase])
+                .map((use) => use.name)
+                .filter((name) => secrets[name] === undefined);
+            if ((testCase.check === 'ui' || testCase.check === 'both') && withoutLogin.length > 0) {
+                const note = `Not run: no ${withoutLogin.join(' or ')} was given.`;
+                io.out(`  skipped: ${note}`);
+                statuses.push('skipped');
+                notes.push(note);
+            }
+            else if (testCase.check === 'ui' || testCase.check === 'both') {
                 if (scripted.includes(testCase)) {
                     browser ??= await (deps.launch ?? launchBrowser)({ headed: options.headed });
                     const ui = await runUiCase(browser, testCase, {
@@ -168,6 +193,7 @@ export async function runCases(acceptance, options, deps) {
                     statuses.push(ui.status);
                     notes.push(...ui.notes);
                     steps = ui.steps;
+                    evidence = ui.evidence;
                 }
                 else {
                     const answer = await askEngineer(io, testCase, 'No UI script');
@@ -214,6 +240,13 @@ export async function runCases(acceptance, options, deps) {
             }
             const status = combine(statuses);
             io.out(`  → ${status}`);
+            let engineerVerdict;
+            if (status === 'failed' || status === 'blocked') {
+                if (!verdictExplained)
+                    io.out(`  ${VERDICT_EXPLAINED}`);
+                verdictExplained = true;
+                engineerVerdict = await askVerdict(io);
+            }
             io.out();
             results.push({
                 caseUuid: testCase.uuid,
@@ -222,6 +255,8 @@ export async function runCases(acceptance, options, deps) {
                 notes: summarize(status, steps, notes),
                 steps,
                 ...(output !== undefined ? { output } : {}),
+                ...(evidence ? { evidence } : {}),
+                ...(engineerVerdict ? { engineerVerdict } : {}),
             });
         }
     }
@@ -284,7 +319,7 @@ export function printSummary(io, results) {
     for (const result of results)
         io.out(`  ${result.status.padEnd(7)}  ${result.id}`);
     const overall = outcome(results);
-    const counts = ['passed', 'failed', 'blocked']
+    const counts = ['passed', 'failed', 'blocked', 'skipped']
         .map((status) => [status, results.filter((r) => r.status === status).length])
         .filter(([, count]) => count > 0)
         .map(([status, count]) => `${count} ${status}`);
@@ -326,13 +361,6 @@ export async function accept(api, args, deps) {
         const results = await loadResults(dir);
         return finish(api, saved, results, dir, args, io);
     }
-    if (!args.url)
-        throw new CliError('--url <test url> is required: the test server the branch is deployed to.');
-    const baseUrl = parseBaseUrl(args.url);
-    const parsedUrl = new URL(baseUrl);
-    if (parsedUrl.pathname !== '/' || parsedUrl.search || parsedUrl.hash) {
-        io.out(`Note: the cases' paths are opened under ${baseUrl}, so the root of the site (${parsedUrl.origin}/) is usually what's wanted.`);
-    }
     let saved;
     let dir;
     if (args.mode === 'run') {
@@ -348,6 +376,18 @@ export async function accept(api, args, deps) {
         saved = await download(api, args.ref);
         dir = await saveCases(args.store, saved);
     }
+    // The project's test server unless --url names another, as for a server of the engineer's own.
+    const url = args.url ?? saved.acceptance.appUrl ?? null;
+    if (!url) {
+        throw new CliError('--url <test url> is required: the test server the branch is deployed to. Or set it once in the project\'s test environment settings.');
+    }
+    const baseUrl = parseBaseUrl(url);
+    if (!args.url)
+        io.out(`Testing ${baseUrl} (the project's test server). Use --url for another.`);
+    const parsedUrl = new URL(baseUrl);
+    if (parsedUrl.search || parsedUrl.hash) {
+        io.out(`Note: the cases' paths are opened under ${baseUrl}, without its query or fragment.`);
+    }
     describeDeploy(saved.acceptance, io);
     const { workdirs, chosen } = defaultWorkdirs(saved.acceptance.cases, args.workdirs ?? {}, gitTopLevel(args.cwd));
     if (chosen)

package/dist/browser.js

@@ -3,10 +3,14 @@ import { CliError } from './api.js';
 const CHANNELS = ['chrome', 'msedge'];
 /** The engineer's own Chrome, else Edge: playwright-core downloads no browser of its own. */
 export async function launchBrowser(options = {}) {
+    return (await launchChannel(options)).browser;
+}
+/** The browser, and which one it is, for a later process (the recorder) to open the same one. */
+export async function launchChannel(options = {}) {
     const failures = [];
     for (const channel of CHANNELS) {
         try {
-            return await chromium.launch({ channel, headless: !options.headed });
+            return { browser: await chromium.launch({ channel, headless: !options.headed }), channel };
         }
         catch (error) {
             failures.push(`${channel}: ${String(error.message ?? error).split('\n')[0]}`);

package/dist/cli.js

@@ -6,6 +6,7 @@ import { Api, ApiError, CliError, DEFAULT_API, normalizeApi } from './api.js';
 import { applyChanges, revertChanges } from './apply.js';
 import { cacheDir, loadCredentials } from './config.js';
 import { login, logout } from './login.js';
+import { recordCase } from './record.js';
 import { LEGACY_STORE } from './store.js';
 const USAGE = `Usage:
   sifur login [--api <url>]
@@ -16,6 +17,8 @@ const USAGE = `Usage:
   sifur accept <KEY|uuid> --download [--apply]
   sifur accept <KEY|uuid> --run --url <test url> [--headed] [--workdir <repo>=<dir> …]
   sifur accept <KEY|uuid> --upload
+  sifur record <KEY|uuid> <TC-n> [--url <test url>] [--headed]
+  sifur record <KEY|uuid> --new "<title>" [--repository <name>] [--url <test url>] [--headed]
 
 Options:
   --api <url>              Sifur's API (default ${DEFAULT_API}, or SIFUR_API_URL)
@@ -29,9 +32,11 @@ Options:
   --download               Save the cases to ${cacheDir()}/<KEY>/ (before a full-tunnel VPN is up)
   --run                    Run the saved cases and save the results (no connection to Sifur)
   --upload                 Send the saved results (after the VPN is down)
+  --new <title>            With record: record a new case with this title
+  --repository <name>      With record --new: the case's repository, when the project has more than one
 
 Secrets: {{NAME}} in a case comes from SIFUR_SECRET_NAME, or is asked for. It never leaves this machine.
-Exit codes: 0 every case passed, 1 any case failed or was blocked, 2 an error.`;
+Exit codes: 0 every case passed (record: uploaded), 1 any case failed or was blocked (record: not uploaded), 2 an error.`;
 function version() {
     try {
         const pkg = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8'));
@@ -75,6 +80,8 @@ export async function main(argv, io, env = process.env, cwd = process.cwd()) {
             upload: { type: 'boolean', default: false },
             apply: { type: 'boolean', default: false },
             revert: { type: 'boolean', default: false },
+            new: { type: 'string' },
+            repository: { type: 'string' },
             help: { type: 'boolean', short: 'h', default: false },
             version: { type: 'boolean', short: 'v', default: false },
         },
@@ -136,6 +143,22 @@ export async function main(argv, io, env = process.env, cwd = process.cwd()) {
                 consoleOrigin: env.SIFUR_CONSOLE_URL ?? credentials?.console ?? null,
             }, { io, env, timeoutMs: timeoutSeconds(values.timeout) * 1000 });
         }
+        case 'record': {
+            const [ref, caseId, ...extra] = rest;
+            if (!ref || extra.length > 0 || (caseId === undefined) === (values.new === undefined)) {
+                throw new CliError('Usage: sifur record <KEY|uuid> <TC-n>, or sifur record <KEY|uuid> --new "<title>"');
+            }
+            const credentials = await loadCredentials(env);
+            return recordCase(signedIn(credentials, explicitApi, userAgent), {
+                ref,
+                caseId,
+                title: values.new,
+                repository: values.repository,
+                url: values.url,
+                headed: values.headed,
+                consoleOrigin: env.SIFUR_CONSOLE_URL ?? credentials?.console ?? null,
+            }, { io, env, timeoutMs: timeoutSeconds(values.timeout) * 1000 });
+        }
         default:
             throw new CliError(`Unknown command "${command}".\n\n${USAGE}`);
     }