verifaied 0.30.0.dev77__tar.gz → 0.31.0.dev78__tar.gz

{verifaied-0.30.0.dev77 → verifaied-0.31.0.dev78}/.gitignore

@@ -25,6 +25,8 @@ frontend/coverage/
 # Environment
 **/.env
 **/.env.*
+# The e2e/bt stacks' template carries no secrets and is what the docs point at.
+!e2e/.env.example
 
 # Docker
 pgdata/
@@ -48,6 +50,9 @@ backend/coverage-e2e.json
 
 # Keys
 *.pem
+# ...except the throwaway GitHub App key the offline test stacks sign with
+# (paired with no App; see its header).
+!e2e/fixtures/github-app/private-key.pem
 
 TODO.txt
 TODO.md

{verifaied-0.30.0.dev77 → verifaied-0.31.0.dev78}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: verifaied
-Version: 0.30.0.dev77
+Version: 0.31.0.dev78
 Summary: Find what's untested in your code — locally, no account required
 Project-URL: Homepage, https://pypi.org/project/verifaied/
 Author: Kyle Richards
@@ -476,9 +476,16 @@ verifaied test run --all --env stage
 # skips the flow's Pause steps and the holds after its checks.
 verifaied test run "Settings/API tokens" --video watch
 
+# On a busy CI machine: replay a failed test once more, from a fresh reset.
+verifaied test run --all --retries 1
+
 # What's recorded, and how each last went on this branch.
 verifaied test list
 
+# Wrote or pulled a spec by hand? Give every spec on disk its row, its
+# chain and its flowchart. Nothing is deleted; --dry-run sends nothing.
+verifaied test sync
+
 # Or leave a recorder running and drive the whole thing from the web.
 verifaied test studio
 ```
@@ -719,6 +726,48 @@ recording of one flow — the composition happens in a throwaway file at run
 time, which is what makes editing `Auth/Log in` fix every test that starts
 by logging in. You can also pick pre-steps on the **New test** page.
 
+### Hand-written specs — `test sync`
+
+A spec doesn't have to come out of the recorder. Every spec names itself
+in a header above its import, and the recorder writes that header on
+every save:
+
+```ts
+// verifaied-test: Settings/API tokens/Create a token
+// verifaied-pre: Login/Paid user
+import { test, expect } from '@playwright/test';
+
+test('Create a token', async ({ page }) => {
+  await page.getByRole('link', { name: 'Settings' }).click();
+  // …
+});
+```
+
+`verifaied-test` is the test's `<feature path>/<name>`, once.
+`verifaied-pre` is a pre-step, one line each, in the order they run. Write
+the file yourself — or pull one somebody else wrote — and `verifaied test
+sync` makes verifAIed agree with it: the test is created if it is new,
+marked recorded, given its chain, and its flowchart is drawn on the
+detail page from the file.
+
+```bash
+verifaied test sync            # create and update rows from .verifaied/tests/
+verifaied test sync --dry-run  # say what would change, send nothing
+```
+
+Everything is checked before anything is sent, and one problem means
+nothing is sent at all: a spec with no header, a spec that isn't at the
+path its name gives (`.verifaied/tests/<feature path>/<name>.spec.ts`,
+slugged), a pre-step that isn't a spec on disk, a loop, or a test whose
+file verifAIed already has somewhere else — a renamed test keeps the file
+it was recorded to. Pre-steps are synced before the tests that name them,
+so one run is always enough. A test that already agrees with its file
+costs no request.
+
+Sync never deletes anything. A test whose spec isn't on disk is listed at
+the end — it may be on another branch, or renamed — so you can delete it
+on the web if it really is gone.
+
 ### Environments
 
 A recorded flow runs against a real app *somewhere*, and somewhere is the
@@ -1052,7 +1101,9 @@ replay and lays over the file.
 - `0` — recorded, or replayed cleanly
 - `*` — the replay's own exit code when the spec failed (`test run --all`
   reports the worst across the run)
-- `2` — the replay was clean but its final coverage upload failed
+- `2` — the replay was clean but its final coverage upload failed (coverage,
+  video and trace uploads are each tried three times, 2 s and 6 s apart, on a
+  dropped connection, a timeout or a 5xx — never on a 4xx)
 - `1` — usage/config error (no token, bad `--repo`, unknown test, no
   Playwright)
 
@@ -1082,10 +1133,24 @@ chain leaves the browser, or at `base_url` itself when there is no chain
 either),
 `--description <text>`, `--pre "<Feature>/<Name>"` (repeatable;
 replaces the stored chain — omit it to keep what the test already has),
-and `--no-replay` to skip the replay. `run` takes `--all` in place of a
-test name. `studio` takes `--no-replay` and takes no test name at all —
+and `--no-replay` to skip the replay. `run` takes `--all` or `--folder
+"<folder path>"` in place of a test name, `--video fast|watch|off`, and
+`--retries <n>` (default 0): a test whose replay **failed** is reset and
+replayed again, up to `n` more times, and the summary says `passed on
+retry 1 of 2` when a later attempt passes. Retries are for infrastructure
+— a sign-in that hung, a page slow to load on a loaded machine — not for a
+flow that is wrong, which fails every attempt. Each attempt is its own
+run; the **last** one is the verdict, the coverage and the video and trace
+the test keeps (a failed attempt that is retried keeps its verdict and its
+failure text but uploads no video or trace). A refusal — a missing spec, a
+variable this machine cannot supply, a reset that failed — is never
+retried, and neither is a clean run whose coverage upload failed (`2`). `studio` takes `--no-replay` and takes no test name at all —
 what it records is whatever the web asks for.
 
+`sync` takes `--repo`, `--branch` (the branch a new test is stamped with),
+`--root`, `--api-url`, `--token` and `--dry-run`. It exits `1` when a spec
+needs fixing (and sends nothing) and `2` when the API refuses a request.
+
 ## `verifaied skill install` — hand the loop to your agent
 
 The CLI ships the agent skill for using it. One command writes it where a

{verifaied-0.30.0.dev77 → verifaied-0.31.0.dev78}/README.md

@@ -430,9 +430,16 @@ verifaied test run --all --env stage
 # skips the flow's Pause steps and the holds after its checks.
 verifaied test run "Settings/API tokens" --video watch
 
+# On a busy CI machine: replay a failed test once more, from a fresh reset.
+verifaied test run --all --retries 1
+
 # What's recorded, and how each last went on this branch.
 verifaied test list
 
+# Wrote or pulled a spec by hand? Give every spec on disk its row, its
+# chain and its flowchart. Nothing is deleted; --dry-run sends nothing.
+verifaied test sync
+
 # Or leave a recorder running and drive the whole thing from the web.
 verifaied test studio
 ```
@@ -673,6 +680,48 @@ recording of one flow — the composition happens in a throwaway file at run
 time, which is what makes editing `Auth/Log in` fix every test that starts
 by logging in. You can also pick pre-steps on the **New test** page.
 
+### Hand-written specs — `test sync`
+
+A spec doesn't have to come out of the recorder. Every spec names itself
+in a header above its import, and the recorder writes that header on
+every save:
+
+```ts
+// verifaied-test: Settings/API tokens/Create a token
+// verifaied-pre: Login/Paid user
+import { test, expect } from '@playwright/test';
+
+test('Create a token', async ({ page }) => {
+  await page.getByRole('link', { name: 'Settings' }).click();
+  // …
+});
+```
+
+`verifaied-test` is the test's `<feature path>/<name>`, once.
+`verifaied-pre` is a pre-step, one line each, in the order they run. Write
+the file yourself — or pull one somebody else wrote — and `verifaied test
+sync` makes verifAIed agree with it: the test is created if it is new,
+marked recorded, given its chain, and its flowchart is drawn on the
+detail page from the file.
+
+```bash
+verifaied test sync            # create and update rows from .verifaied/tests/
+verifaied test sync --dry-run  # say what would change, send nothing
+```
+
+Everything is checked before anything is sent, and one problem means
+nothing is sent at all: a spec with no header, a spec that isn't at the
+path its name gives (`.verifaied/tests/<feature path>/<name>.spec.ts`,
+slugged), a pre-step that isn't a spec on disk, a loop, or a test whose
+file verifAIed already has somewhere else — a renamed test keeps the file
+it was recorded to. Pre-steps are synced before the tests that name them,
+so one run is always enough. A test that already agrees with its file
+costs no request.
+
+Sync never deletes anything. A test whose spec isn't on disk is listed at
+the end — it may be on another branch, or renamed — so you can delete it
+on the web if it really is gone.
+
 ### Environments
 
 A recorded flow runs against a real app *somewhere*, and somewhere is the
@@ -1006,7 +1055,9 @@ replay and lays over the file.
 - `0` — recorded, or replayed cleanly
 - `*` — the replay's own exit code when the spec failed (`test run --all`
   reports the worst across the run)
-- `2` — the replay was clean but its final coverage upload failed
+- `2` — the replay was clean but its final coverage upload failed (coverage,
+  video and trace uploads are each tried three times, 2 s and 6 s apart, on a
+  dropped connection, a timeout or a 5xx — never on a 4xx)
 - `1` — usage/config error (no token, bad `--repo`, unknown test, no
   Playwright)
 
@@ -1036,10 +1087,24 @@ chain leaves the browser, or at `base_url` itself when there is no chain
 either),
 `--description <text>`, `--pre "<Feature>/<Name>"` (repeatable;
 replaces the stored chain — omit it to keep what the test already has),
-and `--no-replay` to skip the replay. `run` takes `--all` in place of a
-test name. `studio` takes `--no-replay` and takes no test name at all —
+and `--no-replay` to skip the replay. `run` takes `--all` or `--folder
+"<folder path>"` in place of a test name, `--video fast|watch|off`, and
+`--retries <n>` (default 0): a test whose replay **failed** is reset and
+replayed again, up to `n` more times, and the summary says `passed on
+retry 1 of 2` when a later attempt passes. Retries are for infrastructure
+— a sign-in that hung, a page slow to load on a loaded machine — not for a
+flow that is wrong, which fails every attempt. Each attempt is its own
+run; the **last** one is the verdict, the coverage and the video and trace
+the test keeps (a failed attempt that is retried keeps its verdict and its
+failure text but uploads no video or trace). A refusal — a missing spec, a
+variable this machine cannot supply, a reset that failed — is never
+retried, and neither is a clean run whose coverage upload failed (`2`). `studio` takes `--no-replay` and takes no test name at all —
 what it records is whatever the web asks for.
 
+`sync` takes `--repo`, `--branch` (the branch a new test is stamped with),
+`--root`, `--api-url`, `--token` and `--dry-run`. It exits `1` when a spec
+needs fixing (and sends nothing) and `2` when the API refuses a request.
+
 ## `verifaied skill install` — hand the loop to your agent
 
 The CLI ships the agent skill for using it. One command writes it where a

{verifaied-0.30.0.dev77 → verifaied-0.31.0.dev78}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "verifaied"
-version = "0.30.0.dev77"
+version = "0.31.0.dev78"
 description = "Find what's untested in your code — locally, no account required"
 readme = "README.md"
 requires-python = ">=3.10"

{verifaied-0.30.0.dev77 → verifaied-0.31.0.dev78}/src/verifaied/cli.py

@@ -67,6 +67,7 @@ from verifaied.testcmd import (
     run_test_record,
     run_test_run,
     run_test_studio,
+    run_test_sync,
 )
 from verifaied.uploader import (
     UploaderError,
@@ -2053,6 +2054,16 @@ def test_run_command(
             "replay_type_delay_ms / replay_pauses for this run."
         ),
     ),
+    retries: int = typer.Option(
+        0,
+        "--retries",
+        min=0,
+        help=(
+            "Replay a test whose run failed up to this many more times, each "
+            "from a fresh reset. For flaky infrastructure; the last attempt "
+            "is the verdict and coverage recorded. Refusals are never retried."
+        ),
+    ),
 ) -> None:
     """Replay a recorded test, capturing video, trace and coverage.
 
@@ -2078,6 +2089,10 @@ def test_run_command(
     reset that fails stops the run; so does a variable this machine
     cannot supply, before anything is reset.
 
+    ``--retries N`` replays a test whose run failed up to N more times,
+    each from a fresh reset. The last attempt is the verdict and the
+    coverage the test keeps; a refusal is never retried.
+
     Exit codes:
 
       0  every test replayed cleanly
@@ -2109,6 +2124,7 @@ def test_run_command(
             proxy_target=proxy_target,
             proxy_port=proxy_port,
             video=video.value if video is not None else None,
+            retries=retries,
         )
     except (RecordedTestError, UploaderError) as e:
         err_console.print(f"[red]VerifAIed error[/red]: {e}")
@@ -2323,6 +2339,87 @@ def test_list_command(
         raise typer.Exit(code=2) from e
 
 
+@test_app.command("sync")
+def test_sync_command(
+    root: Path | None = typer.Option(
+        None,
+        "--root",
+        help="Repo root whose .verifaied/tests/ is synced (default: cwd).",
+    ),
+    dry_run: bool = typer.Option(
+        False,
+        "--dry-run",
+        help="Say what would be created and updated, and send nothing.",
+    ),
+    repo: str | None = typer.Option(
+        None,
+        "--repo",
+        "-r",
+        help=(
+            "Repository to sync. Accepts a UUID, an `owner/name` slug, or "
+            "omit to auto-detect from `git remote get-url origin`."
+        ),
+    ),
+    branch: str | None = typer.Option(
+        None,
+        "--branch",
+        "-b",
+        help=(
+            "Branch a new test is stamped with. Defaults to "
+            "`git branch --show-current`."
+        ),
+    ),
+    api_url: str | None = typer.Option(
+        None,
+        "--api-url",
+        help=f"Backend base URL. Defaults to ${ENV_API_URL} or {DEFAULT_API_URL}.",
+    ),
+    token: str | None = typer.Option(
+        None,
+        "--token",
+        help=f"API token (vr_live_...). Defaults to ${ENV_API_TOKEN}.",
+    ),
+) -> None:
+    """Create or update a test row for every spec under .verifaied/tests/.
+
+    Each spec names itself in a header above its import —
+    ``// verifaied-test: <feature path>/<name>`` — and any tests that
+    replay first, one ``// verifaied-pre: <ref>`` line each, in order. A
+    recorded spec carries the header already; a hand-written one gets its
+    row, its chain and its flowchart from this command. Everything is
+    checked before anything is sent, and a row that already agrees costs
+    no request. Rows are never deleted: one whose spec is missing is
+    listed for you to remove on the web.
+
+    Exit codes:
+
+      0  every spec is synced (or would be, with --dry-run)
+      1  a spec needs fixing first (nothing was sent), or a usage error
+      2  the API refused a request
+    """
+    resolved_url, resolved_token, repo_id = _test_prelude(
+        repo=repo, api_url=api_url, token=token
+    )
+    try:
+        run_test_sync(
+            api_url=resolved_url,
+            token=resolved_token,
+            repo_id=repo_id,
+            root=root or Path.cwd(),
+            branch=branch,
+            dry_run=dry_run,
+            console=console,
+        )
+    except (RecordedTestError, UploaderError) as e:
+        err_console.print(f"[red]VerifAIed error[/red]: {e}")
+        raise typer.Exit(code=1) from e
+    except ApiError as e:
+        err_console.print(
+            f"[red]VerifAIed error[/red]: {_api_error_text(e, resolved_url)}"
+        )
+        raise typer.Exit(code=2) from e
+
+
 @var_app.command("set")
 def var_set_command(
     name: str = typer.Argument(

{verifaied-0.30.0.dev77 → verifaied-0.31.0.dev78}/src/verifaied/client.py

@@ -7,6 +7,8 @@ can be swapped if/when we add a streaming variant for huge uploads.
 
 from __future__ import annotations
 
+import time
+from collections.abc import Callable
 from dataclasses import dataclass
 from pathlib import Path
 from typing import Any
@@ -16,6 +18,36 @@ import httpx
 
 from verifaied.uploader import Payload
 
+# Seconds to wait before the second and third attempt of an upload. A
+# replay's evidence is posted once, at the end, over whatever network the
+# machine has — a laptop roaming between access points, a CI runner
+# sharing one uplink with five others — and one dropped connection there
+# would otherwise turn a passing run into exit 2.
+UPLOAD_BACKOFF: tuple[float, ...] = (2.0, 6.0)
+# Indirected so tests can skip the waits.
+_sleep: Callable[[float], None] = time.sleep
+
+
+def _send_with_retry(send: Callable[[], httpx.Response]) -> httpx.Response:
+    """Send an upload, retrying what a retry can fix.
+
+    A connection that failed or timed out, or a 5xx, is tried again after
+    each :data:`UPLOAD_BACKOFF` delay; the last attempt's answer — or its
+    exception — is what the caller sees. A 4xx is returned at once: the
+    request itself is wrong (a bad token, a payload too big) and sending
+    it again only repeats the refusal.
+    """
+    for delay in UPLOAD_BACKOFF:
+        try:
+            response = send()
+        except httpx.TransportError:
+            _sleep(delay)
+            continue
+        if response.status_code < 500:
+            return response
+        _sleep(delay)
+    return send()
+
 
 class RepoNotFoundError(Exception):
     """The owner/name slug isn't connected to verifAIed under this
@@ -72,7 +104,8 @@ def upload(
 
     Times out at 60s by default — the synchronous endpoint persists rows
     and re-runs the analyzer inline, so very large repos may push past
-    the default httpx 5s. Callers can override for CI use.
+    the default httpx 5s. Callers can override for CI use. A dropped
+    connection, a timeout or a 5xx is tried twice more, after 2s and 6s.
 
     ``session_id`` attaches the upload to a session in the evidence ledger;
     ``driver`` says who was at the wheel for the implicit session a
@@ -101,7 +134,9 @@ def upload(
     headers = {"Authorization": f"Bearer {token}"}
 
     try:
-        response = httpx.post(url, json=body, headers=headers, timeout=timeout)
+        response = _send_with_retry(
+            lambda: httpx.post(url, json=body, headers=headers, timeout=timeout)
+        )
     except httpx.HTTPError as e:
         raise ApiError(0, f"could not reach {url}: {e}") from e
 
@@ -292,21 +327,27 @@ def upload_artifact(
 
     Streamed as multipart with a generous timeout: a trace archive is tens
     of megabytes and the server hashes it on the way through, so the
-    default 60s used elsewhere is too tight to be safe.
+    default 60s used elsewhere is too tight to be safe. Retried like
+    :func:`upload` on a dropped connection, a timeout or a 5xx.
     """
     url = f"{api_url.rstrip('/')}/sessions/{session_id}/artifacts"
     data: dict[str, str] = {"kind": kind}
     if label is not None:
         data["label"] = label
-    try:
+
+    def send() -> httpx.Response:
+        # Reopened per attempt: a retry has to stream the file from the top.
         with path.open("rb") as handle:
-            response = httpx.post(
+            return httpx.post(
                 url,
                 data=data,
                 files={"file": (path.name, handle, _media_type(path))},
                 headers={"Authorization": f"Bearer {token}"},
                 timeout=timeout,
             )
+
+    try:
+        response = _send_with_retry(send)
     except OSError as e:
         raise ApiError(0, f"could not read {path}: {e}") from e
     except httpx.HTTPError as e:

{verifaied-0.30.0.dev77 → verifaied-0.31.0.dev78}/src/verifaied/skill/reference/commands.md

@@ -135,6 +135,12 @@ a normal `@playwright/test` file: read it, edit it, commit it.
   steps, then rebuild from there; `--from 0` starts again, `--from <step
   count>` appends
 - `verifaied test studio` — leave a recorder running for the web UI to drive
+- `verifaied test sync` — create or update the row for every spec under
+  `.verifaied/tests/`, read off each spec's header (`// verifaied-test:
+  <ref>` and one `// verifaied-pre: <ref>` per pre-step, above the import).
+  How a hand-written spec gets its row, chain and flowchart. Checks every
+  spec before sending anything, never deletes a row, and `--dry-run` sends
+  nothing
 
 `--env <name>` points any of them at another environment from
 `.verifaied/config.toml`; `--no-replay` skips the replay that a recording