ordel-cli 0.4.0__tar.gz → 0.4.2__tar.gz

{ordel_cli-0.4.0 → ordel_cli-0.4.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ordel-cli
-Version: 0.4.0
+Version: 0.4.2
 Summary: Ordel free CLI — local QA-automation for individual devs. Drives the deterministic ordel-engine and exposes it to a BYO coding agent over MCP. No account, no DB, no Ordel LLM.
 Project-URL: Homepage, https://ordel.io
 Author: Ordel
@@ -14,8 +14,8 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Topic :: Software Development :: Testing
 Requires-Python: >=3.11
 Requires-Dist: mcp<2.0,>=1.0
-Requires-Dist: ordel-engine==0.4.0
-Requires-Dist: ordel-tools==0.4.0
+Requires-Dist: ordel-engine==0.4.2
+Requires-Dist: ordel-tools==0.4.2
 Requires-Dist: typer>=0.12
 Provides-Extra: dev
 Requires-Dist: hatchling; extra == 'dev'

{ordel_cli-0.4.0 → ordel_cli-0.4.2}/ordel_cli/SKILL.md

@@ -68,7 +68,7 @@ failure means. Three rules hold throughout:
 | `pass` | ran, and its assertions (listed in `evidence.checks`) held | passed those checks on that target |
 | `fail` | ran and failed; `message` says why | a failure to triage: app, expectation, test, data or environment |
 | `flaky` | failed, then passed on retry | unstable; never a stable pass |
-| `unjudged` | green in Playwright but NOT proven: no assertion ran, no evidence was captured, or it depends on a heal no human has reviewed (`reasons` says which) | **not proven**: never "passed" |
+| `unjudged` | green in Playwright but NOT proven: no assertion ran, no evidence was captured, it depends on a heal no human has reviewed, or what it found after a sign-in a visitor who never signed in sees too (`reasons` says which) | **not proven**: never "passed" |
 | `skipped` | did not run | skipped, and why |
 
 A run whose `status` is `no_tests` checked nothing, and `error` means the runner or its setup
@@ -80,15 +80,21 @@ failed: neither is "all green". Before you call a new test done, run it and conf
 These are where agents most often go wrong. The references expand on them.
 
 - **Identity is an expectation, not an observation.** For a flow that signs in,
-  `generate_scenarios` adds an identity check (the signed-in page names the account used) and a
+  `generate_scenarios` adds an identity check: the sign-in took effect (its form is gone) and
+  the signed-in page names the account used. A name that a visitor who never signed in also
+  sees at that address proves no session, so the check reports `unjudged`: assert an element
+  only a signed-in page shows. When sign-in lands somewhere other than the site root, it adds a
   signed-out check (a visitor who never signed in is not served that page). If identity fails
   because the app shows a display name, set THIS account's name from the user or the
   requirements; never copy whatever name the page shows ("Welcome back, Dana Smith" for user
   `agent` is a finding, not the expected value).
-- **A rejection needs the app's own rejection.** A negative test passes on an error message, a
-  field marked invalid, a 4xx, or staying on the form; a 5xx or an error page is a crash, which
-  is a finding, never a correct rejection. The browser blocking a form is not the server
-  validating it.
+- **A rejection needs the app's own rejection, and its effect.** A negative test passes on an
+  error message, a field marked invalid, a 4xx, or staying on the form; a 5xx or an error page
+  is a crash, which is a finding, never a correct rejection. Staying on the form is not enough
+  for a sign-in: rejected means no session, so the signed-in page must stay closed, or, when
+  any visitor may open it, must not name the account (the generated sign-in negatives check
+  this). A denial that crashes (5xx) is not a denial. The
+  browser blocking a form is not the server validating it.
 - **Only confirmed rules are defects.** A failure against a rule you inferred (a format you
   expected, a limit you guessed) is a question for the user, listed separately from defects.
 - **Never silence a failure.** Do not `test.skip`, delete an assertion or widen a pattern to get
@@ -98,7 +104,14 @@ These are where agents most often go wrong. The references expand on them.
   add it to `.ordel/secrets.env` (gitignored; `run_test` loads it). If the user is not
   available and the app itself shows the credential publicly (a demo account printed on its
   sign-in page), you may add it there yourself and say so. Never put a secret in a spec,
-  another file or the chat, and never print it in a report.
+  another file or the chat, and never print it in a report. `run_test` lists what it finds:
+  `secret_leaks` (a secrets-file value in a file) and `hardcoded_secrets` (a spec line that
+  types a literal into a password field); move those to `.ordel/secrets.env`.
+  For interactive fills, pass `fill(env="ORDEL_SECRET_AGENT_PASSWORD", testid="password-input")`
+  instead of a literal `value`. The server resolves the variable from its environment or
+  `.ordel/secrets.env`; never read the value into an agent-visible command or tool argument.
+  For recording, an env-only fill step (`{"op": "fill", "testid": "password-input",
+  "env": "ORDEL_SECRET_AGENT_PASSWORD"}`) resolves on the server too. Leave out `value`.
 - **Coverage is not correctness.** `coverage_report` counts which mapped elements some spec
   touches; it says nothing about whether those checks pass.
 

{ordel_cli-0.4.0 → ordel_cli-0.4.2}/ordel_cli/adopt.py

@@ -180,7 +180,17 @@ def adopt(store: OrdelStore, path: str = "", runs: int = 3,
     report["ingest"] = {k: v for k, v in ingest(store, path).items() if k != "ascii"}
 
     playwright = find_playwright(store.root) is not None
-    history = [r for r in _judged_runs(store) if r.get("schema") == RUN_SCHEMA]
+    # The selection bounds both what runs and which runs count as its history: a run of the
+    # whole project (or of another path) says nothing about this suite's flakiness, and running
+    # everything executed tests the user deliberately left out (the 0.4.0 audit's G2).
+    selection = Path(path).as_posix() if path else ""
+    report["selection"] = selection or "(whole project)"
+
+    def _selected(runs: list[dict[str, Any]]) -> list[dict[str, Any]]:
+        return [r for r in runs if r.get("schema") == RUN_SCHEMA
+                and str(r.get("spec") or "") == selection]
+
+    history = _selected(_judged_runs(store))
     executed: list[str] = []
     if not playwright:
         report["notes"].append("no @playwright/test in this project: runs, flake and heal "
@@ -189,20 +199,22 @@ def adopt(store: OrdelStore, path: str = "", runs: int = 3,
     else:
         if runs and len(history) >= runs:
             report["notes"].append(
-                f".ordel/ already holds {len(history)} judged run(s), so nothing was run: --runs "
-                f"builds history up to that many. Pass --runs {len(history) + 1} to run once more")
+                f".ordel/ already holds {len(history)} judged run(s) of {report['selection']}, "
+                f"so nothing was run: --runs builds that selection's history up to that many. "
+                f"Pass --runs {len(history) + 1} to run once more")
         for _ in range(max(0, runs - len(history))):
-            doc = run_and_record(store, base_url=base_url)
+            doc = run_and_record(store, spec=selection or None, base_url=base_url)
             if doc["status"] == "error":
                 report["notes"].append(f"run failed: {doc.get('error')}")
                 break
             executed.append(str(doc["run_id"]))
     report["runs_executed"] = executed
 
-    all_runs = _judged_runs(store)
+    all_runs = _selected(_judged_runs(store))
+    report["history"] = [str(r.get("run_id")) for r in all_runs]
     report["flake"] = build_flake_report(all_runs)
 
-    latest = next((r for r in reversed(all_runs) if r.get("schema") == RUN_SCHEMA), None)
+    latest = next(reversed(all_runs), None)
     heal_items: list[dict[str, Any]] = []
     failures = [v for v in (latest or {}).get("verdicts", []) or [] if v.get("status") == "fail"]
     if latest is not None:
@@ -224,7 +236,8 @@ def adopt(store: OrdelStore, path: str = "", runs: int = 3,
 
 def render_adopt(report: dict[str, Any]) -> str:
     ing = report["ingest"]
-    lines = [f"Adopted: {ing.get('specs_ingested', 0)} functional spec(s) -> "
+    lines = [f"Adopted {report.get('selection', '(whole project)')}: "
+             f"{ing.get('specs_ingested', 0)} functional spec(s) -> "
              f"{ing.get('flows_created', 0)} flow(s)"
              + (f" ({ing['cypress_specs']} Cypress)" if ing.get("cypress_specs") else "")]
     if ing.get("unparsed_actions"):
@@ -235,7 +248,9 @@ def render_adopt(report: dict[str, Any]) -> str:
     c = fl["counts"]
     just_ran = (f", {len(report['runs_executed'])} just executed" if report["runs_executed"]
                 else "")
-    lines += ["", f"Flake report ({fl['runs_considered']} judged run(s){just_ran}):",
+    runs_used = ", ".join(report.get("history") or []) or "none"
+    lines += ["", (f"Flake report ({fl['runs_considered']} judged run(s) of this selection"
+                   f"{just_ran}: {runs_used}):"),
               (f"  flaky {c['flaky']}  regressed {c['regressed']}  failing {c['failing']}  "
                f"recovered {c['recovered']}  stable {c['stable']}  "
                f"too few runs {c['insufficient']}")]

{ordel_cli-0.4.0 → ordel_cli-0.4.2}/ordel_cli/cli.py

@@ -15,6 +15,7 @@ import json
 import os
 import shutil
 import sys
+from datetime import UTC, datetime
 from importlib.metadata import version as _dist_version
 from pathlib import Path
 from typing import Annotated, Any
@@ -321,11 +322,14 @@ def coverage(fmt: str = typer.Option("ascii", help="ascii | markdown | mermaid")
 
 
 @app.command()
-def scenarios(flow_name: str, show: bool = typer.Option(False, help="print the spec source")) -> None:
+def scenarios(flow_name: str, show: bool = typer.Option(False, help="print the spec source"),
+              repeat_submit: str | None = typer.Option(
+                  None, help="what a second submit of the same record must do: refused | "
+                             "same-record | new-record (kept on the flow)")) -> None:
     """From a recorded flow, emit happy+NEGATIVE+boundary+replay tests that hunt bugs."""
     from ordel_cli.scenarios import generate_scenarios
     store = _store()
-    res = generate_scenarios(store, flow_name)
+    res = generate_scenarios(store, flow_name, repeat_submit=repeat_submit)
     if res["status"] != "generated":
         typer.echo(res.get("note", res["status"]))
         raise typer.Exit(1)
@@ -404,7 +408,7 @@ def plan() -> None:
 def regression(
     pages: Annotated[list[str] | None, typer.Argument(help="changed page_ids (omit=all)")] = None,
 ) -> None:
-    """The exact specs to re-run after a change (impact selection)."""
+    """Candidate specs to re-run after a change: those touching the changed pages."""
     from ordel_cli.qa_plan import regression_set
     res = regression_set(_store(), list(pages) if pages else None)
     typer.echo(res.get("ascii", res.get("note", res["status"])))
@@ -416,8 +420,10 @@ def report(
 ) -> None:
     """Emit a copy-pasteable Markdown report (Slack/Teams/Jira) of what `.ordel/` can prove.
 
-    Every line is a number read from the local store: the map, the last recorded run's
-    verdicts and the heals still waiting for review. It makes no claim the store can't back."""
+    Every line is read from the local store: the map, the last recorded run's verdicts with
+    links to its evidence, the heals still waiting for review, the mapped pages no spec
+    touches, and a pointer to the QA context your agent keeps (.ordel/qa-context.md), which
+    Ordel shows but does not verify. It makes no claim the store can't back."""
     store = _store()
     if not store.exists:
         typer.echo("no .ordel/ here — run `ordel init`")
@@ -441,9 +447,49 @@ def report(
     if last is not None:
         doc["last_run"] = {k: last.get(k) for k in (
             "run_id", "at", "status", "passed", "failed", "flaky", "unjudged", "skipped")}
+        doc["last_run"]["record"] = f".ordel/runs/{runs[-1]}.json"
+        verdicts = last.get("verdicts", []) or []
+
+        def _where(v: dict[str, Any]) -> str:
+            # Playwright names files relative to its testDir; the report links project paths
+            f = str(v.get("file") or "")
+            if f and not (store.root / f).exists() and (store.root / "tests" / f).exists():
+                f = f"tests/{f}"
+            return f"{f}:{v.get('line')}"
+
         doc["last_run"]["unjudged_tests"] = [
             {"file": v.get("file"), "title": v.get("title"), "reasons": v.get("reasons", [])}
-            for v in last.get("verdicts", []) or [] if v.get("status") == "unjudged"]
+            for v in verdicts if v.get("status") == "unjudged"]
+        doc["last_run"]["failed_tests"] = [
+            {"where": _where(v), "title": v.get("title"),
+             "message": str(v.get("message") or "").splitlines()[0][:160] if v.get("message") else ""}
+            for v in verdicts if v.get("status") == "fail"]
+        doc["last_run"]["skipped_tests"] = [
+            {"where": _where(v), "title": v.get("title"),
+             "reasons": v.get("reasons", [])}
+            for v in verdicts if v.get("status") == "skipped"]
+    doc["pending_heals"] = [
+        {"id": h.get("id"), "spec": h.get("spec"),
+         "change": f"{h.get('element_id')} -> {h.get('healed_to')}", "exact": bool(h.get("exact"))}
+        for h in pending]
+    from ordel_cli.coverage import coverage_report as _cov
+    cov = _cov(store, force=True)
+    doc["untested_pages"] = list((cov.get("summary") or {}).get("untested_pages") or [])
+    ctx = store.dir / "qa-context.md"
+    if ctx.is_file():
+        text = ctx.read_text(encoding="utf-8", errors="replace")
+        doc["qa_context"] = {
+            "path": ".ordel/qa-context.md",
+            "updated": datetime.fromtimestamp(ctx.stat().st_mtime, UTC).isoformat(
+                timespec="seconds"),
+            "sections": [ln.lstrip("# ").strip() for ln in text.splitlines()
+                         if ln.startswith("## ")]}
+    else:
+        doc["qa_context"] = None
+    doc["limits"] = ("a pass means its listed checks held on that run's target and build; "
+                     "mapped pages are what was captured, not the whole app; expectations, "
+                     "readiness and selections live in the QA context, which Ordel does not "
+                     "verify")
     if as_json:
         typer.echo(json.dumps(doc, indent=2))
         return
@@ -470,10 +516,35 @@ def report(
         lines.append(f"| Tests flaky | {lr['flaky']} |")
         lines.append(f"| Tests unjudged | {lr['unjudged']} |")
     lines.append(f"| Heals pending human review | {len(pending)} of {len(heals)} |")
+    if lr and lr.get("at"):
+        lines.append(f"| Last run at | {lr['at']} (evidence: `{lr['record']}`) |")
+    qc = doc["qa_context"]
+    lines.append(f"| QA context | `{qc['path']}`, updated {qc['updated']} |" if qc else
+                 "| QA context | none saved: expectations, readiness and selections are not "
+                 "recorded |")
+    failed = (lr or {}).get("failed_tests") or []
+    if failed:
+        lines += ["", "Failed (triage before calling any of these a defect):"]
+        lines += [f"- `{f['where']}` {f['title']}: {f['message']}" for f in failed]
     unjudged = (lr or {}).get("unjudged_tests") or []
     if unjudged:
         lines += ["", "Unjudged (green in Playwright, not proven by Ordel):"]
         lines += [f"- `{u['file']}` {u['title']}: {'; '.join(u['reasons'])}" for u in unjudged]
+    skipped = (lr or {}).get("skipped_tests") or []
+    if skipped:
+        lines += ["", "Skipped (not run):"]
+        lines += [f"- `{k['where']}` {k['title']}"
+                  + (f": {'; '.join(k['reasons'])}" if k["reasons"] else "") for k in skipped]
+    if doc["pending_heals"]:
+        lines += ["", "Heals waiting for review (`ordel adopt --apply|--reject <id>`):"]
+        lines += [f"- {h['id']} `{h['spec']}`: {h['change']}"
+                  + (" (exact match)" if h["exact"] else "") for h in doc["pending_heals"]]
+    if doc["untested_pages"]:
+        lines += ["", f"Mapped pages no spec touches: {', '.join(doc['untested_pages'])}"]
+    if qc and qc["sections"]:
+        lines += ["", (f"QA context sections (written by your agent, not verified by Ordel): "
+                       f"{', '.join(qc['sections'])}")]
+    lines += ["", f"_Limits: {doc['limits']}._"]
     typer.echo("\n".join(lines))
 
 
@@ -693,7 +764,8 @@ def record(
             typer.echo("  no assertion recorded: `ordel run` will report this test unjudged; "
                        "Alt+click the result you expect next time")
         for var in res["needs_env"]:
-            typer.echo(f"  set {var} to run it: a password was typed and is never stored")
+            typer.echo(f"  add {var}=<value> to .ordel/secrets.env to run it: a password was "
+                       "typed and is never stored")
     raise typer.Exit(0 if res["status"] == "recorded" else 1)
 
 
@@ -811,6 +883,8 @@ def _render_run(doc: dict[str, Any]) -> str:
               f"skipped={doc.get('skipped', 0)}")]
     if status == "no_tests":
         lines.append("  no test executed - nothing was checked, so nothing passed")
+    for hit in doc.get("hardcoded_secrets") or []:
+        lines.append(f"  SECRET IN A SPEC: {hit}: read it from .ordel/secrets.env instead")
     if doc.get("secret_leaks"):
         lines.append(f"  SECRET IN A FILE: a value from .ordel/secrets.env is written in "
                      f"{', '.join(doc['secret_leaks'])}: read it from the environment instead")
@@ -824,6 +898,9 @@ def _render_run(doc: dict[str, Any]) -> str:
         where = f"{v.get('file')}:{v.get('line')} › {v.get('title')}"
         detail = v.get("message") if v.get("status") == "fail" else "; ".join(v.get("reasons", []))
         lines.append(f"  {mark} {where}" + (f": {detail}" if detail else ""))
+        if v.get("status") == "fail" and v.get("unresolved_locator"):
+            lines.append(f"      locator never found: {v['unresolved_locator']} (heal_selector can "
+                         "look for its replacement; a person reviews the heal)")
     checked = sum((v.get("evidence") or {}).get("assertions") or 0
                   for v in verdicts if v.get("status") == "pass")
     if doc.get("passed"):

{ordel_cli-0.4.0 → ordel_cli-0.4.2}/ordel_cli/ingest.py

@@ -47,6 +47,7 @@ def ingest(store: OrdelStore, path: str = "") -> dict[str, Any]:
                         if fn.lower().endswith(_SPEC_SUFFIXES)]
 
     ingested = flows_created = unparsed = skipped_unit = skipped_ordel = cypress_specs = 0
+    kept_asserts = dropped_asserts = 0
     created: list[str] = []
     capped = False
     for f in sorted(targets):
@@ -79,10 +80,15 @@ def ingest(store: OrdelStore, path: str = "") -> dict[str, Any]:
                 n += 1
                 uniq = f"{name[:60]}-{n}"
             start = next((s["url"] for s in t["steps"] if s.get("op") == "goto"), "")
+            kept = int(t.get("assertions_kept", 0))
+            dropped = int(t.get("assertions_dropped", 0))
+            kept_asserts += kept
+            dropped_asserts += dropped
             store.save_flow(uniq, {"name": uniq, "source": "ingested",
                                    "framework": "cypress" if cypress else "playwright",
                                    "spec": f.name, "spec_path": f.relative_to(root).as_posix(),
-                                   "start": start, "final_url": "", "steps": t["steps"]})
+                                   "start": start, "final_url": "", "steps": t["steps"],
+                                   "assertions_kept": kept, "assertions_dropped": dropped})
             created.append(uniq)
             flows_created += 1
         if capped:
@@ -98,6 +104,9 @@ def ingest(store: OrdelStore, path: str = "") -> dict[str, Any]:
         lines.append(f"  skipped {skipped_ordel} Ordel-generated spec(s) (already ours)")
     if unparsed:
         lines.append(f"  {unparsed} action line(s) used idioms we couldn't parse — re-record those flows")
+    lines.append(f"  assertions carried into the flows: {kept_asserts}"
+                 + (f"; NOT carried: {dropped_asserts} (the original specs still check them, "
+                    "the generated scenarios do not)" if dropped_asserts else ""))
     if capped:
         lines.append(f"  stopped at the {_MAX_FLOWS}-flow cap — ingest a subdir at a time for the rest")
     if flows_created:
@@ -106,4 +115,5 @@ def ingest(store: OrdelStore, path: str = "") -> dict[str, Any]:
     return {"status": "ok", "specs_ingested": ingested, "flows_created": flows_created,
             "flows": created, "cypress_specs": cypress_specs,
             "skipped_unit": skipped_unit, "skipped_ordel": skipped_ordel,
-            "unparsed_actions": unparsed, "ascii": "\n".join(lines)}
+            "unparsed_actions": unparsed, "assertions_kept": kept_asserts,
+            "assertions_dropped": dropped_asserts, "ascii": "\n".join(lines)}