@andresmassello/uscha 1.85.0 → 1.86.0

package/README.md

@@ -8,7 +8,7 @@ never what was claimed.
 > The tool executes · the method governs · evidence decides · the human approves.
 
 **[uscha.dev](https://uscha.dev)** — the method, the five rules, the skills, the library
-(essay, 2-day dev course, reference, paper).
+(the diamond thesis, how-it-works diagrams, essay, 2-day dev course, reference, paper).
 
 ```bash
 npx --yes @andresmassello/uscha@latest install --target claude   # Claude Code
@@ -40,12 +40,77 @@ Requires **Python 3.8+** on the machine (the engine is Python stdlib — no pip
 runtime dependencies). The npm package is a thin router; the canonical installer is
 `uscha-kit/install-uscha.py`.
 
-**Kit v1.85.0** <!-- uscha:version --> · [uscha.dev](https://uscha.dev) ·
+**Kit v1.86.0** <!-- uscha:version --> · [uscha.dev](https://uscha.dev) ·
 [changelog](https://github.com/andresmassello/uscha/blob/main/uscha-kit/CHANGELOG.md)
 (the per-release changelogs live in the repo, not in the npm tarball)
 
 ---
 
+## The diamond — specs are the source code, end to end
+
+Uscha closes a cycle most spec-driven tools only walk halfway. The **spec package plus its
+behavior ledger** is the canonical asset of a system; the code is a regenerable build artifact.
+An LLM compiles the package into code under a validated contract; reverse discovery decompiles
+existing code back into *curated* specs — passing, mandatorily, through the one step no
+automatic tool can perform: a human verdict.
+
+```
+                THE ASSET (solid) ── it appreciates with every model generation
+          ┌───────────────────────────────────────────────────────────────┐
+          │   SPEC PACKAGE  +  BEHAVIOR LEDGER  +  IR                     │
+          │   SPEC · ADRs · ACCEPTANCE · CONSTITUTION                     │
+          │   verdicts: preserve · fix · undefined                        │
+          └──────────────┬─────────────────────────────▲──────────────────┘
+                         │                             │
+        FORWARD          │                             │          REVERSE
+        the LLM compiles │                             │   reverse discovery
+                         ▼                             │
+   ┌─────────────────────────────────────┐   ┌─────────┴─────────────────────┐
+   │ compile-validate — output contract, │   │ CURATION · the human gate     │
+   │   mechanical only, model-blind      │   │ candidate ─▶ verdict ─▶ ledger │
+   │ withheld ORACLE — authored BEFORE   │   │ no verdict → PR blocked,      │
+   │   compiling, never in the prompt    │   │   the candidate is NAMED      │
+   └─────────────────┬───────────────────┘   └─────────▲─────────────────────┘
+                     │                                 │
+                     ▼                                 │
+   ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐   ┌─────────────┴─────────────────────┐
+     CODE (dashed) — build artifact,      │ discover · golden capture         │
+     regenerable, disposable          ───▶│ candidates: typed evidence        │
+   └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘   │   + confidence                    │
+                                          └───────────────────────────────────┘
+                                            ▲ also enters here: any LEGACY
+                                              system (= 100% drift)
+
+   round trip · bench-roundtrip — how much of the asset the reverse organs re-anchor
+   from the compiled code: 0.062 measured (12 archetypes) — names, not yet semantics
+```
+
+**What each arrow is, in the engine (kit 1.86.0, 52 subcommands, all measured):**
+
+| Leg | Subcommands | What it establishes |
+|---|---|---|
+| Asset → typed graph | `ir-extract`, `ir-render` | the whole package becomes one canonical IR (M2, ADR-015) — deterministic, `UNTYPED` is a measurement not an error |
+| Forward, the compiler | `compile-validate`, `compile-ingest` | any model produces code; the engine validates the output contract and never compiles (M3, ADR-016) |
+| Forward, is it the *same* system? | `bootstrap-oracle`, `bootstrap-variance`, `bench` | a withheld oracle judges blind compilations — **12 archetypes, 9 PASS · 3 PARTIAL**, three models, JS included (M4/M5, ADR-017/018/028/029) |
+| Reverse, facts | `discover`, `golden-diff` (+ the `/uscha-characterize` skill) | system map + mechanically captured golden; typed candidate observations with evidence class (M1, ADR-013) |
+| Reverse, the human gate | `curate`, `promote`, `curation-check`, `bench-curate` | one verdict per candidate, append-only ledger verified against git; unjudged → `pr-ready` blocked naming it (ADR-009/010, INV-CURATION-01) |
+| Fidelity, honestly | `fidelity`, `roundtrip`, `bench-roundtrip`, `bench-r2` | per-compiler fidelity vector, id-level round trip, recoverability **0.062**, and the **noise floor** under every variance claim (ADR-014/022/027/030) |
+
+**Read the numbers the way the repo does.** 9 of 12 archetypes regenerate to the same system
+under an oracle the compilers never saw — that is the closed loop working. 0.062 is the mean
+*recoverability* of the asset from compiled code counting only static and behavioural
+footing, with the behaviour dimension still `UNMEASURED` — it says the reverse organs anchor
+**names, not yet semantics**, and it is published rather than smoothed. And `bench-r2` measured
+that same-model reruns differ structurally about as much as different models do (aggregate
+`NOISY`) — so one earlier variance narrative was **retracted**. Every claim above is a subcommand
+you can run; every unmeasured part is labeled. That honesty is the method applied to itself.
+
+→ The full thesis, with before/after diagrams and the REAL vs VISION table:
+**[uscha.dev/diamond](https://uscha.dev/diamond)** · the mechanism, in three diagrams:
+**[uscha.dev/how](https://uscha.dev/how)**
+
+---
+
 ## The problem it solves
 
 An agent will tell you the tests pass. It will tell you the feature is done. It is often
@@ -68,7 +133,7 @@ and see which file, which test, and when.
 |---|---|
 | `/uscha-discovery` | Idea → spec package (CONTEXT, SPEC, ADRs, CONSTITUTION, ACCEPTANCE) |
 | `/uscha-adr-refine` | Known feature → ADR + ACCEPTANCE, by interrogating you first |
-| `/uscha-reverse-discovery` | Existing system → extracted facts (brownfield migrations) |
+| `/uscha-reverse-discovery` | Existing system → facts + typed candidates in quarantine; never promoted without a human verdict (brownfield) |
 | `/uscha-characterize` | Capture a golden suite of current behavior before touching it |
 | `/uscha-devloop` | Plan → build → severity-gated QA loop → PR (stops at the merge) |
 | `/uscha-rubric` | Grade the non-testable (conventions, ergonomics) against a versioned rubric |
@@ -76,7 +141,7 @@ and see which file, which test, and when.
 | `/uscha-mirador` | Bird's-eye HTML dashboard: readiness, trail, acceptance, loops |
 | `/uscha-status` | One-line progress readout, in chat |
 
-**A measurement engine** (`qa_ledger.py`, 51 subcommands, Python stdlib) that ingests
+**A measurement engine** (`qa_ledger.py`, 52 subcommands, Python stdlib) that ingests
 evidence from **11 language stacks** — maven, gradle, ant, python, node, go, rust, dotnet,
 cpp, swift, flutter — and computes a readiness score with hard caps and visible provenance.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@andresmassello/uscha",
-  "version": "1.85.0",
+  "version": "1.86.0",
   "description": "Spec-driven development for LLM coding agents: 9 skills + a stdlib evidence engine. Facts block, guesses advise; the human approves.",
   "author": {
     "name": "Andres Massello",

package/uscha-kit/.claude/skills/uscha-devloop/qa_ledger.py

@@ -7616,6 +7616,14 @@ def _mirador_adrs(adr_dir):
     return out
 
 
+def _project_name(cfg):
+    """Project label: DECLARED by the human in config (project/name); otherwise derived by
+    joining the configured repo names; None when there is nothing to join. One derivation,
+    shared by `dashboard` and `top` -- two readouts must never disagree about the name."""
+    names = [r.get("name") for r in cfg.get("repos", []) if r.get("name")]
+    return cfg.get("project") or cfg.get("name") or (" + ".join(names) if names else None)
+
+
 def cmd_dashboard(args):
     """mirador — vista bird's-eye del estado. Agrega SOLO hechos que el ledger ya tiene
     al contrato DATA del template. Read-only, determinista, cero narracion. Campos sin
@@ -7780,10 +7788,9 @@ def cmd_dashboard(args):
                   "reached": _reached_index(h.get("score"))}
                  for h in ledger.get("readiness_history", [])]
 
-    names = [r.get("name") for r in cfg.get("repos", []) if r.get("name")]
     # nombre de proyecto: lo declara el humano en config (project/name); si no,
     # se deriva juntando los repos. Truth-pass: nombre puesto si existe, si no derivado.
-    project = cfg.get("project") or cfg.get("name") or (" + ".join(names) if names else None)
+    project = _project_name(cfg)
     adrs = _mirador_adrs(getattr(args, "adr_dir", "docs/adr"))
 
     # evidence (kit 1.50.0): RECEIPTS. The template shipped a click-a-milestone drawer
@@ -8003,6 +8010,260 @@ def cmd_dashboard(args):
           f"{len(snapshots)} snapshot(s) en el time-lapse")
 
 
+TOP_SCHEMA = "uscha-top/v0.1"
+
+# `uscha top` state ladder (ADR-032). TRACED and TAGGED are declared here and NEVER emitted
+# in v0.1: no general-project source exists for either (the only "does source name this AC"
+# scan is bench-wired, and JUnit has no "written but unexecuted" case). They keep their names
+# so the renderer can class them gray instead of inventing them into PASS (INV-TOP-02).
+TOP_STATES = ("UNMEASURED", "TRACED", "TAGGED", "MEASURED_PASS", "MEASURED_FAIL",
+              "QUARANTINE")
+
+
+def _top_dt(iso):
+    """Tolerant ISO-8601 -> datetime, or None. Never raises: a malformed timestamp in one
+    ledger record must degrade that record, not the read-only readout."""
+    if not isinstance(iso, str) or not iso.strip():
+        return None
+    txt = iso.strip()
+    if txt.endswith("Z"):                 # py3.8's fromisoformat does not take the Z suffix
+        txt = txt[:-1] + "+00:00"
+    try:
+        return datetime.fromisoformat(txt)
+    except ValueError:
+        return None
+
+
+def _top_loop_median_min(ledger):
+    """medians.loop_min: the median gap in MINUTES between consecutive QA iterations, over
+    the timestamps the ledger already carries (repos[r].iterations[*].at, grouped by
+    iteration number). Fewer than two iterations -> None: an honest absence, never a zero
+    (audit A/medians.loop_min)."""
+    gaps = []
+    for node in (ledger.get("repos") or {}).values():
+        first = {}
+        for s in node.get("iterations") or []:
+            it, at = s.get("iteration"), _top_dt(s.get("at"))
+            if it is None or at is None:
+                continue
+            if it not in first or at < first[it]:
+                first[it] = at
+        ordered = [first[k] for k in sorted(first)]
+        for a, b in zip(ordered, ordered[1:]):
+            gaps.append((b - a).total_seconds() / 60.0)
+    if not gaps:
+        return None
+    gaps.sort()
+    mid = len(gaps) // 2
+    return int(round(gaps[mid] if len(gaps) % 2 else (gaps[mid - 1] + gaps[mid]) / 2.0))
+
+
+def _top_checks(ledger):
+    """checks{pass,fail,total} from the LATEST snapshot per repo -- the whole suite's run,
+    NOT the AC-tagged subset (which travels per obligation as cases_pass/cases_total).
+    None when no repo carries an ingested report: no evidence, no number."""
+    got = False
+    tot = {"pass": 0, "fail": 0, "total": 0}
+    for node in (ledger.get("repos") or {}).values():
+        snaps = node.get("snapshots") or []
+        if not snaps:
+            continue
+        t = snaps[-1].get("tests") or {}
+        if not t.get("report_found"):
+            continue
+        got = True
+        tot["pass"] += t.get("passed") or 0
+        tot["fail"] += (t.get("failures") or 0) + (t.get("errors") or 0)
+        tot["total"] += t.get("executed") or 0
+    return tot if got else None
+
+
+def _top_spec_pin(ledger):
+    """spec_pin (v0.1): git HEAD of the FIRST configured repo, labelled NOT clean-room
+    verified unless a clean_room GREEN record exists at that exact sha. There is no pinned-
+    spec concept in the engine yet (a designed pin is ADR-035); this is the honest interim
+    proxy, and a non-git tree returns None so the TUI renders an em dash rather than a
+    fabricated sha (INV-TOP-05)."""
+    repos = (ledger.get("config", {}) or {}).get("repos") or []
+    if not repos:
+        return None
+    name, path = repos[0].get("name"), repos[0].get("path", ".")
+    sha = (_evidence_origin(path) or {}).get("commit")
+    if not sha:
+        return None
+    cr = _cr_latest(ledger, name, sha) if name else None
+    return {"sha": sha[:7],
+            "clean_room_verified": bool(cr and cr.get("status") == "GREEN")}
+
+
+def _top_pct(done, total):
+    """A whole-number percentage with INV-TOP-01 enforced AT THE SOURCE: 999 of 1000 rounds
+    to 100, and a board reading 100% while one obligation sits outside MEASURED_PASS is the
+    exact lie the invariant forbids. Capped at 99 until the last one is really measured --
+    in the engine, so no renderer can be the place the rounding happens. The same cap covers
+    the honesty ratio: "100% measured" with one criterion unmeasured is the same lie."""
+    if not total:
+        return 0
+    pct = int(round(done * 100.0 / total))
+    return 99 if (done < total and pct >= 100) else pct
+
+
+def _top_ac_num(cid):
+    """Stable numeric order for the normalized 'AC-<n>' ids _parse_acceptance_items emits."""
+    try:
+        return int(str(cid).split("-")[1])
+    except (IndexError, ValueError):
+        return 0
+
+
+def cmd_top(args):
+    """`uscha top` — the WHOLE projection of the ledger as one read-only JSON (ADR-032).
+
+    Single derivation: every state, cardinality, median and percentage the TUI shows is
+    computed HERE, from the same helpers readiness/dashboard use (_parse_acceptance_items,
+    _ac_tags, _delta_state, _cr_latest, _evidence_origin). The renderer (uscha_top.py) is a
+    pure function of this object and computes no KPI of its own (ADR-034, AC-T-24).
+
+    Read-only and truth-pass: it never writes, never runs tests, never calls a model, and a
+    field with no honest source is null -- eta_min, medians.verdict_min, drift_pct, every
+    age_hours, and every trace[] are null/empty in v0.1 BY DESIGN, each with its deferred
+    wiring recorded in ADR-035. Under-claim, then wire, then re-claim."""
+    ledger = _load(args.ledger)
+    cfg = ledger.get("config", {}) or {}
+    acc_path = args.acceptance or (cfg.get("defaults") or {}).get("acceptance_file")
+    acc_items, _acc_found = _parse_acceptance_items(acc_path, args.section)
+
+    # measured evidence: the SAME helper readiness closes criteria with (_ac_tags), summed
+    # across repos exactly as cmd_readiness sums it. Re-deriving it here would be the 1.48.1
+    # mirador sin (two derivations of one number, free to disagree).
+    ac_tags = {}
+    for rcfg in cfg.get("repos", []):
+        rtags, _stale = _ac_tags(rcfg.get("path", "."), rcfg.get("type", "maven"))
+        for cid, v in rtags.items():
+            d = ac_tags.setdefault(cid, {"green": 0, "red": 0})
+            d["green"] += v.get("green", 0)
+            d["red"] += v.get("red", 0)
+
+    # quarantine: UNCURATED observations whose statement literally names an AC id. The link
+    # is `canonical_match`, a heuristic TEXT match (_match_canonical) -- not a designed
+    # AC<->OBS field -- so an observation that matches nothing carries ac: null rather than
+    # a guessed criterion (audit A/quarantine_obs).
+    quarantine, observations = {}, []
+    for rname in (ledger.get("repos") or {}):
+        try:
+            path = _scope_path(ledger, rname)
+            dstate = _delta_state(ledger, rname, path)
+        except SystemExit as exc:
+            # a repo the scan cannot reach is NAMED on stderr, never dropped in silence: a
+            # missing quarantine row would read as "nothing to curate here" (the one lie a
+            # debtor column must never tell). stdout stays pure JSON.
+            print("[qa_ledger] top: repo %s skipped for quarantine scan: %s"
+                  % (rname, exc or "unknown scope"), file=sys.stderr)
+            continue
+        if not dstate:
+            continue
+        delta, errors = _load_delta(path)
+        if errors or not delta:
+            continue                       # a malformed delta is named by the gates, not here
+        uncurated = set(dstate.get("uncurated") or [])
+        for o in delta.get("observations") or []:
+            if o.get("id") not in uncurated:
+                continue
+            cid = o.get("canonical_match")
+            if cid and cid not in quarantine:
+                quarantine[cid] = o["id"]
+            observations.append({
+                "id": o.get("id"), "ac": cid,
+                # no separate short label exists on an observation; `statement` is the only
+                # prose field, so `title` is null and the TUI may head-truncate candidate[0]
+                "title": None,
+                "candidate": [o.get("statement")],
+                "evidence": list((o.get("provenance") or {}).get("files") or []),
+                "age_hours": None})
+    observations.sort(key=lambda o: o.get("id") or "")
+
+    # obligations: one row per DISTINCT tagged criterion of the acceptance file. kind is
+    # "AC" for all of them -- there is no per-INV ledger in the general path (the mirador's
+    # INV list is a fixed hand-mapped set, audit A/obligations), so no INV row is invented.
+    ids, seen = [], set()
+    for it in acc_items:
+        if it.get("id") and it["id"] not in seen:
+            seen.add(it["id"])
+            ids.append(it["id"])
+    obligations = []
+    for cid in sorted(ids, key=_top_ac_num):
+        tag, obs = ac_tags.get(cid), quarantine.get(cid)
+        # red evidence VETOES (fail-closed, the same rule _ac_closed applies). Measured
+        # evidence outranks the lateral QUARANTINE rung so the four buckets partition the
+        # board exactly once: done + machine + you + untagged == total.
+        if tag and tag["red"] >= 1:
+            state, gate = "MEASURED_FAIL", "junit"
+        elif tag and tag["green"] >= 1:
+            state, gate = "MEASURED_PASS", "junit"
+        elif obs:
+            state, gate = "QUARANTINE", "curation"
+        else:
+            state, gate = "UNMEASURED", "junit"
+        obligations.append({
+            "id": cid, "kind": "AC", "state": state,
+            # never "oracle": for a general project the gate that closes a criterion is
+            # JUnit-tag ingestion or curation. "oracle" is Diamond-bench vocabulary and
+            # would mislead here (audit A/gate, ADR-032).
+            "gate": gate,
+            "cases_pass": (tag or {}).get("green", 0),
+            "cases_total": (tag or {}).get("green", 0) + (tag or {}).get("red", 0),
+            "trace": [],                  # no general AC->implementation map yet (ADR-035/5)
+            "quarantine_obs": obs,
+            "ac": None,                   # contract slot with no second honest meaning here
+            "age_hours": None})           # no first-seen timestamp exists (ADR-035/1)
+
+    def _n(st):
+        return sum(1 for o in obligations if o["state"] == st)
+
+    total = len(obligations)
+    done, fail, quar = _n("MEASURED_PASS"), _n("MEASURED_FAIL"), _n("QUARANTINE")
+    unmeasured = _n("UNMEASURED") + _n("TRACED")
+    pct = _top_pct(done, total)
+    measured = done + fail
+    out = {
+        "schema": TOP_SCHEMA,
+        "project": _project_name(cfg),
+        "spec_pin": _top_spec_pin(ledger),
+        # the engine's GLOBAL step counter -- not a build number and not a QA-loop pass
+        # count (that is _repo_loop_count); the TUI labels it `step #N` (audit A/run).
+        "step": ledger.get("step_counter"),
+        "generated_at": _now(),
+        "obligations": obligations,
+        "observations": observations,
+        "events_tail": [],                # the live feed is M2; the key ships empty, not absent
+        "counts": {"measured_pass": done, "measured_fail": fail, "quarantine": quar,
+                   "unmeasured": _n("UNMEASURED"), "traced": 0, "tagged": 0, "total": total},
+        "terminado": {"done": done, "total": total, "pct": pct, "unmeasured": unmeasured},
+        "debtors": {"machine": fail, "you": quar, "untagged": unmeasured},
+        "honesty": {"measured": measured, "total": total,
+                    "pct": _top_pct(measured, total)},
+        # ETA = you x median_verdict + machine x median_loop. median_verdict is null in v0.1
+        # (no per-OBS first-seen timestamp), so the product is null and the header reads
+        # `ETA -`. A partial ETA computed from half the formula would be a fabrication.
+        "eta_min": None,
+        "medians": {"verdict_min": None, "loop_min": _top_loop_median_min(ledger)},
+        "checks": _top_checks(ledger),
+        "drift_pct": None,                # spec_drift is per-file; an aggregate is ADR-035/3
+        # the ONLY real series is the readiness SCORE history; an obligation-count burn-up
+        # needs new persistence (ADR-035/2), so `kind` is emitted for the TUI to label it a
+        # score trend and never as a count of closed obligations.
+        "burnup": {"kind": "score",
+                   "weeks": [h.get("score") for h in ledger.get("readiness_history", [])
+                             if isinstance(h.get("score"), (int, float))]},
+    }
+    if getattr(args, "json", False):
+        print(json.dumps(out, indent=2, ensure_ascii=False))
+        return
+    print("TOP %s: DONE %d/%d (%d%%) · %d unmeasured — `top --json` prints the full "
+          "contract; `uscha top` renders it live"
+          % (out["project"] or "?", done, total, pct, unmeasured))
+
+
 def cmd_readiness(args):
     ledger = _load(args.ledger)
     defaults = ledger["config"].get("defaults", {})
@@ -8231,8 +8492,15 @@ def cmd_readiness(args):
     # hacia adelante (no backfillea). El dashboard lo consume, nunca escribe.
     if getattr(args, "record", False):
         now = _now()
+        # issue #1 (field note 001, limitation 1): persist the CAUSES per entry, not only the
+        # score -- both facts are already computed above; nothing is re-derived. Additive keys:
+        # every existing reader uses .get("at")/.get("score"). None/0 mean "no cap"/"none
+        # discarded", stated rather than absent.
         ledger.setdefault("readiness_history", []).append(
-            {"at": now, "score": round(final, 1)})
+            {"at": now, "score": round(final, 1),
+             "cap_applied": ({"reason": cap_reason, "key": cap_key} if cap_reason else None),
+             "stale_discarded": {"count": len(stale_reports),
+                                 "reports": list(stale_reports)[:8]}})
         # kit 1.46.1: persist a compact MEASURED summary so the statusline (uscha_progress.py)
         # shows MEASURED acceptance -- the same truth this readiness computed -- instead of
         # counting checkboxes (narrated). Write-once/read-many: the fast Stop hook reads this
@@ -11153,6 +11421,17 @@ def build_parser():
     pdash.add_argument("--json", action="store_true")
     pdash.set_defaults(func=cmd_dashboard)
 
+    ptop = sub.add_parser("top",
+                          help="uscha top: the whole projection of the ledger as one "
+                               "read-only JSON (obligations, debtors, medians) — the "
+                               "contract the terminal view renders (ADR-032)")
+    add_ledger(ptop)
+    ptop.add_argument("--acceptance", default=None,
+                      help="acceptance task list (markdown); overrides config default")
+    ptop.add_argument("--section", default=None)
+    ptop.add_argument("--json", action="store_true")
+    ptop.set_defaults(func=cmd_top)
+
     pb = sub.add_parser("rebuild",
                         help="rebuild test: is the SPEC complete enough to "
                              "regenerate the system? (completeness, not correctness)")