@myelinbridge/cli 0.9.0 → 0.10.0

package/README.md

@@ -50,17 +50,36 @@ npx @myelinbridge/cli push  ./run_042 --dataset onco1-wes --submit
 - **Read the contract before you build the delivery.** `contract` prints what the
   client expects — every check as one plain sentence, grouped by what it answers
   (completeness, structure, validity, consistency, integrity, privacy), and marked
-  `!` when a failure blocks validation. It also tells you which checks `check` can
+  `!` when a failure blocks validation or `?` when the client's reviewer confirms
+  it rather than asking you to fix it. It also tells you which checks `check` can
   verify locally and which only run once the files are uploaded, so nothing about
   the bar is a surprise at review time.
 
 - **`check` costs nothing.** It evaluates your local file list against the dataset's
   quality checks server-side — same engine, same verdicts as submit — without
-  uploading. Exit code 2 means a blocking rule fails.
+  uploading. Exit code 2 means the delivery would not be validatable as it stands:
+  a blocking rule fails, **or** a blocking rule could not be evaluated at all.
+- **A check has five answers, not three** (since 0.10.0). Alongside *passed*,
+  *flagged* and *failed*, the engine can now say it **could not check** something
+  (a missing precondition, or a rule it cannot evaluate) and that a rule **does
+  not apply** to your delivery. This matters to you in one specific way: a rule
+  the client wrote badly used to come back as a *failure on your delivery*. It
+  now says the rule could not be evaluated, and tells you it is not yours to fix.
+  `check` prints the count it could not check and what each one needs — supply it
+  and those checks run on your next delivery, which is fewer rejections later.
+- **Some checks are confirmed, not fixed.** A rule marked *needs confirmation*
+  asks nothing of you: the client's reviewer records what they decided and the
+  delivery proceeds. `check` reports these and never exits 2 on them.
 - **The fix loop is machine-readable.** On `changes_requested`,
   `myelin status <batch> --json` returns the failed files, reviewer comments, and
   rule remediation hints; fix, re-`push --submit`, unchanged files keep their
   review votes.
+- **Limits.** `push` declares 500 files per call and chunks automatically — nothing
+  for you to split. `check` sends your whole file list in one preflight
+  call, which accepts at most 10 000 files — a larger delivery cannot be preflighted
+  by `check` today (the API answers `400 too_many_files`); its checks still run at
+  submit. A delivery holds at most 25 000 files (submit refuses above that with
+  `422 submit_blocked`), and a checksum manifest is inspected up to 16 MiB.
 
 ## If you are the client, not the partner
 
@@ -104,7 +123,23 @@ A partner key calling these gets `403 wrong_key_side`, and vice versa.
 ## Machine mode
 
 Every command takes `--json`. Exit codes: `0` ok · `1` error · `2` blocked
-(blocking preflight failure, locked delivery, blocked submit, locked sample depth).
+(blocking preflight failure, **a blocking rule the engine could not evaluate**,
+locked delivery, blocked submit, locked sample depth).
+
+`check --json` carries the numbers to branch on rather than the prose:
+
+| Field | |
+|---|---|
+| `blocking_failures` | Blocking rules that were evaluated and failed. Unchanged meaning since the first release. |
+| `blocking_not_evaluated` | Blocking rules the engine could not evaluate. **New in 0.10.0** — these stop the reviewer validating just as surely, which is why `check` now exits `2` on them too. |
+| `must_acknowledge_failures` | Rules the client asked to be told about. Reported, never exited on. |
+| `counts` | `passed`, `flagged`, `failed`, `not_evaluated`, `not_applicable` (which sum to `evaluated`), plus `deferred`, `manual` and `checks` (`evaluated + deferred + manual === checks`). |
+
+Every per-check result carries `details.abstained` when the engine did not
+conclude: `rule` means **the client's rule** could not be evaluated and there is
+nothing on your side to fix; `data` means something the check needs is missing
+from the delivery; `nothing_to_evaluate` means the rule matched none of your
+files. Branch on that before you page anyone.
 
 API errors carry structure, not just prose: in `--json` mode an error is
 `{ "error", "code", "status", "request_id" }` — branch on `code`, and quote the

package/bin/myelin.js

@@ -7,6 +7,16 @@
 //
 // Exit codes: 0 ok · 1 error · 2 blocked (failing checks / locked delivery).
 // Every command accepts --json for machine-readable output.
+//
+// 0.10.0 — the engine learned to abstain, so a check has five answers now, not
+// three (`not_evaluated` = "we could not look", `not_applicable` = "this does
+// not apply here"). Two consequences live in this file. Nothing renders an
+// unknown verdict as ✗ any more: 0.9.1 did, so a check nobody ran printed as a
+// failure the partner had caused. And `check` exits 2 on a BLOCKING rule the
+// engine could not evaluate as well as on one it failed — the client's reviewer
+// cannot validate either without deciding by hand, and hearing "all checks
+// pass" before uploading is the green-before-sending / red-after trap this
+// release exists to close.
 
 import { readdirSync, statSync, createReadStream, readFileSync } from 'node:fs'
 import { resolve, join, relative, sep, basename } from 'node:path'
@@ -63,6 +73,35 @@ const die = (message, code = 1, extra = null) => {
 
 const sleep = (ms) => new Promise((r) => setTimeout(r, ms))
 
+// ── The verdict vocabulary, as a partner reads it ──────────────────────────
+//
+// ⚠ NEVER print the raw token. `not_evaluated` on a terminal reads as an
+// internal enum and, worse, as an accusation: a partner seeing it next to their
+// delivery assumes their delivery is what failed. It is not — in the `rule`
+// case it is the client's rule that could not be evaluated.
+//
+// ⚠ And never `✗` for a verdict we do not recognise. That is exactly what 0.9.1
+// does (`verdict === 'passed' ? '✓' : verdict === 'flagged' ? '⚠' : '✗'`), so
+// every check that abstains prints to an installed old client as a failure. A
+// value from a newer server than this binary falls through to `?` and its own
+// raw name — visibly unknown, never falsely red.
+const VERDICT_MARK = {
+  passed: '✓',
+  flagged: '⚠',
+  failed: '✗',
+  not_evaluated: '?',
+  not_applicable: '–',
+}
+const VERDICT_LABEL = {
+  passed: 'passed',
+  flagged: 'flagged',
+  failed: 'failed',
+  not_evaluated: 'not checked',
+  not_applicable: 'does not apply here',
+}
+const verdictMark = (v) => VERDICT_MARK[v] ?? '?'
+const verdictLabel = (v) => VERDICT_LABEL[v] ?? v
+
 // Column widths come from the content. padEnd() alone silently ran a long
 // value into the next column — a 28-character dataset slug swallowed the
 // STATUS header's gutter. Pass headers = null for an unheadered list.
@@ -479,6 +518,9 @@ async function cmdContract() {
   out(`${dataset.name} — delivery contract v${j.version}`)
   out(
     `${s.total} checks · ${s.blocking} block validation · ` +
+      // Only when there are any: a contract with no rule at this rung must read
+      // exactly as it did before 0.10.0, not gain a "0 confirmed".
+      (s.must_acknowledge ? `${s.must_acknowledge} confirmed by your reviewer · ` : '') +
       `${s.checkable_before_upload} checkable before upload` +
       (s.needs_reviewer ? ` · ${s.needs_reviewer} reviewed by a person` : ''),
   )
@@ -496,9 +538,18 @@ async function cmdContract() {
     if (items.length === 0) continue
     out(`${d.label.toUpperCase()} — ${d.question}`)
     for (const c of items) {
-      const gate = c.severity === 'blocking' ? 'must' : 'should'
+      // THREE rungs, not two. The ternary this replaces called everything that
+      // was not `blocking` a "should", so a `must_acknowledge` rule — the one
+      // that demands a named decision from the reviewer — printed identically
+      // to a warning nobody has to read.
+      const mark =
+        c.severity === 'blocking' ? '!' : c.severity === 'must_acknowledge' ? '?' : '·'
       const when = c.runs_at === 'preflight' ? '' : ' (checked at submission)'
-      out(`  ${gate === 'must' ? '!' : '·'} ${c.assertion ?? c.name}${when}`)
+      const note =
+        c.severity === 'must_acknowledge'
+          ? ' — your client’s reviewer records a decision on this; nothing for you to fix'
+          : ''
+      out(`  ${mark} ${c.assertion ?? c.name}${when}${note}`)
     }
     out('')
   }
@@ -538,19 +589,92 @@ async function cmdCheck() {
       ...j.manual.map((m) => m.name.length),
     )
     for (const c of j.evaluated) {
-      const mark = c.verdict === 'passed' ? '✓' : c.verdict === 'flagged' ? '⚠' : '✗'
-      out(`${mark} ${c.check_type.padEnd(w)}  ${c.verdict}${c.severity === 'blocking' && c.verdict === 'failed' ? ' — BLOCKING' : ''}`)
-      if (c.verdict !== 'passed' && c.remediation) out(`    hint: ${c.remediation}`)
+      const gate =
+        c.severity === 'blocking' && c.verdict === 'failed' ? ' — BLOCKING'
+        : c.severity === 'blocking' && c.verdict === 'not_evaluated' ? ' — BLOCKING, and nobody looked'
+        : c.severity === 'must_acknowledge' && c.verdict === 'failed' ? ' — your reviewer confirms this'
+        : ''
+      out(`${verdictMark(c.verdict)} ${c.check_type.padEnd(w)}  ${verdictLabel(c.verdict)}${gate}`)
+      // Why we could not look, in the partner's terms. `abstained: 'rule'` is
+      // the one that matters most: it says the fault is in the client's rule,
+      // not in this delivery, and without the sentence the partner spends a
+      // morning re-checking files that were never the problem.
+      if (c.verdict === 'not_evaluated') {
+        out(
+          c.details?.abstained === 'rule'
+            ? '    this rule could not be evaluated — nothing to fix on your side; ask your client to check it'
+            : '    the delivery is missing something this check needs — see the hint below, or your client’s contract',
+        )
+      }
+      // A remediation hint answers "how do I fix this?", which only has an
+      // answer when there IS something on the partner's side to fix: a verdict
+      // the engine reached, or a missing precondition they can supply. Never
+      // under a `rule` abstention (the client's rule is what broke) and never
+      // under `not_applicable` (there is nothing to do) — printing one there is
+      // an instruction to fix nothing.
+      const fixable =
+        c.verdict === 'failed' ||
+        c.verdict === 'flagged' ||
+        (c.verdict === 'not_evaluated' && c.details?.abstained !== 'rule')
+      if (fixable && c.remediation) out(`    hint: ${c.remediation}`)
     }
     for (const d of j.deferred) out(`… ${d.check_type.padEnd(w)}  ${d.reason}`)
     for (const m of j.manual) out(`○ ${m.name.padEnd(w)}  ${m.reason}`)
   }
+
+  // ── The verdict of the run ────────────────────────────────────────────────
+  //
+  // Two things block the client's reviewer from validating, not one, and this
+  // command has to report both or it is telling a partner their delivery is
+  // clear when it is not (lib/quality/readiness.ts is the gate; the API's
+  // `blocking_not_evaluated` is the same number computed the same way).
+  //
+  // `must_acknowledge` deliberately does NOT exit 2. It is the rung that asks
+  // the reviewer for a decision and the partner for nothing, and a pipeline
+  // halting on it would put back exactly the over-blocking it was invented to
+  // remove — "to the CRO we block nothing" is the point of it.
+  const blockingAbstained = (j.evaluated ?? []).filter(
+    (c) => c.severity === 'blocking' && c.verdict === 'not_evaluated',
+  )
   if (j.blocking_failures > 0) {
     out(`${j.blocking_failures} blocking issue(s). Fix before pushing to avoid a review round-trip.`)
+  }
+  if (blockingAbstained.length > 0) {
+    out(
+      `${blockingAbstained.length} blocking check(s) could not be evaluated — nobody looked, so your ` +
+        `client's reviewer has to decide on each one by hand before this delivery can be validated.`,
+    )
+    const rule = blockingAbstained.filter((c) => c.details?.abstained === 'rule')
+    if (rule.length > 0) {
+      out(`  ${rule.length} of them because the RULE could not be evaluated, not your delivery: ${rule.map((c) => c.check_type).join(', ')}`)
+    }
+  }
+  if (j.blocking_failures + blockingAbstained.length > 0) {
     process.exitCode = 2
     return
   }
-  out('All checks that run before upload pass.')
+  if (j.must_acknowledge_failures > 0) {
+    out(`${j.must_acknowledge_failures} check(s) your client asked to be told about — nothing to fix; their reviewer records a decision. Not a blocker.`)
+  }
+
+  const k = j.counts
+  // "All checks that run before upload pass" was written when every check
+  // reached a conclusion. It is a green claim over rules nobody ran the moment
+  // one abstains, so it is now reserved for the run where that is literally
+  // true, and the coverage is stated the rest of the time — framed as a
+  // capability to gain, because it is one: more checks running before you send
+  // means fewer rejections after.
+  if (!k || k.passed + k.not_applicable === k.evaluated) {
+    out('All checks that run before upload pass.')
+    return
+  }
+  const rest = []
+  if (k.flagged + k.failed > 0) rest.push(`${k.flagged + k.failed} worth a look`)
+  if (k.not_evaluated > 0) rest.push(`${k.not_evaluated} we could not check`)
+  out(`Nothing blocking. ${k.passed}/${k.evaluated} pass${rest.length ? ` — ${rest.join(', ')}` : ''}.`)
+  if (k.not_evaluated > 0) {
+    out(`Supply what ${k.not_evaluated === 1 ? 'it needs' : 'they need'} and ${k.not_evaluated === 1 ? 'it runs' : 'they run'} on your next delivery.`)
+  }
 }
 
 async function cmdPush() {
@@ -649,8 +773,22 @@ async function cmdPush() {
       // success instead of failing, and so do we — nothing to redo.
       out(`✓ Batch already submitted (${j.status}) — nothing to redo. Track: myelin status ${batchId} --watch`)
     } else {
+      // ⚠ This line used to print three numbers and no denominator, which was
+      // exhaustive until the engine could abstain — after that, an abstention
+      // was in none of the three and "7 passed, 1 flagged, 0 failed" on a
+      // ten-rule contract looked complete while two rules had never been
+      // looked at. `of N` is the denominator that makes the omission visible.
       const a = j.auto_checks
-      out(`Submitting… auto-checks: ${a.passed} passed, ${a.flagged} flagged, ${a.failed} failed.`)
+      const tally = [`${a.passed} passed`, `${a.flagged} flagged`, `${a.failed} failed`]
+      if (a.not_evaluated) tally.push(`${a.not_evaluated} not checked`)
+      if (a.not_applicable) tally.push(`${a.not_applicable} not applicable`)
+      out(`Submitting… auto-checks: ${tally.join(', ')}${a.total ? ` of ${a.total}` : ''}.`)
+      if (a.blocking_not_evaluated) {
+        out(`  ${a.blocking_not_evaluated} blocking check(s) could not be evaluated — your client's reviewer decides on those by hand.`)
+      }
+      if (a.must_acknowledge_failures) {
+        out(`  ${a.must_acknowledge_failures} check(s) your client asked to be told about — nothing to fix.`)
+      }
       out(`✓ Batch submitted for review. Track: myelin status ${batchId} --watch`)
     }
   } else {
@@ -687,6 +825,20 @@ async function cmdStatus() {
       for (const qf of f.quality_findings.filter((x) => x.verdict === 'failed')) {
         out(`  ✗ ${qf.name ?? qf.rule_key}${qf.hint ? `: ${qf.hint}` : ''}`)
       }
+      // ⚠ The filter above is "what to fix", and until 0.10.0 it was the whole
+      // list. A rule the engine could not evaluate is not a failure and never
+      // matched it — so a delivery bounced back partly BECAUSE two checks could
+      // not run showed the partner only the failures, and they resubmitted into
+      // the same wall. An abstention that a partner can clear (`abstained`
+      // anything but 'rule') belongs on this list; a `rule` one belongs to the
+      // client and is printed as such rather than as work owed.
+      for (const qf of f.quality_findings.filter((x) => x.verdict === 'not_evaluated')) {
+        out(
+          qf.details?.abstained === 'rule'
+            ? `  ? ${qf.name ?? qf.rule_key}: this rule could not be evaluated — ask your client to check it`
+            : `  ? ${qf.name ?? qf.rule_key}: we could not check this${qf.hint ? ` — ${qf.hint}` : ''}`,
+        )
+      }
     }
     return batch
   }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@myelinbridge/cli",
-  "version": "0.9.0",
+  "version": "0.10.0",
   "description": "Myelin Partner Ingestion CLI — push R&D data deliveries from a pipeline: preflight against the client's quality rules, resumable upload, submit, track review outcomes.",
   "type": "module",
   "bin": {