@wrangled/cli 0.3.165 → 0.3.166

package/dist/guide.js

@@ -125,7 +125,7 @@ Write it as JSON, such as \`${JSON.stringify(funnelTestExample)}\`, and run \`${
 
 A test that already ran, such as one from before ${brand.name}, goes in with \`logged_after: true\` and its \`started_on\` and \`ended_on\`; it goes straight from idea to decided with \`${cli} tests decide <test_id> --decision <d> --outcome <o>\`.
 
-\`${cli} tests results <test_id>\` reads a started test: each side's people, how many moved on to the next step, how many of those people bought (each once, however many times they bought) and what they paid, spend and the test's metric, over the days since it started (a \`before_after\` test compares the same number of days before it started). People are counted by their first visit: the landing page they arrived on, or the Adzara ad (by \`asset_version\` and the test's \`test_id\`) that brought them; a page no one's first visit was recorded on is counted by its visits instead, and a plan whose \`unit\` is sessions counts its pages by visits (a visit is one visitor on the page on one day). A side counted by people is never called against one counted by visits. Each side's numbers per day are in \`cro_test_results_daily\` (a person counts on their first-visit day; spend on the day it was spent), refreshed after each sync, so you can chart a test with \`${cli} warehouse query\`. Once a test is decided, its results are saved with it and \`${cli} tests results\` shows them as they were at the decision. Then it says plainly whether the test can be called yet: "Not enough people yet: 210 of the 400 each side needs.", "Video at the top is ahead, 36% vs 30%, and that gap is big enough to trust." or "No real difference after 900 people each side". A gap counts when a two-proportion test says it is real at 95%; cost per sale and revenue per visitor are judged on sales per visitor under them, and cost per lead on sign-ups per visitor.
+\`${cli} tests results <test_id>\` reads a started test: each side's people, how many moved on to the next step, how many of those people bought (each once, however many times they bought) and what they paid, spend and the test's metric, over the days since it started (a \`before_after\` test compares the same number of days before it started). People are counted by their first visit: the landing page they arrived on, or the Adzara ad (by \`asset_version\` and the test's \`test_id\`) that brought them; a page no one's first visit was recorded on is counted by its visits instead, and a plan whose \`unit\` is sessions counts its pages by visits (a visit is one visitor on the page on one day). A side counted by people is never called against one counted by visits. Each side's numbers per day are in \`cro_test_results_daily\` (a person counts on their first-visit day; spend on the day it was spent), refreshed after each sync, so you can chart a test with \`${cli} warehouse query\`. Once a test is decided, its results are saved with it and \`${cli} tests results\` shows them as they were at the decision. Then it says plainly whether the test can be called yet, with how likely the new version is better: "Too early: day 6 of 14.", "Not enough people yet: 210 of the 400 each side needs.", "Video at the top wins: 97% likely better than letter at the top ..." (95% or more, at least 30 conversions on the smaller side, and \`people_per_arm\` reached or the most days passed), "Leaning toward video at the top, but not proven: 88% likely better ..." (85% to 95%: a hunch to retest, never a win), "No difference found ..." or "The split looks broken ..." (people did not split as planned, so the result is not to be trusted). No call comes before \`min_days\` (14 unless set), and once they pass only whole weeks from the start count. With 2 or 3 variants each needs a higher chance. Every call ends with an evidence grade, A to D: before_after and adaptive tests and numbers another tool counted are never above C. Cost per sale and revenue per visitor are judged on sales per visitor under them, and cost per lead on sign-ups per visitor. When a test is planned, ${brand.name} estimates from the control's last 28 days how many people each side needs and how many weeks that takes; \`${cli} tests show\` prints it as "Needs", and a plan with no \`people_per_arm\` takes the estimate's.
 
 \`${cli} tests decide <test_id> --decision <d> --outcome <won|lost|inconclusive|invalid> --note "<why>"\` records the decision on a running test: \`scale\` (the new version won; use it everywhere), \`continue\` (keep running), \`iterate\` (promising; it also writes down a sharper version linked to this one), \`hold\` (pause it; something outside the test got in the way) or \`stop\` (it lost or made no difference). scale, iterate and stop need \`--outcome\`. Each decision writes a lesson with the test, its pages and its step as proof, in one go with the decision. Deciding changes nothing live: switching a page or ads still goes through approval.
 

package/dist/main.js

@@ -3258,10 +3258,10 @@ const commands = {
             if (args.length !== 1 || !options.approvedBy)
                 throw new CliError('Usage: tests plan <test_id> --approved-by "<owner>" [--note "<what they said>"]', exitCodes.usage);
             const body = { approved_by: options.approvedBy, ...(options.note ? { note: options.note } : {}) };
-            const { test } = await api(await resolveConnection(options), `${testPath(args[0])}/plan`, body);
+            const planned = await api(await resolveConnection(options), `${testPath(args[0])}/plan`, body);
             if (options.format === "json")
-                return printJson({ test });
-            console.log(`Test ${test.test_id} is planned; ${options.approvedBy} said yes.`);
+                return printJson(planned);
+            console.log(`Test ${planned.test.test_id} is planned; ${options.approvedBy} said yes, ${planned.test.people_per_arm} people each side.\nNeeds: ${planned.estimate.words}`);
         },
     },
     "tests start": {
@@ -4843,6 +4843,7 @@ function printTest(t, events) {
         ...t.arms.filter((a) => a.content !== null).map((a) => `  ${a.arm} shows: ${excerpt(a.content, 100)}`),
         ...line("Split", t.design_type && `${t.design_type}${t.design_note ? `, ${t.design_note}` : ""}`),
         ...line("Judged by", t.metric && `${t.metric} by ${t.unit ?? "people"}${t.people_per_arm ? `, ${t.people_per_arm} each side` : ""}, ${t.min_days} to ${t.max_days} days`),
+        ...line("Needs", plannedEstimate(events)?.words),
         ...line("Today", pct(t.baseline_rate) && `${pct(t.baseline_rate)} from ${t.baseline_from} to ${t.baseline_to}; smallest lift worth finding ${pct(t.min_lift)}`),
         ...line("Must not get worse", t.guardrails.join(", ")),
         ...line("Decision rule", t.decision_rule),
@@ -4860,6 +4861,10 @@ function printTest(t, events) {
         lines.push("History:", ...events.map((e) => `  ${e.at.slice(0, 16).replace("T", " ")}  ${eventWords(e)}  (${e.author})`));
     console.log(lines.join("\n"));
 }
+/** The estimate the test was planned with, from its latest move to planned. */
+function plannedEstimate(events) {
+    return events.findLast((e) => e.kind === "status" && e.detail.to === "planned" && e.detail.estimate !== undefined)?.detail.estimate;
+}
 function eventWords(e) {
     const d = e.detail;
     switch (e.kind) {

package/dist/playbooks.js

@@ -2134,11 +2134,13 @@ ${cli} tests results <test_id> --format json > results-1.json
 
 Without a shell: MCP \`get_funnel_test_results\` with \`{"id": "<test_id>"}\`.
 
-\`calls\` has one call for each changed version against the control: \`not_enough_yet\` (with the \`people\` each side has and the \`need\`), \`variant_ahead\`, \`control_ahead\` or \`no_real_difference\`, and \`words\` says it plainly. \`caveats\` names what weakens the call.
+\`calls\` has one call for each changed version against the control: \`split_broken\`, \`too_early\` (with the \`day\` and the \`of\`), \`not_enough_yet\` (with the \`people\` each side has and the \`need\`), \`variant_wins\` or \`control_wins\` (95% or more likely better), \`variant_directional\` or \`control_directional\` (85% to 95%: leaning, not proven) or \`no_difference_found\`, and \`words\` says it plainly. \`grade\` (A to D) says how much to trust it and \`grade_reason\` why. \`caveats\` names what weakens the call.
 
-- **Every call is \`not_enough_yet\`:** leave the test running; that needs no command. Only when something outside the test got in the way (the page broke, a price changed, the ads stopped), suggest \`hold\` to the owner.
+- **Every call is \`too_early\` or \`not_enough_yet\`:** leave the test running; that needs no command.
+- **A call is \`split_broken\`:** people did not split between the sides as planned. Tell the owner the result cannot be trusted, and suggest marking the test invalid (\`${cli} tests invalidate <test_id> --note "<what went wrong>"\`) once the split is checked.
+- **A call is directional:** it is a hunch, not a win. Suggest keeping the test running to its most days, or \`iterate\` to retest a sharper version; never \`scale\` on it. Only when something outside the test got in the way (the page broke, a price changed, the ads stopped), suggest \`hold\` to the owner.
 - **The split runs in another tool** (a \`tool_split\` in GoHighLevel, a Meta A/B test): read each side's numbers in that tool and enter them with \`${cli} tests report <test_id> --file counts.json\` (MCP \`report_test_numbers\`), then read the results again; their words start "Counted by <tool>". Never decide a test judged on sales on Meta's purchase count alone.
-- **A call is ready:** suggest \`scale\` (the new version won), \`stop\` (it lost or made no difference) or \`iterate\` (promising; it writes down a sharper version). \`iterate\` writes a new test, so suggest it only while the week is under its 3; otherwise suggest \`stop\` and draft the sharper version next week.
+- **A call is ready** (a win, or no difference found): suggest \`scale\` (the new version won), \`stop\` (it lost or made no difference) or \`iterate\` (promising; it writes down a sharper version). \`iterate\` writes a new test, so suggest it only while the week is under its 3; otherwise suggest \`stop\` and draft the sharper version next week.
 
 Put each suggestion to the owner with the ideas from step 4, then record the owner's decision:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@wrangled/cli",
-  "version": "0.3.165",
+  "version": "0.3.166",
   "description": "Command-line access to your marketing data warehouse, for people and coding agents",
   "type": "module",
   "bin": {