@wrangled/cli 0.3.164 → 0.3.166

package/dist/guide.js

@@ -117,7 +117,7 @@ The \`weekly-loop\` playbook (\`${cli} skill install weekly-loop\` installs it a
 
 ## Trying one change: tests
 
-A test writes down one change to a page or funnel step, tried against what runs now. Each test is kept in the workspace's warehouse, in \`cro_tests\` (one row per test), \`cro_test_arms\` (its sides) and \`cro_test_events\` (its history), so you can join tests with visits, sign-ups and sales in one query: \`${cli} warehouse query "SELECT test_id, status, elements FROM cro_tests"\`.
+A test writes down one change to a page or funnel step, tried against what runs now. Each test is kept in the workspace's warehouse, in \`cro_tests\` (one row per test), \`cro_test_arms\` (its sides), \`cro_test_events\` (its history), \`cro_test_results_daily\` (each side's numbers per day) and \`cro_test_reported\` (numbers other tools counted), so you can join tests with visits, sign-ups and sales in one query: \`${cli} warehouse query "SELECT test_id, status, elements FROM cro_tests"\`.
 
 A test moves through these stages: \`idea\` (only the one change and where are needed), \`planned\` (the full plan is in and the owner said yes), \`running\` (its plan is frozen), then \`decided\`, \`abandoned\` or \`invalid\` (the tracking broke or traffic did not split as planned). Where is a funnel step (\`funnel_key\` and \`step_key\` from the funnels map), a page by its address (\`page_url\`, even one not on the map yet), or both. The full plan is: \`because\` (what led to the idea), \`audience\`, \`expected_lift\` (relative, 0.18 for 18%), \`prediction_confidence\` (low, medium or high), \`disproved_if\`, \`design_type\` (how traffic is split: ad_split_randomized, Meta's own A/B test; ad_split_duplicated, two copies of a campaign, one to each page, which is not a true random split; tool_split, inside a page builder or webinar tool named in a \`design_note\`; link_split; before_after; or adaptive), \`traffic\` (\`{campaigns, optimization_goal, note}\`: which campaigns feed it and what they optimize for, frozen with the plan), \`metric\` (step_conversion, which needs a step, sales_per_visitor, revenue_per_visitor, cost_per_sale or cost_per_lead, spend per sign-up) and \`unit\` (people or sessions), \`guardrails\` (numbers that must not get worse), \`baseline_rate\` with \`baseline_from\` and \`baseline_to\`, \`min_lift\`, \`people_per_arm\`, \`min_days\` and \`max_days\` (14 and 56 unless set), \`decision_rule\`, \`concurrent_changes\`, and the control and 1 to 3 \`variants\`, each with the exact \`content\` it shows and, when a tool runs the split, its \`platform_ref\` (the tool's own id for the side). Tags come from fixed lists: \`elements\` (what changes: headline, subheadline, hero, call_to_action, form, offer, price, guarantee, proof, urgency, page_length, layout, checkout, order_bump, follow_up, tool_or_platform, other) and \`levers\` (why it should work: clarity, relevance, trust, value, urgency, less_effort, motivation, other). \`ice_impact\`, \`ice_confidence\` and \`ice_ease\` (1 to 10) sort the idea list. \`evidence\` quotes the proof, each \`{kind, ref, note}\`, kind one of leak, lesson, customer_words, page, ad or other. A test changes one thing: a \`change\` naming two is refused.
 
@@ -125,10 +125,12 @@ Write it as JSON, such as \`${JSON.stringify(funnelTestExample)}\`, and run \`${
 
 A test that already ran, such as one from before ${brand.name}, goes in with \`logged_after: true\` and its \`started_on\` and \`ended_on\`; it goes straight from idea to decided with \`${cli} tests decide <test_id> --decision <d> --outcome <o>\`.
 
-\`${cli} tests results <test_id>\` reads a started test: each side's people, how many moved on to the next step, how many of those people bought (each once, however many times they bought) and what they paid, spend and the test's metric, over the days since it started (a \`before_after\` test compares the same number of days before it started). People are counted by their first visit: the landing page they arrived on, or the Adzara ad (by \`asset_version\` and the test's \`test_id\`) that brought them; a page no one's first visit was recorded on is counted by its site visitors instead. Then it says plainly whether the test can be called yet: "Not enough people yet: 210 of the 400 each side needs.", "Video at the top is ahead, 36% vs 30%, and that gap is big enough to trust." or "No real difference after 900 people each side". A gap counts when a two-proportion test says it is real at 95%; cost per sale and revenue per visitor are judged on sales per visitor under them, and cost per lead on sign-ups per visitor.
+\`${cli} tests results <test_id>\` reads a started test: each side's people, how many moved on to the next step, how many of those people bought (each once, however many times they bought) and what they paid, spend and the test's metric, over the days since it started (a \`before_after\` test compares the same number of days before it started). People are counted by their first visit: the landing page they arrived on, or the Adzara ad (by \`asset_version\` and the test's \`test_id\`) that brought them; a page no one's first visit was recorded on is counted by its visits instead, and a plan whose \`unit\` is sessions counts its pages by visits (a visit is one visitor on the page on one day). A side counted by people is never called against one counted by visits. Each side's numbers per day are in \`cro_test_results_daily\` (a person counts on their first-visit day; spend on the day it was spent), refreshed after each sync, so you can chart a test with \`${cli} warehouse query\`. Once a test is decided, its results are saved with it and \`${cli} tests results\` shows them as they were at the decision. Then it says plainly whether the test can be called yet, with how likely the new version is better: "Too early: day 6 of 14.", "Not enough people yet: 210 of the 400 each side needs.", "Video at the top wins: 97% likely better than letter at the top ..." (95% or more, at least 30 conversions on the smaller side, and \`people_per_arm\` reached or the most days passed), "Leaning toward video at the top, but not proven: 88% likely better ..." (85% to 95%: a hunch to retest, never a win), "No difference found ..." or "The split looks broken ..." (people did not split as planned, so the result is not to be trusted). No call comes before \`min_days\` (14 unless set), and once they pass only whole weeks from the start count. With 2 or 3 variants each needs a higher chance. Every call ends with an evidence grade, A to D: before_after and adaptive tests and numbers another tool counted are never above C. Cost per sale and revenue per visitor are judged on sales per visitor under them, and cost per lead on sign-ups per visitor. When a test is planned, ${brand.name} estimates from the control's last 28 days how many people each side needs and how many weeks that takes; \`${cli} tests show\` prints it as "Needs", and a plan with no \`people_per_arm\` takes the estimate's.
 
 \`${cli} tests decide <test_id> --decision <d> --outcome <won|lost|inconclusive|invalid> --note "<why>"\` records the decision on a running test: \`scale\` (the new version won; use it everywhere), \`continue\` (keep running), \`iterate\` (promising; it also writes down a sharper version linked to this one), \`hold\` (pause it; something outside the test got in the way) or \`stop\` (it lost or made no difference). scale, iterate and stop need \`--outcome\`. Each decision writes a lesson with the test, its pages and its step as proof, in one go with the decision. Deciding changes nothing live: switching a page or ads still goes through approval.
 
+When the split runs where ${brand.name} cannot count it (GoHighLevel's own split-test stats, a Mailchimp subject test, a Meta A/B test), enter each side's numbers as the tool shows them with \`${cli} tests report <test_id> --file counts.json\`, for a running test or one written down after it ran. The file is \`{"source_tool": "GoHighLevel", "unit": "people", "from": "2026-09-01", "to": "2026-09-30", "arms": [{"arm": "control", "people": 400, "conversions": 80}, {"arm": "variant_1", "people": 410, "conversions": 110}]}\`: conversions is what the test is judged on, as the tool counts it (moved on, bought or signed up), and \`revenue\`, \`spend\`, \`currency\`, \`tool_confidence\` (the tool's own confidence, such as Meta's 87% as 0.87) and \`note\` are optional. They are kept in \`cro_test_reported\`, apart from what ${brand.name} counts, and \`${cli} tests results\` calls them on their own with words starting "Counted by <tool>". To correct them, report the same tool, sides and dates again; the latest wins. A tool's own confidence is shown as its figure, never as the chance the variant is better, and Meta's purchase count alone never calls a test judged on sales: compare it with the sales in the client's own records.
+
 The \`cro\` playbook (\`${cli} skill install cro\` installs it as the \`${CRO_SKILL_NAME}\` skill; \`${cli} docs cro\` prints it) runs this once a week: it turns the costliest leaks from \`${cli} funnels leaks\` into at most 3 test ideas, each with one change, its full plan and its proof, for the owner to approve in the agent's own session, and reads the tests already running. It never publishes, edits a live page, or touches ads or messages.
 
 ## Content: search before SQL
@@ -589,7 +591,7 @@ ${cli} number-alerts [--since <date>]   the key numbers that swung far from norm
 ${cli} facts propose --file <fact.json>   suggest a fact for a person to confirm
 ${cli} lessons [--status active|retired] [--about ads|pages|funnel|messages|cross_channel] [--dimension <d>] | lessons add --file <lesson.json> | lessons retire <id> --note "<why>"   lessons from winners and flops, writing one, retiring one
 ${cli} tests p1   this brand's P1 test ads by verdict, judged on its own sales against its P1 thresholds, with Meta's count beside
-${cli} tests [--status <status>] [--funnel <key>] [--limit <n>] | tests show <id> | tests draft --file <test.json> | tests start <id> | tests abandon <id> [--note "<why>"] | tests results <id> | tests decide <id> --decision <d> [--note "<why>"]   funnel tests, one change each
+${cli} tests [--status <status>] [--funnel <key>] [--limit <n>] | tests show <id> | tests draft --file <test.json> | tests start <id> | tests abandon <id> [--note "<why>"] | tests results <id> | tests decide <id> --decision <d> [--note "<why>"] | tests report <id> --file <counts.json>   funnel tests, one change each
 ${cli} ads tags [--ad <id>] [--status <status>] [--limit <n>] | ads tag --file <tags.json>   the marketing tags of ads not made here, and suggesting them
 ${cli} ads names [--unparsed]   Meta ad names and the tags each gives; --unparsed lists those that fit no pattern
 ${cli} ads tag-formats   tag every untagged ad's format (image, video or carousel) from Meta's records, free; run it before ads autotag

package/dist/main.js

@@ -3258,10 +3258,10 @@ const commands = {
             if (args.length !== 1 || !options.approvedBy)
                 throw new CliError('Usage: tests plan <test_id> --approved-by "<owner>" [--note "<what they said>"]', exitCodes.usage);
             const body = { approved_by: options.approvedBy, ...(options.note ? { note: options.note } : {}) };
-            const { test } = await api(await resolveConnection(options), `${testPath(args[0])}/plan`, body);
+            const planned = await api(await resolveConnection(options), `${testPath(args[0])}/plan`, body);
             if (options.format === "json")
-                return printJson({ test });
-            console.log(`Test ${test.test_id} is planned; ${options.approvedBy} said yes.`);
+                return printJson(planned);
+            console.log(`Test ${planned.test.test_id} is planned; ${options.approvedBy} said yes, ${planned.test.people_per_arm} people each side.\nNeeds: ${planned.estimate.words}`);
         },
     },
     "tests start": {
@@ -3342,6 +3342,18 @@ const commands = {
                 console.log(`Wrote down the next version as ${next.test_id}.`);
         },
     },
+    "tests report": {
+        usage: "tests report <test_id> --file <counts.json> | < counts.json   enter each side's numbers as another tool counted them (GoHighLevel, Meta, Mailchimp...), for a running test or one written down after it ran; reporting the same tool, side and dates again corrects them",
+        async run(args, options) {
+            if (args.length !== 1)
+                throw new CliError("Usage: tests report <test_id> --file <counts.json> | < counts.json", exitCodes.usage);
+            const reported = await api(await resolveConnection(options), `${testPath(args[0])}/reported`, await readToolInput(options));
+            if (options.format === "json")
+                return printJson(reported);
+            console.log(`Kept the numbers for test ${reported.test.test_id}.`);
+            printReported(reported.reported);
+        },
+    },
     "lessons retire": {
         usage: 'lessons retire <id> --note "<why it is no longer true>"   retire a lesson the numbers no longer back; it is kept with why',
         async run(args, options) {
@@ -4831,6 +4843,7 @@ function printTest(t, events) {
         ...t.arms.filter((a) => a.content !== null).map((a) => `  ${a.arm} shows: ${excerpt(a.content, 100)}`),
         ...line("Split", t.design_type && `${t.design_type}${t.design_note ? `, ${t.design_note}` : ""}`),
         ...line("Judged by", t.metric && `${t.metric} by ${t.unit ?? "people"}${t.people_per_arm ? `, ${t.people_per_arm} each side` : ""}, ${t.min_days} to ${t.max_days} days`),
+        ...line("Needs", plannedEstimate(events)?.words),
         ...line("Today", pct(t.baseline_rate) && `${pct(t.baseline_rate)} from ${t.baseline_from} to ${t.baseline_to}; smallest lift worth finding ${pct(t.min_lift)}`),
         ...line("Must not get worse", t.guardrails.join(", ")),
         ...line("Decision rule", t.decision_rule),
@@ -4848,6 +4861,10 @@ function printTest(t, events) {
         lines.push("History:", ...events.map((e) => `  ${e.at.slice(0, 16).replace("T", " ")}  ${eventWords(e)}  (${e.author})`));
     console.log(lines.join("\n"));
 }
+/** The estimate the test was planned with, from its latest move to planned. */
+function plannedEstimate(events) {
+    return events.findLast((e) => e.kind === "status" && e.detail.to === "planned" && e.detail.estimate !== undefined)?.detail.estimate;
+}
 function eventWords(e) {
     const d = e.detail;
     switch (e.kind) {
@@ -4861,6 +4878,8 @@ function eventWords(e) {
             return `note: ${d.note}`;
         case "decided":
             return `decided ${d.decision}${d.outcome ? ` (${d.outcome})` : ""}${d.note ? `: ${d.note}` : ""}`;
+        case "reported":
+            return `${d.source_tool}'s numbers for ${(d.arms ?? []).join(", ")}, ${d.from} to ${d.to}`;
     }
 }
 const RESULT_COLUMNS = ["side", "label", "counted by", "people", "moved on", "sign-ups", "buyers", "revenue", "spend", "metric", "dates"];
@@ -4879,10 +4898,33 @@ function printResults(r) {
         metric: a.metric_value === null ? "" : r.test.metric === "cost_per_sale" || r.test.metric === "cost_per_lead" || r.test.metric === "revenue_per_visitor" ? money(a.metric_value, a.currency) : `${(a.metric_value * 100).toFixed(1)}%`,
         dates: `${a.window.from} to ${a.window.to}`,
     }));
-    const lines = [`${r.test.test_id} (${r.test.status})   judged by ${r.test.metric ?? "no main number"}, ${r.test.people_per_arm ?? "no count of"} people each side`, renderTable(RESULT_COLUMNS, rows), "", ...r.calls.map((c) => c.words)];
+    const saved = r.saved_at === null ? "" : `   numbers as saved at the decision, ${r.saved_at.slice(0, 10)}`;
+    const lines = [`${r.test.test_id} (${r.test.status})   judged by ${r.test.metric ?? "no main number"}, ${r.test.people_per_arm ?? "no count of"} people each side${saved}`];
+    if (r.arms.length > 0)
+        lines.push(renderTable(RESULT_COLUMNS, rows), "", ...r.calls.map((c) => c.words));
+    lines.push(...(r.words && r.arms.every((a) => a.people === 0) ? ["", r.words] : []));
     if (r.caveats.length > 0)
         lines.push("", ...r.caveats.map((c) => `Note: ${c}`));
     console.log(lines.join("\n"));
+    printReported(r.reported);
+}
+const REPORTED_COLUMNS = ["side", "label", "counted", "conversions", "revenue", "spend", "tool's confidence", "dates", "entered"];
+function printReported(reported) {
+    const money = (n, currency) => (n === null ? "" : `${n.toFixed(2)}${currency ? ` ${currency}` : ""}`);
+    for (const r of reported) {
+        const rows = r.arms.map((a) => ({
+            side: a.arm,
+            label: excerpt(a.label, 30),
+            counted: `${a.people} ${r.unit}`,
+            conversions: a.conversions ?? "",
+            revenue: money(a.revenue, a.currency),
+            spend: money(a.spend, a.currency),
+            "tool's confidence": a.tool_confidence === null ? "" : `${(a.tool_confidence * 100).toFixed(0)}%`,
+            dates: `${a.from} to ${a.to}`,
+            entered: a.entered_at.slice(0, 10),
+        }));
+        console.log(["", `Reported by ${r.source_tool}`, renderTable(REPORTED_COLUMNS, rows), "", r.words, ...r.caveats.map((c) => `Note: ${c}`)].join("\n"));
+    }
 }
 const LESSON_COLUMNS = ["id", "seen", "last week", "dimension", "lesson"];
 function lessonRow(l) {

package/dist/playbooks.js

@@ -2134,10 +2134,13 @@ ${cli} tests results <test_id> --format json > results-1.json
 
 Without a shell: MCP \`get_funnel_test_results\` with \`{"id": "<test_id>"}\`.
 
-\`calls\` has one call for each changed version against the control: \`not_enough_yet\` (with the \`people\` each side has and the \`need\`), \`variant_ahead\`, \`control_ahead\` or \`no_real_difference\`, and \`words\` says it plainly. \`caveats\` names what weakens the call.
+\`calls\` has one call for each changed version against the control: \`split_broken\`, \`too_early\` (with the \`day\` and the \`of\`), \`not_enough_yet\` (with the \`people\` each side has and the \`need\`), \`variant_wins\` or \`control_wins\` (95% or more likely better), \`variant_directional\` or \`control_directional\` (85% to 95%: leaning, not proven) or \`no_difference_found\`, and \`words\` says it plainly. \`grade\` (A to D) says how much to trust it and \`grade_reason\` why. \`caveats\` names what weakens the call.
 
-- **Every call is \`not_enough_yet\`:** leave the test running; that needs no command. Only when something outside the test got in the way (the page broke, a price changed, the ads stopped), suggest \`hold\` to the owner.
-- **A call is ready:** suggest \`scale\` (the new version won), \`stop\` (it lost or made no difference) or \`iterate\` (promising; it writes down a sharper version). \`iterate\` writes a new test, so suggest it only while the week is under its 3; otherwise suggest \`stop\` and draft the sharper version next week.
+- **Every call is \`too_early\` or \`not_enough_yet\`:** leave the test running; that needs no command.
+- **A call is \`split_broken\`:** people did not split between the sides as planned. Tell the owner the result cannot be trusted, and suggest marking the test invalid (\`${cli} tests invalidate <test_id> --note "<what went wrong>"\`) once the split is checked.
+- **A call is directional:** it is a hunch, not a win. Suggest keeping the test running to its most days, or \`iterate\` to retest a sharper version; never \`scale\` on it. Only when something outside the test got in the way (the page broke, a price changed, the ads stopped), suggest \`hold\` to the owner.
+- **The split runs in another tool** (a \`tool_split\` in GoHighLevel, a Meta A/B test): read each side's numbers in that tool and enter them with \`${cli} tests report <test_id> --file counts.json\` (MCP \`report_test_numbers\`), then read the results again; their words start "Counted by <tool>". Never decide a test judged on sales on Meta's purchase count alone.
+- **A call is ready** (a win, or no difference found): suggest \`scale\` (the new version won), \`stop\` (it lost or made no difference) or \`iterate\` (promising; it writes down a sharper version). \`iterate\` writes a new test, so suggest it only while the week is under its 3; otherwise suggest \`stop\` and draft the sharper version next week.
 
 Put each suggestion to the owner with the ideas from step 4, then record the owner's decision:
 
@@ -2168,6 +2171,7 @@ ${cli} tests plan <test_id> --approved-by "<owner>"        record the owner's ye
 ${cli} tests start <test_id>                               record that a planned, live test is running
 ${cli} tests abandon <test_id> --note "<why>"              drop a test the owner rejected
 ${cli} tests results <test_id> --format json              a running test's numbers per side and whether it can be called
+${cli} tests report <test_id> --file <counts.json>         enter each side's numbers from the tool that runs the split
 ${cli} tests decide <test_id> --decision <d> --outcome <o> --note "<why>"   record the owner's decision; it writes the lesson
 \`\`\`
 `;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@wrangled/cli",
-  "version": "0.3.164",
+  "version": "0.3.166",
   "description": "Command-line access to your marketing data warehouse, for people and coding agents",
   "type": "module",
   "bin": {