@wrangled/cli 0.3.161 → 0.3.163
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/guide.js +33 -16
- package/dist/main.js +156 -48
- package/dist/playbooks.js +48 -24
- package/package.json +1 -1
package/dist/guide.js
CHANGED
|
@@ -7,17 +7,30 @@ import { PULL_STATE_FILE } from "./pull.js";
|
|
|
7
7
|
const cli = brand.slug;
|
|
8
8
|
/** The funnel test the guide shows; the app's parser must accept it (src/lib/cli-consistency.test.ts). */
|
|
9
9
|
export const funnelTestExample = {
|
|
10
|
-
|
|
10
|
+
name: "Video at the top of the sign-up page",
|
|
11
11
|
funnel_key: "webinar",
|
|
12
12
|
step_key: "signup",
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
split_method: "adzara_ads",
|
|
13
|
+
change: "top of the page: letter to video",
|
|
14
|
+
because: "Buyers on sales calls say they did not know what the webinar was about, and sign-ups per visitor fell from 30% to 22% this month.",
|
|
15
|
+
audience: "new visitors from Facebook ads",
|
|
16
|
+
expected_lift: 0.18,
|
|
17
|
+
prediction_confidence: "medium",
|
|
18
|
+
disproved_if: "sign-ups per visitor rise less than 5% after 400 people each side",
|
|
20
19
|
evidence: [{ kind: "customer_words", ref: "sales call 2026-09-30", note: "I didn't know what the webinar was about." }],
|
|
20
|
+
design_type: "ad_split_duplicated",
|
|
21
|
+
metric: "step_conversion",
|
|
22
|
+
unit: "people",
|
|
23
|
+
guardrails: ["cost per sale"],
|
|
24
|
+
baseline_rate: 0.22,
|
|
25
|
+
baseline_from: "2026-09-01",
|
|
26
|
+
baseline_to: "2026-09-30",
|
|
27
|
+
min_lift: 0.1,
|
|
28
|
+
people_per_arm: 400,
|
|
29
|
+
decision_rule: "Scale the video if sign-ups per visitor rise at least 10% with cost per sale no worse.",
|
|
30
|
+
elements: ["hero"],
|
|
31
|
+
levers: ["clarity"],
|
|
32
|
+
control: { label: "letter at the top", content: "A letter from the founder: why this webinar exists.", url: "https://example.com/signup" },
|
|
33
|
+
variants: [{ label: "video at the top", content: "A 90-second video of the founder saying what the webinar covers.", url: "https://example.com/signup-video" }],
|
|
21
34
|
};
|
|
22
35
|
/** The dashboard the guide shows; the app's parser must accept it (src/lib/cli-consistency.test.ts). */
|
|
23
36
|
export const dashboardExample = {
|
|
@@ -102,17 +115,21 @@ Write one as JSON, \`{"text": "...", "dimension": "hook", "evidence": [{"ad": "<
|
|
|
102
115
|
|
|
103
116
|
The \`weekly-loop\` playbook (\`${cli} skill install weekly-loop\` installs it as the \`${WEEKLY_LOOP_SKILL_NAME}\` skill; \`${cli} docs weekly-loop\` prints it) is a sample to borrow from, not a schedule, for a local agent that runs a client's Facebook ads with its owner: its steps can run daily, several times a week or weekly, and the winners report covers whatever dates are asked for. It names the command and the MCP tool for each step: the winners and flops report, lessons, fresh customer words, the brief, the offers and funnels map, making ads (Exodus, paid tools or the agent's own skills), publishing approved ads, pauses and budget changes with ${brand.name}'s Meta commands, and the results of what was published (Adzara's labels for ads it published before the switch). Judge winners by ${brand.name}'s Winners and flops report, which judges each client on its goal (\`${cli} winners\`); Meta's own results are Meta's count. Approvals and decisions happen in the agent's own session with the owner; ${brand.name} records none. Ads are published and changed through ${brand.name}'s Meta commands, only after the owner approves and only inside the client's control list.
|
|
104
117
|
|
|
105
|
-
## Trying one change:
|
|
118
|
+
## Trying one change: tests
|
|
119
|
+
|
|
120
|
+
A test writes down one change to a page or funnel step, tried against what runs now. Each test is kept in the workspace's warehouse, in \`cro_tests\` (one row per test), \`cro_test_arms\` (its sides) and \`cro_test_events\` (its history), so you can join tests with visits, sign-ups and sales in one query: \`${cli} warehouse query "SELECT test_id, status, elements FROM cro_tests"\`.
|
|
121
|
+
|
|
122
|
+
A test moves through these stages: \`idea\` (only the one change and where are needed), \`planned\` (the full plan is in and the owner said yes), \`running\` (its plan is frozen), then \`decided\`, \`abandoned\` or \`invalid\` (the tracking broke or traffic did not split as planned). Where is a funnel step (\`funnel_key\` and \`step_key\` from the funnels map), a page by its address (\`page_url\`, even one not on the map yet), or both. The full plan is: \`because\` (what led to the idea), \`audience\`, \`expected_lift\` (relative, 0.18 for 18%), \`prediction_confidence\` (low, medium or high), \`disproved_if\`, \`design_type\` (how traffic is split: ad_split_randomized, Meta's own A/B test; ad_split_duplicated, two copies of a campaign, one to each page, which is not a true random split; tool_split, inside a page builder or webinar tool named in a \`design_note\`; link_split; before_after; or adaptive), \`traffic\` (\`{campaigns, optimization_goal, note}\`: which campaigns feed it and what they optimize for, frozen with the plan), \`metric\` (step_conversion, which needs a step, sales_per_visitor, revenue_per_visitor, cost_per_sale or cost_per_lead, spend per sign-up) and \`unit\` (people or sessions), \`guardrails\` (numbers that must not get worse), \`baseline_rate\` with \`baseline_from\` and \`baseline_to\`, \`min_lift\`, \`people_per_arm\`, \`min_days\` and \`max_days\` (14 and 56 unless set), \`decision_rule\`, \`concurrent_changes\`, and the control and 1 to 3 \`variants\`, each with the exact \`content\` it shows and, when a tool runs the split, its \`platform_ref\` (the tool's own id for the side). Tags come from fixed lists: \`elements\` (what changes: headline, subheadline, hero, call_to_action, form, offer, price, guarantee, proof, urgency, page_length, layout, checkout, order_bump, follow_up, tool_or_platform, other) and \`levers\` (why it should work: clarity, relevance, trust, value, urgency, less_effort, motivation, other). \`ice_impact\`, \`ice_confidence\` and \`ice_ease\` (1 to 10) sort the idea list. \`evidence\` quotes the proof, each \`{kind, ref, note}\`, kind one of leak, lesson, customer_words, page, ad or other. A test changes one thing: a \`change\` naming two is refused.
|
|
106
123
|
|
|
107
|
-
|
|
124
|
+
Write it as JSON, such as \`${JSON.stringify(funnelTestExample)}\`, and run \`${cli} tests draft --file test.json\`. It gets Adzara-style labels from the change and this month (\`hyp-<slug>\` and \`test-<yyyy-mm>-<slug>\`); give \`hypothesis_id\` and \`test_id\` to reuse Adzara's own, so the results of the ads Adzara published for the test (\`ad_test_results_daily\`, \`ad_hypothesis_results_daily\`) join on the same label. \`${cli} tests edit <test_id> --file changes.json\` changes some fields (null clears one). ${brand.name} shows no approval screen: ask the owner in your own session, then record the yes with \`${cli} tests plan <test_id> --approved-by "<owner>"\`, which refuses a plan with a gap and names every missing field. \`${cli} tests start <test_id>\` marks a planned test as running from today and freezes its plan; an edit after that is still saved, and its history and results say so. Starting changes nothing live: the page still goes live through its own approval, and the ads through ${brand.name}'s Meta commands. Until #2202, an ad split test can be measured only on ads Adzara made before the switch, because its sides are counted from Adzara's labels; a \`before_after\` test still works. \`${cli} tests abandon <test_id> --note "<why>"\` drops one that has not ended, \`${cli} tests invalidate <test_id> --note "<what went wrong>"\` ends a running one whose numbers cannot be trusted, and \`${cli} tests note <test_id> --note "<note>"\` adds to its history. \`${cli} tests [--status <status>] [--funnel <funnel_key>] [--limit <n>]\` lists them, newest first (50 unless \`--limit\`, at most 200), and \`${cli} tests show <test_id>\` shows one with its history.
|
|
108
125
|
|
|
109
|
-
|
|
126
|
+
A test that already ran, such as one from before ${brand.name}, goes in with \`logged_after: true\` and its \`started_on\` and \`ended_on\`; it goes straight from idea to decided with \`${cli} tests decide <test_id> --decision <d> --outcome <o>\`.
|
|
110
127
|
|
|
111
|
-
\`${cli} tests results <test_id>\` reads a started test: each side's people, how many moved on to the next step, how many of those people bought (each once, however many times they bought) and what they paid, spend and the test's metric, over the days since it started (a \`before_after\` test compares the same number of days before it started). People are counted by their first visit: the landing page they arrived on, or the Adzara ad (by \`asset_version\` and the test's \`test_id\`) that brought them; a page no one's first visit was recorded on is counted by its site visitors instead. Then it says plainly whether the test can be called yet: "Not enough people yet: 210 of the 400 each side needs.", "Video at the top is ahead, 36% vs 30%, and that gap is big enough to trust." or "No real difference after 900 people each side". A gap counts when a two-proportion test says it is real at 95%; cost per sale and revenue per visitor are judged on sales per visitor under them.
|
|
128
|
+
\`${cli} tests results <test_id>\` reads a started test: each side's people, how many moved on to the next step, how many of those people bought (each once, however many times they bought) and what they paid, spend and the test's metric, over the days since it started (a \`before_after\` test compares the same number of days before it started). People are counted by their first visit: the landing page they arrived on, or the Adzara ad (by \`asset_version\` and the test's \`test_id\`) that brought them; a page no one's first visit was recorded on is counted by its site visitors instead. Then it says plainly whether the test can be called yet: "Not enough people yet: 210 of the 400 each side needs.", "Video at the top is ahead, 36% vs 30%, and that gap is big enough to trust." or "No real difference after 900 people each side". A gap counts when a two-proportion test says it is real at 95%; cost per sale and revenue per visitor are judged on sales per visitor under them, and cost per lead on sign-ups per visitor.
|
|
112
129
|
|
|
113
|
-
\`${cli} tests decide <test_id> --decision <d> --note "<why>"\` records the decision on a running test: \`scale\` (the new version won; use it everywhere), \`continue\` (keep running), \`iterate\` (promising; it also
|
|
130
|
+
\`${cli} tests decide <test_id> --decision <d> --outcome <won|lost|inconclusive|invalid> --note "<why>"\` records the decision on a running test: \`scale\` (the new version won; use it everywhere), \`continue\` (keep running), \`iterate\` (promising; it also writes down a sharper version linked to this one), \`hold\` (pause it; something outside the test got in the way) or \`stop\` (it lost or made no difference). scale, iterate and stop need \`--outcome\`. Each decision writes a lesson with the test, its pages and its step as proof, in one go with the decision. Deciding changes nothing live: switching a page or ads still goes through approval.
|
|
114
131
|
|
|
115
|
-
The \`cro\` playbook (\`${cli} skill install cro\` installs it as the \`${CRO_SKILL_NAME}\` skill; \`${cli} docs cro\` prints it) runs this once a week: it turns the costliest leaks from \`${cli} funnels leaks\` into at most 3 test
|
|
132
|
+
The \`cro\` playbook (\`${cli} skill install cro\` installs it as the \`${CRO_SKILL_NAME}\` skill; \`${cli} docs cro\` prints it) runs this once a week: it turns the costliest leaks from \`${cli} funnels leaks\` into at most 3 test ideas, each with one change, its full plan and its proof, for the owner to approve in the agent's own session, and reads the tests already running. It never publishes, edits a live page, or touches ads or messages.
|
|
116
133
|
|
|
117
134
|
## Content: search before SQL
|
|
118
135
|
|
|
@@ -324,7 +341,7 @@ Some clients review every batch before it goes out. \`${cli} review settings\` s
|
|
|
324
341
|
|
|
325
342
|
Each workspace keeps one map of what the client sells (the offer stack) and the steps a person walks from first seeing an ad to buying (its funnels). \`${cli} funnels show\` prints it in plain words: the version and who confirmed it, each offer in stack order with its tier, price and delivery, then each funnel's numbered steps with the offer it sells, its assets (marked \`(not found)\` when the id is not in synced data yet) and what counts as reaching it. It ends with the draft on hold, if any, printed in full. \`--version <n>\` prints an older version; \`--format json\` prints the whole map, which is the shape a draft takes.
|
|
326
343
|
|
|
327
|
-
To change the map, write the whole map as JSON (\`{"offers": [...], "funnels": [...]}\`, the \`map\` of \`${cli} funnels show --format json\`) and run \`${cli} funnels draft --file map.json --note "<what changed and why>"\`. Add \`--evidence '<json>'\` or \`@file.json\` with what you looked at: counts and ids, never names, emails or phones. It is checked against synced data, and when every step's rule is filled in it goes live at once, confirmed by \`system:auto\`; a person can undo it in the app. A map with a rule not filled in stays on hold as the draft and counts nothing; saving replaces the draft on hold. Never invent an id or a number to fill a rule. Pass \`--based-on <n>\` with the version you read (the draft on hold, else the confirmed map): if someone saved since, the save is refused as \`draft_conflict\`, and you show again and redo your change on top. Leave out \`--based-on\` only for the first map, when there is neither a map nor a draft; once there is one, a save without it is refused as \`draft_conflict\`. A map that does not fit is refused with every problem by path, such as \`funnels[0].steps[1].offer_key: no offer has the key x\`. Saving prints whether the map went live, what is left to fill in (an asset not synced yet, a rule with no calendar chosen), and the parts synced data does not back up, such as an offer with fewer than 3 buyers in a year. Because a save can change the client's numbers at once, run it first with \`--dry-run\`: that saves nothing and prints what saving would do (the version it would make, whether it would go live or wait on hold and why, the problems, warnings and questions), and refuses a stale \`--based-on\` the same way. When you are unsure the map is right, save with \`--hold\`: it is kept as the draft on hold, counting nothing, until a person puts it live in the app. \`--dry-run\` and \`--hold\` cannot be used together.
|
|
344
|
+
To change the map, write the whole map as JSON (\`{"offers": [...], "funnels": [...]}\`, the \`map\` of \`${cli} funnels show --format json\`) and run \`${cli} funnels draft --file map.json --note "<what changed and why>"\`. For a small change, send only the change instead: \`--patch '<json>'\` or \`@patch.json\` with JSON Patch ops on the version you read, named by key, such as \`[{"op": "replace", "path": "/offers/course/price", "value": 997}]\`, and \`--based-on\` that version; the patched map is checked and saved the same way, and an op that no longer applies is refused as \`patch_not_applicable\`. Add \`--evidence '<json>'\` or \`@file.json\` with what you looked at: counts and ids, never names, emails or phones. It is checked against synced data, and when every step's rule is filled in it goes live at once, confirmed by \`system:auto\`; a person can undo it in the app. A map with a rule not filled in stays on hold as the draft and counts nothing; saving replaces the draft on hold. Never invent an id or a number to fill a rule. Pass \`--based-on <n>\` with the version you read (the draft on hold, else the confirmed map): if someone saved since, the save is refused as \`draft_conflict\`, and you show again and redo your change on top. Leave out \`--based-on\` only for the first map, when there is neither a map nor a draft; once there is one, a save without it is refused as \`draft_conflict\`. A map that does not fit is refused with every problem by path, such as \`funnels[0].steps[1].offer_key: no offer has the key x\`. Saving prints whether the map went live, what is left to fill in (an asset not synced yet, a rule with no calendar chosen), and the parts synced data does not back up, such as an offer with fewer than 3 buyers in a year. Because a save can change the client's numbers at once, run it first with \`--dry-run\`: that saves nothing and prints what saving would do (the version it would make, whether it would go live or wait on hold and why, the problems, warnings and questions), and refuses a stale \`--based-on\` the same way. When you are unsure the map is right, save with \`--hold\`: it is kept as the draft on hold, counting nothing, until a person puts it live in the app. \`--dry-run\` and \`--hold\` cannot be used together.
|
|
328
345
|
|
|
329
346
|
\`--counts-from fix\` (the default) makes the map count for every past date the current map counted, as a fix; \`--counts-from <YYYY-MM-DD>\` makes it a change in the business that counts from that day (today or earlier). Save a fix and a change as two drafts, the fix first. \`--questions @questions.json\` asks a person what only they can settle: an array of \`{question_id, text, kind, evidence, options: [{label, patch}], default_option}\`, where the map already holds the default option and each other option's patch is JSON Patch ops on the map, naming offers, funnels and steps by key (\`/offers/course/matches/-\`). A question whose options all count the same sales and people, or that was answered already, is left out; at most 7 stay open, biggest money at stake first, and the checks add a yes/no question for each weak part no question covers. People answer them in the app; an answer saves the map as a fix, and later drafts keep it. \`${cli} funnels show\` lists the open and answered questions and says whether the map counts as a fix or from a date.
|
|
330
347
|
|
|
@@ -566,7 +583,7 @@ ${cli} funnels show [--version <n>] | funnels versions [--diff <a> <b>] the of
|
|
|
566
583
|
${cli} funnels numbers [--funnel <key>] [--from <date>] [--to <date>] how many reached each funnel step and moved on
|
|
567
584
|
${cli} funnels leaks [--funnel <key>] [--to <date>] the steps doing worse than their normal, costliest first
|
|
568
585
|
${cli} funnels evidence prices paid, buyer paths, GoHighLevel funnels, calendars and tags, webinars, pages and campaigns, to draft the map from
|
|
569
|
-
${cli} funnels draft --file <map.json> [--note <why>] [--evidence '<json>' | @file.json] [--based-on <n>] [--counts-from fix|<date>] [--questions @file.json] [--dry-run | --hold] save the whole map; it goes live when every step can be counted, unless --hold keeps it for a person; --dry-run saves nothing and says what saving would do
|
|
586
|
+
${cli} funnels draft --file <map.json> | --patch @patch.json [--note <why>] [--evidence '<json>' | @file.json] [--based-on <n>] [--counts-from fix|<date>] [--questions @file.json] [--dry-run | --hold] save the whole map, or a patch on version --based-on; it goes live when every step can be counted, unless --hold keeps it for a person; --dry-run saves nothing and says what saving would do
|
|
570
587
|
${cli} facts [--kind <kind>] [--subject <type>[:<ref>]] [--status <status>] [--json] | facts show <id> settled facts about the data, to treat as true
|
|
571
588
|
${cli} number-alerts [--since <date>] the key numbers that swung far from normal, day by day
|
|
572
589
|
${cli} facts propose --file <fact.json> suggest a fact for a person to confirm
|
package/dist/main.js
CHANGED
|
@@ -2819,22 +2819,24 @@ const commands = {
|
|
|
2819
2819
|
},
|
|
2820
2820
|
},
|
|
2821
2821
|
"funnels draft": {
|
|
2822
|
-
usage: "funnels draft --file <map.json> [--note <why>] [--evidence '<json>' | @file.json] [--based-on <n>] [--counts-from fix|<YYYY-MM-DD>] [--questions '<json>' | @file.json] [--dry-run | --hold] save a whole map; it counts at once when every step can be counted, unless --hold; --dry-run saves nothing and says what saving would do",
|
|
2822
|
+
usage: "funnels draft --file <map.json> | --patch '<json>' | @patch.json [--note <why>] [--evidence '<json>' | @file.json] [--based-on <n>] [--counts-from fix|<YYYY-MM-DD>] [--questions '<json>' | @file.json] [--dry-run | --hold] save a whole map, or a patch on version --based-on; it counts at once when every step can be counted, unless --hold; --dry-run saves nothing and says what saving would do",
|
|
2823
2823
|
async run(args, options) {
|
|
2824
|
-
const usage = "Usage: funnels draft --file <map.json> [--note <why>] [--evidence '<json>' | @file.json] [--based-on <n>] [--counts-from fix|<YYYY-MM-DD>] [--questions '<json>' | @file.json] [--dry-run | --hold]";
|
|
2825
|
-
if (args.length > 0 ||
|
|
2824
|
+
const usage = "Usage: funnels draft --file <map.json> | --patch '<json>' | @patch.json [--note <why>] [--evidence '<json>' | @file.json] [--based-on <n>] [--counts-from fix|<YYYY-MM-DD>] [--questions '<json>' | @file.json] [--dry-run | --hold]";
|
|
2825
|
+
if (args.length > 0 || Boolean(options.file) === (options.patch !== undefined))
|
|
2826
2826
|
throw new CliError(usage, exitCodes.usage);
|
|
2827
|
+
if (options.patch !== undefined && options.basedOn === undefined)
|
|
2828
|
+
throw new CliError("--patch needs --based-on: the version it changes.", exitCodes.usage);
|
|
2827
2829
|
if (options.dryRun && options.hold)
|
|
2828
2830
|
throw new CliError("Give --dry-run or --hold, not both: --dry-run saves nothing, --hold saves the map on hold.", exitCodes.usage);
|
|
2829
2831
|
if (options.countsFrom !== undefined && options.countsFrom !== "fix" && !/^\d{4}-\d{2}-\d{2}$/.test(options.countsFrom)) {
|
|
2830
2832
|
throw new CliError("--counts-from is fix or a date, YYYY-MM-DD", exitCodes.usage);
|
|
2831
2833
|
}
|
|
2832
|
-
const
|
|
2834
|
+
const change = options.patch === undefined ? { map: await parseJson(`@${options.file}`, "--file") } : { patch: await parseJson(options.patch, "--patch") };
|
|
2833
2835
|
const evidence = options.evidence === undefined ? undefined : await parseJson(options.evidence, "--evidence");
|
|
2834
2836
|
const questions = options.questions === undefined ? undefined : await parseJson(options.questions, "--questions");
|
|
2835
2837
|
const connection = await resolveConnection(options);
|
|
2836
2838
|
const basedOn = options.basedOn === undefined ? null : wholeNumber(options.basedOn, "--based-on");
|
|
2837
|
-
const saved = await api(connection, `${FUNNELS_PATH}/draft`, {
|
|
2839
|
+
const saved = await api(connection, `${FUNNELS_PATH}/draft`, { ...change, note: options.note, evidence, based_on: basedOn, counts_from: options.countsFrom, questions, apply: options.dryRun ? "check" : options.hold ? "hold" : "auto" }, "PUT");
|
|
2838
2840
|
if (options.format === "json")
|
|
2839
2841
|
return printJson(saved);
|
|
2840
2842
|
const based = basedOn === null ? "" : `, based on version ${basedOn}`;
|
|
@@ -3218,32 +3220,55 @@ const commands = {
|
|
|
3218
3220
|
},
|
|
3219
3221
|
},
|
|
3220
3222
|
"tests show": {
|
|
3221
|
-
usage: "tests show <
|
|
3223
|
+
usage: "tests show <test_id> one funnel test in full, with its history",
|
|
3222
3224
|
async run(args, options) {
|
|
3223
3225
|
if (args.length !== 1)
|
|
3224
|
-
throw new CliError("Usage: tests show <
|
|
3225
|
-
const
|
|
3226
|
+
throw new CliError("Usage: tests show <test_id>", exitCodes.usage);
|
|
3227
|
+
const shown = await api(await resolveConnection(options), testPath(args[0]));
|
|
3226
3228
|
if (options.format === "json")
|
|
3227
|
-
return printJson(
|
|
3228
|
-
printTest(test);
|
|
3229
|
+
return printJson(shown);
|
|
3230
|
+
printTest(shown.test, shown.events);
|
|
3229
3231
|
},
|
|
3230
3232
|
},
|
|
3231
3233
|
"tests draft": {
|
|
3232
|
-
usage: "tests draft --file <test.json> | < test.json write down a
|
|
3234
|
+
usage: "tests draft --file <test.json> | < test.json write down a test idea: the one change and where (a funnel step or a page); the rest of the plan may come later with tests edit",
|
|
3233
3235
|
async run(args, options) {
|
|
3234
3236
|
if (args.length > 0)
|
|
3235
3237
|
throw new CliError("Usage: tests draft --file <test.json> | < test.json", exitCodes.usage);
|
|
3236
3238
|
const { test } = await api(await resolveConnection(options), TESTS_PATH, await readToolInput(options));
|
|
3237
3239
|
if (options.format === "json")
|
|
3238
3240
|
return printJson({ test });
|
|
3239
|
-
console.log(`
|
|
3241
|
+
console.log(`Wrote down test ${test.test_id} as an idea.`);
|
|
3242
|
+
},
|
|
3243
|
+
},
|
|
3244
|
+
"tests edit": {
|
|
3245
|
+
usage: "tests edit <test_id> --file <changes.json> | < changes.json change some of a test's fields (null clears one); a change to the plan after the start is kept and marked in its history",
|
|
3246
|
+
async run(args, options) {
|
|
3247
|
+
if (args.length !== 1)
|
|
3248
|
+
throw new CliError("Usage: tests edit <test_id> --file <changes.json> | < changes.json", exitCodes.usage);
|
|
3249
|
+
const { test } = await api(await resolveConnection(options), testPath(args[0]), await readToolInput(options), "PATCH");
|
|
3250
|
+
if (options.format === "json")
|
|
3251
|
+
return printJson({ test });
|
|
3252
|
+
console.log(`Changed test ${test.test_id} (${test.status}).`);
|
|
3253
|
+
},
|
|
3254
|
+
},
|
|
3255
|
+
"tests plan": {
|
|
3256
|
+
usage: 'tests plan <test_id> --approved-by "<owner>" [--note "<what they said>"] record the owner\'s yes to a test\'s full plan, after asking them in your session',
|
|
3257
|
+
async run(args, options) {
|
|
3258
|
+
if (args.length !== 1 || !options.approvedBy)
|
|
3259
|
+
throw new CliError('Usage: tests plan <test_id> --approved-by "<owner>" [--note "<what they said>"]', exitCodes.usage);
|
|
3260
|
+
const body = { approved_by: options.approvedBy, ...(options.note ? { note: options.note } : {}) };
|
|
3261
|
+
const { test } = await api(await resolveConnection(options), `${testPath(args[0])}/plan`, body);
|
|
3262
|
+
if (options.format === "json")
|
|
3263
|
+
return printJson({ test });
|
|
3264
|
+
console.log(`Test ${test.test_id} is planned; ${options.approvedBy} said yes.`);
|
|
3240
3265
|
},
|
|
3241
3266
|
},
|
|
3242
3267
|
"tests start": {
|
|
3243
|
-
usage: "tests start <
|
|
3268
|
+
usage: "tests start <test_id> mark a planned test as running from today and freeze its plan; nothing goes live",
|
|
3244
3269
|
async run(args, options) {
|
|
3245
3270
|
if (args.length !== 1)
|
|
3246
|
-
throw new CliError("Usage: tests start <
|
|
3271
|
+
throw new CliError("Usage: tests start <test_id>", exitCodes.usage);
|
|
3247
3272
|
const { test } = await api(await resolveConnection(options), `${testPath(args[0])}/start`, {});
|
|
3248
3273
|
if (options.format === "json")
|
|
3249
3274
|
return printJson({ test });
|
|
@@ -3251,21 +3276,43 @@ const commands = {
|
|
|
3251
3276
|
},
|
|
3252
3277
|
},
|
|
3253
3278
|
"tests abandon": {
|
|
3254
|
-
usage: 'tests abandon <
|
|
3279
|
+
usage: 'tests abandon <test_id> [--note "<why>"] drop a test that has not ended',
|
|
3255
3280
|
async run(args, options) {
|
|
3256
3281
|
if (args.length !== 1)
|
|
3257
|
-
throw new CliError('Usage: tests abandon <
|
|
3282
|
+
throw new CliError('Usage: tests abandon <test_id> [--note "<why>"]', exitCodes.usage);
|
|
3258
3283
|
const { test } = await api(await resolveConnection(options), `${testPath(args[0])}/abandon`, options.note ? { note: options.note } : {});
|
|
3259
3284
|
if (options.format === "json")
|
|
3260
3285
|
return printJson({ test });
|
|
3261
3286
|
console.log(`Abandoned test ${test.test_id}.`);
|
|
3262
3287
|
},
|
|
3263
3288
|
},
|
|
3289
|
+
"tests invalidate": {
|
|
3290
|
+
usage: 'tests invalidate <test_id> --note "<what went wrong>" end a running test whose tracking broke or whose traffic did not split as planned',
|
|
3291
|
+
async run(args, options) {
|
|
3292
|
+
if (args.length !== 1 || !options.note)
|
|
3293
|
+
throw new CliError('Usage: tests invalidate <test_id> --note "<what went wrong>"', exitCodes.usage);
|
|
3294
|
+
const { test } = await api(await resolveConnection(options), `${testPath(args[0])}/invalidate`, { note: options.note });
|
|
3295
|
+
if (options.format === "json")
|
|
3296
|
+
return printJson({ test });
|
|
3297
|
+
console.log(`Test ${test.test_id} is invalid from ${test.ended_on}.`);
|
|
3298
|
+
},
|
|
3299
|
+
},
|
|
3300
|
+
"tests note": {
|
|
3301
|
+
usage: 'tests note <test_id> --note "<note>" add a note to a test\'s history',
|
|
3302
|
+
async run(args, options) {
|
|
3303
|
+
if (args.length !== 1 || !options.note)
|
|
3304
|
+
throw new CliError('Usage: tests note <test_id> --note "<note>"', exitCodes.usage);
|
|
3305
|
+
const { events } = await api(await resolveConnection(options), `${testPath(args[0])}/notes`, { note: options.note });
|
|
3306
|
+
if (options.format === "json")
|
|
3307
|
+
return printJson({ events });
|
|
3308
|
+
console.log(`Noted on test ${args[0]}.`);
|
|
3309
|
+
},
|
|
3310
|
+
},
|
|
3264
3311
|
"tests results": {
|
|
3265
|
-
usage: "tests results <
|
|
3312
|
+
usage: "tests results <test_id> a started test's numbers per side and whether it can be called yet",
|
|
3266
3313
|
async run(args, options) {
|
|
3267
3314
|
if (args.length !== 1)
|
|
3268
|
-
throw new CliError("Usage: tests results <
|
|
3315
|
+
throw new CliError("Usage: tests results <test_id>", exitCodes.usage);
|
|
3269
3316
|
const { results } = await api(await resolveConnection(options), `${testPath(args[0])}/results`);
|
|
3270
3317
|
if (options.format === "json")
|
|
3271
3318
|
return printJson({ results });
|
|
@@ -3273,20 +3320,26 @@ const commands = {
|
|
|
3273
3320
|
},
|
|
3274
3321
|
},
|
|
3275
3322
|
"tests decide": {
|
|
3276
|
-
usage: 'tests decide <
|
|
3323
|
+
usage: 'tests decide <test_id> --decision <scale|continue|iterate|hold|stop> [--outcome <won|lost|inconclusive|invalid>] [--note "<why>"] [--learning "<what it taught>"] record the decision, with its lesson; scale, iterate and stop need --outcome; nothing goes live',
|
|
3277
3324
|
async run(args, options) {
|
|
3278
|
-
|
|
3279
|
-
|
|
3280
|
-
|
|
3281
|
-
const
|
|
3325
|
+
const usage = 'Usage: tests decide <test_id> --decision <scale|continue|iterate|hold|stop> [--outcome <won|lost|inconclusive|invalid>] [--note "<why>"] [--learning "<what it taught>"]';
|
|
3326
|
+
if (args.length !== 1 || !options.decision)
|
|
3327
|
+
throw new CliError(usage, exitCodes.usage);
|
|
3328
|
+
const body = {
|
|
3329
|
+
decision: options.decision,
|
|
3330
|
+
...(options.outcome ? { outcome: options.outcome } : {}),
|
|
3331
|
+
...(options.note ? { note: options.note } : {}),
|
|
3332
|
+
...(options.learning ? { learning: options.learning } : {}),
|
|
3333
|
+
};
|
|
3334
|
+
const decided = await api(await resolveConnection(options), `${testPath(args[0])}/decide`, body);
|
|
3282
3335
|
if (options.format === "json")
|
|
3283
3336
|
return printJson(decided);
|
|
3284
3337
|
const { test, lesson, next_test: next } = decided;
|
|
3285
3338
|
console.log(test.decision === options.decision
|
|
3286
|
-
? `Test ${test.test_id}: ${test.decision}, now ${test.status}. Lesson ${lesson.id}.`
|
|
3339
|
+
? `Test ${test.test_id}: ${test.decision}${test.outcome ? ` (${test.outcome})` : ""}, now ${test.status}. Lesson ${lesson.id}.`
|
|
3287
3340
|
: `Test ${test.test_id}: ${options.decision} recorded; it stays ${test.status} (${test.decision}). Lesson ${lesson.id}.`);
|
|
3288
3341
|
if (next)
|
|
3289
|
-
console.log(`
|
|
3342
|
+
console.log(`Wrote down the next version as ${next.test_id}.`);
|
|
3290
3343
|
},
|
|
3291
3344
|
},
|
|
3292
3345
|
"lessons retire": {
|
|
@@ -4744,32 +4797,73 @@ const RULE_LIST_IDS = ["banned_claims", "words_to_avoid", "required_disclaimers"
|
|
|
4744
4797
|
const VOICE_SUGGESTIONS_PATH = "/api/v1/voice-lists/suggestions";
|
|
4745
4798
|
const LESSONS_PATH = "/api/v1/lessons";
|
|
4746
4799
|
const TESTS_PATH = "/api/v1/funnel-tests";
|
|
4747
|
-
const TEST_COLUMNS = ["test_id", "status", "
|
|
4800
|
+
const TEST_COLUMNS = ["test_id", "status", "where", "change", "metric"];
|
|
4748
4801
|
const testPath = (ref) => `${TESTS_PATH}/${encodeURIComponent(ref)}`;
|
|
4749
4802
|
function testRow(t) {
|
|
4750
|
-
return { test_id: t.test_id, status: t.status,
|
|
4751
|
-
}
|
|
4752
|
-
|
|
4753
|
-
|
|
4803
|
+
return { test_id: t.test_id, status: t.status, where: testWhere(t), change: excerpt(t.change, 50), metric: t.metric ?? "" };
|
|
4804
|
+
}
|
|
4805
|
+
const testWhere = (t) => [t.step_key && `${t.funnel_key} / ${t.step_key}`, t.page_key].filter(Boolean).join(", ");
|
|
4806
|
+
function printTest(t, events) {
|
|
4807
|
+
const arm = (a) => [
|
|
4808
|
+
a.label,
|
|
4809
|
+
a.page_key ?? a.url,
|
|
4810
|
+
a.item_id && `item ${a.item_id}`,
|
|
4811
|
+
a.asset_version && `version ${a.asset_version}`,
|
|
4812
|
+
a.platform_ref && `tool id ${a.platform_ref}`,
|
|
4813
|
+
a.weight !== null && `${Math.round(a.weight * 100)}% of traffic`,
|
|
4814
|
+
]
|
|
4815
|
+
.filter(Boolean)
|
|
4816
|
+
.join(" · ");
|
|
4817
|
+
const pct = (n) => (n === null ? null : `${Math.round(n * 1000) / 10}%`);
|
|
4818
|
+
const line = (label, value) => (value === null || value === undefined || value === "" ? [] : [`${label}: ${value}`]);
|
|
4819
|
+
const variants = t.arms.filter((a) => a.arm !== "control");
|
|
4754
4820
|
const lines = [
|
|
4755
|
-
`${t.test_id} (${t.status}) idea ${t.hypothesis_id}
|
|
4756
|
-
|
|
4757
|
-
`Where: ${t
|
|
4758
|
-
`The one change: ${t.
|
|
4759
|
-
|
|
4760
|
-
...
|
|
4761
|
-
|
|
4762
|
-
|
|
4763
|
-
"
|
|
4764
|
-
...
|
|
4821
|
+
`${t.test_id} (${t.status}) idea ${t.hypothesis_id}${t.logged_after ? " written down after it ran" : ""}`,
|
|
4822
|
+
...line("Name", t.name === t.change ? null : t.name),
|
|
4823
|
+
`Where: ${testWhere(t)}`,
|
|
4824
|
+
`The one change: ${t.change}`,
|
|
4825
|
+
...line("Because", t.because),
|
|
4826
|
+
...line("Who sees it", t.audience),
|
|
4827
|
+
...line("Expected lift", pct(t.expected_lift) && `${pct(t.expected_lift)}${t.prediction_confidence ? `, ${t.prediction_confidence} confidence` : ""}`),
|
|
4828
|
+
...line("Wrong if", t.disproved_if),
|
|
4829
|
+
...t.arms.filter((a) => a.arm === "control").map((a) => `Stays the same: ${arm(a)}`),
|
|
4830
|
+
...variants.map((v, i) => `${variants.length === 1 ? "Changed" : `Changed ${i + 1}`}: ${arm(v)}`),
|
|
4831
|
+
...t.arms.filter((a) => a.content !== null).map((a) => ` ${a.arm} shows: ${excerpt(a.content, 100)}`),
|
|
4832
|
+
...line("Split", t.design_type && `${t.design_type}${t.design_note ? `, ${t.design_note}` : ""}`),
|
|
4833
|
+
...line("Judged by", t.metric && `${t.metric} by ${t.unit ?? "people"}${t.people_per_arm ? `, ${t.people_per_arm} each side` : ""}, ${t.min_days} to ${t.max_days} days`),
|
|
4834
|
+
...line("Today", pct(t.baseline_rate) && `${pct(t.baseline_rate)} from ${t.baseline_from} to ${t.baseline_to}; smallest lift worth finding ${pct(t.min_lift)}`),
|
|
4835
|
+
...line("Must not get worse", t.guardrails.join(", ")),
|
|
4836
|
+
...line("Decision rule", t.decision_rule),
|
|
4837
|
+
...line("Also changing", t.concurrent_changes),
|
|
4838
|
+
...line("Traffic", t.traffic && [t.traffic.campaigns?.length && `campaigns ${t.traffic.campaigns.join(", ")}`, t.traffic.optimization_goal && `optimized for ${t.traffic.optimization_goal}`, t.traffic.note].filter(Boolean).join("; ")),
|
|
4839
|
+
...line("Tags", [...t.elements, ...t.levers].join(", ")),
|
|
4840
|
+
...line("ICE", t.ice_impact !== null || t.ice_confidence !== null || t.ice_ease !== null ? `${t.ice_impact ?? "-"} / ${t.ice_confidence ?? "-"} / ${t.ice_ease ?? "-"}` : null),
|
|
4841
|
+
...(t.evidence.length > 0 ? ["Evidence:", ...t.evidence.map((e) => ` ${e.kind}: ${e.ref}${e.note ? ` (${e.note})` : ""}`)] : []),
|
|
4842
|
+
...line("Started", t.started_on && `${t.started_on}${t.ended_on ? `, ended ${t.ended_on}` : ""}`),
|
|
4843
|
+
...line("Decided", t.decision && `${t.decision}${t.outcome ? `, ${t.outcome}` : ""}`),
|
|
4844
|
+
...line("Note", t.decision_note),
|
|
4845
|
+
...line("Learned", t.learning),
|
|
4765
4846
|
];
|
|
4766
|
-
if (
|
|
4767
|
-
lines.push(
|
|
4768
|
-
if (t.decision_note)
|
|
4769
|
-
lines.push(`Note: ${t.decision_note}`);
|
|
4847
|
+
if (events.length > 0)
|
|
4848
|
+
lines.push("History:", ...events.map((e) => ` ${e.at.slice(0, 16).replace("T", " ")} ${eventWords(e)} (${e.author})`));
|
|
4770
4849
|
console.log(lines.join("\n"));
|
|
4771
4850
|
}
|
|
4772
|
-
|
|
4851
|
+
function eventWords(e) {
|
|
4852
|
+
const d = e.detail;
|
|
4853
|
+
switch (e.kind) {
|
|
4854
|
+
case "created":
|
|
4855
|
+
return "written down";
|
|
4856
|
+
case "edited":
|
|
4857
|
+
return `changed ${Object.keys(d.fields ?? {}).join(", ")}${d.after_start ? " after it started" : ""}`;
|
|
4858
|
+
case "status":
|
|
4859
|
+
return `${d.from} to ${d.to}${d.approved_by ? `, ${d.approved_by} said yes` : ""}${d.note ? `: ${d.note}` : ""}`;
|
|
4860
|
+
case "note":
|
|
4861
|
+
return `note: ${d.note}`;
|
|
4862
|
+
case "decided":
|
|
4863
|
+
return `decided ${d.decision}${d.outcome ? ` (${d.outcome})` : ""}${d.note ? `: ${d.note}` : ""}`;
|
|
4864
|
+
}
|
|
4865
|
+
}
|
|
4866
|
+
const RESULT_COLUMNS = ["side", "label", "counted by", "people", "moved on", "sign-ups", "buyers", "revenue", "spend", "metric", "dates"];
|
|
4773
4867
|
function printResults(r) {
|
|
4774
4868
|
const money = (n, currency) => (n === null ? "" : `${n.toFixed(2)}${currency ? ` ${currency}` : ""}`);
|
|
4775
4869
|
const rows = r.arms.map((a) => ({
|
|
@@ -4778,13 +4872,14 @@ function printResults(r) {
|
|
|
4778
4872
|
"counted by": a.counted_by,
|
|
4779
4873
|
people: a.people,
|
|
4780
4874
|
"moved on": a.moved_on ?? "",
|
|
4875
|
+
"sign-ups": a.leads ?? "",
|
|
4781
4876
|
buyers: a.buyers ?? "",
|
|
4782
4877
|
revenue: money(a.revenue, a.currency),
|
|
4783
4878
|
spend: money(a.spend, a.currency),
|
|
4784
|
-
metric: a.metric_value === null ? "" : r.test.metric === "cost_per_sale" || r.test.metric === "revenue_per_visitor" ? money(a.metric_value, a.currency) : `${(a.metric_value * 100).toFixed(1)}%`,
|
|
4879
|
+
metric: a.metric_value === null ? "" : r.test.metric === "cost_per_sale" || r.test.metric === "cost_per_lead" || r.test.metric === "revenue_per_visitor" ? money(a.metric_value, a.currency) : `${(a.metric_value * 100).toFixed(1)}%`,
|
|
4785
4880
|
dates: `${a.window.from} to ${a.window.to}`,
|
|
4786
4881
|
}));
|
|
4787
|
-
const lines = [`${r.test.test_id} (${r.test.status}) judged by ${r.test.metric}, ${r.test.
|
|
4882
|
+
const lines = [`${r.test.test_id} (${r.test.status}) judged by ${r.test.metric ?? "no main number"}, ${r.test.people_per_arm ?? "no count of"} people each side`, renderTable(RESULT_COLUMNS, rows), "", ...r.calls.map((c) => c.words)];
|
|
4788
4883
|
if (r.caveats.length > 0)
|
|
4789
4884
|
lines.push("", ...r.caveats.map((c) => `Note: ${c}`));
|
|
4790
4885
|
console.log(lines.join("\n"));
|
|
@@ -5194,9 +5289,13 @@ Options:
|
|
|
5194
5289
|
the sync matches several kinds, so its next read joins the item
|
|
5195
5290
|
--notes <text> what changed in the version skill publish adds
|
|
5196
5291
|
--note <why> why: what content create and edit changed, why lessons retire says a lesson is no longer true,
|
|
5197
|
-
|
|
5292
|
+
why tests abandon drops one, what tests invalidate says went wrong, the note tests note adds, or what the
|
|
5293
|
+
owner said with tests plan; for profile upload, what the files are, in the client's words;
|
|
5198
5294
|
for meta ad promote into a campaign, a few words on the new ad set's audience;
|
|
5199
5295
|
for scorecard record, what was checked and what was found; for repairs open and update, a progress line
|
|
5296
|
+
--approved-by <name> tests plan: who said yes to the test's plan, as the owner gave their name in your session
|
|
5297
|
+
--outcome <o> tests decide: what the test found, won, lost, inconclusive or invalid; needed with scale, iterate and stop
|
|
5298
|
+
--learning <text> tests decide: what the test taught, in a sentence, kept on the test
|
|
5200
5299
|
--about <a> with lessons, only the lessons with a proof about ads, pages, funnel (steps and tests) or messages
|
|
5201
5300
|
(emails, texts and sequences), or cross_channel: proven on two or more of ads, pages and messages
|
|
5202
5301
|
--dimension <d> with lessons, only the lessons along that dimension (hook, angle, format, offer, persona, page, funnel_step, follow_up, other); repeat for several
|
|
@@ -5207,6 +5306,7 @@ Options:
|
|
|
5207
5306
|
--counts-from fix|<YYYY-MM-DD> how funnels draft counts: fix (default) for past dates too, or a change from that day
|
|
5208
5307
|
--questions <json | @file.json> the questions funnels draft asks a person, as an array; with map, no value: only
|
|
5209
5308
|
whether each everyday question can be answered
|
|
5309
|
+
--patch <json | @file.json> funnels draft: JSON Patch ops on version --based-on, instead of a whole map in --file
|
|
5210
5310
|
--person <name> with profile confirm, the name of the person in the chat who gave the answer
|
|
5211
5311
|
--diff <a> <b> with funnels versions, what changed from version a to version b
|
|
5212
5312
|
--funnel <key> only this funnel's steps in funnels numbers and funnels leaks (default: every funnel in the confirmed map);
|
|
@@ -5433,6 +5533,9 @@ function parseOptions(argv) {
|
|
|
5433
5533
|
"merged-at": { type: "string" },
|
|
5434
5534
|
probes: { type: "string" },
|
|
5435
5535
|
decision: { type: "string" },
|
|
5536
|
+
outcome: { type: "string" },
|
|
5537
|
+
learning: { type: "string" },
|
|
5538
|
+
"approved-by": { type: "string" },
|
|
5436
5539
|
harness: { type: "string" },
|
|
5437
5540
|
platform: { type: "string" },
|
|
5438
5541
|
library: { type: "string" },
|
|
@@ -5459,6 +5562,7 @@ function parseOptions(argv) {
|
|
|
5459
5562
|
evidence: { type: "string" },
|
|
5460
5563
|
"counts-from": { type: "string" },
|
|
5461
5564
|
questions: { type: "string" },
|
|
5565
|
+
patch: { type: "string" },
|
|
5462
5566
|
person: { type: "string" },
|
|
5463
5567
|
about: { type: "string" },
|
|
5464
5568
|
dimension: { type: "string", multiple: true },
|
|
@@ -5660,6 +5764,9 @@ function parseOptions(argv) {
|
|
|
5660
5764
|
mergedAt: values["merged-at"],
|
|
5661
5765
|
probes: values.probes,
|
|
5662
5766
|
decision: values.decision,
|
|
5767
|
+
outcome: values.outcome,
|
|
5768
|
+
learning: values.learning,
|
|
5769
|
+
approvedBy: values["approved-by"],
|
|
5663
5770
|
notes: values.notes,
|
|
5664
5771
|
harness: values.harness,
|
|
5665
5772
|
platform: values.platform,
|
|
@@ -5689,6 +5796,7 @@ function parseOptions(argv) {
|
|
|
5689
5796
|
evidence: values.evidence,
|
|
5690
5797
|
countsFrom: values["counts-from"],
|
|
5691
5798
|
questions: values.questions,
|
|
5799
|
+
patch: values.patch,
|
|
5692
5800
|
person: values.person,
|
|
5693
5801
|
about: values.about,
|
|
5694
5802
|
dimension: values.dimension,
|
package/dist/playbooks.js
CHANGED
|
@@ -701,8 +701,8 @@ Without a shell: MCP \`get_funnel_evidence\`, then MCP \`get_funnel_map\`.
|
|
|
701
701
|
|
|
702
702
|
\`evidence.json\` is \`{"evidence": {...}, "text": "..."}\`. The parts below are under \`.evidence\`, each either \`{"status": "ok", "rows": [...]}\` or \`{"status": "not_connected", "note": "..."}\`. Amounts are in whole currency units (1500 is 1,500.00), over the last \`.evidence.windowDays\` days unless a part says otherwise.
|
|
703
703
|
|
|
704
|
-
- **prices**: the prices people pay, on each platform, with how many buyers and payments. A GoHighLevel or Shopify row has the product's \`name\`, a Kajabi row the offer's title, with the Kajabi offer id as \`productId\`; a Stripe charge has none. For GoHighLevel, \`payments\` counts orders, not instalments. A GoHighLevel or Kajabi sale taken through Stripe shows twice, as a row naming the product and a Stripe row with only the amount, and the two amounts can differ a little (1497 and 1500).
|
|
705
|
-
- **buyerPaths**: the most common orders in which one person paid, with the median days between each payment. A small payment minutes after a big one is an order bump or upsell; a large payment weeks later is a back-end offer. A path such as \`[1500, 497, 3500]\` with medians \`[0, 60]\` reads: course, then an upsell the same day, then an event about two months later. A GoHighLevel checkout with an order bump is one payment of both prices together: 1994 is 1497 + 497. Before you take an amount for a new offer, check whether it is a sum of prices you already know.
|
|
704
|
+
- **prices**: the prices people pay, on each platform, with how many buyers and payments. A GoHighLevel or Shopify row has the product's \`name\`, a Kajabi row the offer's title, with the Kajabi offer id as \`productId\`; a Stripe charge has none. For GoHighLevel, \`payments\` counts orders, not instalments. A GoHighLevel payment with no order lines, such as an invoice, has only its amount, like a Stripe charge; match it with a \`ghl\` \`amount\` match. A GoHighLevel or Kajabi sale taken through Stripe shows twice, as a row naming the product and a Stripe row with only the amount, and the two amounts can differ a little (1497 and 1500).
|
|
705
|
+
- **buyerPaths**: the most common orders in which one person paid, with the median days between each payment. A path needs at least two payments by one person matched to a known person, so it is empty when no matched buyer paid twice in the window, even when prices has rows. A small payment minutes after a big one is an order bump or upsell; a large payment weeks later is a back-end offer. A path such as \`[1500, 497, 3500]\` with medians \`[0, 60]\` reads: course, then an upsell the same day, then an event about two months later. A GoHighLevel checkout with an order bump is one payment of both prices together: 1994 is 1497 + 497. Before you take an amount for a new offer, check whether it is a sum of prices you already know.
|
|
706
706
|
- **funnels**, **calendars**, **tags**: what GoHighLevel holds. Tags often name purchases and memberships.
|
|
707
707
|
- **pipelineStages**: each stage of the sales pipelines in GoHighLevel, HubSpot and Attio (\`source\` is \`ghl\`, \`hubspot\` or \`attio\`; \`accountId\` is the GoHighLevel location id, the HubSpot account id or the Attio workspace id), with the deals in it now and, for GoHighLevel only, how many moved into it in the window; \`entered\` is null for HubSpot and Attio, which keep no stage history, so null means unknown, not none. An Attio \`pipelineId\` is a list id, or \`deals\` for the stage on the deals themselves. Only GoHighLevel stages can be a step's rule (\`ghl_stage\`); HubSpot and Attio stages are evidence of how deals move, not something a rule can count.
|
|
708
708
|
- **webinars**: sign-ups, viewers and offer clicks per webinar id.
|
|
@@ -1994,21 +1994,33 @@ ${pagesPlaybook}`;
|
|
|
1994
1994
|
export const CRO_SKILL_NAME = `${cli}-cro`;
|
|
1995
1995
|
/** The test the cro playbook shows, exported so a test checks the server accepts it. */
|
|
1996
1996
|
export const croTestExample = {
|
|
1997
|
-
hypothesis: "A text 15 minutes before the webinar will get more sign-ups to show up, because people on sales calls say they did not know it was live.",
|
|
1998
1997
|
funnel_key: "webinar_to_course",
|
|
1999
1998
|
step_key: "attend",
|
|
2000
|
-
|
|
2001
|
-
|
|
2002
|
-
|
|
2003
|
-
|
|
2004
|
-
|
|
2005
|
-
|
|
2006
|
-
split_method: "other",
|
|
2007
|
-
split_note: "The reminder workflow sends the text to every other new sign-up.",
|
|
1999
|
+
change: "a reminder text 15 minutes before the start",
|
|
2000
|
+
because: "Attendance fell from 38% to 30% of sign-ups, and people on sales calls say they did not know the webinar was live.",
|
|
2001
|
+
audience: "everyone who signs up for the webinar",
|
|
2002
|
+
expected_lift: 0.15,
|
|
2003
|
+
prediction_confidence: "medium",
|
|
2004
|
+
disproved_if: "attendance rises less than 5% after 300 sign-ups each side",
|
|
2008
2005
|
evidence: [
|
|
2009
2006
|
{ kind: "leak", ref: "webinar_to_course.attend", note: "Attendance fell from 38% to 30% of sign-ups, about $2,300 a month." },
|
|
2010
2007
|
{ kind: "customer_words", ref: "sales call 2026-10-02", note: "I didn't know the webinar was live." },
|
|
2011
2008
|
],
|
|
2009
|
+
design_type: "link_split",
|
|
2010
|
+
design_note: "The reminder workflow sends the text to every other new sign-up.",
|
|
2011
|
+
metric: "step_conversion",
|
|
2012
|
+
unit: "people",
|
|
2013
|
+
guardrails: ["unsubscribes"],
|
|
2014
|
+
baseline_rate: 0.3,
|
|
2015
|
+
baseline_from: "2026-09-01",
|
|
2016
|
+
baseline_to: "2026-09-30",
|
|
2017
|
+
min_lift: 0.1,
|
|
2018
|
+
people_per_arm: 300,
|
|
2019
|
+
decision_rule: "Keep the text if attendance rises at least 10% with unsubscribes no higher.",
|
|
2020
|
+
elements: ["follow_up"],
|
|
2021
|
+
levers: ["urgency"],
|
|
2022
|
+
control: { label: "today's reminders: an email the day before and an hour before", content: "Email the day before and an hour before; no text." },
|
|
2023
|
+
variants: [{ label: "the same reminders plus a text 15 minutes before", content: "We're live in 15 minutes: {link}" }],
|
|
2012
2024
|
};
|
|
2013
2025
|
export const croPlaybook = `# Funnel test playbook
|
|
2014
2026
|
|
|
@@ -2039,7 +2051,7 @@ Without a shell: MCP \`get_funnel_leaks\`, then MCP \`list_funnel_tests\` with \
|
|
|
2039
2051
|
|
|
2040
2052
|
In \`leaks.json\`, \`leaks.leaks\` lists every step doing worse than its normal, costliest first. Each has its \`funnel_key\` and \`step_key\`, its \`recent\` and \`baseline\` rates, \`people_lost_vs_baseline\`, \`money_at_stake\` per currency (an estimate; never add currencies together) and, for some steps, \`splits\` by channel, campaign and ad. An empty list means no step is leaking: write no ideas and go to step 5.
|
|
2041
2053
|
|
|
2042
|
-
\`tests.json\` holds the tests already written, newest first, each with its \`status\`, \`funnel_key\`, \`step_key\` and \`created_at\`. Pick the 1 to 3 costliest leaks you can act on whose step has no \`
|
|
2054
|
+
\`tests.json\` holds the tests already written, newest first, each with its \`status\`, \`funnel_key\`, \`step_key\` and \`created_at\`. Pick the 1 to 3 costliest leaks you can act on whose step has no \`idea\`, \`planned\` or \`running\` test. Count every test created since Monday toward the 3, whatever its status now, abandoned ones included: with 3 already, write no new ideas this week.
|
|
2043
2055
|
|
|
2044
2056
|
## 2. Gather proof for each leak
|
|
2045
2057
|
|
|
@@ -2066,9 +2078,11 @@ Write each test to its own file with your file tool, such as \`test-1.json\`:
|
|
|
2066
2078
|
${JSON.stringify(croTestExample)}
|
|
2067
2079
|
\`\`\`
|
|
2068
2080
|
|
|
2069
|
-
- \`
|
|
2070
|
-
- \`
|
|
2071
|
-
- \`
|
|
2081
|
+
- \`change\` is the one change. \`because\` is what led to it, \`audience\` who sees it, \`expected_lift\` how much it should move (0.15 for 15%) and \`prediction_confidence\` how sure you are; \`disproved_if\` says what result would prove it wrong.
|
|
2082
|
+
- \`control\` is what runs now; \`variants\` is the changed version, each \`{label, content, page?, url?, item_id?, asset_version?}\`. \`content\` is the exact words or change each side shows.
|
|
2083
|
+
- \`metric\` is the number that should move: \`step_conversion\` (only with a \`step_key\`), \`sales_per_visitor\`, \`revenue_per_visitor\`, \`cost_per_sale\` or \`cost_per_lead\` (spend per sign-up), counted by \`unit\` (people or sessions). \`baseline_rate\` is today's rate over \`baseline_from\` to \`baseline_to\`, \`min_lift\` the smallest lift worth finding, \`people_per_arm\` how many each side needs, \`guardrails\` the numbers that must not get worse, and \`decision_rule\` how you will call it. \`min_days\` and \`max_days\` are 14 and 56 unless set.
|
|
2084
|
+
- \`design_type\` is how traffic is split: \`ad_split_randomized\` (Meta's own A/B test), \`ad_split_duplicated\` (two copies of a campaign, one to each page; not a true random split), \`tool_split\` (inside a page builder or webinar tool, named in \`design_note\`), \`link_split\`, \`before_after\` or \`adaptive\`. \`traffic\` says which campaigns feed it and what they optimize for. Until #2202, an ad split test is counted only from Adzara's labels, so it can be measured only on ads Adzara made before the switch; for a new test, use \`before_after\`.
|
|
2085
|
+
- \`elements\` and \`levers\` tag what changes and why it should work, from \`${cli} guide\`'s fixed lists.
|
|
2072
2086
|
- \`evidence\` quotes the proof, each \`{kind, ref, note}\` with kind \`leak\`, \`lesson\`, \`customer_words\`, \`page\`, \`ad\` or \`other\`. Start with the leak itself.
|
|
2073
2087
|
- Give \`hypothesis_id\` and \`test_id\` only to reuse Adzara's own labels; left out, they are made for you.
|
|
2074
2088
|
|
|
@@ -2080,13 +2094,21 @@ ${cli} tests draft --file test-1.json --format json > drafted-1.json
|
|
|
2080
2094
|
|
|
2081
2095
|
Without a shell: MCP \`draft_funnel_test\` with the same JSON.
|
|
2082
2096
|
|
|
2083
|
-
A test that changes two things is refused: split it into two tests and keep the stronger one.
|
|
2097
|
+
It is written down as an \`idea\`. A test that changes two things is refused: split it into two tests and keep the stronger one.
|
|
2084
2098
|
|
|
2085
2099
|
## 4. Put the ideas to the owner
|
|
2086
2100
|
|
|
2087
|
-
Approvals happen in your own session with the owner. ${brand.name}
|
|
2101
|
+
Approvals happen in your own session with the owner. ${brand.name} shows no approval screen. For each idea, give the owner the leak and what it costs, the one change, the number that should move, and the proof in a line. The owner says yes or no.
|
|
2102
|
+
|
|
2103
|
+
When the owner says yes, record who said it. It refuses a plan with a gap and names each missing field; fill them in with \`${cli} tests edit <test_id> --file changes.json\` and try again.
|
|
2104
|
+
|
|
2105
|
+
\`\`\`sh
|
|
2106
|
+
${cli} tests plan <test_id> --approved-by "<the owner's name>"
|
|
2107
|
+
\`\`\`
|
|
2108
|
+
|
|
2109
|
+
Without a shell: MCP \`plan_funnel_test\` with \`{"id": "<test_id>", "approved_by": "<the owner's name>"}\`.
|
|
2088
2110
|
|
|
2089
|
-
|
|
2111
|
+
Then the change goes live through its own path: a page through its own approval (the pages playbook), ads through ${brand.name}'s Meta commands (an \`adzara_ads\` split cannot be measured on those new ads until #2202). Once it is actually live, and no other test runs on the same step, mark it running. This changes nothing live; it records the start day and freezes the plan.
|
|
2090
2112
|
|
|
2091
2113
|
\`\`\`sh
|
|
2092
2114
|
${cli} tests start <test_id>
|
|
@@ -2115,15 +2137,15 @@ Without a shell: MCP \`get_funnel_test_results\` with \`{"id": "<test_id>"}\`.
|
|
|
2115
2137
|
\`calls\` has one call for each changed version against the control: \`not_enough_yet\` (with the \`people\` each side has and the \`need\`), \`variant_ahead\`, \`control_ahead\` or \`no_real_difference\`, and \`words\` says it plainly. \`caveats\` names what weakens the call.
|
|
2116
2138
|
|
|
2117
2139
|
- **Every call is \`not_enough_yet\`:** leave the test running; that needs no command. Only when something outside the test got in the way (the page broke, a price changed, the ads stopped), suggest \`hold\` to the owner.
|
|
2118
|
-
- **A call is ready:** suggest \`scale\` (the new version won), \`stop\` (it lost or made no difference) or \`iterate\` (promising; it
|
|
2140
|
+
- **A call is ready:** suggest \`scale\` (the new version won), \`stop\` (it lost or made no difference) or \`iterate\` (promising; it writes down a sharper version). \`iterate\` writes a new test, so suggest it only while the week is under its 3; otherwise suggest \`stop\` and draft the sharper version next week.
|
|
2119
2141
|
|
|
2120
2142
|
Put each suggestion to the owner with the ideas from step 4, then record the owner's decision:
|
|
2121
2143
|
|
|
2122
2144
|
\`\`\`sh
|
|
2123
|
-
${cli} tests decide <test_id> --decision <scale|iterate|stop|hold> --note "<why, in a sentence>"
|
|
2145
|
+
${cli} tests decide <test_id> --decision <scale|iterate|stop|hold> --outcome <won|lost|inconclusive> --note "<why, in a sentence>"
|
|
2124
2146
|
\`\`\`
|
|
2125
2147
|
|
|
2126
|
-
Without a shell: MCP \`decide_funnel_test\` with \`{"id": "<test_id>", "decision": "<scale|iterate|stop|hold>", "note": "<why, in a sentence>"}\`.
|
|
2148
|
+
Without a shell: MCP \`decide_funnel_test\` with \`{"id": "<test_id>", "decision": "<scale|iterate|stop|hold>", "outcome": "<won|lost|inconclusive>", "note": "<why, in a sentence>"}\`. scale, iterate and stop need the outcome; hold does not.
|
|
2127
2149
|
|
|
2128
2150
|
Deciding changes nothing live. It writes the lesson itself, with the test, its pages and its step as proof, so write no lesson of your own for it.
|
|
2129
2151
|
|
|
@@ -2140,11 +2162,13 @@ ${cli} lessons [--about pages|funnel] --format json what earlier tests ta
|
|
|
2140
2162
|
${cli} voice --since <YYYY-MM-DD> --sort useful --json fresh customer words
|
|
2141
2163
|
${cli} pages results --format json which landing pages bring real sales
|
|
2142
2164
|
${cli} pages brief --step <key> [--funnel <key>] --format json the step's current page and the ads that send people there
|
|
2143
|
-
${cli} tests draft --file <test.json> write down a test
|
|
2144
|
-
${cli} tests
|
|
2165
|
+
${cli} tests draft --file <test.json> write down a test idea
|
|
2166
|
+
${cli} tests edit <test_id> --file <changes.json> fill in or change its plan
|
|
2167
|
+
${cli} tests plan <test_id> --approved-by "<owner>" record the owner's yes to the full plan
|
|
2168
|
+
${cli} tests start <test_id> record that a planned, live test is running
|
|
2145
2169
|
${cli} tests abandon <test_id> --note "<why>" drop a test the owner rejected
|
|
2146
2170
|
${cli} tests results <test_id> --format json a running test's numbers per side and whether it can be called
|
|
2147
|
-
${cli} tests decide <test_id> --decision <d> --note "<why>" record the owner's decision; it writes the lesson
|
|
2171
|
+
${cli} tests decide <test_id> --decision <d> --outcome <o> --note "<why>" record the owner's decision; it writes the lesson
|
|
2148
2172
|
\`\`\`
|
|
2149
2173
|
`;
|
|
2150
2174
|
export const croSkill = `---
|