@wrangled/cli 0.3.165 → 0.3.167
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/guide.js +6 -5
- package/dist/main.js +59 -10
- package/dist/map.js +14 -2
- package/dist/playbooks.js +8 -5
- package/package.json +1 -1
package/dist/guide.js
CHANGED
|
@@ -27,7 +27,8 @@ export const funnelTestExample = {
|
|
|
27
27
|
min_lift: 0.1,
|
|
28
28
|
people_per_arm: 400,
|
|
29
29
|
decision_rule: "Scale the video if sign-ups per visitor rise at least 10% with cost per sale no worse.",
|
|
30
|
-
|
|
30
|
+
category: "design",
|
|
31
|
+
subcategory: "hero_image",
|
|
31
32
|
levers: ["clarity"],
|
|
32
33
|
control: { label: "letter at the top", content: "A letter from the founder: why this webinar exists.", url: "https://example.com/signup" },
|
|
33
34
|
variants: [{ label: "video at the top", content: "A 90-second video of the founder saying what the webinar covers.", url: "https://example.com/signup-video" }],
|
|
@@ -117,15 +118,15 @@ The \`weekly-loop\` playbook (\`${cli} skill install weekly-loop\` installs it a
|
|
|
117
118
|
|
|
118
119
|
## Trying one change: tests
|
|
119
120
|
|
|
120
|
-
A test writes down one change to a page or funnel step, tried against what runs now. Each test is kept in the workspace's warehouse, in \`cro_tests\` (one row per test), \`cro_test_arms\` (its sides), \`cro_test_events\` (its history), \`cro_test_results_daily\` (each side's numbers per day) and \`cro_test_reported\` (numbers other tools counted), so you can join tests with visits, sign-ups and sales in one query: \`${cli} warehouse query "SELECT test_id, status,
|
|
121
|
+
A test writes down one change to a page or funnel step, tried against what runs now. Each test is kept in the workspace's warehouse, in \`cro_tests\` (one row per test), \`cro_test_arms\` (its sides), \`cro_test_events\` (its history), \`cro_test_results_daily\` (each side's numbers per day) and \`cro_test_reported\` (numbers other tools counted), so you can join tests with visits, sign-ups and sales in one query: \`${cli} warehouse query "SELECT test_id, status, category, subcategory FROM cro_tests"\`.
|
|
121
122
|
|
|
122
|
-
A test moves through these stages: \`idea\` (only the one change and where are needed), \`planned\` (the full plan is in and the owner said yes), \`running\` (its plan is frozen), then \`decided\`, \`abandoned\` or \`invalid\` (the tracking broke or traffic did not split as planned). Where is a funnel step (\`funnel_key\` and \`step_key\` from the funnels map), a page by its address (\`page_url\`, even one not on the map yet), or both. The full plan is: \`because\` (what led to the idea), \`audience\`, \`expected_lift\` (relative, 0.18 for 18%), \`prediction_confidence\` (low, medium or high), \`disproved_if\`, \`design_type\` (how traffic is split: ad_split_randomized, Meta's own A/B test; ad_split_duplicated, two copies of a campaign, one to each page, which is not a true random split; tool_split, inside a page builder or webinar tool named in a \`design_note\`; link_split; before_after; or adaptive), \`traffic\` (\`{campaigns, optimization_goal, note}\`: which campaigns feed it and what they optimize for, frozen with the plan), \`metric\` (step_conversion, which needs a step, sales_per_visitor, revenue_per_visitor, cost_per_sale or cost_per_lead, spend per sign-up) and \`unit\` (people or sessions), \`guardrails\` (numbers that must not get worse), \`baseline_rate\` with \`baseline_from\` and \`baseline_to\`, \`min_lift\`, \`people_per_arm\`, \`min_days\` and \`max_days\` (14 and 56 unless set), \`decision_rule\`, \`concurrent_changes\`, and the control and 1 to 3 \`variants\`, each with the exact \`content\` it shows and, when a tool runs the split, its \`platform_ref\` (the tool's own id for the side).
|
|
123
|
+
A test moves through these stages: \`idea\` (only the one change and where are needed), \`planned\` (the full plan is in and the owner said yes), \`running\` (its plan is frozen), then \`decided\`, \`abandoned\` or \`invalid\` (the tracking broke or traffic did not split as planned). Where is a funnel step (\`funnel_key\` and \`step_key\` from the funnels map), a page by its address (\`page_url\`, even one not on the map yet), or both. The full plan is: \`because\` (what led to the idea), \`audience\`, \`expected_lift\` (relative, 0.18 for 18%), \`prediction_confidence\` (low, medium or high), \`disproved_if\`, \`design_type\` (how traffic is split: ad_split_randomized, Meta's own A/B test; ad_split_duplicated, two copies of a campaign, one to each page, which is not a true random split; tool_split, inside a page builder or webinar tool named in a \`design_note\`; link_split; before_after; or adaptive), \`traffic\` (\`{campaigns, optimization_goal, note}\`: which campaigns feed it and what they optimize for, frozen with the plan), \`metric\` (step_conversion, which needs a step, sales_per_visitor, revenue_per_visitor, cost_per_sale or cost_per_lead, spend per sign-up) and \`unit\` (people or sessions), \`guardrails\` (numbers that must not get worse), \`baseline_rate\` with \`baseline_from\` and \`baseline_to\`, \`min_lift\`, \`people_per_arm\`, \`min_days\` and \`max_days\` (14 and 56 unless set), \`decision_rule\`, \`concurrent_changes\`, and the control and 1 to 3 \`variants\`, each with the exact \`content\` it shows and, when a tool runs the split, its \`platform_ref\` (the tool's own id for the side). What a test is mainly about is a \`category\` and its \`subcategory\`, from a fixed list: copy (headline, subheadline, body, call_to_action, bullets, story, other), offer (price, bonus, guarantee, payment_plan, order_bump, upsell, deadline_or_scarcity, other), proof (testimonials, case_study, numbers_and_results, logos_or_press, reviews, other), design (layout, hero_image, video, colors, page_length, mobile, other), form_checkout (form_fields, number_of_steps, checkout_page, other), webinar_event (length, presenter, offer_timing, replay, reminders, other), follow_up (email, text_message, thank_you_page, other), audience_traffic (who_sees_it, ad_angle, optimization_goal, other), tool_platform (webinar_platform, page_builder, checkout_tool, other), and custom, which has no subcategory. Subcategory \`other\` and category \`custom\` need a \`label\` naming what was tested, such as "exit pop-up", kept in lower case. \`also_changed\` lists anything else the test changed, each \`copy.subheadline\`, \`offer.other:<label>\` or \`custom:<label>\`. \`${cli} tests categories\` prints the list and the labels this workspace already uses: reuse one that fits before writing a new one. \`${cli} tests draft\` and \`${cli} tests edit\` take them as flags too: \`--category copy --subcategory headline --also copy.subheadline\`. \`levers\` tag why it should work, from a fixed list: clarity, relevance, trust, value, urgency, less_effort, motivation, other. \`ice_impact\`, \`ice_confidence\` and \`ice_ease\` (1 to 10) sort the idea list. \`evidence\` quotes the proof, each \`{kind, ref, note}\`, kind one of leak, lesson, customer_words, page, ad or other. A test changes one thing: a \`change\` naming two is refused.
|
|
123
124
|
|
|
124
125
|
Write it as JSON, such as \`${JSON.stringify(funnelTestExample)}\`, and run \`${cli} tests draft --file test.json\`. It gets Adzara-style labels from the change and this month (\`hyp-<slug>\` and \`test-<yyyy-mm>-<slug>\`); give \`hypothesis_id\` and \`test_id\` to reuse Adzara's own, so the results of the ads Adzara published for the test (\`ad_test_results_daily\`, \`ad_hypothesis_results_daily\`) join on the same label. \`${cli} tests edit <test_id> --file changes.json\` changes some fields (null clears one). ${brand.name} shows no approval screen: ask the owner in your own session, then record the yes with \`${cli} tests plan <test_id> --approved-by "<owner>"\`, which refuses a plan with a gap and names every missing field. \`${cli} tests start <test_id>\` marks a planned test as running from today and freezes its plan; an edit after that is still saved, and its history and results say so. Starting changes nothing live: the page still goes live through its own approval, and the ads through ${brand.name}'s Meta commands. Until #2202, an ad split test can be measured only on ads Adzara made before the switch, because its sides are counted from Adzara's labels; a \`before_after\` test still works. \`${cli} tests abandon <test_id> --note "<why>"\` drops one that has not ended, \`${cli} tests invalidate <test_id> --note "<what went wrong>"\` ends a running one whose numbers cannot be trusted, and \`${cli} tests note <test_id> --note "<note>"\` adds to its history. \`${cli} tests [--status <status>] [--funnel <funnel_key>] [--limit <n>]\` lists them, newest first (50 unless \`--limit\`, at most 200), and \`${cli} tests show <test_id>\` shows one with its history.
|
|
125
126
|
|
|
126
127
|
A test that already ran, such as one from before ${brand.name}, goes in with \`logged_after: true\` and its \`started_on\` and \`ended_on\`; it goes straight from idea to decided with \`${cli} tests decide <test_id> --decision <d> --outcome <o>\`.
|
|
127
128
|
|
|
128
|
-
\`${cli} tests results <test_id>\` reads a started test: each side's people, how many moved on to the next step, how many of those people bought (each once, however many times they bought) and what they paid, spend and the test's metric, over the days since it started (a \`before_after\` test compares the same number of days before it started). People are counted by their first visit: the landing page they arrived on, or the Adzara ad (by \`asset_version\` and the test's \`test_id\`) that brought them; a page no one's first visit was recorded on is counted by its visits instead, and a plan whose \`unit\` is sessions counts its pages by visits (a visit is one visitor on the page on one day). A side counted by people is never called against one counted by visits. Each side's numbers per day are in \`cro_test_results_daily\` (a person counts on their first-visit day; spend on the day it was spent), refreshed after each sync, so you can chart a test with \`${cli} warehouse query\`. Once a test is decided, its results are saved with it and \`${cli} tests results\` shows them as they were at the decision. Then it says plainly whether the test can be called yet: "Not enough people yet: 210 of the 400 each side needs.", "Video at the top
|
|
129
|
+
\`${cli} tests results <test_id>\` reads a started test: each side's people, how many moved on to the next step, how many of those people bought (each once, however many times they bought) and what they paid, spend and the test's metric, over the days since it started (a \`before_after\` test compares the same number of days before it started). People are counted by their first visit: the landing page they arrived on, or the Adzara ad (by \`asset_version\` and the test's \`test_id\`) that brought them; a page no one's first visit was recorded on is counted by its visits instead, and a plan whose \`unit\` is sessions counts its pages by visits (a visit is one visitor on the page on one day). A side counted by people is never called against one counted by visits. Each side's numbers per day are in \`cro_test_results_daily\` (a person counts on their first-visit day; spend on the day it was spent), refreshed after each sync, so you can chart a test with \`${cli} warehouse query\`. Once a test is decided, its results are saved with it and \`${cli} tests results\` shows them as they were at the decision. Then it says plainly whether the test can be called yet, with how likely the new version is better: "Too early: day 6 of 14.", "Not enough people yet: 210 of the 400 each side needs.", "Video at the top wins: 97% likely better than letter at the top ..." (95% or more, at least 30 conversions on the smaller side, and \`people_per_arm\` reached or the most days passed), "Leaning toward video at the top, but not proven: 88% likely better ..." (85% to 95%: a hunch to retest, never a win), "No difference found ..." or "The split looks broken ..." (people did not split as planned, so the result is not to be trusted). No call comes before \`min_days\` (14 unless set), and once they pass only whole weeks from the start count. With 2 or 3 variants each needs a higher chance. Every call ends with an evidence grade, A to D: before_after and adaptive tests and numbers another tool counted are never above C. Cost per sale and revenue per visitor are judged on sales per visitor under them, and cost per lead on sign-ups per visitor. When a test is planned, ${brand.name} estimates from the control's last 28 days how many people each side needs and how many weeks that takes; \`${cli} tests show\` prints it as "Needs", and a plan with no \`people_per_arm\` takes the estimate's.
|
|
129
130
|
|
|
130
131
|
\`${cli} tests decide <test_id> --decision <d> --outcome <won|lost|inconclusive|invalid> --note "<why>"\` records the decision on a running test: \`scale\` (the new version won; use it everywhere), \`continue\` (keep running), \`iterate\` (promising; it also writes down a sharper version linked to this one), \`hold\` (pause it; something outside the test got in the way) or \`stop\` (it lost or made no difference). scale, iterate and stop need \`--outcome\`. Each decision writes a lesson with the test, its pages and its step as proof, in one go with the decision. Deciding changes nothing live: switching a page or ads still goes through approval.
|
|
131
132
|
|
|
@@ -591,7 +592,7 @@ ${cli} number-alerts [--since <date>] the key numbers that swung far from norm
|
|
|
591
592
|
${cli} facts propose --file <fact.json> suggest a fact for a person to confirm
|
|
592
593
|
${cli} lessons [--status active|retired] [--about ads|pages|funnel|messages|cross_channel] [--dimension <d>] | lessons add --file <lesson.json> | lessons retire <id> --note "<why>" lessons from winners and flops, writing one, retiring one
|
|
593
594
|
${cli} tests p1 this brand's P1 test ads by verdict, judged on its own sales against its P1 thresholds, with Meta's count beside
|
|
594
|
-
${cli} tests [--status <status>] [--funnel <key>] [--limit <n>] | tests show <id> | tests draft --file <test.json> | tests start <id> | tests abandon <id> [--note "<why>"] | tests results <id> | tests decide <id> --decision <d> [--note "<why>"] | tests report <id> --file <counts.json> funnel tests, one change each
|
|
595
|
+
${cli} tests [--status <status>] [--funnel <key>] [--limit <n>] | tests show <id> | tests categories | tests draft --file <test.json> | tests start <id> | tests abandon <id> [--note "<why>"] | tests results <id> | tests decide <id> --decision <d> [--note "<why>"] | tests report <id> --file <counts.json> funnel tests, one change each
|
|
595
596
|
${cli} ads tags [--ad <id>] [--status <status>] [--limit <n>] | ads tag --file <tags.json> the marketing tags of ads not made here, and suggesting them
|
|
596
597
|
${cli} ads names [--unparsed] Meta ad names and the tags each gives; --unparsed lists those that fit no pattern
|
|
597
598
|
${cli} ads tag-formats tag every untagged ad's format (image, video or carousel) from Meta's records, free; run it before ads autotag
|
package/dist/main.js
CHANGED
|
@@ -41,6 +41,7 @@ const AD_CHANGE_COLUMNS = ["platform", ...GOOGLE_ADS_CHANGE_COLUMNS];
|
|
|
41
41
|
const SERVER_WAIT_CAP = 50;
|
|
42
42
|
const REVIEW_WAIT_SECONDS = 1800;
|
|
43
43
|
const PROFILE_MESSAGING_USAGE = `profile messaging [--section ${PROFILE_MESSAGING_KINDS.join("|")}] [--offer <offer key>] [--json]`;
|
|
44
|
+
const TEST_CHANGE_FLAGS = '[--category <category> [--subcategory <subcategory>] [--label "<words>"]] [--also <change>]...';
|
|
44
45
|
const PROFILE_ANSWERS_USAGE = "profile answers (--category <category> | --ids <id,id,...>) [--json]";
|
|
45
46
|
const RULES_ADD_USAGE = 'rules add banned_claims|words_to_avoid|required_disclaimers "<text>" [--note <why>]';
|
|
46
47
|
const PROFILE_CHECK_USAGE = "profile check --file ad.json | --input '<json>' | < ad.json [--json]";
|
|
@@ -3230,23 +3231,34 @@ const commands = {
|
|
|
3230
3231
|
printTest(shown.test, shown.events);
|
|
3231
3232
|
},
|
|
3232
3233
|
},
|
|
3234
|
+
"tests categories": {
|
|
3235
|
+
usage: "tests categories what a test can be about: each category and its subcategories, and the labels this workspace's tests already use",
|
|
3236
|
+
async run(args, options) {
|
|
3237
|
+
if (args.length > 0)
|
|
3238
|
+
throw new CliError("Usage: tests categories", exitCodes.usage);
|
|
3239
|
+
const list = await api(await resolveConnection(options), `${TESTS_PATH}/categories`);
|
|
3240
|
+
if (options.format === "json")
|
|
3241
|
+
return printJson(list);
|
|
3242
|
+
printTestCategories(list);
|
|
3243
|
+
},
|
|
3244
|
+
},
|
|
3233
3245
|
"tests draft": {
|
|
3234
|
-
usage:
|
|
3246
|
+
usage: `tests draft --file <test.json> | < test.json ${TEST_CHANGE_FLAGS} write down a test idea: the one change and where (a funnel step or a page); the rest of the plan may come later with tests edit`,
|
|
3235
3247
|
async run(args, options) {
|
|
3236
3248
|
if (args.length > 0)
|
|
3237
|
-
throw new CliError(
|
|
3238
|
-
const { test } = await api(await resolveConnection(options), TESTS_PATH, await readToolInput(options));
|
|
3249
|
+
throw new CliError(`Usage: tests draft --file <test.json> | < test.json ${TEST_CHANGE_FLAGS}`, exitCodes.usage);
|
|
3250
|
+
const { test } = await api(await resolveConnection(options), TESTS_PATH, withTestChange(await readToolInput(options), options));
|
|
3239
3251
|
if (options.format === "json")
|
|
3240
3252
|
return printJson({ test });
|
|
3241
3253
|
console.log(`Wrote down test ${test.test_id} as an idea.`);
|
|
3242
3254
|
},
|
|
3243
3255
|
},
|
|
3244
3256
|
"tests edit": {
|
|
3245
|
-
usage:
|
|
3257
|
+
usage: `tests edit <test_id> [--file <changes.json> | < changes.json] ${TEST_CHANGE_FLAGS} change some of a test's fields (null clears one); a change to the plan after the start is kept and marked in its history`,
|
|
3246
3258
|
async run(args, options) {
|
|
3247
3259
|
if (args.length !== 1)
|
|
3248
|
-
throw new CliError(
|
|
3249
|
-
const { test } = await api(await resolveConnection(options), testPath(args[0]), await readToolInput(options), "PATCH");
|
|
3260
|
+
throw new CliError(`Usage: tests edit <test_id> [--file <changes.json> | < changes.json] ${TEST_CHANGE_FLAGS}`, exitCodes.usage);
|
|
3261
|
+
const { test } = await api(await resolveConnection(options), testPath(args[0]), withTestChange(await readToolInput(options), options), "PATCH");
|
|
3250
3262
|
if (options.format === "json")
|
|
3251
3263
|
return printJson({ test });
|
|
3252
3264
|
console.log(`Changed test ${test.test_id} (${test.status}).`);
|
|
@@ -3258,10 +3270,10 @@ const commands = {
|
|
|
3258
3270
|
if (args.length !== 1 || !options.approvedBy)
|
|
3259
3271
|
throw new CliError('Usage: tests plan <test_id> --approved-by "<owner>" [--note "<what they said>"]', exitCodes.usage);
|
|
3260
3272
|
const body = { approved_by: options.approvedBy, ...(options.note ? { note: options.note } : {}) };
|
|
3261
|
-
const
|
|
3273
|
+
const planned = await api(await resolveConnection(options), `${testPath(args[0])}/plan`, body);
|
|
3262
3274
|
if (options.format === "json")
|
|
3263
|
-
return printJson(
|
|
3264
|
-
console.log(`Test ${test.test_id} is planned; ${options.approvedBy} said yes
|
|
3275
|
+
return printJson(planned);
|
|
3276
|
+
console.log(`Test ${planned.test.test_id} is planned; ${options.approvedBy} said yes, ${planned.test.people_per_arm} people each side.\nNeeds: ${planned.estimate.words}`);
|
|
3265
3277
|
},
|
|
3266
3278
|
},
|
|
3267
3279
|
"tests start": {
|
|
@@ -4810,6 +4822,27 @@ const VOICE_SUGGESTIONS_PATH = "/api/v1/voice-lists/suggestions";
|
|
|
4810
4822
|
const LESSONS_PATH = "/api/v1/lessons";
|
|
4811
4823
|
const TESTS_PATH = "/api/v1/funnel-tests";
|
|
4812
4824
|
const TEST_COLUMNS = ["test_id", "status", "where", "change", "metric"];
|
|
4825
|
+
/** The test body with the change flags on top: --category, --subcategory and --label give the main change, each --also one more. */
|
|
4826
|
+
function withTestChange(body, options) {
|
|
4827
|
+
const flags = {
|
|
4828
|
+
...(options.category === undefined ? {} : { category: options.category }),
|
|
4829
|
+
...(options.subcategory === undefined ? {} : { subcategory: options.subcategory }),
|
|
4830
|
+
...(options.label === undefined ? {} : { label: options.label }),
|
|
4831
|
+
...(options.also === undefined ? {} : { also_changed: options.also }),
|
|
4832
|
+
};
|
|
4833
|
+
if (Object.keys(flags).length === 0)
|
|
4834
|
+
return body;
|
|
4835
|
+
if (body === null || typeof body !== "object" || Array.isArray(body))
|
|
4836
|
+
throw new CliError("The test file must hold a JSON object to add --category, --subcategory, --label or --also to.", exitCodes.usage);
|
|
4837
|
+
return { ...body, ...flags };
|
|
4838
|
+
}
|
|
4839
|
+
/** A change as the API returns it, written as also_changed takes it: copy.headline, offer.other:<label> or custom:<label>. */
|
|
4840
|
+
const changeText = (c) => (c.category === "custom" ? `custom:${c.label}` : c.label === undefined ? `${c.category}.${c.subcategory}` : `${c.category}.other:${c.label}`);
|
|
4841
|
+
function printTestCategories(list) {
|
|
4842
|
+
const lines = list.categories.map((c) => `${c.category} (${c.words}): ${c.subcategories.map((s) => (s.words === s.subcategory.replaceAll("_", " ") ? s.subcategory : `${s.subcategory} (${s.words})`)).join(", ") || "no subcategories; give --label"}`);
|
|
4843
|
+
console.log(lines.join("\n"));
|
|
4844
|
+
console.log(list.labels.length === 0 ? "\nNo labels used yet." : `\nLabels already used (reuse one that fits):\n${renderTable(["category", "label", "tests"], list.labels)}`);
|
|
4845
|
+
}
|
|
4813
4846
|
const testPath = (ref) => `${TESTS_PATH}/${encodeURIComponent(ref)}`;
|
|
4814
4847
|
function testRow(t) {
|
|
4815
4848
|
return { test_id: t.test_id, status: t.status, where: testWhere(t), change: excerpt(t.change, 50), metric: t.metric ?? "" };
|
|
@@ -4843,12 +4876,15 @@ function printTest(t, events) {
|
|
|
4843
4876
|
...t.arms.filter((a) => a.content !== null).map((a) => ` ${a.arm} shows: ${excerpt(a.content, 100)}`),
|
|
4844
4877
|
...line("Split", t.design_type && `${t.design_type}${t.design_note ? `, ${t.design_note}` : ""}`),
|
|
4845
4878
|
...line("Judged by", t.metric && `${t.metric} by ${t.unit ?? "people"}${t.people_per_arm ? `, ${t.people_per_arm} each side` : ""}, ${t.min_days} to ${t.max_days} days`),
|
|
4879
|
+
...line("Needs", plannedEstimate(events)?.words),
|
|
4846
4880
|
...line("Today", pct(t.baseline_rate) && `${pct(t.baseline_rate)} from ${t.baseline_from} to ${t.baseline_to}; smallest lift worth finding ${pct(t.min_lift)}`),
|
|
4847
4881
|
...line("Must not get worse", t.guardrails.join(", ")),
|
|
4848
4882
|
...line("Decision rule", t.decision_rule),
|
|
4849
4883
|
...line("Also changing", t.concurrent_changes),
|
|
4850
4884
|
...line("Traffic", t.traffic && [t.traffic.campaigns?.length && `campaigns ${t.traffic.campaigns.join(", ")}`, t.traffic.optimization_goal && `optimized for ${t.traffic.optimization_goal}`, t.traffic.note].filter(Boolean).join("; ")),
|
|
4851
|
-
...line("
|
|
4885
|
+
...line("Category", t.main_change && changeText(t.main_change)),
|
|
4886
|
+
...line("The test also changes", t.also_changed.map(changeText).join(", ")),
|
|
4887
|
+
...line("Tags", t.levers.join(", ")),
|
|
4852
4888
|
...line("ICE", t.ice_impact !== null || t.ice_confidence !== null || t.ice_ease !== null ? `${t.ice_impact ?? "-"} / ${t.ice_confidence ?? "-"} / ${t.ice_ease ?? "-"}` : null),
|
|
4853
4889
|
...(t.evidence.length > 0 ? ["Evidence:", ...t.evidence.map((e) => ` ${e.kind}: ${e.ref}${e.note ? ` (${e.note})` : ""}`)] : []),
|
|
4854
4890
|
...line("Started", t.started_on && `${t.started_on}${t.ended_on ? `, ended ${t.ended_on}` : ""}`),
|
|
@@ -4860,6 +4896,10 @@ function printTest(t, events) {
|
|
|
4860
4896
|
lines.push("History:", ...events.map((e) => ` ${e.at.slice(0, 16).replace("T", " ")} ${eventWords(e)} (${e.author})`));
|
|
4861
4897
|
console.log(lines.join("\n"));
|
|
4862
4898
|
}
|
|
4899
|
+
/** The estimate the test was planned with, from its latest move to planned. */
|
|
4900
|
+
function plannedEstimate(events) {
|
|
4901
|
+
return events.findLast((e) => e.kind === "status" && e.detail.to === "planned" && e.detail.estimate !== undefined)?.detail.estimate;
|
|
4902
|
+
}
|
|
4863
4903
|
function eventWords(e) {
|
|
4864
4904
|
const d = e.detail;
|
|
4865
4905
|
switch (e.kind) {
|
|
@@ -5333,6 +5373,11 @@ Options:
|
|
|
5333
5373
|
--approved-by <name> tests plan: who said yes to the test's plan, as the owner gave their name in your session
|
|
5334
5374
|
--outcome <o> tests decide: what the test found, won, lost, inconclusive or invalid; needed with scale, iterate and stop
|
|
5335
5375
|
--learning <text> tests decide: what the test taught, in a sentence, kept on the test
|
|
5376
|
+
--category <c> tests draft and edit: what the test is mainly about, such as copy (tests categories lists them)
|
|
5377
|
+
--subcategory <s> tests draft and edit: the part of the category it changes, such as headline
|
|
5378
|
+
--label <words> tests draft and edit: for subcategory other or category custom, what was tested, such as "exit pop-up"
|
|
5379
|
+
--also <change> tests draft and edit: something else the test changed, such as copy.subheadline,
|
|
5380
|
+
offer.other:<label> or custom:<label>; repeat for several
|
|
5336
5381
|
--about <a> with lessons, only the lessons with a proof about ads, pages, funnel (steps and tests) or messages
|
|
5337
5382
|
(emails, texts and sequences), or cross_channel: proven on two or more of ads, pages and messages
|
|
5338
5383
|
--dimension <d> with lessons, only the lessons along that dimension (hook, angle, format, offer, persona, page, funnel_step, follow_up, other); repeat for several
|
|
@@ -5620,6 +5665,8 @@ function parseOptions(argv) {
|
|
|
5620
5665
|
awareness: { type: "string" },
|
|
5621
5666
|
confidence: { type: "string" },
|
|
5622
5667
|
category: { type: "string" },
|
|
5668
|
+
subcategory: { type: "string" },
|
|
5669
|
+
also: { type: "string", multiple: true },
|
|
5623
5670
|
ids: { type: "string" },
|
|
5624
5671
|
section: { type: "string" },
|
|
5625
5672
|
offer: { type: "string" },
|
|
@@ -5849,6 +5896,8 @@ function parseOptions(argv) {
|
|
|
5849
5896
|
entryTags: Object.fromEntries(Object.entries(entryTagFlags).flatMap(([field, flag]) => (values[flag] === undefined ? [] : [[field, values[flag]]]))),
|
|
5850
5897
|
confidence: values.confidence,
|
|
5851
5898
|
category: values.category,
|
|
5899
|
+
subcategory: values.subcategory,
|
|
5900
|
+
also: values.also,
|
|
5852
5901
|
ids: values.ids,
|
|
5853
5902
|
section: values.section,
|
|
5854
5903
|
offer: values.offer,
|
package/dist/map.js
CHANGED
|
@@ -83,8 +83,20 @@ export function renderQuestions(map) {
|
|
|
83
83
|
const note = unmeasured ? [" Worked out from the nightly counts, so every question shows as not yet until the first nightly run."] : [];
|
|
84
84
|
return `${["Questions", ...note, ...map.questions.flatMap(questionLines)].join("\n")}\n`;
|
|
85
85
|
}
|
|
86
|
+
/** Wrangled's own tables and views, grouped by owner, one line each with its kinds. */
|
|
87
|
+
export function renderOwn(own) {
|
|
88
|
+
const lines = [`${brand.name}'s own data (not counted yet)`];
|
|
89
|
+
own.forEach((row, i) => {
|
|
90
|
+
if (row.owner !== own[i - 1]?.owner)
|
|
91
|
+
lines.push(` ${row.owner}`);
|
|
92
|
+
lines.push(` ${row.table} ${row.kinds.join(", ")}`);
|
|
93
|
+
});
|
|
94
|
+
return `${lines.join("\n")}\n`;
|
|
95
|
+
}
|
|
86
96
|
export function renderDataMap(map) {
|
|
97
|
+
// A server from before the own section sends none.
|
|
98
|
+
const own = map.own?.length ? [renderOwn(map.own)] : [];
|
|
87
99
|
if (map.sources.length === 0)
|
|
88
|
-
return "No sources connected.\n";
|
|
89
|
-
return `${map.sources.map((source) => sourceLines(source).join("\n")).join("\n\n")}\n
|
|
100
|
+
return ["No sources connected.\n", ...own].join("\n");
|
|
101
|
+
return [`${map.sources.map((source) => sourceLines(source).join("\n")).join("\n\n")}\n`, ...own, renderQuestions(map)].join("\n");
|
|
90
102
|
}
|
package/dist/playbooks.js
CHANGED
|
@@ -2017,7 +2017,8 @@ export const croTestExample = {
|
|
|
2017
2017
|
min_lift: 0.1,
|
|
2018
2018
|
people_per_arm: 300,
|
|
2019
2019
|
decision_rule: "Keep the text if attendance rises at least 10% with unsubscribes no higher.",
|
|
2020
|
-
|
|
2020
|
+
category: "follow_up",
|
|
2021
|
+
subcategory: "text_message",
|
|
2021
2022
|
levers: ["urgency"],
|
|
2022
2023
|
control: { label: "today's reminders: an email the day before and an hour before", content: "Email the day before and an hour before; no text." },
|
|
2023
2024
|
variants: [{ label: "the same reminders plus a text 15 minutes before", content: "We're live in 15 minutes: {link}" }],
|
|
@@ -2082,7 +2083,7 @@ ${JSON.stringify(croTestExample)}
|
|
|
2082
2083
|
- \`control\` is what runs now; \`variants\` is the changed version, each \`{label, content, page?, url?, item_id?, asset_version?}\`. \`content\` is the exact words or change each side shows.
|
|
2083
2084
|
- \`metric\` is the number that should move: \`step_conversion\` (only with a \`step_key\`), \`sales_per_visitor\`, \`revenue_per_visitor\`, \`cost_per_sale\` or \`cost_per_lead\` (spend per sign-up), counted by \`unit\` (people or sessions). \`baseline_rate\` is today's rate over \`baseline_from\` to \`baseline_to\`, \`min_lift\` the smallest lift worth finding, \`people_per_arm\` how many each side needs, \`guardrails\` the numbers that must not get worse, and \`decision_rule\` how you will call it. \`min_days\` and \`max_days\` are 14 and 56 unless set.
|
|
2084
2085
|
- \`design_type\` is how traffic is split: \`ad_split_randomized\` (Meta's own A/B test), \`ad_split_duplicated\` (two copies of a campaign, one to each page; not a true random split), \`tool_split\` (inside a page builder or webinar tool, named in \`design_note\`), \`link_split\`, \`before_after\` or \`adaptive\`. \`traffic\` says which campaigns feed it and what they optimize for. Until #2202, an ad split test is counted only from Adzara's labels, so it can be measured only on ads Adzara made before the switch; for a new test, use \`before_after\`.
|
|
2085
|
-
- \`
|
|
2086
|
+
- \`category\` and \`subcategory\` say what the test is mainly about, and \`also_changed\` anything else it changed: run \`${cli} tests categories\` for the lists and the labels already used, and reuse a label before writing a new one. \`levers\` tag why it should work, from \`${cli} guide\`'s fixed list.
|
|
2086
2087
|
- \`evidence\` quotes the proof, each \`{kind, ref, note}\` with kind \`leak\`, \`lesson\`, \`customer_words\`, \`page\`, \`ad\` or \`other\`. Start with the leak itself.
|
|
2087
2088
|
- Give \`hypothesis_id\` and \`test_id\` only to reuse Adzara's own labels; left out, they are made for you.
|
|
2088
2089
|
|
|
@@ -2134,11 +2135,13 @@ ${cli} tests results <test_id> --format json > results-1.json
|
|
|
2134
2135
|
|
|
2135
2136
|
Without a shell: MCP \`get_funnel_test_results\` with \`{"id": "<test_id>"}\`.
|
|
2136
2137
|
|
|
2137
|
-
\`calls\` has one call for each changed version against the control: \`not_enough_yet\` (with the \`people\` each side has and the \`need\`), \`
|
|
2138
|
+
\`calls\` has one call for each changed version against the control: \`split_broken\`, \`too_early\` (with the \`day\` and the \`of\`), \`not_enough_yet\` (with the \`people\` each side has and the \`need\`), \`variant_wins\` or \`control_wins\` (95% or more likely better), \`variant_directional\` or \`control_directional\` (85% to 95%: leaning, not proven) or \`no_difference_found\`, and \`words\` says it plainly. \`grade\` (A to D) says how much to trust it and \`grade_reason\` why. \`caveats\` names what weakens the call.
|
|
2138
2139
|
|
|
2139
|
-
- **Every call is \`not_enough_yet\`:** leave the test running; that needs no command.
|
|
2140
|
+
- **Every call is \`too_early\` or \`not_enough_yet\`:** leave the test running; that needs no command.
|
|
2141
|
+
- **A call is \`split_broken\`:** people did not split between the sides as planned. Tell the owner the result cannot be trusted, and suggest marking the test invalid (\`${cli} tests invalidate <test_id> --note "<what went wrong>"\`) once the split is checked.
|
|
2142
|
+
- **A call is directional:** it is a hunch, not a win. Suggest keeping the test running to its most days, or \`iterate\` to retest a sharper version; never \`scale\` on it. Only when something outside the test got in the way (the page broke, a price changed, the ads stopped), suggest \`hold\` to the owner.
|
|
2140
2143
|
- **The split runs in another tool** (a \`tool_split\` in GoHighLevel, a Meta A/B test): read each side's numbers in that tool and enter them with \`${cli} tests report <test_id> --file counts.json\` (MCP \`report_test_numbers\`), then read the results again; their words start "Counted by <tool>". Never decide a test judged on sales on Meta's purchase count alone.
|
|
2141
|
-
- **A call is ready
|
|
2144
|
+
- **A call is ready** (a win, or no difference found): suggest \`scale\` (the new version won), \`stop\` (it lost or made no difference) or \`iterate\` (promising; it writes down a sharper version). \`iterate\` writes a new test, so suggest it only while the week is under its 3; otherwise suggest \`stop\` and draft the sharper version next week.
|
|
2142
2145
|
|
|
2143
2146
|
Put each suggestion to the owner with the ideas from step 4, then record the owner's decision:
|
|
2144
2147
|
|