skilldrop-cli 0.16.28 → 0.16.30

package/packs/product-manager/skills/feedback-synthesis/reference.md

@@ -0,0 +1,89 @@
+# feedback-synthesis — reference
+
+## Counting
+
+- **The unit is a person or an account, stated once.** B2B with account IDs: accounts. B2C,
+  or B2B with no account IDs: people. Say which in the headline or the theme table header.
+- **Same ID, same person, across sources.** An interviewee identified as account A-1042 and the
+  tickets from A-1042 are one account. Use the same `person_id` in every row.
+- **No ID, own person.** A row with a blank `person_id` counts as its own person; the script
+  labels these. When a theme is mostly anonymous rows, its count is a count of mentions and the
+  reply says so.
+- **Display names are weak IDs.** App-store reviewers with the same name and a consistent
+  story (same device, same complaint, "updating my review") are treated as one; record it as
+  an assumption. Two different stories under a common first name are two people.
+- **Anonymous beside identified.** "3 accounts, plus one anonymous comment that may be one of
+  them" — never 4.
+- **A reporter speaking for others is one source, not many.** "Two of our clients in France
+  couldn't pay" is one account reporting; the count of affected end users goes in the evidence
+  as what the account said, not in the people count.
+- **Repeat filing is evidence of cost, not of breadth.** Six tickets from one account say the
+  problem is costing that account; they don't say six customers have it. Report the repeat
+  filing as severity evidence ("raised six times in two months").
+
+## Severity scale
+
+Rate each item from what happened, not tone.
+
+| Level | Rule | Edge cases |
+|---|---|---|
+| S1 — blocker | A core job can't be done, or money, data, security, privacy or compliance is at stake | A payment that fails and is later made another way is still S1: the job failed in the product. A message sent to the wrong recipient is S2 unless the person says it exposed sensitive data. |
+| S2 — major | The job is done only through a costly recurring workaround, or the person says they will leave over it | "Thinking about moving to another tool" lifts an otherwise S3 item to S2. A one-off workaround that took an hour is S3. |
+| S3 — minor | Slower, confusing or annoying; the workaround is quick | A how-to question is S3 when the person couldn't find something that exists; it is not a severity at all when they just asked. |
+| S4 — cosmetic | Appearance, wording or preference with no stated cost | "Looks unprofessional" stays S4 unless the person says it cost them a client. |
+
+A theme's severity is the worst level at least two people reached (that level or worse). A
+theme raised by one person takes that person's level, labelled *one report — confirm before
+acting*. Report the spread ("S1 for 3
+accounts; S3 for 1") when it differs.
+
+## Sampling bias by source
+
+| Source | Who is in it | What it can't support |
+|---|---|---|
+| Support tickets | People who hit a problem and chose to write in; heavy users and the angriest over-represented | How common a problem is across all customers; what satisfied customers think |
+| NPS / CSAT verbatims | Respondents who left a comment; strong opinions at both ends over-respond, and detractors write more | Satisfaction of the base; comparing themes by share of responses as if it were share of customers |
+| App-store reviews | People moved to review, often right after a crash or a prompt; no IDs | Anything per user; retention; why non-reviewers stay |
+| Interviews | Whoever agreed to talk, often recruited from engaged customers; small n | Frequency of anything; "4 of 5 interviewees" is a story count, not a rate |
+| Sales-call notes | Prospects, filtered through a rep's summary; skews to those who asked for a demo and to what reps listen for | What existing customers need; prospect words, unless in quotation marks |
+| Churn / exit surveys | Leavers who answered; the first reason ticked, often the easiest | Why people stay; reasons of segments that didn't churn in the period |
+| Community / social posts | The vocal few; public posts perform | Anything per customer without IDs |
+
+Always name, per source used, who is in it and one conclusion it can't support. Never turn a
+self-selected count into a percentage of the user base. A share of the sample is allowed when
+labelled as such: ✅ *"5 of the 10 exit-survey respondents"*.
+
+## Rewriting a request into a problem
+
+Ask of each request: *what was the person trying to do when they wanted this, and what does
+not having it cost them?* Use only what they said.
+
+| Request as given | Problem statement (when the item says why) | When the item doesn't say why |
+|---|---|---|
+| "Please add bulk edit for recurring invoices." + "had to open 58 recurring invoices one at a time" | Changing a rate means editing every recurring invoice by hand | — |
+| "Add dark mode." | — | Request with no stated need; interview question: *"When do you use the app, and what's hard about it then?"* |
+| "We need SSO with Entra." | Enterprise IT won't approve a rollout without central sign-in | — |
+| "Make it work offline." + "half our jobs are in plant rooms" | Technicians lose access to job details where there's no signal | — |
+
+The request stays in the evidence as a quote. Two different requests can sit under one problem,
+and one request can point at two problems; code both.
+
+## Codebook rules
+
+- 4–12 problem themes plus the valued (`+`) themes. Fewer than 4 from 30+ items means themes are
+  too broad; more than 12 means they are too narrow to act on.
+- One sentence each: who, what happens, what it costs. Code is short kebab-case.
+- An item can carry several codes. An item that fits no theme stays unthemed; don't stretch a
+  theme to swallow it.
+- How-to questions are coded only when several point at one gap ("can't find reports after the
+  update"); otherwise they are unthemed and listed by count.
+- Contradictions are their own finding, not a reason to drop a theme: code both sides.
+
+## Interview questions
+
+Tied to a theme, open, about past behaviour, not hypothetical, not leading:
+
+- ✅ *"Tell me about the last time you changed your prices. What did you do in the product?"*
+- ✅ Disconfirming: *"Which clients pay by card without trouble? What's different about them?"*
+- ❌ *"Would you pay more for bulk edit?"* — hypothetical and leading
+- ❌ *"Is dark mode important to you?"* — yes/no, and the answer is always yes

package/packs/product-manager/skills/feedback-synthesis/scripts/theme_counts.py

@@ -0,0 +1,355 @@
+#!/usr/bin/env python3
+"""Count coded customer feedback by distinct person, not by mention.
+
+Reads a coding sheet: one row per feedback item (a ticket, a survey verbatim, an interview
+note, a review) that an agent or analyst has already tagged with theme codes. Prints, per
+theme:
+  - distinct people (or accounts) and mentions (items), and which people filed several
+  - the sources it came from, with distinct people per source
+  - the segment breakdown for every segment column, and a flag when a theme lives in one
+    segment value only
+  - distinct people at each severity level, and the worst level reported
+and then the items with no theme, the people who filed most, and the rows whose person ID
+is missing (those count as one person each, so their numbers are mentions).
+
+Columns (header names, case-insensitive):
+  item_id     required  ticket / response / note ID, quoted back as the source of a quote
+  person_id   optional  person or account ID; blank -> the row counts as its own person
+  source      optional  tickets, nps, interview, app-store, sales-call ...
+  themes      required  theme codes separated by ";". A code may carry its own severity
+                        as code:S1, overriding the row's severity. A code starting with "+"
+                        is something people value (no severity).
+  severity    optional  S1 (blocker) .. S4 (cosmetic); blank -> unrated
+  any other column is a segment (plan, role, region, tenure ...) unless named in --ignore.
+  Columns called quote, text, notes, comment, date, created or summary are ignored by default.
+
+Usage:
+    python3 theme_counts.py coded.csv
+    python3 theme_counts.py coded.csv --person-col account_id -o counts.md
+    python3 theme_counts.py coded.csv --segment plan --segment region --json counts.json
+
+Stdlib only; Python 3.9+. Reads only the CSV it is given; writes only the paths named with
+-o / --json.
+"""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+import sys
+from collections import Counter, defaultdict
+from pathlib import Path
+
+SEVERITIES = ["S1", "S2", "S3", "S4"]
+SEVERITY_ALIASES = {
+    "s1": "S1", "1": "S1", "blocker": "S1", "critical": "S1",
+    "s2": "S2", "2": "S2", "major": "S2", "high": "S2",
+    "s3": "S3", "3": "S3", "minor": "S3", "medium": "S3",
+    "s4": "S4", "4": "S4", "cosmetic": "S4", "low": "S4", "trivial": "S4",
+}
+DEFAULT_IGNORE = {"quote", "text", "notes", "note", "comment", "comments", "date", "created",
+                  "summary", "body", "description", "subject", "title", "score"}
+CORE = {"item_id", "person_id", "source", "themes", "severity"}
+NO_ID = "(no id: "
+
+
+def die(msg: str) -> None:
+    sys.stderr.write("error: " + msg + "\n")
+    sys.exit(2)
+
+
+def norm_sev(value: str, where: str, warnings: list) -> str:
+    v = (value or "").strip()
+    if not v:
+        return ""
+    s = SEVERITY_ALIASES.get(v.lower())
+    if not s:
+        warnings.append(f"{where}: unknown severity '{v}' treated as unrated")
+        return ""
+    return s
+
+
+def load(path: Path, person_col: str, ignore: set, segments_wanted: list):
+    try:
+        text = path.read_text(encoding="utf-8-sig")
+    except OSError as e:
+        die(f"cannot read {path}: {e}")
+    reader = csv.DictReader(text.splitlines())
+    if not reader.fieldnames:
+        die(f"{path} has no header row")
+    header = {h.strip().lower(): h for h in reader.fieldnames if h}
+    person_key = person_col.lower()
+    for need in ("item_id", "themes"):
+        if need not in header:
+            die(f"{path} needs a '{need}' column (found: {', '.join(reader.fieldnames)})")
+    if person_col != "person_id" and person_key not in header:
+        die(f"--person-col '{person_col}' is not a column in {path}")
+    core = (CORE - {"person_id"}) | {person_key}
+    if segments_wanted:
+        segs = [s.lower() for s in segments_wanted]
+        for s in segs:
+            if s not in header:
+                die(f"segment column '{s}' is not in {path}")
+    else:
+        segs = [h for h in header if h not in core and h not in ignore and h != "person_id"]
+
+    warnings: list = []
+    rows = []
+    seen_ids = set()
+    for n, raw in enumerate(reader, start=2):
+        r = {k.strip().lower(): (v or "").strip() for k, v in raw.items() if k}
+        item = r.get("item_id", "")
+        if not item:
+            warnings.append(f"line {n}: no item_id; row skipped")
+            continue
+        if item in seen_ids:
+            warnings.append(f"line {n}: item_id {item} appears twice; both rows counted")
+        seen_ids.add(item)
+        pid = r.get(person_key, "")
+        row_sev = norm_sev(r.get("severity", ""), f"{item}", warnings)
+        codes = []
+        for c in r.get("themes", "").split(";"):
+            c = c.strip()
+            if not c:
+                continue
+            sev = row_sev
+            if ":" in c:
+                c, _, s = c.partition(":")
+                c = c.strip()
+                sev = norm_sev(s, f"{item} theme {c}", warnings)
+            if c.startswith("+"):
+                sev = ""
+            if c:
+                codes.append((c, sev))
+        rows.append({
+            "item": item,
+            "person": pid or f"{NO_ID}{item})",
+            "has_id": bool(pid),
+            "source": r.get("source", "") or "(none)",
+            "segments": {s: r.get(s, "") for s in segs},
+            "codes": codes,
+        })
+    return rows, segs, warnings
+
+
+def analyse(rows, segs):
+    people_seg = defaultdict(dict)          # person -> seg -> value (first non-empty)
+    seg_conflicts = defaultdict(set)
+    for r in rows:
+        for s, v in r["segments"].items():
+            if not v:
+                continue
+            have = people_seg[r["person"]].get(s)
+            if have is None:
+                people_seg[r["person"]][s] = v
+            elif have != v:
+                seg_conflicts[(r["person"], s)].update({have, v})
+
+    all_values = {s: Counter() for s in segs}
+    for p, d in people_seg.items():
+        for s, v in d.items():
+            all_values[s][v] += 1
+
+    themes = defaultdict(lambda: {"items": [], "people": Counter(), "sources": defaultdict(set),
+                                  "sev_people": defaultdict(set)})
+    for r in rows:
+        for code, sev in r["codes"]:
+            t = themes[code]
+            t["items"].append(r["item"])
+            t["people"][r["person"]] += 1
+            t["sources"][r["source"]].add(r["person"])
+            t["sev_people"][sev or "unrated"].add(r["person"])
+
+    out = []
+    for code, t in themes.items():
+        people = list(t["people"])
+        seg_break = {}
+        single = []
+        for s in segs:
+            c = Counter(people_seg[p].get(s, "(blank)") for p in people)
+            seg_break[s] = dict(sorted(c.items(), key=lambda kv: (-kv[1], kv[0])))
+            known = [v for v in c if v != "(blank)"]
+            known_people = sum(n for v, n in c.items() if v != "(blank)")
+            total = sum(all_values[s].values())
+            # Flag a theme that sits in one segment value only, unless that value is most of
+            # the whole sample anyway (a UK-only theme in a 90% UK sample says nothing).
+            if (known_people >= 2 and len(known) == 1 and len(all_values[s]) >= 2
+                    and all_values[s][known[0]] / total < 2 / 3):
+                blank = c.get("(blank)", 0)
+                single.append(f"{s}={known[0]} ({all_values[s][known[0]]} of {total} overall"
+                              + (f"; {blank} blank here" if blank else "") + ")")
+        # A person's severity for a theme is the worst level they reported for it.
+        worst_by_person = {}
+        for sev in SEVERITIES + ["unrated"]:
+            for p in t["sev_people"].get(sev, ()):
+                worst_by_person.setdefault(p, sev)
+        sev_counts = Counter(worst_by_person.values())
+        worst = next((s for s in SEVERITIES if sev_counts.get(s)), "")
+        # Ranking severity: the worst level that at least two people reached (that level or
+        # worse). A theme raised by one person takes that person's level, marked "1 report".
+        rank_sev, running = "", 0
+        for s in SEVERITIES:
+            running += sev_counts.get(s, 0)
+            if running >= 2:
+                rank_sev = s
+                break
+        if not rank_sev and len(people) == 1 and worst:
+            rank_sev = worst + " (1 report)"
+        elif not rank_sev and worst:
+            rank_sev = worst + " (1 rated)"
+        top_person, top_n = t["people"].most_common(1)[0]
+        no_id = [p for p in people if p.startswith(NO_ID)]
+        out.append({
+            "theme": code,
+            "valued": code.startswith("+"),
+            # people = distinct people with an ID. Rows with no ID are reported beside it,
+            # never added to it: an anonymous comment may come from someone already counted.
+            "people": len(people) - len(no_id),
+            "no_id_items": len(no_id),
+            "mentions": len(t["items"]),
+            "repeat_filers": {p: n for p, n in t["people"].most_common() if n > 1},
+            "top_person_share": round(top_n / len(t["items"]), 2),
+            "top_person": top_person,
+            "sources": {s: {"people": sum(1 for p in ps if not p.startswith(NO_ID)),
+                            "no_id_items": sum(1 for p in ps if p.startswith(NO_ID))}
+                        for s, ps in sorted(t["sources"].items())},
+            "segments": seg_break,
+            "single_segment": single,
+            "severity_people": {s: sev_counts[s] for s in SEVERITIES + ["unrated"] if sev_counts.get(s)},
+            "worst_severity": worst,
+            "ranking_severity": "" if code.startswith("+") else rank_sev,
+            "items": t["items"],
+        })
+    sev_rank = {s: i for i, s in enumerate(SEVERITIES)}
+    out.sort(key=lambda d: (d["valued"], sev_rank.get(d["ranking_severity"][:2], 9), -d["people"],
+                            -d["no_id_items"], d["theme"]))
+
+    per_person = Counter(r["person"] for r in rows)
+    summary = {
+        "items": len(rows),
+        "people": len({r["person"] for r in rows if r["has_id"]}),
+        "items_without_person_id": sum(1 for r in rows if not r["has_id"]),
+        "sources": {s: {"items": n, "people": len({r["person"] for r in rows if r["source"] == s and r["has_id"]}),
+                        "no_id_items": sum(1 for r in rows if r["source"] == s and not r["has_id"])}
+                    for s, n in sorted(Counter(r["source"] for r in rows).items())},
+        "segment_values": {s: dict(all_values[s]) for s in segs},
+        "most_items_per_person": {p: n for p, n in per_person.most_common(5) if n > 1 and not p.startswith(NO_ID)},
+        "unthemed_items": [r["item"] for r in rows if not r["codes"]],
+        "segment_conflicts": {f"{p} {s}": sorted(v) for (p, s), v in seg_conflicts.items()},
+    }
+    return summary, out
+
+
+def fmt_counts(d: dict) -> str:
+    return ", ".join(f"{k} {v}" for k, v in d.items()) or "-"
+
+
+def fmt_sources(d: dict) -> str:
+    parts = []
+    for s, v in d.items():
+        bits = []
+        if v["people"]:
+            bits.append(str(v["people"]))
+        if v["no_id_items"]:
+            bits.append(f"+{v['no_id_items']} no-ID")
+        parts.append(f"{s} {' '.join(bits)}")
+    return ", ".join(parts) or "-"
+
+
+def render(summary, themes, person_label: str, warnings) -> str:
+    plural = "people" if person_label == "person" else person_label + "s"
+    L = []
+    L.append("# Theme counts")
+    L.append("")
+    L.append(f"- Items: {summary['items']}; distinct {plural} with an ID: {summary['people']}")
+    if summary["items_without_person_id"]:
+        L.append(f"- {summary['items_without_person_id']} item(s) have no {person_label} ID. They are counted "
+                 f"in the No-ID column, never added to the {person_label} count: each may come from someone "
+                 "already counted, so they are mentions, not people.")
+    L.append("- By source: " + "; ".join(
+        f"{s} {v['items']} items / {v['people']} {plural}" + (f" + {v['no_id_items']} no-ID" if v["no_id_items"] else "")
+        for s, v in summary["sources"].items()))
+    for s, vals in summary["segment_values"].items():
+        L.append(f"- {s} across all {plural} and no-ID items: {fmt_counts(dict(sorted(vals.items())))}")
+    if summary["most_items_per_person"]:
+        L.append(f"- Most items from one {person_label}: {fmt_counts(summary['most_items_per_person'])}")
+    L.append("")
+    L.append("Ranked by ranking severity (the worst level at least two people reached), then by "
+             f"distinct {plural}. Themes starting with + are things people value.")
+    L.append("")
+    L.append(f"| Theme | {plural.capitalize()} | No-ID items | Mentions | Ranking severity | Worst | "
+             f"By severity | Sources | One segment only |")
+    L.append("|---|---|---|---|---|---|---|---|---|")
+    for t in themes:
+        sev_cells = ("-", "-", "-") if t["valued"] else (
+            t["ranking_severity"] or "-", t["worst_severity"] or "-", fmt_counts(t["severity_people"]))
+        L.append(f"| {t['theme']} | {t['people']} | {t['no_id_items']} | {t['mentions']} | {sev_cells[0]} | "
+                 f"{sev_cells[1]} | {sev_cells[2]} | {fmt_sources(t['sources'])} | "
+                 f"{', '.join(t['single_segment']) or '-'} |")
+    L.append("")
+    for t in themes:
+        L.append(f"## {t['theme']}")
+        L.append(f"- {t['people']} {person_label}(s) with an ID"
+                 + (f", plus {t['no_id_items']} item(s) with no ID" if t["no_id_items"] else "")
+                 + f"; {t['mentions']} mention(s)")
+        L.append(f"- Ranking severity: {t['ranking_severity'] or '-'}" if not t["valued"] else "- Valued (no severity)")
+        if t["repeat_filers"]:
+            L.append(f"- Repeat filers: {fmt_counts(t['repeat_filers'])}")
+        if t["mentions"] >= 3 and t["top_person_share"] > 0.5:
+            L.append(f"- WARNING: {t['top_person']} accounts for {int(t['top_person_share'] * 100)}% of this theme's mentions")
+        for s, br in t["segments"].items():
+            L.append(f"- {s}: {fmt_counts(br)}")
+        if t["single_segment"]:
+            L.append(f"- Only in: {', '.join(t['single_segment'])}")
+        if not t["valued"]:
+            L.append(f"- Severity (worst level each {person_label} or no-ID item reported): "
+                     f"{fmt_counts(t['severity_people'])}")
+        L.append(f"- Items: {', '.join(t['items'])}")
+        L.append("")
+    L.append("## Items with no theme")
+    L.append(", ".join(summary["unthemed_items"]) or "None")
+    if summary["segment_conflicts"]:
+        L.append("")
+        L.append("## Segment conflicts (same ID, different values; first value used)")
+        for k, v in summary["segment_conflicts"].items():
+            L.append(f"- {k}: {', '.join(v)}")
+    if warnings:
+        L.append("")
+        L.append("## Warnings")
+        L.extend(f"- {w}" for w in warnings)
+    return "\n".join(L) + "\n"
+
+
+def main(argv=None) -> int:
+    ap = argparse.ArgumentParser(description="Count coded feedback by distinct person per theme.")
+    ap.add_argument("csv", type=Path, help="coding sheet: item_id, person_id, source, themes, severity, segment columns")
+    ap.add_argument("--person-col", default="person_id",
+                    help="column that identifies a person or account (default person_id)")
+    ap.add_argument("--person-label", default=None, help="word for the counted unit, e.g. account (default: person, or the column name)")
+    ap.add_argument("--segment", action="append", default=[], help="segment column to break down by (repeatable; default: every other column)")
+    ap.add_argument("--ignore", action="append", default=[], help="column to leave out of segments (repeatable)")
+    ap.add_argument("-o", "--output", type=Path, help="write the markdown report here instead of stdout")
+    ap.add_argument("--json", type=Path, help="also write the counts as JSON to this path")
+    a = ap.parse_args(argv)
+
+    ignore = DEFAULT_IGNORE | {c.lower() for c in a.ignore}
+    rows, segs, warnings = load(a.csv, a.person_col, ignore, a.segment)
+    if not rows:
+        die(f"{a.csv} has no feedback rows")
+    label = a.person_label or ("person" if a.person_col == "person_id" else a.person_col.replace("_id", "").replace("_", " "))
+    summary, themes = analyse(rows, segs)
+    report = render(summary, themes, label, warnings)
+    if a.output:
+        a.output.write_text(report, encoding="utf-8")
+        print(f"wrote {a.output}")
+    else:
+        sys.stdout.write(report)
+    if a.json:
+        a.json.write_text(json.dumps({"summary": summary, "themes": themes, "warnings": warnings}, indent=2),
+                          encoding="utf-8")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/packs/product-manager/skills/feedback-synthesis/templates/coding-sheet.csv

@@ -0,0 +1,6 @@
+item_id,person_id,source,plan,region,themes,severity
+T-101,A-17,tickets,Pro,UK,card-payment-fails,S1
+T-102,A-17,tickets,Pro,UK,card-payment-fails,S1
+N-04,,nps,Growth,,+reminders;logo-blurry:S4,
+I-2,A-30,interview,Pro,US,rate-change-manual;+reminders,S2
+T-108,A-41,tickets,Starter,US,,

package/packs/product-manager/skills/feedback-synthesis/templates/synthesis.md

@@ -0,0 +1,42 @@
+# {What customers are telling us about {product}, {period}}
+
+{Headline, 2–4 sentences: what to act on first and why. If the data contradicts the premise in
+the request, say so here with the count that shows it. Name the counting unit once:
+"Counts are distinct accounts, not tickets."}
+
+## Themes, ranked
+
+| # | Problem | Who | Severity | Sources |
+|---|---|---|---|---|
+| 1 | {Who hits what, and what it costs them} | {N accounts (M mentions); segment, or "only {segment}"} | {Blocker / Major / Minor / Cosmetic — one-line reason} | {tickets, interviews…} |
+
+Ranked by severity, then by distinct people. {One sentence where this differs from volume order.}
+
+## Evidence
+
+### 1. {Problem statement}
+- **Who:** {N accounts: segment breakdown. Repeat filing, if any.}
+- **What it costs them:** {from the items, not inferred}
+- **In their words:** "{verbatim}" ({ID}); "{verbatim}" ({ID})
+- **What they asked for:** "{request, verbatim}" ({ID})
+- **What would change this:** {the counter-evidence or the gap}
+
+## Requests with no stated need
+- {"Add dark mode" — N comments (IDs), no reason given. Asked about below.}
+
+## What not to break
+- {What people value, N people, "{verbatim}" ({ID})}
+
+## Where the sources disagree
+- {Source A says X (N); source B says Y (N). What would resolve it.}
+
+## What this can't tell us
+- **{Source}:** {who is in it}; {the conclusion it can't support}.
+- {No share of the customer base: these are self-selected samples.}
+
+## Questions for the next interviews
+1. {Open, past-behaviour question} — tests theme {#}
+2. {A question that could disconfirm theme 1}
+
+{Next step, one plain sentence: e.g. "Next: turn themes 1 and 2 into a brief with the open
+questions as risks."}

package/packs/research/loops/research/LOOP.md

@@ -0,0 +1,123 @@
+---
+name: research
+description: Take a question that turns on why something happened to an answer the evidence can carry: a plan tied to the decision it serves, a synthesis of the sources with every claim cited, and rival explanations ranked by how little evidence contradicts them, with a gate that sends the work back for the observation that would tell the leaders apart. Use when the user says "research why…", "find out what's causing…", "run the research loop", or needs a decision-ready answer on which explanation the evidence supports.
+---
+
+# research
+
+One question, from scoping to an answer someone can decide on. The three skills each work
+alone; in order, each one constrains the next. The plan fixes what would change the decision
+before anyone reads a source, so the findings can't be read to fit. The synthesis cites every
+claim, so the comparison rates only evidence that exists. The gate then asks whether that
+evidence separates the explanations, which none of the three can judge about its own output.
+
+## Stages
+
+| # | Stage | Type | Skills | Gate |
+|---|---|---|---|---|
+| 1 | `plan` | generate | [`research-plan`](../../skills/research-plan/SKILL.md) | — |
+| 2 | `synthesise` | generate | [`source-synthesis`](../../skills/source-synthesis/SKILL.md) | — |
+| 3 | `compare` | verify | [`hypothesis-comparison`](../../skills/hypothesis-comparison/SKILL.md) | **G7** review |
+
+## How to run this loop
+
+1. **Plan before reading.** Run `research-plan` on the question. It asks what decision the
+   research serves, who makes it and by when; don't skip that. Save the plan as `plan.md`.
+   Carry three things from it into every later stage: the decision, the findings that would
+   change it, and the stopping rule.
+2. **Gather the sources the plan names.** The person supplies them, or you collect them under
+   the plan's inclusion and exclusion criteria with the search strings it wrote. A source the
+   criteria exclude stays out, however convenient. List each as `S1: …` in `sources.md`.
+3. **Synthesise, cited.** Run `source-synthesis` in its cited form (an evidence summary with
+   `[S1:p4]` citations), on the plan's question. It counts a source that repeats another once,
+   and its citation check runs before you hand anything on. Save it as `synthesis.md`.
+4. **Compare the explanations.** Run `hypothesis-comparison` with the observation from the
+   plan as the thing to explain and the synthesis's findings as the evidence, keeping their
+   citations. Every evidence row cites a source from `sources.md` or is marked
+   `[assumption]`. The comparison names the leader, the diagnostic evidence, the linchpins and
+   the next observation that would tell the top two apart.
+5. **Judge the evidence at G7.** Ask four questions, then choose the verdict below:
+   - Does at least one diagnostic row separate the leader from the runner-up?
+   - Does any linchpin rest on a single source or an `[assumption]`?
+   - Is the plan's stopping rule met?
+   - Would the finding move the decision past a threshold the plan set?
+6. **On `REVISE`, get the observation, not more of the same.** Return to `synthesise` with the
+   one observation the comparison named as discriminating. Add it to `sources.md` as a new
+   source, re-synthesise, and compare again. More sources that the top two both predict
+   equally add nothing. Each return counts against the cap. When only a person can get the
+   observation (a data pull, an interview, a log nobody gave you), stop here: say what to
+   get, from whom, and what each result would mean, and resume when they add it.
+7. **Hand over the answer, then the verdict.** Lead with the explanation the evidence favours
+   and its confidence, then what that means for the decision named in the plan, then the
+   open conditions. The matrix and the synthesis go underneath, for anyone who wants to
+   check. Close with the gate on its own line, `G7: <VERDICT>`, and one sentence of reasons.
+   The skills keep their own terms out of a reply; the loop's verdict is the exception,
+   because the run log and the next round both read it.
+
+## Verdicts
+
+Defined in [`contracts/terminals.json`](../../../../contracts/terminals.json). G7 emits:
+
+- `PROCEED`: diagnostic evidence separates the leader, no linchpin rests on one source or an
+  assumption, and the stopping rule is met.
+- `PROCEED WITH CONDITIONS`: the leader is clear, but a named linchpin is unchecked or a
+  sub-question is still open. The conditions travel with the answer: each one says what to
+  check and how the decision changes if it comes out the other way.
+- `REVISE`: the top two are not separated by any diagnostic row, or the leader rests on one
+  weak row. Return to `synthesise` with the discriminating observation.
+- `RECONSIDER`: every hypothesis is contradicted by strong evidence, or the observation
+  itself turned out wrong (a tracking change, a definition change). The question needs
+  re-planning, so the loop ends here and a new run starts at `plan`.
+- `BLOCKED`: no decision to serve, no observation to explain, or no sources.
+
+On the cap-th `REVISE`, emit `PROCEED WITH CONDITIONS` with the leaders still tied and say
+plainly that the evidence doesn't settle it: the decision will be made under that
+uncertainty, and the discriminating observation is the first thing to get.
+
+## Non-interactive runs
+
+With no person to name the decision, `research-plan` either infers it as an `[assumption]` or
+emits `BLOCKED: need the decision this research informs, who makes it, and by when`. Carry
+that `BLOCKED` through as the loop's verdict. Run no further stages on an invented decision:
+a stopping rule set against a guessed decision stops at the wrong place. With no sources and
+nothing to collect them from, emit `BLOCKED: need the sources, or access to collect them
+under the plan's criteria`.
+
+## When a stage's skill is not installed
+
+Name the missing skill and skilldrop as its source, do the minimal inline version of that
+stage, and record that it ran degraded. A degraded `compare` still builds a matrix that
+ranks by inconsistency and names its linchpins, because G7 can't be judged without them.
+Never skip G7.
+
+## Run log (opt-in)
+
+Only when the environment variable `SKILLDROP_LOOP_LOG` is set to a file path: after each
+gate's verdict, append one JSON line to that file. Nothing else is recorded, and nothing is
+sent anywhere. When the variable isn't set, or you can't read it, skip this section.
+
+```bash
+printf '%s\n' '{"ts":"2026-10-19T14:03:00Z","loop":"research","stage":"compare","gate":"G7","verdict":"REVISE","round":1}' >> "$SKILLDROP_LOOP_LOG"
+```
+
+Fields: `ts` (UTC), `loop`, `stage`, `gate` (`null` for a stage without one), `verdict` (the
+gate's exact word), and `round` (1 for the first attempt, counting returns under the cap).
+The log holds verdicts only: never the question, the sources or the findings.
+`skilldrop loop-stats` summarises the file.
+
+## Quality bar
+
+- **What would change the decision is written before any source is read.** Thresholds set after the findings are read get set to fit them.
+- **Every evidence row in the matrix cites a source in `sources.md` or is marked `[assumption]`.** A matrix of uncited evidence ranks a favourite theory.
+- **G7 is judged on diagnostic evidence.** Rows that every hypothesis predicts equally are named and set aside, however many there are.
+- **A `REVISE` brings back the one observation that discriminates.** It does not bring back more sources of the same kind.
+- **The answer leads with the explanation and what it means for the decision.** The matrix is the working, not the reply.
+
+## Anti-patterns to avoid
+
+- ❌ **Synthesising before the plan exists.** The sources then decide the question, and the stopping rule is whenever the reading runs out.
+- ❌ **Rating evidence the synthesis didn't cite.** An observation from memory or a plausible industry figure is an `[assumption]` row, weighted as one.
+- ❌ **Counting a press release and the three articles that quote it as four sources.** The synthesis counts them once, and so does the matrix.
+- ❌ **Passing G7 on volume.** Ten consistent rows that every rival also explains don't separate anything.
+- ❌ **Looping past the cap in search of certainty.** At the cap, say that the evidence doesn't settle it and name the next check.
+- ❌ **Re-planning inside the loop after `RECONSIDER`.** A new question is a new run, with its own plan and stopping rule.

package/packs/research/loops/research/evals/eval_queries.json

@@ -0,0 +1,9 @@
+[
+  {"query": "research why our trial-to-paid conversion dropped in September", "should_trigger": true},
+  {"query": "run the research loop on what's causing the rise in support tickets", "should_trigger": true},
+  {"query": "find out which explanation the evidence supports for the churn spike, so I can decide what to fix", "should_trigger": true},
+  {"query": "I need a decision-ready answer on why the pilot underperformed", "should_trigger": true},
+  {"query": "summarise these three papers for me", "should_trigger": false},
+  {"query": "help me scope a research plan for entering the German market", "should_trigger": false},
+  {"query": "build an ACH matrix from this evidence list", "should_trigger": false}
+]