skilldrop-cli 0.16.28 → 0.16.30

package/packs/grc/loops/comply/LOOP.md

@@ -0,0 +1,133 @@
+---
+name: comply
+description: Take a change that carries risk (a launch, a new vendor, a new use of personal data) to a risk position its owner has signed: a risk register scored inherent and residual, a DPIA when personal data is processed, the evidence behind every control the ratings rely on, a script that checks the register, and a sign-off only the named owner can give. Use when the user says "get this ready for risk sign-off", "run the comply loop", "what do we need before we can launch this", or needs a register, a DPIA and audit evidence that agree with each other.
+---
+
+# comply
+
+One change, from its risks to a signature. Each of the three documents is useful alone. The
+loop exists for the places where they disagree. A register can rate a risk low because of a
+control that leaves no record. A DPIA can find a high risk to people that never reaches the
+register its owner reads. An evidence map can show a gap that nobody connects to the rating
+that relied on it. The loop makes the three agree. Then the register's own script checks
+them, and the person who owns the risk decides.
+
+This loop prepares material for a qualified reviewer: a DPO, a risk committee, an auditor.
+It is not legal or audit advice, and every document it produces says so once, at the top.
+
+## Stages
+
+| # | Stage | Type | Skills | Gate |
+|---|---|---|---|---|
+| 1 | `register` | generate | [`risk-register`](../../skills/risk-register/SKILL.md) | — |
+| 2 | `assess` | generate | [`dpia`](../../skills/dpia/SKILL.md) | — |
+| 3 | `evidence` | generate | [`soc2-evidence-map`](../../skills/soc2-evidence-map/SKILL.md) | — |
+| 4 | `check` | verify | [`risk-register`](../../skills/risk-register/SKILL.md) | **G8** mechanical |
+| 5 | `sign-off` | gate | [`doc-critique`](../../../core/skills/doc-critique/SKILL.md) | **G8.1** human |
+
+## How to run this loop
+
+1. **Register the change's risks.** Run `risk-register` with the change as its scope and the
+   objective it threatens. Save `risk-register.csv`. Owners come from the person; a missing
+   owner stays an error to resolve, never a guessed name.
+2. **Screen and assess personal data.** Run `dpia` on the processing the change introduces.
+   - When the screening finds no criteria that apply, keep its short screening record and go on.
+   - When a DPIA is needed, complete it. Then give each risk to people still at medium or high
+     residual its own register row, with the harm to people as the consequence, the same
+     owner as the DPIA measure, and `DPIA` in its controls cell. The DPIA keeps scoring harm to
+     people; the register row is how its owner sees it.
+3. **Evidence every control the ratings rely on.** List the controls named in the register's
+   residual ratings and the DPIA's in-place measures, and run `soc2-evidence-map` on that
+   list. Use the organisation's SOC 2 scope when it has one; otherwise use the skill's
+   default, Security only, tagged `[assumption]`.
+4. **Put the evidence gaps back into the register.** A residual rating can count only
+   controls that operate today. For each control the map marks `gap`:
+   - set the residual likelihood and impact of every risk that relied on it back to their
+     inherent values;
+   - move the control to a dated action under the treatment.
+
+   A `partial` control keeps its credit only for what its evidence covers; say what that is
+   in the summary for the owner. Don't lower a rating to make a heat map look better.
+5. **Check at G8.** Run the register's script on the final CSV:
+
+   ```bash
+   # Claude Code (the loop and risk-register installed side by side)
+   python3 "${CLAUDE_SKILL_DIR}/../risk-register/scripts/check_register.py" risk-register.csv --top 5 --out register-check.txt
+   # Other IDEs (from the folder the skills were installed into)
+   python3 risk-register/scripts/check_register.py risk-register.csv --top 5 --out register-check.txt
+   ```
+
+   The exit code is the verdict: `0` is `READY`, `1` is `NOT READY`, `2` is `BLOCKED`.
+   `NOT READY` returns to `register` to fix what the script names, then runs `check` again.
+   Report its warnings to the person; don't edit scores to silence them.
+6. **Critique, then put the decision to the owner at G8.1.** Run `doc-critique` on the
+   summary the owner will read. Then show them:
+   - the residual heat map from the script, as printed;
+   - every risk they are asked to accept;
+   - the evidence gaps and their dates;
+   - the DPIA's view on prior consultation.
+
+   Each owner accepts their own risks. A high or critical residual risk that its own owner
+   accepts needs a sign-off from someone above them, named in the summary.
+
+## Verdicts
+
+Defined in [`contracts/terminals.json`](../../../../contracts/terminals.json).
+
+- G8 emits `READY`, `NOT READY` or `BLOCKED`, from the script's exit code alone.
+- G8.1 emits:
+  - `PROCEED`: every residual risk is accepted or treated, by its named owner.
+  - `PROCEED WITH CONDITIONS`: accepted, subject to named actions with dates, each recorded
+    as a control and `due_date` on its risk.
+  - `REVISE`: the owner rejects a rating or a treatment. Return to `register` with their reasons.
+  - `RECONSIDER`: the change shouldn't go ahead as designed, for example when a residual risk
+    to people stays high and the design has to change before prior consultation. The loop
+    ends here.
+  - `BLOCKED`: no owner is available, or the DPO's advice is required and not yet given.
+
+## Non-interactive runs
+
+With no person present, G8.1 cannot be satisfied. Run every stage through G8, deliver the
+documents, and emit `BLOCKED: need sign-off from <the named owners>`. Never accept a risk
+on someone's behalf, never fill an owner, a retention period or a vendor location with a
+plausible guess, and leave the DPIA's DPO advice and decision fields blank.
+
+## When a stage's skill is not installed
+
+Name the missing skill and skilldrop as its source, do the minimal inline version of that
+stage, and record that it ran degraded. Without `risk-register`, G8 has no script. Say so,
+check the register by hand against the script's rules (every required field, owners that
+name a person, residual never above inherent, dates that are dates), and mark G8 degraded.
+Never skip G8.1.
+
+## Run log (opt-in)
+
+Only when the environment variable `SKILLDROP_LOOP_LOG` is set to a file path: after each
+gate's verdict, append one JSON line to that file. Nothing else is recorded, and nothing is
+sent anywhere. When the variable isn't set, or you can't read it, skip this section.
+
+```bash
+printf '%s\n' '{"ts":"2026-10-19T14:03:00Z","loop":"comply","stage":"check","gate":"G8","verdict":"READY","round":1}' >> "$SKILLDROP_LOOP_LOG"
+```
+
+Fields: `ts` (UTC), `loop`, `stage`, `gate` (`null` for a stage without one), `verdict` (the
+gate's exact word), and `round` (1 for the first attempt, counting returns under the cap).
+The log holds verdicts only: never a risk, an owner's name or any personal data.
+`skilldrop loop-stats` summarises the file.
+
+## Quality bar
+
+- **The three documents agree.** Every DPIA risk at medium or high residual has a register row, and every control a residual rating relies on appears in the evidence map.
+- **A residual rating counts only evidenced controls.** A control with an evidence gap gives no credit until the evidence exists, and the gap has a dated action.
+- **G8 is the script's exit code.** The register reaches its owner only with zero errors.
+- **Only a named owner accepts a risk.** High and critical residual risks accepted by their own owner carry a second, named sign-off.
+- **Not legal or audit advice, said once.** The DPO's advice, the decision and the sign-off fields are left for the people who give them.
+
+## Anti-patterns to avoid
+
+- ❌ **Filing the DPIA beside the register instead of into it.** A high risk to people that isn't on the owner's list doesn't get signed off; it gets missed.
+- ❌ **Leaving a residual rating on a control the evidence map marked `gap`.** That is a planned control scored as a real one, the exact thing the register bans.
+- ❌ **Editing the register to turn G8 green.** Fix the risk the script names; don't delete the row or soften the score.
+- ❌ **Running the evidence map on the whole SOC 2 scope.** The loop needs evidence for the controls this change relies on; a full readiness assessment is a separate piece of work.
+- ❌ **Self-signing at G8.1.** A clean check says the register is well-formed. It cannot say the risk is acceptable.
+- ❌ **Writing a guessed vendor location or retention period** to complete the DPIA. It stays `[to confirm: …]` and becomes a DPO question.

package/packs/grc/loops/comply/evals/eval_queries.json

@@ -0,0 +1,9 @@
+[
+  {"query": "get the new payroll vendor ready for risk sign-off", "should_trigger": true},
+  {"query": "run the comply loop on our launch of in-store facial recognition", "should_trigger": true},
+  {"query": "what do we need before we can launch this feature that profiles customers?", "should_trigger": true},
+  {"query": "I need a risk register, a DPIA and the audit evidence for this change, and they have to agree", "should_trigger": true},
+  {"query": "score these five risks on a 5x5", "should_trigger": false},
+  {"query": "do we need a DPIA for the new CRM?", "should_trigger": false},
+  {"query": "fill in this customer's security questionnaire", "should_trigger": false}
+]

package/packs/grc/loops/comply/evals/evals.json

@@ -0,0 +1,30 @@
+{
+  "loop_name": "comply",
+  "evals": [
+    {
+      "id": "evidence-gap-reverts-the-rating",
+      "prompt": "Run the comply loop. We're moving payroll to a new SaaS vendor, Paywell, on 1 December. It will hold names, bank details, salaries and sickness absence for 420 UK staff; Paywell's support team is in India. The HR director, Priya Shah, owns the change. Our controls: SSO with MFA for HR users (Okta), quarterly access reviews (we've done one, in March), and an annual vendor security review (not yet done for Paywell).",
+      "assertions": [
+        "Writes each risk as cause, event and consequence with a named owner, inherent and residual scores on a stated scale",
+        "Finds that a DPIA is needed (special category health data, transfer outside the UK) and completes it, leaving the DPO advice and decision blank",
+        "Adds a register row for each DPIA risk to people left at medium or high residual",
+        "Marks the quarterly access review as partial or a gap for a Type II period (one review since March) and the Paywell vendor review as a gap",
+        "Sets the residual rating of any risk relying on the missing vendor review back to its inherent rating",
+        "Runs check_register.py and reports G8 from its exit code",
+        "Stops at G8.1 with BLOCKED or a request for Priya Shah's decision; does not accept any risk on the owner's behalf",
+        "Does not invent Paywell's data centre location or a retention period; marks them [to confirm: ...]"
+      ]
+    },
+    {
+      "id": "script-failure-returns-to-register",
+      "prompt": "Continuing the comply loop. The check exited 1 and printed: \"line 5 [R-004]: owner 'IT' is not a named person or role\" and \"line 8 [R-007]: residual score 12 is higher than inherent score 9\". Fix it.",
+      "assertions": [
+        "Treats the output as G8 NOT READY and returns to the register stage",
+        "Asks for or names a person or role to own R-004 rather than inventing one",
+        "Corrects R-007 by revisiting its scores or controls, not by deleting the row",
+        "Re-runs the script and reports the new exit code before going to sign-off",
+        "Counts this as round 2 under the cap of 3"
+      ]
+    }
+  ]
+}

package/packs/grc/loops/comply/loop.json

@@ -0,0 +1,54 @@
+{
+  "$comment": "RFC-0052. A change to a risk position its owner has signed. The register's own script decides G8, so a register with a guessed owner or a residual above its inherent cannot reach the person signing; that person decides G8.1, because accepting a risk is theirs alone.",
+  "name": "comply",
+  "version": "0.1.0",
+  "description": "Take a change that carries risk (a launch, a new vendor, a new use of personal data) to a risk position its owner has signed: a risk register scored inherent and residual, a DPIA when personal data is processed, the evidence behind every control the ratings rely on, a script that checks the register, and a sign-off only the named owner can give. Use when the user says \"get this ready for risk sign-off\", \"run the comply loop\", \"what do we need before we can launch this\", or needs a register, a DPIA and audit evidence that agree with each other.",
+  "entrypoint": "LOOP.md",
+  "kind": "loop",
+  "cap": 3,
+  "stages": [
+    {
+      "id": "register",
+      "type": "generate",
+      "intent": "Write the change's risks as cause, event and consequence, scored inherent and residual on a stated scale, each with a named owner and a treatment.",
+      "skills": ["risk-register"]
+    },
+    {
+      "id": "assess",
+      "type": "generate",
+      "intent": "Screen for a DPIA and, where one is needed, score the harm to people; each risk to people left at medium or high gets its own row in the register.",
+      "skills": ["dpia"]
+    },
+    {
+      "id": "evidence",
+      "type": "generate",
+      "intent": "For every control the residual ratings and the DPIA's measures rely on, name the evidence that it operates today, and mark the gaps.",
+      "skills": ["soc2-evidence-map"]
+    },
+    {
+      "id": "check",
+      "type": "verify",
+      "intent": "Run the register's own check, after evidence gaps have put residual ratings back where the evidence leaves them.",
+      "skills": ["risk-register"],
+      "gate": {
+        "id": "G8",
+        "kind": "mechanical",
+        "script": "packs/grc/skills/risk-register/scripts/check_register.py",
+        "verdicts": ["READY", "NOT READY", "BLOCKED"],
+        "revise_to": "register"
+      }
+    },
+    {
+      "id": "sign-off",
+      "type": "gate",
+      "intent": "The named owner accepts each residual risk, or doesn't. The critique informs that decision; it does not make it.",
+      "skills": ["doc-critique"],
+      "gate": {
+        "id": "G8.1",
+        "kind": "human",
+        "verdicts": ["PROCEED", "PROCEED WITH CONDITIONS", "REVISE", "RECONSIDER", "BLOCKED"],
+        "revise_to": "register"
+      }
+    }
+  ]
+}

package/packs/product-manager/loops/discover/LOOP.md

@@ -13,7 +13,7 @@ script: no check can tell you that you solved the wrong problem well.
 
 | # | Stage | Type | Skills | Gate |
 |---|---|---|---|---|
-| 1 | `gather` | generate | [`requirements-interview`](../../skills/requirements-interview/SKILL.md), [`user-journey-map`](../../skills/user-journey-map/SKILL.md), [`strategy-analysis`](../../skills/strategy-analysis/SKILL.md) | — |
+| 1 | `gather` | generate | [`requirements-interview`](../../skills/requirements-interview/SKILL.md), [`user-journey-map`](../../skills/user-journey-map/SKILL.md), [`strategy-analysis`](../../skills/strategy-analysis/SKILL.md), [`feedback-synthesis`](../../skills/feedback-synthesis/SKILL.md) | — |
 | 2 | `structure` | generate | [`brief-intake`](../../../core/skills/brief-intake/SKILL.md) | — |
 | 3 | `specify` | generate | [`prd-draft`](../../skills/prd-draft/SKILL.md) | — |
 | 4 | `ratify` | gate | [`doc-critique`](../../../core/skills/doc-critique/SKILL.md) | **G0** human |
@@ -22,8 +22,9 @@ script: no check can tell you that you solved the wrong problem well.
 
 1. **Gather from whichever direction the signal arrives.** A stakeholder conversation →
    `requirements-interview`. An existing experience with known friction → `user-journey-map`.
-   An open strategic question → `strategy-analysis`. Run one; running all three on the same
-   input produces three views of the same thing and no more information.
+   An open strategic question → `strategy-analysis`. A pile of tickets, survey verbatims or
+   interview notes already collected → `feedback-synthesis`. Run one; running several on the
+   same input produces several views of the same thing and no more information.
 2. **Structure once.** `brief-intake` tags every field `[explicit]` / `[implied]` /
    `[inferred]` / `[missing]` with verbatim quotes. The tags are the deliverable — an
    untagged brief hides which parts are somebody's assumption.
@@ -75,7 +76,7 @@ which end in `BLOCKED`.
 
 ## Anti-patterns to avoid
 
-- ❌ **Running all three `gather` skills to be thorough.** Three lenses on one input is not three inputs; pick the one that matches where the signal came from.
+- ❌ **Running every `gather` skill to be thorough.** Three lenses on one input is not three inputs; pick the one that matches where the signal came from.
 - ❌ **Letting `prd-draft` fill a `[missing]` field.** That converts an open question into a fact nobody agreed to.
 - ❌ **Treating `doc-critique`'s `SHIP IT` as ratification.** The critique judges the document; G0 judges the problem.
 - ❌ **Returning `REVISE` to `specify`.** Re-writing the PRD from the same thin brief reproduces the same gaps in better prose.

package/packs/product-manager/loops/discover/loop.json

@@ -1,7 +1,7 @@
 {
   "$comment": "RFC-0028. The first loop of the lifecycle: nothing is designed or built until a human has ratified what problem is being solved. Cheapest loop to redo and the most expensive to skip.",
   "name": "discover",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "Take raw signal — interviews, journeys, a strategy question — to a requirement a human has ratified, so design starts from an agreed problem rather than an assumed one. Use when the user is deciding what to build, scoping a feature, or turning stakeholder conversations into requirements.",
   "entrypoint": "LOOP.md",
   "kind": "loop",
@@ -11,7 +11,7 @@
       "id": "gather",
       "type": "generate",
       "intent": "Collect the raw signal from whichever direction it arrives: a stakeholder conversation, an existing user journey, or a strategic question.",
-      "skills": ["requirements-interview", "user-journey-map", "strategy-analysis"]
+      "skills": ["requirements-interview", "user-journey-map", "strategy-analysis", "feedback-synthesis"]
     },
     {
       "id": "structure",

package/packs/product-manager/skills/feedback-synthesis/SKILL.md

@@ -0,0 +1,182 @@
+---
+name: feedback-synthesis
+description: Turn raw customer feedback — interview notes, support tickets, NPS or CSAT verbatims, app-store reviews, sales-call notes — into ranked themes a product decision can rest on, each a problem statement rather than a feature request, counted by distinct people or accounts not mentions, with segments, severity kept separate from frequency, verbatim quotes with source IDs, what people value, where sources disagree, what the sample cannot tell, and the questions for the next interviews. A stdlib script computes the counts. Use when the user says "what are customers telling us", "synthesise this feedback", "find the themes in these tickets / reviews / NPS comments", "affinity map these interview notes", or "what should we fix first based on feedback".
+---
+
+# feedback-synthesis
+
+Feedback arrives as a pile of tickets, survey comments, reviews and notes, and the loudest
+thing in the pile is rarely the most important. This skill turns the pile into a short list of
+problems worth acting on, with numbers that are counted rather than guessed and quotes that can
+be traced to the item they came from. It feeds discovery: the themes go to `brief-intake` to
+become a brief, and the open questions go into the next round of interviews.
+
+## How to respond
+
+**Five rules come before the steps and outrank them:**
+
+- **Answer what was asked, first.** Open with what to act on and why, in plain words. If the
+  data contradicts the user's premise ("the CSV export is our biggest problem"), say so in the
+  opening lines, with the count that shows it.
+- **Count people, not mentions.** One customer filing six tickets is one customer. Every number
+  in the reply is a count of distinct people or accounts, computed, and says which.
+- **Use only what the input contains.** No quote, count, segment, customer name, revenue figure
+  or reason that isn't in the feedback. What is missing goes in **What this can't tell us** and
+  the interview questions, never into a theme.
+- **Deliver from what there is.** Feedback without IDs, segments or a stated decision still gets
+  a synthesis, with the limit named once. Ask first only when there is no feedback at all.
+- **Write for someone who has never heard of this skill.** No skill names, file paths, script
+  output, theme codes or "S2" shorthand without its meaning. A next step is one plain sentence
+  at the end.
+
+1. **Read every item before coding any.** List the sources and, for each, what an item is
+   (a ticket, a response, an interview, a review), what identifies the person (account ID,
+   respondent ID, reviewer name, nothing), and what segment fields exist (plan, role, region,
+   tenure, company size, score). Decide the counting unit: **accounts** for B2B feedback where
+   account IDs exist (two staff from one customer are one account), **people** otherwise.
+   Match people across sources only on a shared ID: an interviewee whose account ID appears in
+   the tickets is the same account. Anonymous rows are never merged by guesswork. A rep's or
+   interviewer's guess about what someone needs ("they'll probably need SSO") is not that
+   person's feedback: leave it uncoded and say so if it matters to the answer.
+
+2. **Write the codebook as problem statements.** 4–12 themes, each one sentence saying who hits
+   what and what it costs them, plus a short code. A theme must be something the next interview
+   could prove wrong.
+   - ✅ *"Clients in the EU can't complete card payments; the error says authentication failed
+     and payment arrives weeks later by transfer."*
+   - ❌ *"Payment issues"* / *"Users want a better experience"* / *"Improve usability"* — banned:
+     nothing could disprove them.
+   - **A feature request is evidence of a need, not the need.** Rewrite it into the job behind
+     it and keep the request as a quote. ✅ *"Changing a rate means editing every recurring
+     invoice by hand — 30 to 58 invoices per price change"*, with "Please add bulk edit"
+     quoted as evidence. ❌ Theme title *"Bulk edit"*. Never let the loudest proposed solution
+     become the title.
+   - **A request with no stated reason is not a theme.** Three "add dark mode" comments with no
+     context go under **Requests with no stated need**, and the reason becomes an interview
+     question. Don't invent the need.
+   - Code what people value as themes too, prefixed `+` (`+reminders`). These become **What
+     not to break**.
+
+3. **Rate severity per item, separately from frequency.** From what happened to the person, not
+   how angry they sound; capitals and exclamation marks don't raise it.
+
+   | Level | Meaning | Example |
+   |---|---|---|
+   | **S1 — blocker** | A core job can't be done, or money, data, security or compliance is at stake | Clients can't pay; data lost; can't log in |
+   | **S2 — major** | Done only through a costly recurring workaround, or the person says they'll leave over it | Reformatting an export by hand every month |
+   | **S3 — minor** | Slower or annoying; the workaround is quick | An extra click to find a report |
+   | **S4 — cosmetic** | Looks, wording, preference, no stated cost | Blurry logo on a PDF |
+
+   More rules and edge cases in [`reference.md`](reference.md).
+
+4. **Fill the coding sheet and run the counts.** One row per item in the shape of
+   [`templates/coding-sheet.csv`](templates/coding-sheet.csv): `item_id`, `person_id` (blank
+   when the source has none), `source`, the segment columns, `themes` separated by `;` (a code
+   may carry its own severity as `code:S1`), and `severity`. When a raw segment is too fine to
+   show a pattern, add a derived column beside it (country → `market` EU / UK / US; employee
+   count → `size` band). Then run:
+
+   ```bash
+   # Claude Code
+   python3 "${CLAUDE_SKILL_DIR}/scripts/theme_counts.py" coded.csv --person-label account -o counts.md
+   # Other IDEs (from the skill folder)
+   python3 scripts/theme_counts.py coded.csv --person-label account -o counts.md
+   ```
+
+   It prints per theme the distinct people with an ID, the items with no ID (beside the count,
+   never added to it), mentions, repeat filers, sources, the segment breakdown with a flag when
+   a theme sits in one segment only, the ranking severity, and the items with no theme. If no script can run, count by hand from the sheet and show the
+   tally per theme. Above about 300 items, code the most recent full period or a stated sample,
+   and say the counts cover that sample.
+
+5. **Read the counts, then fix the codebook.** Re-read the unthemed items: three or more about
+   the same thing is a missing theme. Where one person drives most of a theme's mentions, the
+   theme is ranked on its people count. Where a theme lives in one segment, that is a finding
+   (✅ *"Only EU accounts: DE, FR, NL"*), not a footnote. Rerun after any recode.
+
+6. **Rank: severity first, then distinct people.** A theme's severity is the worst level at
+   least two people reached; a theme raised by one person takes that person's level and is
+   labelled *one report — confirm before acting*. The script's ranking severity column applies
+   this rule. A rare issue that stops payment outranks a common cosmetic gripe: say so in those
+   words when the ranking differs from the volume order.
+
+7. **Write the reply in the shape of [`templates/synthesis.md`](templates/synthesis.md)**,
+   sized to the ask (a few bullets when asked for bullets):
+   - **Headline** — what to act on first and why, correcting the premise if needed.
+   - **Themes, ranked** — problem statement; distinct people or accounts, plus mentions when
+     they differ; segments; severity in words; sources.
+   - **Evidence per theme** — 1–3 verbatim quotes, each followed by its source ID:
+     *"Took most of a Friday." (I-2)*. The quote is an exact substring of the item; cut with
+     "…", never reword, never stitch two items into one quote. An interviewer's or rep's summary
+     is reported without quotation marks and attributed to them. The requests behind the theme
+     are quoted here too.
+   - **What not to break** — what people value, with counts and quotes.
+   - **Where the sources disagree** — ✅ *"Interviewees call reminders the reason they stay;
+     four accounts report reminders going to a former contact."* Both sides, counted.
+   - **What this can't tell us** — each source's sampling bias in one line, and the
+     conclusions it rules out. No percentage of the user base from a self-selected sample:
+     ✅ *"7 of the 29 accounts that raised a ticket"* — ❌ *"24% of customers"*. Anonymous rows
+     that may overlap with identified ones are reported beside the count, never added to it:
+     ✅ *"3 accounts, plus one anonymous NPS comment that may be one of them"*.
+   - **Questions for the next interviews** — 3–6, each tied to a theme, open, non-leading, and
+     at least one that could disconfirm the top theme. ✅ *"Walk me through the last time a
+     client couldn't pay you. What happened next?"* — ❌ *"Would a fix for card payments help?"*
+
+8. **Keep the machinery out of the reply.** Theme codes, the coding sheet and the script's report
+   are working files; save them only when the user asked. The reply carries the answer.
+
+**Non-interactive runs** (subagent, CI, headless): with no feedback items at all, emit
+`BLOCKED: need the feedback itself (tickets, survey verbatims, interview notes or reviews)`,
+because a synthesis of what customers "usually say" is fabrication. A missing decision or
+audience becomes an `[assumption]` stated at the top (default: *"feeds next quarter's
+prioritisation"*). Missing person IDs or segment fields are not blockers: proceed, and state
+that the counts are mentions or unsegmented.
+
+## Useful references in this skill
+
+- [`reference.md`](reference.md) — the severity scale with edge cases, sampling bias by source, rewriting requests into problems, codebook rules, counting across sources
+- [`templates/coding-sheet.csv`](templates/coding-sheet.csv) — the CSV the script reads
+- [`templates/synthesis.md`](templates/synthesis.md) — the reply skeleton
+- [`scripts/theme_counts.py`](scripts/theme_counts.py) — distinct-person counts per theme, segments, severity, unthemed items (stdlib, Python 3.9+)
+- [`examples/booking-app-feedback.md`](examples/booking-app-feedback.md) — worked example: 16 items from three sources, the coding sheet, the real script output, and the finished reply
+
+## Quality bar
+
+- **Every count is distinct people or accounts,** computed from the coded items, and says which unit; mentions appear only beside it.
+- **The ranking puts severity before frequency** and says so where the two disagree.
+- **Every theme title is a falsifiable problem statement** naming who and what it costs; no theme is titled with a proposed feature.
+- **Every quote is verbatim with a source ID;** paraphrase never sits inside quotation marks.
+- **Segments are reported per theme,** and a one-segment theme is called out.
+- **Each source's sampling bias is named,** and no percentage of the whole user base is drawn from a self-selected sample.
+- **What people value and where the sources disagree** are both present.
+- **Interview questions are tied to themes,** open, and include one that could disconfirm the top theme.
+- **Nothing is invented:** no quote, count, segment, reason or customer that isn't in the input.
+
+## When to use this skill
+
+- ✅ A pile of support tickets, NPS or CSAT comments, reviews or interview notes, and the question "what are customers telling us?"
+- ✅ Checking a stakeholder's claim ("everyone asks for SSO") against the feedback before it goes into a brief
+- ✅ Affinity-mapping interview notes after a discovery round
+- ✅ Choosing what to fix first from a quarter's feedback
+
+## When NOT to use this skill
+
+- ❌ Planning or running the interviews themselves, or writing the question guide — `requirements-interview`
+- ❌ Mapping the stages of a journey a customer goes through — `user-journey-map`
+- ❌ Answering a research question from papers, reports or articles — `source-synthesis`
+- ❌ Ranking rival explanations for a metric change ("why did churn rise in March?") — `hypothesis-comparison`
+- ❌ Turning agreed findings into a brief for a PRD or design doc — `brief-intake`
+- ❌ Cleaning up a tracker backlog: duplicates, stale tickets, ordering — `backlog-triage`
+
+## Anti-patterns to avoid
+
+- ❌ **Counting tickets as customers.** "CSV export: 7 tickets, our biggest issue" when six came from one account.
+- ❌ **Ranking by volume alone.** Seven blurry-logo complaints above three accounts whose clients can't pay.
+- ❌ **The feature as the theme.** "Users want bulk edit" — the theme is the hours lost re-pricing recurring invoices.
+- ❌ **Inventing the need behind a bare request.** "Users want dark mode for eye strain" when nobody said why.
+- ❌ **Quotation marks around a summary.** A rep's note or the agent's paraphrase presented as the customer's words.
+- ❌ **Percentages of the base from a self-selected sample.** "40% of users want an Apple Watch app" from fifteen reviews.
+- ❌ **Merging anonymous rows into identified counts** by guessing who wrote them.
+- ❌ **Only problems.** Leaving out what people value invites the fix that breaks it.
+- ❌ **Vague themes.** "Pricing concerns", "usability", "better experience": name who, what happened, and what it cost.
+- ❌ **Leading interview questions.** "Would you use bulk edit?" confirms nothing.

package/packs/product-manager/skills/feedback-synthesis/evals/eval_queries.json

@@ -0,0 +1,46 @@
+[
+  {
+    "query": "Here are 200 support tickets and last quarter's NPS comments. What are customers actually telling us?",
+    "should_trigger": true
+  },
+  {
+    "query": "Affinity map these eight interview notes into themes with quotes",
+    "should_trigger": true
+  },
+  {
+    "query": "Find the themes in our app store reviews and tell me what to fix first",
+    "should_trigger": true
+  },
+  {
+    "query": "Sales says every prospect wants SSO. Check that against the call notes and churn survey before it goes in the brief",
+    "should_trigger": true
+  },
+  {
+    "query": "Synthesise this CSAT export into the top customer pain points by segment",
+    "should_trigger": true
+  },
+  {
+    "query": "Write me an interview guide for talking to churned customers next week",
+    "should_trigger": false
+  },
+  {
+    "query": "Map our customer's onboarding journey from sign-up to first invoice, with pains at each stage",
+    "should_trigger": false
+  },
+  {
+    "query": "Summarise what these five industry reports say about four-day weeks",
+    "should_trigger": false
+  },
+  {
+    "query": "Churn jumped 3 points in August. Which of these explanations fits the evidence best?",
+    "should_trigger": false
+  },
+  {
+    "query": "Clean up our Jira backlog: duplicates, stale tickets and what to pull next",
+    "should_trigger": false
+  },
+  {
+    "query": "Turn this Slack thread about the export bug into a brief for the PRD",
+    "should_trigger": false
+  }
+]

package/packs/product-manager/skills/feedback-synthesis/evals/evals.json

@@ -0,0 +1,76 @@
+{
+  "skill_name": "feedback-synthesis",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "I'm the PM for Tallyhouse, our invoicing app for small businesses. Q4 planning is next week and I need to know what customers actually told us in Q3. It's all in this folder: support-tickets.csv (35 tickets), nps-verbatims.csv (the 12 NPS responses that left a comment; the survey is anonymous) and interview-notes.md (5 customer interviews). What are the themes, and which should we act on first? The loudest thing in support is the CSV export, so I'm assuming that's number one.",
+      "assertions": [
+        "It corrects the premise: the CSV export problem comes from 2 accounts, and 6 of its 7 tickets are from one account (A-1042), so it is not ranked first on volume",
+        "It ranks card payments failing for clients (3 accounts: A-1107, A-1220, A-1315) above the blurry PDF logo (7 accounts), stating that payment failure blocks getting paid while the logo is cosmetic",
+        "It says the card payment failures come only from EU accounts (Germany, France, Netherlands), and reports the anonymous NPS comment N-06 beside the 3 accounts rather than adding it to make 4",
+        "It states the bulk-edit requests as the problem behind them (changing a rate means editing each recurring invoice by hand, 30 to 58 invoices) and keeps 'bulk edit' only as a quoted request",
+        "It does not make dark mode a theme with an invented reason: it reports the three anonymous NPS requests as having no stated need and turns the reason into an interview question",
+        "It names automatic payment reminders as something customers value and should not be broken, and notes the conflict with the accounts reporting reminders sent to a former contact",
+        "It names the limits of each source (tickets come from people with problems, 12 self-selected anonymous NPS comments, 5 interviews) and gives no percentage of the customer base",
+        "Every quotation is verbatim from the files with its ticket, response or interview ID, interviewer summaries are not put in quotation marks, and it invents no quotes or counts not in the input"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "Here are our 15 App Store reviews for Stridewell (running app) from September. Which 3 features should we build next, and roughly what % of our users want each? Bullets please, it's going in Slack.\n\n1. RunnerJo, 2 stars: \"Please sync with Strava! I have to upload every run twice.\"\n2. Mo_K, 5 stars: \"The training plans are brilliant. Got me from 5k to 10k.\"\n3. hilltrotter, 1 star: \"Lost my whole 21k long run when I went through the woods with no signal. Recording just stopped. Gutted.\"\n4. Bex R, 4 stars: \"Need an Apple Watch app so I can leave my phone at home.\"\n5. RunnerJo, 2 stars: \"Still no Strava sync. Updating my review.\"\n6. dawnpatrol, 3 stars: \"Apple Watch app please\"\n7. Sam T, 5 stars: \"Training plans adapt when I miss a session, love it.\"\n8. trailmix, 1 star: \"Run recording cuts out in the hills where there's no reception. Lost two runs this month.\"\n9. Kez, 4 stars: \"Would be 5 stars with Apple Watch support.\"\n10. NightOwl, 3 stars: \"Font on the run screen is too small to read mid-run.\"\n11. pacepal, 3 stars: \"Strava sync would be nice.\"\n12. Lou, 4 stars: \"Apple Watch app!!\"\n13. GreenMile, 5 stars: \"Best plans of any running app I've tried.\"\n14. ahmedruns, 2 stars: \"Make it work offline.\"\n15. Fenn, 4 stars: \"Great app, but add dark mode.\"",
+      "assertions": [
+        "It gives no percentage of users for any item, says 15 self-selected reviews cannot support one, and gives counts of reviewers instead",
+        "It counts the Strava request as 2 reviewers, not 3, because RunnerJo wrote two of the reviews",
+        "It ranks runs being lost where there is no signal (hilltrotter, trailmix) at or near the top as a data-loss problem, ahead of more often requested items that block nothing",
+        "It states the lost-runs and Apple Watch items as problems (recording stops without signal; running without the phone) rather than as 'offline mode' and 'Apple Watch app', and notes only one of the four Apple Watch reviews says why",
+        "It names the training plans as what reviewers value and should not be broken",
+        "It keeps to short bullets suitable for a Slack message, without a long report",
+        "Quotations are verbatim from the reviews, and it invents no reviews, reasons or counts not in the prompt"
+      ]
+    },
+    {
+      "id": 3,
+      "prompt": "Our Head of Sales keeps saying every enterprise prospect asks for SSO and it's costing us deals, and she wants it at the top of the Q1 brief for Fieldnote. Before I write the brief, pull the themes out of sales-call-notes.md (eight calls from September) and churn-exit-survey.csv (the ten exit surveys from Q3). I need something I can hand straight to whoever writes the brief.",
+      "assertions": [
+        "It counts SSO by distinct prospect: Harrowgate Logistics' two calls (C-01, C-03) count once, and the rep's own note on Norland (C-06) is not counted as a prospect asking, giving 2 prospects (Harrowgate and Brightwater)",
+        "It reports that SSO came only from prospects with 650 or more staff, that Norland (900 staff) did not raise it, and that no churned customer mentions it while noting that no churned respondent is in that size band",
+        "It identifies double-booking or clashes when jobs are reassigned as the most common churn theme (5 of the 10 exit responses: X-02, X-03, X-05, X-06, X-09) and notes that two prospects (C-02, C-07) raised the same problem",
+        "It states themes as problems (enterprise IT won't approve a rollout without central sign-in; reassigning a sick technician's jobs creates double bookings) rather than as feature names",
+        "It names the sampling limits: sales notes are prospects' words filtered through reps, and the exit survey covers leavers only",
+        "It does not put a rep's summary (such as C-08's pricing note or C-06's SSO note) inside quotation marks as the prospect's words, and every quotation carries its call or response ID",
+        "It states plainly how the evidence compares with the claim that every enterprise prospect asks for SSO, and ends with questions for the next interviews tied to the themes",
+        "It invents no quotes, deal values or counts not in the files; the only deal value cited is the rep's £84k estimate for Harrowgate, attributed to the rep"
+      ]
+    },
+    {
+      "id": 4,
+      "held_out": "2026-10-20",
+      "prompt": "I'm the PM for Tallyweave (invoicing + expenses for small businesses). I've dropped our Q3 support ticket export in support_tickets_q3.csv and the CSAT survey comments for the same quarter in csat_comments_q3.csv. Engineering has told me we can fund exactly one fix next quarter. Can you work out what customers are actually telling us and what we should fix first? I'm taking this to quarterly planning on Thursday, so I need something I can defend in the room, not just a list of the most common ticket subjects.",
+      "assertions": [
+        "Counts the export-timeout issue by account, not by ticket: 10 tickets but only 3 accounts. It says outright that one account (ACC-1042 / Brightfen Logistics) filed 8 of those 10, and it does not rank export timeouts first because of ticket volume.",
+        "Groups the tickets worded as different feature requests ('Excel export', 'bulk edit', 'notes field', 'bank integration', the wrong balance on a statement, the overdue reminder sent after a deposit) into one problem: Tallyweave cannot record or show part-payments or instalments against an invoice. It counts this across 9 accounts in the tickets (10 distinct accounts if CSAT is included, with ACC-1023 counted once), and states it as a problem, not as a request for an Excel export.",
+        "Breaks themes down by segment and notes that the part-payment problem sits mostly in the Starter plan (7 of 9 ticket accounts). It also notes the SSO lockout affects only Enterprise accounts.",
+        "Calls out the SSO lockout (TW-4130, TW-4168: 2 Enterprise accounts, whole organisation unable to log in, no fallback) as a severe blocker. It rates severity separately from how often the issue appears, and does not bury the lockout or drop it because only 2 accounts raised it. It also says the lockout may need handling outside the one-fix-per-quarter budget, or names it as an explicit risk.",
+        "Spots that the sources disagree about receipt scanning. The tickets say it misreads amounts, while CSAT comments call it the best part of the product. It explains the gap by platform: all 7 scanning tickets (6 accounts) come from Android, and the 5 CSAT comments praising the scanner come from iOS.",
+        "Makes one clear recommendation for the single funded fix. The case rests on distinct accounts affected plus severity and business impact (e.g. churn threats, client-facing embarrassment), not raw ticket counts. It also says what evidence would change the call (e.g. account revenue, Brightfen's churn risk).",
+        "States what this sample cannot tell. Tickets only capture customers who contact support. CSAT covers a subset of accounts. The data has no revenue or seat-count weighting. It then lists specific follow-up questions or interviews to close those gaps.",
+        "Every quote is verbatim text from the two CSV files and carries its ticket_id or response_id. No count, percentage or account total is invented or fails to match the files (e.g. 48 tickets from 36 accounts; 16 CSAT responses)."
+      ]
+    },
+    {
+      "id": 5,
+      "held_out": "2026-10-20",
+      "prompt": "I'm the product designer on Basilbox, a meal-planning app. Attached are the last 60 days of App Store and Google Play reviews (app_reviews_last_60_days.csv, exported from our review tool, so reviewers only have display names) and my typed notes from the six discovery interviews I ran (interview_notes_round1.md). Could you affinity map all of this and tell me what users are telling us? I'm running round 2 of interviews in two weeks and want to know who to recruit and what to ask.",
+      "assertions": [
+        "Counts reviewers, not reviews. It notices that KitchenKween88 wrote 4 of the reviews complaining about the grocery-list paywall and ads. It reports that theme as roughly 4 to 6 distinct reviewers (depending on whether the ads-only and price-only reviews are included), not as 9 people.",
+        "Reframes the 'calendar view' requests as the problem underneath them: the number of people eating changes from night to night or week to week, so portions are wrong and food is wasted. The evidence is the review text (lowtide_lou, Priya_in_Leeds, Teo M.) and interviews P2, P4 and P5. It does not present 'build a calendar view' as the finding.",
+        "Flags the data loss after Sign in with Apple (2 App Store reviews on version 4.2.0; all saved recipes gone) as high severity despite being rare. It keeps severity separate from frequency, and may link this to P6 saying losing recipes would make them quit.",
+        "Notes that URL recipe-import failures are concentrated on one platform: all 5 such reviews are from Google Play, mostly on 4.2.0/4.2.1. It does not treat P3's smooth import on iPhone as evidence against the bug.",
+        "Surfaces the disagreement on price. Reviews call Premium too expensive and the paywall a bait-and-switch, while every interviewee calls the price fine or worth it. It explains the gap by recruitment bias: interviewees were active annual Premium subscribers. It does not declare either source right.",
+        "States what the sample cannot tell. App reviews self-select toward extremes. No free-tier or Android users were interviewed. Reviewers cannot be matched to interviewees, so people cannot be added up across the two sources. Six interviews are too few to estimate prevalence.",
+        "Gives round-2 recruitment and questions tied to the gaps found. Examples: recruit free-tier or lapsed users and Android users; ask how household headcount varies across a week and how people adjust portions now; probe the sign-in or update experience. Generic questions like 'what do you like about the app' do not count.",
+        "Every quote is verbatim text from the CSV or the interview notes, with its source (review author/date, or P1 to P6). No counts or percentages are invented or contradict the files (31 reviews, 28 distinct display names, 6 interviewees)."
+      ]
+    }
+  ]
+}