@ckelsoe/prompt-architect 3.3.1 → 3.5.0

package/skills/prompt-architect/references/frameworks/chain-of-density.md

@@ -34,12 +34,12 @@ Entity identification is done by the model itself inside the loop — there is n
 
 ## The Method
 
-Five iterations, issued as a **single prompt** that returns all five summaries at once (not five separate turns):
+`[N]` iterations, issued as a **single prompt** that returns all `[N]` summaries at once (not `[N]` separate turns). The paper uses five, and that is the template's default when the user gives no number:
 
-1. **Write a deliberately sparse first summary.** Long (4-5 sentences, ~80 words) but highly non-specific, padded with filler like "this article discusses." The padding is intentional — it creates the slack that later steps convert into entities.
+1. **Write a deliberately sparse first summary.** Long (4-5 sentences, ~`[TARGET]` words) but highly non-specific, padded with filler like "this article discusses." The padding is intentional — it creates the slack that later steps convert into entities.
 2. **Identify 1-3 missing entities** from the article that are absent from the current summary.
 3. **Rewrite at identical length**, covering everything from the previous summary *plus* the new entities. Make room through fusion, compression, and removal of uninformative phrases.
-4. **Repeat steps 2-3** until five summaries exist.
+4. **Repeat steps 2-3** until `[N]` summaries exist.
 5. **Choose the step you want.** Later is not automatically better — human raters preferred steps 2-3.
 
 **Never drop entities from the previous summary.** If space cannot be made, add fewer new entities. Under pressure, density yields — length and prior content do not.
@@ -57,49 +57,71 @@ Five iterations, issued as a **single prompt** that returns all five summaries a
 - Progressive shortening to a target word count → use **Iterative Compression**
 - Non-summarization tasks. CoD is summarization-specific; the paper evaluates it only on news articles (CNN/DailyMail).
 
-## The Prompt
+## Template Structure
 
-This is the paper's actual prompt, lightly reformatted. It is reproduced closely because the specific wording — especially the fixed-length instruction and the entity criteria — is what makes the technique work.
+The block below is the template's prompt region as authored — what you fill in, not what you send. (Its divider rules are omitted; they mark the guidance/prompt boundary inside the template file and never ship.) Section headers are stripped at emission, so every slot's meaning is carried by the prose around and inside it rather than by the header above it. `SOURCE MATERIAL:` is such a header — it goes, and the unbracketed sentence below it is what tells the model the pasted text is the article to summarize. `Step 1.`, `Step 2.` and `Guidelines:` are the exception: they are literal labels inside the paper's protocol, not document structure, and they ship verbatim.
+
+`[TARGET]` is the fixed word budget and `[N]` the number of summaries, both as bare numerals. Substitute the same numeral at every occurrence of each — in the prompt body below, `[TARGET]` appears three times and `[N]` three times. (The template file names each twice more in its guidance region; those are descriptions of the fields, not slots, and never ship.) The paper's settings are `[TARGET]` = 80 and `[N]` = 5; use those if the user gave no numbers of their own.
 
 ```
-Article: {{ARTICLE}}
+SOURCE MATERIAL:
+[Paste the full document to be summarized here.]
+Use the material above as the article to be summarized for the work described below.
 
-You will generate increasingly concise, entity-dense summaries of the above Article.
+You will generate increasingly concise, entity-dense summaries of the article above.
+Produce [N] summaries in total, each denser than the last, all in a single response.
+Every summary must be exactly [TARGET] words long.
 
-Repeat the following 2 steps 5 times.
+Repeat the following 2 steps [N] times.
 
-Step 1. Identify 1-3 informative Entities (";" delimited) from the Article which are
-missing from the previously generated summary.
-Step 2. Write a new, denser summary of identical length which covers every entity and
-detail from the previous summary plus the Missing Entities.
+Step 1. Identify 1-3 informative entities (";" delimited) from the article which
+are missing from the previously generated summary.
+Step 2. Write a new, denser summary of identical length which covers every entity
+and detail from the previous summary plus the missing entities.
 
-A Missing Entity is:
+A missing entity is:
 - Relevant: to the main story.
 - Specific: descriptive yet concise (5 words or fewer).
 - Novel: not in the previous summary.
-- Faithful: present in the Article.
-- Anywhere: located anywhere in the Article.
+- Faithful: present in the article.
+- Anywhere: located anywhere in the article.
 
 Guidelines:
-- The first summary should be long (4-5 sentences, ~80 words) yet highly non-specific,
-  containing little information beyond the entities marked as missing. Use overly verbose
-  language and fillers (e.g., "this article discusses") to reach ~80 words.
-- Make every word count: re-write the previous summary to improve flow and make space for
-  additional entities.
-- Make space with fusion, compression, and removal of uninformative phrases like "the
-  article discusses".
-- The summaries should become highly dense and concise yet self-contained, e.g., easily
-  understood without the Article.
+- The first summary should be long (4-5 sentences, ~[TARGET] words) yet highly
+  non-specific, containing little information beyond the entities marked as missing.
+  Use overly verbose language and fillers (e.g., "this article discusses") to reach
+  ~[TARGET] words.
+- Make every word count: re-write the previous summary to improve flow and make
+  space for additional entities.
+- Make space with fusion, compression, and removal of uninformative phrases like
+  "the article discusses".
+- The summaries should become highly dense and concise yet self-contained, e.g.,
+  easily understood without the article.
 - Missing entities can appear anywhere in the new summary.
-- Never drop entities from the previous summary. If space cannot be made, add fewer new
-  entities.
+- Never drop entities from the previous summary. If space cannot be made, add
+  fewer new entities.
 
 Remember, use the exact same number of words for each summary.
 
-Answer in JSON. The JSON should be a list (length 5) of dictionaries whose keys are
+Answer in JSON. The JSON should be a list (length [N]) of dictionaries whose keys are
 "Missing_Entities" and "Denser_Summary".
+
+After the JSON, and outside it, add one short paragraph naming which summary you
+recommend and why. Do not default to the last one: in the paper's human evaluation,
+summary 2 (30.8% of first-place votes) and summary 3 (23.0%) were preferred over the
+sparse summary 1 (8.3%), because past a point density costs readability.
 ```
 
+### Relationship to the paper's original prompt
+
+The body above is the paper's prompt, reproduced closely on purpose — the specific wording, especially the fixed-length instruction and the entity criteria, is what makes the technique work. Three deliberate departures:
+
+- **`SOURCE MATERIAL:` replaces the paper's `Article: {{ARTICLE}}` field.** Same job, house convention, and the trailing sentence survives header-stripping where a bare `Article:` label would not.
+- **`[TARGET]` and `[N]` are parameterized** where the paper hardcodes 80 and 5, so the template serves a user-supplied budget and pass count.
+- **The closing recommendation paragraph is an addition, not from the paper.** The paper reports the preference numbers; it does not instruct the model to act on them. This template does, so the user gets a pick rather than a default-to-last.
+
+Sentence-case "missing entity" and "article" replace the paper's mid-sentence capitals. That is formatting only — no wording changed.
+
 ## Worked Example
 
 **Source:** a 1,200-word news article about a municipal transit funding decision.

package/skills/prompt-architect/references/frameworks/chain-of-thought.md

@@ -4,6 +4,8 @@
 
 Chain of Thought is a prompting technique that encourages step-by-step reasoning and makes the thinking process explicit. Instead of jumping to answers, it guides Claude to break down complex problems, show intermediate steps, and verify logic along the way.
 
+When prompt-architect emits a Chain-of-Thought prompt, its framework section headers — `PROBLEM / QUESTION`, `SOURCE MATERIAL`, `REASONING INSTRUCTION` — are stripped and the model receives a flat block. The reasoning instruction is written as a complete sentence so it survives on its own, while the numbered step labels the prompt requests (`Step 1:`, `STEP 1 - REQUIREMENTS:`) are literal labels inside the reasoning protocol and stay in place. An optional `SOURCE MATERIAL` block holds the code, data, or document being reasoned over and is deleted when the problem is self-contained.
+
 **Research basis:** "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (Wei et al., Google Research, arXiv 2201.11903, NeurIPS 2022). Prompting PaLM 540B with eight chain-of-thought exemplars raised GSM8K accuracy from 17.9% with standard prompting to 56.9%, beating the prior fine-tuned state of the art of 55% (GPT-3 plus a verifier, Cobbe et al. 2021).
 
 ## Core Concept
@@ -168,6 +170,16 @@ Poor:
 
 ## Complete Examples
 
+Every example below is shown in emitted form. Two kinds of label appear, and they behave
+differently at emission. The framework's own section headers — `SOURCE MATERIAL:`,
+`PROBLEM / QUESTION:`, `REASONING INSTRUCTION:` — are scaffolding and are stripped, so each
+is written so its instruction still reads once the header is gone. The numbered
+reasoning-step labels the prompt asks for (`Step 1:`, `STEP 1 - REQUIREMENTS:`) are literal
+labels inside the reasoning protocol, not section headers, and they survive into the emitted
+prompt exactly as written — they are what the model is being told to produce. Where the
+reasoning runs over an existing artifact, the `SOURCE MATERIAL` block carries it; where the
+task starts from a blank page, that block is deleted.
+
 ### Example 1: Debugging
 
 **Without CoT:**
@@ -176,14 +188,19 @@ Why isn't this code working?
 [code snippet]
 ```
 
-**With CoT:**
+**With CoT** (source material supplied):
 ```
-Debug this code by thinking through it step-by-step:
+SOURCE MATERIAL:
+[Paste the code you want debugged here]
+The reasoning that follows must work from the material above, not from
+assumptions about it.
 
-[code snippet]
+PROBLEM / QUESTION:
+Debug the code above by thinking through it step-by-step.
 
+REASONING INSTRUCTION:
 Use this process:
-1. Read the code and state what it's supposed to do
+1. Read the code and state what it is supposed to do
 2. Identify the input and expected output
 3. Trace through execution line by line
 4. Note any suspicious patterns or red flags
@@ -201,7 +218,9 @@ Use this process:
 Design a URL shortener.
 ```
 
-**With CoT:**
+**With CoT** (no source material — the design is worked out from scratch, so no
+`SOURCE MATERIAL` block is needed; the `STEP N -` labels are literal protocol labels and
+survive header stripping):
 ```
 Design a URL shortener by working through these steps:
 
@@ -244,7 +263,9 @@ For each step, explain your reasoning.
 Should we use MongoDB or PostgreSQL?
 ```
 
-**With CoT:**
+**With CoT** (no source material — the decision is reasoned out from scratch, so no
+`SOURCE MATERIAL` block is needed; the `STEP N -` labels are literal protocol labels and
+survive header stripping):
 ```
 Decide between MongoDB and PostgreSQL by reasoning through:
 
@@ -306,7 +327,7 @@ STEPS:
 ...
 ```
 
-### CoT + RISE
+### CoT + RISE-IE
 ```
 ROLE: [Analyst]
 INPUT: [Data]

package/skills/prompt-architect/references/frameworks/chain-of-verification.md

@@ -0,0 +1,231 @@
+# CoVe (Chain-of-Verification)
+
+## Overview
+
+Chain-of-Verification (CoVe) is a verification framework that reduces hallucinated facts by making the model check its own draft against answers it produces without looking at that draft. The model first writes a baseline response, then plans a set of targeted verification questions aimed at the discrete factual claims the draft makes, then answers those questions independently — crucially without reference to the baseline — and finally regenerates the response, correcting anything the independent answers contradicted. The whole procedure fits inside a single pasted prompt: one response that walks through all four steps in order.
+
+**Origin:** Dhuliawala et al., "Chain-of-Verification Reduces Hallucination in Large Language Models" (arXiv 2309.11495). First posted as a preprint in September 2023, it was published in the **Findings of the Association for Computational Linguistics: ACL 2024** (anthology id 2024.findings-acl.212) — so cite the venue as Findings of ACL **2024**, not 2023. The work is a collaboration: the first author holds a dual affiliation with **ETH Zürich** alongside Meta AI, so it is not accurate to credit Meta AI alone. Note also that the paper's **abstract contains no numbers** — it states qualitatively that CoVe decreases hallucinations across tasks. The frequently quoted precision improvement on list-based Wikidata questions (0.17 to 0.36 for Llama 65B) is a **body figure from the experiments**, not an abstract claim; only present it as a body/table result, and do not attach it to the abstract.
+
+## Why CoVe Works
+
+Hallucinated facts in a baseline draft are internally consistent and fluent, which is exactly why re-reading the draft does not catch them — the model that wrote the error will happily reaffirm it when asked to review its own text. CoVe breaks that loop by structural means rather than by asking harder:
+
+- The draft is decomposed into discrete factual claims, each of which becomes a short, answerable verification question.
+- Each verification question is answered **on its own**, not conditioned on the draft, so the model retrieves the fact fresh instead of echoing what it already wrote.
+- The regenerated answer is forced to reconcile with those independent answers, so contradictions surface as edits.
+
+**The independence of the verification step is the entire mechanism.** If the model answers the verification questions while looking at the baseline, it tends to repeat the baseline's errors — the check collapses into agreement with the thing being checked. The paper studies factored variants (answering verifications separately from the original draft) precisely to preserve this independence, and reports the largest gains when the verification answers are not conditioned on the initial response.
+
+**Limitation:** CoVe targets hallucinated facts — fabricated names, dates, quantities, citations, list members. It is weaker on reasoning that is factually grounded but logically flawed, and it cannot verify a claim the model has no better knowledge of on the second pass than the first.
+
+## Distinction from RCoT
+
+CoVe and RCoT are both verification frameworks in this package, and they address different failure modes with different mechanisms. Keep them separate:
+
+- **RCoT verifies reasoning.** It reconstructs the original question from a candidate answer and cross-checks the reconstruction against the actual question to find overlooked conditions, misinterpreted constraints, and unstated assumptions. Its failure mode is a *missed condition*.
+- **CoVe verifies facts.** It drafts an answer, generates targeted verification questions about the claims in that draft, answers them independently, and revises. Its failure mode is a *hallucinated fact*.
+
+Put plainly: RCoT asks "what question would produce this answer, and does it match what was actually asked?" CoVe asks "which specific claims did I just assert, are they individually true when I check each one cold, and what must change?" Reach for RCoT on multi-condition logic and constraint problems; reach for CoVe on fact-dense output like biographies, entity lists, and cited summaries.
+
+## Components
+
+### Baseline Response
+**Purpose:** Draft an initial answer to the question. This draft is expected to be fluent and may contain hallucinated facts — it is the material to be checked, not the final output.
+
+### Plan Verification Questions
+**Purpose:** Break the draft into its discrete factual claims and turn each into a short, self-contained fact-checking question. Good verification questions are answerable on their own and target one claim each ("In what year was X founded?"), not the draft's conclusion as a whole.
+
+**Trigger:** *"List the specific factual claims your draft makes, then write one short verification question for each claim."*
+
+### Answer Verifications Independently
+**Purpose:** Answer every verification question on its own terms, **without** consulting the baseline draft. This is the step that does the work: answering cold prevents the model from biasing the verification into simply confirming what it already wrote.
+
+**Trigger:** *"Answer each verification question independently. Do not look back at your draft while answering — answer as if seeing each question fresh."*
+
+### Final Verified Response
+**Purpose:** Regenerate the answer, keeping what the independent answers confirmed and correcting anything they contradicted. Discrepancies between the draft and the independent answers become explicit edits.
+
+## Template Structure
+
+Section headers are stripped at emission, so every instruction below has to carry its meaning in prose — the model receives flat text, not a labeled form. The independence requirement is the one thing most easily lost when the `ANSWER VERIFICATIONS` header disappears, so it is written as an explicit "Do not…" sentence inside the step itself, where it survives stripping.
+
+```
+SOURCE MATERIAL:
+[Optional — paste the draft answer you want fact-checked here, OR the factual
+question you want answered with verification built in. If you are asking a fresh
+question rather than checking an existing draft, delete this bracketed block and
+the sentence directly below it.]
+Treat the material above as the subject to be verified in the steps below.
+
+Work through the following four steps in order, in a single response.
+
+First, write a baseline answer to the question. Do not polish it — this draft is
+what you will fact-check.
+
+Second, list the specific factual claims your draft makes (names, dates,
+quantities, entities, list members), and write one short verification question
+for each claim.
+
+Third, answer each verification question independently. Do not look back at your
+draft while answering these questions — answer each one on its own, as if seeing
+it fresh, so your checks are not biased into agreeing with the draft.
+
+Fourth, write a final verified answer. Keep every claim your independent answers
+confirmed, correct every claim they contradicted, and drop any claim you could
+not verify. Note briefly what changed from the draft.
+```
+
+The `SOURCE MATERIAL` block carries no step number because it is optional and holds either of two things: an existing draft to be checked, or the factual question to be answered from scratch with verification built in. When it holds a draft, the first step reviews that draft rather than writing a new one. When there is no draft, delete the block and the sentence beneath it, and the model produces its own baseline in step one.
+
+## Complete Examples
+
+Every example below is shown in emitted form: the step headers are gone and each instruction carries its own role in prose, so nothing is lost when the scaffolding is deleted.
+
+### Example 1: Fact-Checking an Existing Draft (source material supplied)
+
+**Before CoVe:**
+"Is this bio accurate?"
+
+**After CoVe:**
+```
+SOURCE MATERIAL:
+[Paste the draft biography you want fact-checked here]
+Treat the material above as the subject to be verified in the steps below.
+
+Work through the following four steps in order, in a single response.
+
+First, treat the draft above as your baseline answer.
+
+Second, list the specific factual claims the draft makes — every name, date,
+title, place, and quantity — and write one short verification question for each.
+
+Third, answer each verification question independently. Do not look back at the
+draft while answering — answer each question on its own so your checks are not
+biased into confirming what the draft already says.
+
+Fourth, write a corrected version of the bio: keep every claim your independent
+answers confirmed, fix every claim they contradicted, and remove any claim you
+could not verify. List what you changed.
+```
+
+### Example 2: Named-Entity List Question (no source material)
+
+**Before CoVe:**
+"Name some politicians born in New York City."
+
+**After CoVe** (no source material — the answer is generated from scratch, so the
+`SOURCE MATERIAL` block and the sentence below it are both deleted):
+```
+Work through the following four steps in order, in a single response.
+
+First, write a baseline answer listing politicians born in New York City. Do not
+polish it — this draft is what you will fact-check.
+
+Second, for each name on your list, write one short verification question of the
+form "Was [name] born in New York City?"
+
+Third, answer each verification question independently. Do not look back at your
+list while answering — answer each question on its own, so a wrong name on the
+draft does not get waved through.
+
+Fourth, write a final list that keeps only the names your independent answers
+confirmed were born in New York City, and drop the rest. Note which names you
+removed and why.
+```
+
+### Example 3: Cited Factual Summary (source material supplied)
+
+**Before CoVe:**
+"Summarize this article and make sure the numbers are right."
+
+**After CoVe:**
+```
+SOURCE MATERIAL:
+[Paste the article you want summarized here]
+Treat the material above as the subject to be verified in the steps below.
+
+Work through the following four steps in order, in a single response.
+
+First, write a baseline summary of the article above, including its key figures,
+dates, and named parties.
+
+Second, list every factual claim in your summary — each figure, date, and name —
+and write one short verification question for each that can be checked against
+the article.
+
+Third, answer each verification question independently by locating the supporting
+passage in the article. Do not answer from your summary — answer from the source,
+so a number you misremembered while summarizing gets caught.
+
+Fourth, write a final summary in which every figure, date, and name matches what
+you verified against the article. Flag anything the article does not actually
+state.
+```
+
+## Best Use Cases
+
+1. **Fact-Dense Generation**
+   - Biographies, company profiles, historical summaries
+   - Anything where fabricated names, dates, or quantities are the main risk
+
+2. **List and Entity Questions**
+   - "Name all X that Y" questions, where the paper shows the clearest gains
+   - Each list member becomes its own cheap verification question
+
+3. **Cited or Sourced Summaries**
+   - Summaries whose figures must match a supplied document
+   - Verification questions check each figure against the source
+
+4. **A Verification Layer for Draft Output**
+   - Drop CoVe in after any first-pass factual draft
+   - Especially before publishing or sending fact-heavy text
+
+## Selection Criteria
+
+**Choose CoVe when:**
+- ✅ The output is fact-dense and hallucinated details are the main risk
+- ✅ The claims decompose into discrete, individually checkable questions
+- ✅ You are checking a draft you already have, or generating one you don't trust
+- ✅ The model plausibly knows each fact better in isolation than buried in a paragraph
+
+**Avoid CoVe when:**
+- ❌ The failure mode is a missed condition or constraint → use RCoT
+- ❌ The task is reasoning or arithmetic, not factual recall → use Plan-and-Solve or CoT
+- ❌ Facts genuinely require external lookup the model cannot do → use a tool-grounded method (CRITIC)
+- ❌ The task is creative or generative with no facts to verify
+
+## CoVe vs. Other Verification Frameworks
+
+| | CoVe | RCoT | Self-Refine | CRITIC |
+|---|---|---|---|---|
+| Verifies | Facts | Reasoning conditions | General quality | Facts (tool-grounded) |
+| Failure mode caught | Hallucinated facts | Overlooked conditions | Vague weaknesses | Hallucinated facts |
+| Mechanism | Independent re-answering of claims | Question reconstruction | Self-critique then revise | External tool checks |
+| Key requirement | Verification not conditioned on draft | Reconstruct without re-reading question | Honest self-critique | Access to tools |
+| Fits one prompt? | Yes | Yes | Yes | No (needs tool calls) |
+
+## Common Mistakes
+
+1. **Letting the verification see the draft**
+   - This is the cardinal error — if the model answers verification questions while looking at its draft, it confirms its own hallucinations
+   - The "Do not look back at your draft" instruction is load-bearing; do not drop it
+
+2. **Verification questions that are too broad**
+   - "Is this bio accurate?" is not a verification question — it just re-asks the whole task
+   - Each question must target one discrete claim: one name, one date, one figure
+
+3. **Skipping the correction step**
+   - Finding contradictions is not the goal; fixing them is
+   - The final answer must reconcile with the independent answers, not just report the mismatch
+
+4. **Using CoVe where facts aren't the risk**
+   - On reasoning or constraint problems, CoVe's questions have nothing factual to grip; RCoT fits those
+
+## Quick Reference
+
+| Step | Purpose |
+|------|---------|
+| Baseline Response | Draft an initial answer (may contain hallucinations) |
+| Plan Verification Questions | Turn each discrete claim into a short fact-check question |
+| Answer Independently | Answer each question cold — not conditioned on the draft |
+| Final Verified Response | Regenerate, correcting whatever the checks contradicted |