@ckelsoe/prompt-architect 3.4.0 → 3.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +3 -3
- package/.claude-plugin/plugin.json +2 -2
- package/CHANGELOG.md +15 -0
- package/README.md +7 -3
- package/adapters/system-prompt.md +19 -4
- package/package.json +6 -4
- package/scripts/test.js +6 -0
- package/skills/prompt-architect/SKILL.md +26 -6
- package/skills/prompt-architect/assets/templates/chain-of-verification_template.txt +26 -0
- package/skills/prompt-architect/assets/templates/self-consistency_template.txt +12 -0
- package/skills/prompt-architect/references/frameworks/chain-of-verification.md +231 -0
- package/skills/prompt-architect/references/frameworks/self-consistency.md +195 -0
- package/skills/prompt-architect/references/techniques/few-shot.md +121 -0
|
@@ -5,14 +5,14 @@
|
|
|
5
5
|
"email": "charles@kelsoe.com"
|
|
6
6
|
},
|
|
7
7
|
"metadata": {
|
|
8
|
-
"description": "Prompt engineering skill using
|
|
8
|
+
"description": "Prompt engineering skill using 31 frameworks across 7 intent categories"
|
|
9
9
|
},
|
|
10
10
|
"plugins": [
|
|
11
11
|
{
|
|
12
12
|
"name": "prompt-architect",
|
|
13
13
|
"source": "./",
|
|
14
|
-
"description": "Analyzes and improves prompts using
|
|
15
|
-
"version": "3.
|
|
14
|
+
"description": "Analyzes and improves prompts using 31 frameworks. Intent-based selection routes to the right framework — from ultra-minimal (APE) to agentic (ReAct).",
|
|
15
|
+
"version": "3.5.0",
|
|
16
16
|
"keywords": ["prompt-engineering", "prompt-frameworks", "agent-skills"],
|
|
17
17
|
"category": "productivity"
|
|
18
18
|
}
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "prompt-architect",
|
|
3
|
-
"description": "
|
|
4
|
-
"version": "3.
|
|
3
|
+
"description": "31-framework prompt engineering skill for analyzing and improving prompts across 7 intent categories",
|
|
4
|
+
"version": "3.5.0",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Charles Kelsoe",
|
|
7
7
|
"email": "charles@kelsoe.com"
|
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,21 @@ All notable changes to the Prompt Architect Claude Code skill will be documented
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [3.5.0] - 2026-07-22
|
|
9
|
+
|
|
10
|
+
Adds two research-backed frameworks, one composable technique, and a guardrail against unfalsifiable framework-selection rationale. Framework count is now 31.
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
- **Self-Consistency** (REASON) — sample the same reasoning prompt N times at non-zero temperature and take a plain majority vote over the final answers, aggregated *outside* the model. Cited to Wang et al., "Self-Consistency Improves Chain of Thought Reasoning in Language Models" (arXiv 2203.11171, ICLR 2023). The doc is explicit that this is *not* self-assessment (asking one model to judge its own paths) and that the emitted prompt is a single solve-prompt ending in a parseable `FINAL ANSWER:` line — the N-sample vote is an operational step the analysis section instructs, not something one pasted prompt can do. Absolute figures (56.5% → 74.4%, PaLM 540B / GSM8K) are labeled Table 1 body figures; the +17.9% is the abstract's relative delta.
|
|
14
|
+
- **Chain-of-Verification** (CRITIQUE) — draft an answer, plan verification questions, answer them **independently of the draft**, then revise. That independence is the mechanism: it stops the verification from inheriting the draft's errors. Cited to Dhuliawala et al. (arXiv 2309.11495, Findings of ACL 2024). Distinct from RCoT, which reconstructs the question to catch missed *conditions*; CoVe fact-checks *claims*.
|
|
15
|
+
- **Few-shot / in-context learning** as a **composable technique**, not a routed framework. It lives in `references/techniques/few-shot.md` and layers onto whichever framework was chosen — covering when examples help, how many (2–5), ordering and recency effects (Lu et al., ACL 2022), what examples actually teach (format and label space over per-example correctness — Min et al., EMNLP 2022), and calibration/bias balancing (Zhao et al., ICML 2021). Brown et al. (arXiv 2005.14165) is cited only for the term, never for a number. A new "Composable Techniques" section in `SKILL.md` and the adapter routes to it.
|
|
16
|
+
|
|
17
|
+
### Changed
|
|
18
|
+
- **Anti-false-narration guardrail.** Because section headers are stripped at emission, the framework choice is often invisible in the delivered prompt — so `SKILL.md` and the adapter now instruct: when two frameworks would produce the same prompt, say so, pick the simpler one, and do not manufacture a distinction the output won't show. A new drift-test invariant keeps the rule present in both files.
|
|
19
|
+
- Framework count 29 → 31 across all advertised sites (`package.json` description and `claudeCode.frameworks`, `plugin.json`, `marketplace.json`, `README.md`, `SKILL.md`, adapter).
|
|
20
|
+
|
|
21
|
+
---
|
|
22
|
+
|
|
8
23
|
## [3.4.0] - 2026-07-22
|
|
9
24
|
|
|
10
25
|
This release makes the **emission contract** explicit and consistent across the whole skill. When Prompt Architect delivers the final prompt, framework section headers (`CONTEXT:`, `ROLE:`, `BEFORE:`, …) are stripped — the user pastes a flat block of prose with no labels. Previously the framework docs and templates were written as if those headers would survive, so slots that were bare fragments ("meeting summary", "busy professionals") lost their meaning once the header was removed. Every framework now carries its own meaning in prose.
|
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Prompt Architect
|
|
2
2
|
|
|
3
|
-
Transform vague prompts into expert-level, structured prompts using
|
|
3
|
+
Transform vague prompts into expert-level, structured prompts using 31 frameworks across 7 intent categories.
|
|
4
4
|
|
|
5
5
|
Works with **Claude Code, ChatGPT, Gemini CLI, Cursor, GitHub Copilot, Windsurf, OpenAI Codex**, and [30+ Agent Skills compatible tools](https://agentskills.io).
|
|
6
6
|
|
|
@@ -60,7 +60,7 @@ Prompt Architect is an [Agent Skills](https://agentskills.io) compatible skill t
|
|
|
60
60
|
|
|
61
61
|
## Key Features
|
|
62
62
|
|
|
63
|
-
###
|
|
63
|
+
### 31 Frameworks Across 7 Intent Categories
|
|
64
64
|
|
|
65
65
|
| Framework | Best For | Complexity |
|
|
66
66
|
|-----------|----------|------------|
|
|
@@ -84,6 +84,7 @@ Prompt Architect is an [Agent Skills](https://agentskills.io) compatible skill t
|
|
|
84
84
|
| **Least-to-Most** | Compositional multi-hop problems (simplest first) | Medium |
|
|
85
85
|
| **Plan-and-Solve (PS+)** | Zero-shot numerical/calculation reasoning | Low |
|
|
86
86
|
| **Chain of Thought** | Reasoning, problem-solving | Medium |
|
|
87
|
+
| **Self-Consistency** | Robust answers: sample many reasoning paths, majority-vote | Medium |
|
|
87
88
|
| **Chain of Density** | Summarization at fixed length, maximum information density | Medium |
|
|
88
89
|
| **Iterative Compression** | Shortening text toward a target length | Low |
|
|
89
90
|
| **Self-Refine** | Iterative output quality improvement (any task) | Medium |
|
|
@@ -91,9 +92,12 @@ Prompt Architect is an [Agent Skills](https://agentskills.io) compatible skill t
|
|
|
91
92
|
| **Devil's Advocate** | Strongest opposing argument against a position | Low |
|
|
92
93
|
| **Pre-Mortem** | Assume failure, identify specific causes | Low |
|
|
93
94
|
| **RCoT** | Verify reasoning by reconstructing the question | Medium |
|
|
95
|
+
| **Chain-of-Verification** | Fact-check a draft by verifying each claim independently | Medium |
|
|
94
96
|
| **RPEF** | Recover/reconstruct a prompt from an existing output | Low |
|
|
95
97
|
| **Reverse Role Prompting** | AI interviews you before executing | Low |
|
|
96
98
|
|
|
99
|
+
**Composable technique** (layered onto any framework, not one of the 31): **Few-shot / in-context examples** — the discipline of adding well-chosen, well-ordered worked examples to a prompt.
|
|
100
|
+
|
|
97
101
|
### Quality Scoring System
|
|
98
102
|
|
|
99
103
|
Every prompt is evaluated across:
|
|
@@ -908,7 +912,7 @@ prompt-architect/
|
|
|
908
912
|
|
|
909
913
|
**Core Components:**
|
|
910
914
|
- **SKILL.md** - Main skill logic and instructions (intent-based selection system)
|
|
911
|
-
- **Framework Reference Docs** -
|
|
915
|
+
- **Framework Reference Docs** - 30 files covering all 31 frameworks, with examples (RISE documents two variants)
|
|
912
916
|
- **Templates** - 30 ready-to-use structures: one per framework, plus `hybrid_template.txt` for combinations
|
|
913
917
|
|
|
914
918
|
---
|
|
@@ -36,7 +36,9 @@ Score the prompt *as written*, not as you charitably interpret it — the gap be
|
|
|
36
36
|
|
|
37
37
|
### 2. Intent-Based Framework Selection
|
|
38
38
|
|
|
39
|
-
With
|
|
39
|
+
With 31 frameworks, identify the user's **primary intent** first, then use the discriminating questions within that category.
|
|
40
|
+
|
|
41
|
+
**When two frameworks would produce the same prompt, say so and pick the simpler one.** Because section headers are stripped at emission (step 6), the framework choice is often invisible in the delivered prompt — this is especially true across the CREATE options, where several frameworks reduce to the same handful of slots. When you cannot point to a concrete difference the *emitted* prompt would show, do not manufacture one: name the tie plainly, choose the simpler framework, and move on. A confident rationale for an unobservable choice is exactly the overstatement this skill exists to remove.
|
|
40
42
|
|
|
41
43
|
---
|
|
42
44
|
|
|
@@ -94,6 +96,7 @@ With 29 frameworks, identify the user's **primary intent** first, then use the d
|
|
|
94
96
|
| Multiple distinct approaches to compare | **Tree of Thought** |
|
|
95
97
|
| Verify reasoning didn't overlook conditions | **RCoT** |
|
|
96
98
|
| Linear step-by-step reasoning | **Chain of Thought** |
|
|
99
|
+
| Answer must be robust; sample many paths and majority-vote | **Self-Consistency** |
|
|
97
100
|
|
|
98
101
|
---
|
|
99
102
|
|
|
@@ -106,8 +109,9 @@ With 29 frameworks, identify the user's **primary intent** first, then use the d
|
|
|
106
109
|
| Find the strongest opposing argument | **Devil's Advocate** |
|
|
107
110
|
| Identify failure modes before they happen | **Pre-Mortem** |
|
|
108
111
|
| Verify reasoning didn't miss conditions | **RCoT** |
|
|
112
|
+
| Draft may contain hallucinated facts; verify each claim | **Chain-of-Verification** |
|
|
109
113
|
|
|
110
|
-
*Self-Refine = any quality. CAI =
|
|
114
|
+
*Self-Refine = any quality. CAI = compliance with an **explicitly stated** standard or requirement set (and aligning the artifact to it — e.g. auditing a plan against a brief's constraints). Devil's Advocate = opposing arguments. Pre-Mortem = failure analysis. RCoT = an answer or plan overlooked a condition **implicit in the problem** (units, edge cases, unstated dependencies). Chain-of-Verification = independent fact-checking of a draft's factual claims.*
|
|
111
115
|
|
|
112
116
|
---
|
|
113
117
|
|
|
@@ -129,6 +133,12 @@ Most prompts need exactly one framework. Combine only when the task genuinely ha
|
|
|
129
133
|
|
|
130
134
|
When you combine, state plainly in your analysis which framework owns which phase. Never stack more than two — beyond that the frameworks' instructions start to overlap and contradict, and no single framework clearly owns any phase.
|
|
131
135
|
|
|
136
|
+
#### Composable Techniques
|
|
137
|
+
|
|
138
|
+
Some techniques are not frameworks you choose *between* — they are layers you add *on top of* whichever framework you picked, so they never appear in the routing tables above.
|
|
139
|
+
|
|
140
|
+
- **Few-shot / in-context examples** — showing 2–5 worked input→output examples inside the emitted prompt. This is the highest-leverage technique in prompting and applies to almost any framework, not just the two with a dedicated examples slot (CARE, RISE-IX). After you draft the framework prompt, decide whether examples earn their place; if they do, insert them before the final instruction, in the exact target output format, and end with the actual task. Keep examples to a handful, put the most representative one last (models weight the most recent example most heavily), keep every example in the exact format you want back, and balance the label set. Never invent examples the user or their material did not supply — fall back to a precise instruction instead.
|
|
141
|
+
|
|
132
142
|
---
|
|
133
143
|
|
|
134
144
|
### 3. Framework Quick Reference
|
|
@@ -137,12 +147,14 @@ When you combine, state plainly in your analysis which framework owns which phas
|
|
|
137
147
|
**Medium:** RACE | CARE | BAB | BROKE | CRISPE
|
|
138
148
|
**Comprehensive:** CO-STAR | RISEN | TIDD-EC
|
|
139
149
|
**Data:** RISE-IE | RISE-IX
|
|
140
|
-
**Reasoning:** Plan-and-Solve | Chain of Thought | Least-to-Most | Step-Back | Tree of Thought | RCoT
|
|
150
|
+
**Reasoning:** Plan-and-Solve | Chain of Thought | Least-to-Most | Step-Back | Tree of Thought | RCoT | Self-Consistency
|
|
141
151
|
**Structure/Iteration:** Skeleton of Thought | Chain of Density | Iterative Compression
|
|
142
|
-
**Critique/Quality:** Self-Refine | CAI Critique-Revise | Devil's Advocate | Pre-Mortem
|
|
152
|
+
**Critique/Quality:** Self-Refine | CAI Critique-Revise | Devil's Advocate | Pre-Mortem | Chain-of-Verification
|
|
143
153
|
**Meta/Reverse:** RPEF | Reverse Role Prompting
|
|
144
154
|
**Agentic:** ReAct
|
|
145
155
|
|
|
156
|
+
**Composable technique (layered onto any framework, not selected between):** Few-shot / in-context examples
|
|
157
|
+
|
|
146
158
|
### 4. Clarification Questions
|
|
147
159
|
|
|
148
160
|
Ask targeted questions (3-5 at a time) based on identified gaps:
|
|
@@ -167,6 +179,7 @@ Ask targeted questions (3-5 at a time) based on identified gaps:
|
|
|
167
179
|
**For Least-to-Most**: The full problem in one statement, paste the material the subproblems must reason over, what is the simplest thing that must be answered first, what does the final answer depend on?
|
|
168
180
|
**For Plan-and-Solve**: The problem with every number, unit, and constraint written out, paste the dataset or figures the calculation runs on if any, which values are given and which must be derived?
|
|
169
181
|
**For Chain of Thought**: The problem with all its conditions stated, paste the code, data, or document to reason over if any, what the reasoning steps should be?
|
|
182
|
+
**For Self-Consistency**: The problem with all its conditions stated, paste the data or figures it runs on if any, what the single final answer should look like so every sampled run ends in a comparable `FINAL ANSWER:` line, how many samples to run and majority-vote over (the paper uses 40; 5-10 is usually enough)?
|
|
170
183
|
**For Chain of Density**: Paste the full document to summarize, the fixed word budget every summary must hit, how many densification passes (the paper uses 5)?
|
|
171
184
|
**For Iterative Compression**: Paste the content to compress, where it should end up (word count, reading level, single paragraph), what should improve on each pass, how many passes and when to stop?
|
|
172
185
|
**For Self-Refine**: Paste the actual draft to improve, which dimensions the critique should cover (clarity, completeness, tone), what would make this output wrong or unusable?
|
|
@@ -174,6 +187,7 @@ Ask targeted questions (3-5 at a time) based on identified gaps:
|
|
|
174
187
|
**For Devil's Advocate**: The position, plan, or decision to attack, paste the proposal or memo that sets it out if you have one, which dimensions the attack should cover?
|
|
175
188
|
**For Pre-Mortem**: The project or decision being analyzed with its team, timeline, and goals, paste the plan or proposal document if you have one, how far in the future the imagined failure should be dated?
|
|
176
189
|
**For RCoT**: The question with every condition and constraint written out, paste the document those conditions come from if any, any implicit requirement not yet written into the question (units, deadlines, exclusions, edge cases) that a correct answer must still satisfy?
|
|
190
|
+
**For Chain-of-Verification**: Paste the draft answer to fact-check if you have one, or the factual question to answer carefully, which specific claims are most at risk of being wrong, what a correct final answer must not get wrong?
|
|
177
191
|
**For RPEF**: Paste the actual output sample to reverse-engineer, paste the input that produced it or confirm it is output-only, which details are one-off specifics that should become [PLACEHOLDER] variables?
|
|
178
192
|
**For Reverse Role**: What you want to achieve in one or two sentences, the domain of expertise to consult, questions one at a time or all at once, should it then do the task or synthesize a structured prompt for you to approve?
|
|
179
193
|
|
|
@@ -191,6 +205,7 @@ Using gathered information:
|
|
|
191
205
|
2. Map user's information to framework components
|
|
192
206
|
3. Fill missing elements with reasonable defaults — **with two exceptions, below**
|
|
193
207
|
4. Structure according to framework format
|
|
208
|
+
5. **Decide whether worked examples would materially improve the output**; if so, layer in few-shot examples as described in Composable Techniques above — this applies to any framework, not only the two with a built-in examples slot. Reach for it especially on classification, extraction, strict-format, and style-matching tasks, and only when the user or their material supplies real examples.
|
|
194
209
|
|
|
195
210
|
**Never default a fact about the user's world.** Their business, metrics, history, policies, staff, customers, data, or constraints are things only they know. A plausible-sounding default here is a fabrication the user may not notice before sending — asserting "our first price increase in three years" in an email to paying customers, or inventing a phone number in a published review reply. Where such a slot is unanswered, emit a visible `[you fill this in: <what is needed>]` placeholder and list every placeholder in your analysis section.
|
|
196
211
|
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ckelsoe/prompt-architect",
|
|
3
|
-
"version": "3.
|
|
4
|
-
"description": "Agent skill for analyzing and improving prompts using
|
|
3
|
+
"version": "3.5.0",
|
|
4
|
+
"description": "Agent skill for analyzing and improving prompts using 31 frameworks across 7 intent categories. Works with Claude Code, Gemini CLI, Cursor, Copilot, and 30+ Agent Skills compatible tools.",
|
|
5
5
|
"main": "skills/prompt-architect/SKILL.md",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"claude-code",
|
|
@@ -81,7 +81,7 @@
|
|
|
81
81
|
"type": "skill",
|
|
82
82
|
"skillName": "prompt-architect",
|
|
83
83
|
"skillPath": "skills/prompt-architect",
|
|
84
|
-
"version": "3.
|
|
84
|
+
"version": "3.5.0",
|
|
85
85
|
"compatibility": {
|
|
86
86
|
"claudeCode": ">=1.0.0"
|
|
87
87
|
},
|
|
@@ -123,7 +123,9 @@
|
|
|
123
123
|
"Pre-Mortem",
|
|
124
124
|
"RCoT",
|
|
125
125
|
"RPEF",
|
|
126
|
-
"Reverse-Role-Prompting"
|
|
126
|
+
"Reverse-Role-Prompting",
|
|
127
|
+
"Self-Consistency",
|
|
128
|
+
"Chain-of-Verification"
|
|
127
129
|
],
|
|
128
130
|
"features": {
|
|
129
131
|
"promptAnalysis": true,
|
package/scripts/test.js
CHANGED
|
@@ -198,6 +198,12 @@ test('Adapter has not drifted from SKILL.md', () => {
|
|
|
198
198
|
'Never stack more than two',
|
|
199
199
|
'Never default a fact about the user',
|
|
200
200
|
'Never soften or drop a prohibition',
|
|
201
|
+
// The anti-false-narration guardrail: the model must not invent a rationale
|
|
202
|
+
// for a framework choice the stripped output cannot show. Lives in both files.
|
|
203
|
+
'A confident rationale for an unobservable choice',
|
|
204
|
+
// The few-shot hook must live in the Apply-Framework step of BOTH files, or
|
|
205
|
+
// the composable technique is documented but never reached at build time.
|
|
206
|
+
'Decide whether worked examples would materially improve the output',
|
|
201
207
|
// The worked example must demonstrate the emission rules, not violate them.
|
|
202
208
|
// The adapter previously shipped a BAB example with BEFORE:/AFTER:/BRIDGE: headers.
|
|
203
209
|
'Now rewrite the job posting above.',
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: prompt-architect
|
|
3
|
-
description: Analyzes and improves prompts using
|
|
3
|
+
description: Analyzes and improves prompts using 31 frameworks across 7 intent categories. Use when a user wants to improve, rewrite, structure, or engineer a prompt — including requests like "help me write a better prompt", "improve this prompt", "what framework should I use", "make this prompt more effective", or any prompt engineering task. Recommends the right framework based on intent (create, transform, reason, critique, recover, clarify, agentic), asks targeted questions, and delivers a structured, high-quality result.
|
|
4
4
|
license: MIT
|
|
5
5
|
compatibility: Requires no external dependencies. Works with any Agent Skills compatible tool.
|
|
6
6
|
metadata:
|
|
7
7
|
author: ckelsoe
|
|
8
|
-
version: "3.
|
|
8
|
+
version: "3.5.0"
|
|
9
9
|
homepage: https://github.com/ckelsoe/prompt-architect
|
|
10
10
|
---
|
|
11
11
|
|
|
@@ -40,7 +40,9 @@ Score the prompt *as written*, not as you charitably interpret it — the gap be
|
|
|
40
40
|
|
|
41
41
|
### 2. Intent-Based Framework Selection
|
|
42
42
|
|
|
43
|
-
With
|
|
43
|
+
With 31 frameworks, identify the user's **primary intent** first, then use the discriminating questions within that category.
|
|
44
|
+
|
|
45
|
+
**When two frameworks would produce the same prompt, say so and pick the simpler one.** Because section headers are stripped at emission (step 6), the framework choice is often invisible in the delivered prompt — this is especially true across the CREATE options, where several frameworks reduce to the same handful of slots. When you cannot point to a concrete difference the *emitted* prompt would show, do not manufacture one: name the tie plainly, choose the simpler framework, and move on. A confident rationale for an unobservable choice is exactly the overstatement this skill exists to remove.
|
|
44
46
|
|
|
45
47
|
---
|
|
46
48
|
|
|
@@ -98,6 +100,7 @@ With 29 frameworks, identify the user's **primary intent** first, then use the d
|
|
|
98
100
|
| Multiple distinct approaches to compare | **Tree of Thought** |
|
|
99
101
|
| Verify reasoning didn't overlook conditions | **RCoT** |
|
|
100
102
|
| Linear step-by-step reasoning | **Chain of Thought** |
|
|
103
|
+
| Answer must be robust; sample many paths and majority-vote | **Self-Consistency** |
|
|
101
104
|
|
|
102
105
|
---
|
|
103
106
|
|
|
@@ -110,8 +113,9 @@ With 29 frameworks, identify the user's **primary intent** first, then use the d
|
|
|
110
113
|
| Find the strongest opposing argument | **Devil's Advocate** |
|
|
111
114
|
| Identify failure modes before they happen | **Pre-Mortem** |
|
|
112
115
|
| Verify reasoning didn't miss conditions | **RCoT** |
|
|
116
|
+
| Draft may contain hallucinated facts; verify each claim | **Chain-of-Verification** |
|
|
113
117
|
|
|
114
|
-
*Self-Refine = any quality. CAI =
|
|
118
|
+
*Self-Refine = any quality. CAI = compliance with an **explicitly stated** standard or requirement set (and aligning the artifact to it — e.g. auditing a plan against a brief's constraints). Devil's Advocate = opposing arguments. Pre-Mortem = failure analysis. RCoT = an answer or plan overlooked a condition **implicit in the problem** (units, edge cases, unstated dependencies). Chain-of-Verification = independent fact-checking of a draft's factual claims.*
|
|
115
119
|
|
|
116
120
|
---
|
|
117
121
|
|
|
@@ -133,6 +137,12 @@ Most prompts need exactly one framework. Combine only when the task genuinely ha
|
|
|
133
137
|
|
|
134
138
|
When you combine, load `assets/templates/hybrid_template.txt` and state plainly in your analysis which framework owns which phase. Never stack more than two — beyond that the frameworks' instructions start to overlap and contradict, and no single framework clearly owns any phase.
|
|
135
139
|
|
|
140
|
+
#### Composable Techniques
|
|
141
|
+
|
|
142
|
+
Some techniques are not frameworks you choose *between* — they are layers you add *on top of* whichever framework you picked. They answer "how should this prompt be built?", not "which shape is it?", so they never appear in the routing tables above.
|
|
143
|
+
|
|
144
|
+
- **Few-shot / in-context examples** — showing 2–5 worked input→output examples inside the emitted prompt. This is the highest-leverage technique in prompting and applies to almost any framework, not just the two with a dedicated examples slot (CARE, RISE-IX). After you draft the framework prompt, decide whether examples earn their place; if they do, insert them before the final instruction, in the exact target output format, and end with the actual task. Load `references/techniques/few-shot.md` for when to use it, how many, ordering and recency effects, and the label-space rules — and for the rule that you never invent examples the user or their material did not supply.
|
|
145
|
+
|
|
136
146
|
---
|
|
137
147
|
|
|
138
148
|
### 3. Framework Quick Reference
|
|
@@ -143,12 +153,14 @@ One-line per framework (load `references/frameworks/` for full detail):
|
|
|
143
153
|
**Medium:** RACE | CARE | BAB | BROKE | CRISPE
|
|
144
154
|
**Comprehensive:** CO-STAR | RISEN | TIDD-EC
|
|
145
155
|
**Data:** RISE-IE | RISE-IX
|
|
146
|
-
**Reasoning:** Plan-and-Solve | Chain of Thought | Least-to-Most | Step-Back | Tree of Thought | RCoT
|
|
156
|
+
**Reasoning:** Plan-and-Solve | Chain of Thought | Least-to-Most | Step-Back | Tree of Thought | RCoT | Self-Consistency
|
|
147
157
|
**Structure/Iteration:** Skeleton of Thought | Chain of Density | Iterative Compression
|
|
148
|
-
**Critique/Quality:** Self-Refine | CAI Critique-Revise | Devil's Advocate | Pre-Mortem
|
|
158
|
+
**Critique/Quality:** Self-Refine | CAI Critique-Revise | Devil's Advocate | Pre-Mortem | Chain-of-Verification
|
|
149
159
|
**Meta/Reverse:** RPEF | Reverse Role Prompting
|
|
150
160
|
**Agentic:** ReAct
|
|
151
161
|
|
|
162
|
+
**Composable technique (layered onto any framework, not selected between):** Few-shot / in-context examples
|
|
163
|
+
|
|
152
164
|
### 4. Clarification Questions
|
|
153
165
|
|
|
154
166
|
Ask targeted questions (3-5 at a time) based on identified gaps:
|
|
@@ -173,6 +185,7 @@ Ask targeted questions (3-5 at a time) based on identified gaps:
|
|
|
173
185
|
**For Least-to-Most**: The full problem in one statement, paste the material the subproblems must reason over, what is the simplest thing that must be answered first, what does the final answer depend on?
|
|
174
186
|
**For Plan-and-Solve**: The problem with every number, unit, and constraint written out, paste the dataset or figures the calculation runs on if any, which values are given and which must be derived?
|
|
175
187
|
**For Chain of Thought**: The problem with all its conditions stated, paste the code, data, or document to reason over if any, what the reasoning steps should be?
|
|
188
|
+
**For Self-Consistency**: The problem with all its conditions stated, paste the data or figures it runs on if any, what the single final answer should look like so every sampled run ends in a comparable `FINAL ANSWER:` line, how many samples to run and majority-vote over (the paper uses 40; 5-10 is usually enough)?
|
|
176
189
|
**For Chain of Density**: Paste the full document to summarize, the fixed word budget every summary must hit, how many densification passes (the paper uses 5)?
|
|
177
190
|
**For Iterative Compression**: Paste the content to compress, where it should end up (word count, reading level, single paragraph), what should improve on each pass, how many passes and when to stop?
|
|
178
191
|
**For Self-Refine**: Paste the actual draft to improve, which dimensions the critique should cover (clarity, completeness, tone), what would make this output wrong or unusable?
|
|
@@ -180,6 +193,7 @@ Ask targeted questions (3-5 at a time) based on identified gaps:
|
|
|
180
193
|
**For Devil's Advocate**: The position, plan, or decision to attack, paste the proposal or memo that sets it out if you have one, which dimensions the attack should cover?
|
|
181
194
|
**For Pre-Mortem**: The project or decision being analyzed with its team, timeline, and goals, paste the plan or proposal document if you have one, how far in the future the imagined failure should be dated?
|
|
182
195
|
**For RCoT**: The question with every condition and constraint written out, paste the document those conditions come from if any, any implicit requirement not yet written into the question (units, deadlines, exclusions, edge cases) that a correct answer must still satisfy?
|
|
196
|
+
**For Chain-of-Verification**: Paste the draft answer to fact-check if you have one, or the factual question to answer carefully, which specific claims are most at risk of being wrong, what a correct final answer must not get wrong?
|
|
183
197
|
**For RPEF**: Paste the actual output sample to reverse-engineer, paste the input that produced it or confirm it is output-only, which details are one-off specifics that should become [PLACEHOLDER] variables?
|
|
184
198
|
**For Reverse Role**: What you want to achieve in one or two sentences, the domain of expertise to consult, questions one at a time or all at once, should it then do the task or synthesize a structured prompt for you to approve?
|
|
185
199
|
|
|
@@ -197,6 +211,7 @@ Using gathered information:
|
|
|
197
211
|
2. Map user's information to framework components
|
|
198
212
|
3. Fill missing elements with reasonable defaults — **with two exceptions, below**
|
|
199
213
|
4. Structure according to framework format
|
|
214
|
+
5. **Decide whether worked examples would materially improve the output**; if so, layer in few-shot examples per `references/techniques/few-shot.md` — this applies to any framework, not only the two with a built-in examples slot. Reach for it especially on classification, extraction, strict-format, and style-matching tasks, and only when the user or their material supplies real examples.
|
|
200
215
|
|
|
201
216
|
**Never default a fact about the user's world.** Their business, metrics, history, policies, staff, customers, data, or constraints are things only they know. A plausible-sounding default here is a fabrication the user may not notice before sending — asserting "our first price increase in three years" in an email to paying customers, or inventing a phone number in a published review reply. Where such a slot is unanswered, emit a visible `[you fill this in: <what is needed>]` placeholder and list every placeholder in your analysis section.
|
|
202
217
|
|
|
@@ -263,9 +278,14 @@ Detailed framework docs in `references/frameworks/`:
|
|
|
263
278
|
- `rcot.md` - Reverse Chain-of-Thought: verify by reconstructing the question
|
|
264
279
|
- `rpef.md` - Reverse Prompt Engineering: recover prompt from output (EMNLP 2025)
|
|
265
280
|
- `reverse-role.md` - AI-Led Interview: AI asks you questions first (FATA)
|
|
281
|
+
- `self-consistency.md` - Sample N reasoning paths, majority-vote externally (Wang et al., ICLR 2023)
|
|
282
|
+
- `chain-of-verification.md` - Draft, plan checks, verify independently, revise (Dhuliawala et al., Findings of ACL 2024)
|
|
266
283
|
|
|
267
284
|
Load these when applying specific frameworks for detailed component guidance, selection criteria, and examples.
|
|
268
285
|
|
|
286
|
+
**Composable techniques** (layered onto a framework, not selected from the routing tables) live in `references/techniques/`:
|
|
287
|
+
- `few-shot.md` - In-context examples: when to add them, how many, ordering, label-space rules
|
|
288
|
+
|
|
269
289
|
## Templates
|
|
270
290
|
|
|
271
291
|
Every framework has a fill-in template at `assets/templates/<framework>_template.txt`, named after the framework's reference doc (e.g. `co-star.md` → `co-star_template.txt`). RISE has two: `rise-ie_template.txt` and `rise-ix_template.txt`. One extra template, `hybrid_template.txt`, is for combined-framework prompts (see **Combining Frameworks**).
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
SOURCE MATERIAL:
|
|
2
|
+
[Optional — paste the draft answer you want fact-checked here, OR the factual
|
|
3
|
+
question you want answered with verification built in. If you are asking a fresh
|
|
4
|
+
question rather than checking an existing draft, delete this bracketed block and
|
|
5
|
+
the sentence directly below it.]
|
|
6
|
+
Treat the material above as the subject to be verified in the steps below.
|
|
7
|
+
|
|
8
|
+
Work through the following four steps in order, in a single response.
|
|
9
|
+
|
|
10
|
+
First, write a baseline answer to the question. Do not polish it — this draft is
|
|
11
|
+
what you will fact-check. (If a draft was supplied above, treat that draft as your
|
|
12
|
+
baseline answer instead of writing a new one.)
|
|
13
|
+
|
|
14
|
+
Second, list the specific factual claims your draft makes — every name, date,
|
|
15
|
+
quantity, entity, and list member — and write one short verification question for
|
|
16
|
+
each claim. Each question must target one claim and be answerable on its own; do
|
|
17
|
+
not write a single broad question that re-asks the whole task.
|
|
18
|
+
|
|
19
|
+
Third, answer each verification question independently. Do not look back at your
|
|
20
|
+
draft while answering these questions — answer each one on its own, as if seeing
|
|
21
|
+
it fresh, so your checks are not biased into agreeing with whatever the draft
|
|
22
|
+
already says.
|
|
23
|
+
|
|
24
|
+
Fourth, write a final verified answer. Keep every claim your independent answers
|
|
25
|
+
confirmed, correct every claim they contradicted, and drop any claim you could
|
|
26
|
+
not verify. Note briefly what changed from the draft.
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
PROBLEM:
|
|
2
|
+
[Your question, calculation, or reasoning problem]
|
|
3
|
+
|
|
4
|
+
SOURCE MATERIAL:
|
|
5
|
+
[Optional — paste the problem data the solution runs on here (the word problem, dataset, table, figures, or constraint list). Delete this line and the one below it if the problem above already states every value needed.]
|
|
6
|
+
The figures and constraints in the material above are authoritative — extract from them rather than from assumed or recalled values.
|
|
7
|
+
|
|
8
|
+
Solve the problem above. Work through it step by step, showing your reasoning
|
|
9
|
+
at each stage. When you reach an answer, state it on its own line, in exactly
|
|
10
|
+
this form and with nothing else on that line:
|
|
11
|
+
|
|
12
|
+
FINAL ANSWER: [answer]
|
|
@@ -0,0 +1,231 @@
|
|
|
1
|
+
# CoVe (Chain-of-Verification)
|
|
2
|
+
|
|
3
|
+
## Overview
|
|
4
|
+
|
|
5
|
+
Chain-of-Verification (CoVe) is a verification framework that reduces hallucinated facts by making the model check its own draft against answers it produces without looking at that draft. The model first writes a baseline response, then plans a set of targeted verification questions aimed at the discrete factual claims the draft makes, then answers those questions independently — crucially without reference to the baseline — and finally regenerates the response, correcting anything the independent answers contradicted. The whole procedure fits inside a single pasted prompt: one response that walks through all four steps in order.
|
|
6
|
+
|
|
7
|
+
**Origin:** Dhuliawala et al., "Chain-of-Verification Reduces Hallucination in Large Language Models" (arXiv 2309.11495). First posted as a preprint in September 2023, it was published in the **Findings of the Association for Computational Linguistics: ACL 2024** (anthology id 2024.findings-acl.212) — so cite the venue as Findings of ACL **2024**, not 2023. The work is a collaboration: the first author holds a dual affiliation with **ETH Zürich** alongside Meta AI, so it is not accurate to credit Meta AI alone. Note also that the paper's **abstract contains no numbers** — it states qualitatively that CoVe decreases hallucinations across tasks. The frequently quoted precision improvement on list-based Wikidata questions (0.17 to 0.36 for Llama 65B) is a **body figure from the experiments**, not an abstract claim; only present it as a body/table result, and do not attach it to the abstract.
|
|
8
|
+
|
|
9
|
+
## Why CoVe Works
|
|
10
|
+
|
|
11
|
+
Hallucinated facts in a baseline draft are internally consistent and fluent, which is exactly why re-reading the draft does not catch them — the model that wrote the error will happily reaffirm it when asked to review its own text. CoVe breaks that loop by structural means rather than by asking harder:
|
|
12
|
+
|
|
13
|
+
- The draft is decomposed into discrete factual claims, each of which becomes a short, answerable verification question.
|
|
14
|
+
- Each verification question is answered **on its own**, not conditioned on the draft, so the model retrieves the fact fresh instead of echoing what it already wrote.
|
|
15
|
+
- The regenerated answer is forced to reconcile with those independent answers, so contradictions surface as edits.
|
|
16
|
+
|
|
17
|
+
**The independence of the verification step is the entire mechanism.** If the model answers the verification questions while looking at the baseline, it tends to repeat the baseline's errors — the check collapses into agreement with the thing being checked. The paper studies factored variants (answering verifications separately from the original draft) precisely to preserve this independence, and reports the largest gains when the verification answers are not conditioned on the initial response.
|
|
18
|
+
|
|
19
|
+
**Limitation:** CoVe targets hallucinated facts — fabricated names, dates, quantities, citations, list members. It is weaker on reasoning that is factually grounded but logically flawed, and it cannot verify a claim the model has no better knowledge of on the second pass than the first.
|
|
20
|
+
|
|
21
|
+
## Distinction from RCoT
|
|
22
|
+
|
|
23
|
+
CoVe and RCoT are both verification frameworks in this package, and they address different failure modes with different mechanisms. Keep them separate:
|
|
24
|
+
|
|
25
|
+
- **RCoT verifies reasoning.** It reconstructs the original question from a candidate answer and cross-checks the reconstruction against the actual question to find overlooked conditions, misinterpreted constraints, and unstated assumptions. Its failure mode is a *missed condition*.
|
|
26
|
+
- **CoVe verifies facts.** It drafts an answer, generates targeted verification questions about the claims in that draft, answers them independently, and revises. Its failure mode is a *hallucinated fact*.
|
|
27
|
+
|
|
28
|
+
Put plainly: RCoT asks "what question would produce this answer, and does it match what was actually asked?" CoVe asks "which specific claims did I just assert, are they individually true when I check each one cold, and what must change?" Reach for RCoT on multi-condition logic and constraint problems; reach for CoVe on fact-dense output like biographies, entity lists, and cited summaries.
|
|
29
|
+
|
|
30
|
+
## Components
|
|
31
|
+
|
|
32
|
+
### Baseline Response
|
|
33
|
+
**Purpose:** Draft an initial answer to the question. This draft is expected to be fluent and may contain hallucinated facts — it is the material to be checked, not the final output.
|
|
34
|
+
|
|
35
|
+
### Plan Verification Questions
|
|
36
|
+
**Purpose:** Break the draft into its discrete factual claims and turn each into a short, self-contained fact-checking question. Good verification questions are answerable on their own and target one claim each ("In what year was X founded?"), not the draft's conclusion as a whole.
|
|
37
|
+
|
|
38
|
+
**Trigger:** *"List the specific factual claims your draft makes, then write one short verification question for each claim."*
|
|
39
|
+
|
|
40
|
+
### Answer Verifications Independently
|
|
41
|
+
**Purpose:** Answer every verification question on its own terms, **without** consulting the baseline draft. This is the step that does the work: answering cold prevents the model from biasing the verification into simply confirming what it already wrote.
|
|
42
|
+
|
|
43
|
+
**Trigger:** *"Answer each verification question independently. Do not look back at your draft while answering — answer as if seeing each question fresh."*
|
|
44
|
+
|
|
45
|
+
### Final Verified Response
|
|
46
|
+
**Purpose:** Regenerate the answer, keeping what the independent answers confirmed and correcting anything they contradicted. Discrepancies between the draft and the independent answers become explicit edits.
|
|
47
|
+
|
|
48
|
+
## Template Structure
|
|
49
|
+
|
|
50
|
+
Section headers are stripped at emission, so every instruction below has to carry its meaning in prose — the model receives flat text, not a labeled form. The independence requirement is the one thing most easily lost when the `ANSWER VERIFICATIONS` header disappears, so it is written as an explicit "Do not…" sentence inside the step itself, where it survives stripping.
|
|
51
|
+
|
|
52
|
+
```
|
|
53
|
+
SOURCE MATERIAL:
|
|
54
|
+
[Optional — paste the draft answer you want fact-checked here, OR the factual
|
|
55
|
+
question you want answered with verification built in. If you are asking a fresh
|
|
56
|
+
question rather than checking an existing draft, delete this bracketed block and
|
|
57
|
+
the sentence directly below it.]
|
|
58
|
+
Treat the material above as the subject to be verified in the steps below.
|
|
59
|
+
|
|
60
|
+
Work through the following four steps in order, in a single response.
|
|
61
|
+
|
|
62
|
+
First, write a baseline answer to the question. Do not polish it — this draft is
|
|
63
|
+
what you will fact-check.
|
|
64
|
+
|
|
65
|
+
Second, list the specific factual claims your draft makes (names, dates,
|
|
66
|
+
quantities, entities, list members), and write one short verification question
|
|
67
|
+
for each claim.
|
|
68
|
+
|
|
69
|
+
Third, answer each verification question independently. Do not look back at your
|
|
70
|
+
draft while answering these questions — answer each one on its own, as if seeing
|
|
71
|
+
it fresh, so your checks are not biased into agreeing with the draft.
|
|
72
|
+
|
|
73
|
+
Fourth, write a final verified answer. Keep every claim your independent answers
|
|
74
|
+
confirmed, correct every claim they contradicted, and drop any claim you could
|
|
75
|
+
not verify. Note briefly what changed from the draft.
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
The `SOURCE MATERIAL` block carries no step number because it is optional and holds either of two things: an existing draft to be checked, or the factual question to be answered from scratch with verification built in. When it holds a draft, the first step reviews that draft rather than writing a new one. When there is no draft, delete the block and the sentence beneath it, and the model produces its own baseline in step one.
|
|
79
|
+
|
|
80
|
+
## Complete Examples
|
|
81
|
+
|
|
82
|
+
Every example below is shown in emitted form: the step headers are gone and each instruction carries its own role in prose, so nothing is lost when the scaffolding is deleted.
|
|
83
|
+
|
|
84
|
+
### Example 1: Fact-Checking an Existing Draft (source material supplied)
|
|
85
|
+
|
|
86
|
+
**Before CoVe:**
|
|
87
|
+
"Is this bio accurate?"
|
|
88
|
+
|
|
89
|
+
**After CoVe:**
|
|
90
|
+
```
|
|
91
|
+
SOURCE MATERIAL:
|
|
92
|
+
[Paste the draft biography you want fact-checked here]
|
|
93
|
+
Treat the material above as the subject to be verified in the steps below.
|
|
94
|
+
|
|
95
|
+
Work through the following four steps in order, in a single response.
|
|
96
|
+
|
|
97
|
+
First, treat the draft above as your baseline answer.
|
|
98
|
+
|
|
99
|
+
Second, list the specific factual claims the draft makes — every name, date,
|
|
100
|
+
title, place, and quantity — and write one short verification question for each.
|
|
101
|
+
|
|
102
|
+
Third, answer each verification question independently. Do not look back at the
|
|
103
|
+
draft while answering — answer each question on its own so your checks are not
|
|
104
|
+
biased into confirming what the draft already says.
|
|
105
|
+
|
|
106
|
+
Fourth, write a corrected version of the bio: keep every claim your independent
|
|
107
|
+
answers confirmed, fix every claim they contradicted, and remove any claim you
|
|
108
|
+
could not verify. List what you changed.
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
### Example 2: Named-Entity List Question (no source material)
|
|
112
|
+
|
|
113
|
+
**Before CoVe:**
|
|
114
|
+
"Name some politicians born in New York City."
|
|
115
|
+
|
|
116
|
+
**After CoVe** (no source material — the answer is generated from scratch, so the
|
|
117
|
+
`SOURCE MATERIAL` block and the sentence below it are both deleted):
|
|
118
|
+
```
|
|
119
|
+
Work through the following four steps in order, in a single response.
|
|
120
|
+
|
|
121
|
+
First, write a baseline answer listing politicians born in New York City. Do not
|
|
122
|
+
polish it — this draft is what you will fact-check.
|
|
123
|
+
|
|
124
|
+
Second, for each name on your list, write one short verification question of the
|
|
125
|
+
form "Was [name] born in New York City?"
|
|
126
|
+
|
|
127
|
+
Third, answer each verification question independently. Do not look back at your
|
|
128
|
+
list while answering — answer each question on its own, so a wrong name on the
|
|
129
|
+
draft does not get waved through.
|
|
130
|
+
|
|
131
|
+
Fourth, write a final list that keeps only the names your independent answers
|
|
132
|
+
confirmed were born in New York City, and drop the rest. Note which names you
|
|
133
|
+
removed and why.
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
### Example 3: Cited Factual Summary (source material supplied)
|
|
137
|
+
|
|
138
|
+
**Before CoVe:**
|
|
139
|
+
"Summarize this article and make sure the numbers are right."
|
|
140
|
+
|
|
141
|
+
**After CoVe:**
|
|
142
|
+
```
|
|
143
|
+
SOURCE MATERIAL:
|
|
144
|
+
[Paste the article you want summarized here]
|
|
145
|
+
Treat the material above as the subject to be verified in the steps below.
|
|
146
|
+
|
|
147
|
+
Work through the following four steps in order, in a single response.
|
|
148
|
+
|
|
149
|
+
First, write a baseline summary of the article above, including its key figures,
|
|
150
|
+
dates, and named parties.
|
|
151
|
+
|
|
152
|
+
Second, list every factual claim in your summary — each figure, date, and name —
|
|
153
|
+
and write one short verification question for each that can be checked against
|
|
154
|
+
the article.
|
|
155
|
+
|
|
156
|
+
Third, answer each verification question independently by locating the supporting
|
|
157
|
+
passage in the article. Do not answer from your summary — answer from the source,
|
|
158
|
+
so a number you misremembered while summarizing gets caught.
|
|
159
|
+
|
|
160
|
+
Fourth, write a final summary in which every figure, date, and name matches what
|
|
161
|
+
you verified against the article. Flag anything the article does not actually
|
|
162
|
+
state.
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
## Best Use Cases
|
|
166
|
+
|
|
167
|
+
1. **Fact-Dense Generation**
|
|
168
|
+
- Biographies, company profiles, historical summaries
|
|
169
|
+
- Anything where fabricated names, dates, or quantities are the main risk
|
|
170
|
+
|
|
171
|
+
2. **List and Entity Questions**
|
|
172
|
+
- "Name all X that Y" questions, where the paper shows the clearest gains
|
|
173
|
+
- Each list member becomes its own cheap verification question
|
|
174
|
+
|
|
175
|
+
3. **Cited or Sourced Summaries**
|
|
176
|
+
- Summaries whose figures must match a supplied document
|
|
177
|
+
- Verification questions check each figure against the source
|
|
178
|
+
|
|
179
|
+
4. **A Verification Layer for Draft Output**
|
|
180
|
+
- Drop CoVe in after any first-pass factual draft
|
|
181
|
+
- Especially before publishing or sending fact-heavy text
|
|
182
|
+
|
|
183
|
+
## Selection Criteria
|
|
184
|
+
|
|
185
|
+
**Choose CoVe when:**
|
|
186
|
+
- ✅ The output is fact-dense and hallucinated details are the main risk
|
|
187
|
+
- ✅ The claims decompose into discrete, individually checkable questions
|
|
188
|
+
- ✅ You are checking a draft you already have, or generating one you don't trust
|
|
189
|
+
- ✅ The model plausibly knows each fact better in isolation than buried in a paragraph
|
|
190
|
+
|
|
191
|
+
**Avoid CoVe when:**
|
|
192
|
+
- ❌ The failure mode is a missed condition or constraint → use RCoT
|
|
193
|
+
- ❌ The task is reasoning or arithmetic, not factual recall → use Plan-and-Solve or CoT
|
|
194
|
+
- ❌ Facts genuinely require external lookup the model cannot do → use a tool-grounded method (CRITIC)
|
|
195
|
+
- ❌ The task is creative or generative with no facts to verify
|
|
196
|
+
|
|
197
|
+
## CoVe vs. Other Verification Frameworks
|
|
198
|
+
|
|
199
|
+
| | CoVe | RCoT | Self-Refine | CRITIC |
|
|
200
|
+
|---|---|---|---|---|
|
|
201
|
+
| Verifies | Facts | Reasoning conditions | General quality | Facts (tool-grounded) |
|
|
202
|
+
| Failure mode caught | Hallucinated facts | Overlooked conditions | Vague weaknesses | Hallucinated facts |
|
|
203
|
+
| Mechanism | Independent re-answering of claims | Question reconstruction | Self-critique then revise | External tool checks |
|
|
204
|
+
| Key requirement | Verification not conditioned on draft | Reconstruct without re-reading question | Honest self-critique | Access to tools |
|
|
205
|
+
| Fits one prompt? | Yes | Yes | Yes | No (needs tool calls) |
|
|
206
|
+
|
|
207
|
+
## Common Mistakes
|
|
208
|
+
|
|
209
|
+
1. **Letting the verification see the draft**
|
|
210
|
+
- This is the cardinal error — if the model answers verification questions while looking at its draft, it confirms its own hallucinations
|
|
211
|
+
- The "Do not look back at your draft" instruction is load-bearing; do not drop it
|
|
212
|
+
|
|
213
|
+
2. **Verification questions that are too broad**
|
|
214
|
+
- "Is this bio accurate?" is not a verification question — it just re-asks the whole task
|
|
215
|
+
- Each question must target one discrete claim: one name, one date, one figure
|
|
216
|
+
|
|
217
|
+
3. **Skipping the correction step**
|
|
218
|
+
- Finding contradictions is not the goal; fixing them is
|
|
219
|
+
- The final answer must reconcile with the independent answers, not just report the mismatch
|
|
220
|
+
|
|
221
|
+
4. **Using CoVe where facts aren't the risk**
|
|
222
|
+
- On reasoning or constraint problems, CoVe's questions have nothing factual to grip; RCoT fits those
|
|
223
|
+
|
|
224
|
+
## Quick Reference
|
|
225
|
+
|
|
226
|
+
| Step | Purpose |
|
|
227
|
+
|------|---------|
|
|
228
|
+
| Baseline Response | Draft an initial answer (may contain hallucinations) |
|
|
229
|
+
| Plan Verification Questions | Turn each discrete claim into a short fact-check question |
|
|
230
|
+
| Answer Independently | Answer each question cold — not conditioned on the draft |
|
|
231
|
+
| Final Verified Response | Regenerate, correcting whatever the checks contradicted |
|
|
@@ -0,0 +1,195 @@
|
|
|
1
|
+
# Self-Consistency Framework
|
|
2
|
+
|
|
3
|
+
## Overview
|
|
4
|
+
|
|
5
|
+
Self-Consistency is a reasoning technique that replaces a single greedy answer with a majority vote over many independently sampled reasoning paths. Instead of asking the model to think step by step once and trust the result, you run the *same* prompt several times at non-zero temperature, collect the final answer from each run, and return the answer that appears most often. The reasoning inside each run still looks like Chain-of-Thought; what changes is that the decision is made across runs rather than within one. The insight is that a correct answer tends to be reachable by several different valid lines of reasoning, while the various ways of being wrong scatter — so agreement across independent samples is itself evidence.
|
|
6
|
+
|
|
7
|
+
When prompt-architect emits a Self-Consistency prompt, its framework section headers — `PROBLEM:` and `SOURCE MATERIAL:` — are stripped and the model receives a flat block. The solve instruction is written as a complete sentence so it survives on its own, and the `FINAL ANSWER: [answer]` line is a literal output label the model is being told to produce, not a section header, so it stays in place exactly as written. The optional `SOURCE MATERIAL` block holds the data the solution runs on and is deleted when the problem is self-contained.
|
|
8
|
+
|
|
9
|
+
**Origin:** "Self-Consistency Improves Chain of Thought Reasoning in Language Models" (Wang et al., arXiv 2203.11171, ICLR 2023). The abstract reports the technique's headline gain as a relative delta of **+17.9%** on GSM8K. The corresponding absolute figures — greedy Chain-of-Thought decoding at **56.5%** rising to **74.4%** with a majority vote over 40 sampled paths, using PaLM 540B — are **Table 1** body figures, not abstract figures. A notable finding: the plain, unweighted majority vote matched or beat every weighted aggregation scheme the authors tested, so there is no need to weight votes by probability or confidence.
|
|
10
|
+
|
|
11
|
+
## How This Framework Is Delivered
|
|
12
|
+
|
|
13
|
+
Self-Consistency is operational, not a single instruction: the gain comes from *sampling the same prompt N times and voting across the results*. A single pasted prompt cannot run itself N times, so no one prompt can perform the vote on its own. That constraint is handled honestly rather than hidden:
|
|
14
|
+
|
|
15
|
+
- **The emitted prompt** is a clean solve-prompt. It forces step-by-step reasoning and ends with a machine-parseable `FINAL ANSWER:` line so the outputs of separate runs can be lined up and tallied. It is a single-path prompt by design.
|
|
16
|
+
- **The skill's ANALYSIS section** is where the user is told to run that prompt N times at non-zero temperature and take the majority vote *externally* — in a script, a notebook, or by hand. The voting lives outside the model.
|
|
17
|
+
|
|
18
|
+
Do not pretend one prompt does the voting. Stating plainly that the aggregation happens outside the model is a feature: it is what keeps the technique faithful to the paper instead of collapsing it into something weaker.
|
|
19
|
+
|
|
20
|
+
## Components
|
|
21
|
+
|
|
22
|
+
### The Solve Prompt (emitted)
|
|
23
|
+
**Purpose:** Produce one full reasoning path and one cleanly extractable answer. The prompt demands step-by-step work — the diversity of reasoning across runs is the whole engine, so each run must actually reason rather than guess — and pins the answer to a fixed `FINAL ANSWER: [answer]` line so a program can find it identically in every sample.
|
|
24
|
+
|
|
25
|
+
**What it must contain:**
|
|
26
|
+
- A complete solve instruction that survives header stripping ("Solve the problem above. Work through it step by step…").
|
|
27
|
+
- The exact terminal line `FINAL ANSWER: [answer]`, with nothing else on that line, so every run's answer sits in the same parseable position.
|
|
28
|
+
- Optionally, a `SOURCE MATERIAL` block naming the data the solution runs on.
|
|
29
|
+
|
|
30
|
+
### The Sampling Parameters (analysis-side)
|
|
31
|
+
**Purpose:** Generate the independent reasoning paths. The paper samples **40**; in practice **5-10** is usually enough to see the benefit. Sampling must be at **non-zero temperature** — **~0.7** is a reasonable default — because temperature 0 collapses every run to the same greedy path and destroys the diversity the vote depends on.
|
|
32
|
+
|
|
33
|
+
### The Aggregation (analysis-side)
|
|
34
|
+
**Purpose:** Turn N answers into one. Extract the `FINAL ANSWER:` value from each sample, count how often each distinct answer appears, and return the most frequent one. This is a **plain, unweighted majority vote** — not a confidence-weighted or probability-weighted scheme, both of which the authors found no better. A near-tie is a signal of genuine uncertainty, not an invitation to ask the model to break the tie.
|
|
35
|
+
|
|
36
|
+
## Template Structure
|
|
37
|
+
|
|
38
|
+
Section headers — `PROBLEM:` and `SOURCE MATERIAL:` — are stripped at emission, so the solve instruction that follows is written as a complete sentence and survives on its own. The `FINAL ANSWER:` line is a literal output label, not a header, and survives verbatim. The `SOURCE MATERIAL` block is optional and is deleted when the problem already states every value needed.
|
|
39
|
+
|
|
40
|
+
```
|
|
41
|
+
PROBLEM:
|
|
42
|
+
[Your question, calculation, or reasoning problem]
|
|
43
|
+
|
|
44
|
+
SOURCE MATERIAL:
|
|
45
|
+
[Optional — paste the problem data the solution runs on here (the word problem, dataset, table, figures, or constraint list). Delete this line and the one below it if the problem above already states every value needed.]
|
|
46
|
+
The figures and constraints in the material above are authoritative — extract from them rather than from assumed or recalled values.
|
|
47
|
+
|
|
48
|
+
Solve the problem above. Work through it step by step, showing your reasoning
|
|
49
|
+
at each stage. When you reach an answer, state it on its own line, in exactly
|
|
50
|
+
this form and with nothing else on that line:
|
|
51
|
+
|
|
52
|
+
FINAL ANSWER: [answer]
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
## Complete Examples
|
|
56
|
+
|
|
57
|
+
Every example below is shown in emitted form: the solve instruction and the `FINAL ANSWER:` line are the same in each — they are what survives header stripping. Where the reasoning runs on a pasted dataset or constraint list, the `SOURCE MATERIAL` block carries it and the problem refers to "the problem above"; where every value is already in the problem statement, that block is deleted. In every case the emitted artifact is a single-path solve-prompt, and the vote is run afterward per the analysis note.
|
|
58
|
+
|
|
59
|
+
### Example 1: Math / Word Problem
|
|
60
|
+
|
|
61
|
+
**Before Self-Consistency:**
|
|
62
|
+
"A shipment has 3 crates of 24 units and 5 crates of 18 units. If 12% are defective, how many good units are there?"
|
|
63
|
+
|
|
64
|
+
**After Self-Consistency** (no source material — every value is stated in the problem, so the `SOURCE MATERIAL` block is deleted):
|
|
65
|
+
```
|
|
66
|
+
PROBLEM:
|
|
67
|
+
A shipment has 3 crates of 24 units and 5 crates of 18 units. 12% of all
|
|
68
|
+
units are defective. How many good (non-defective) units are there?
|
|
69
|
+
|
|
70
|
+
Solve the problem above. Work through it step by step, showing your reasoning
|
|
71
|
+
at each stage. When you reach an answer, state it on its own line, in exactly
|
|
72
|
+
this form and with nothing else on that line:
|
|
73
|
+
|
|
74
|
+
FINAL ANSWER: [answer]
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
**Analysis note handed to the user:** Run this prompt 5-10 times at temperature ~0.7. Extract each run's `FINAL ANSWER:` value, count them, and return the most frequent. If two answers tie, treat that as real uncertainty and inspect the reasoning rather than asking the model to pick.
|
|
78
|
+
|
|
79
|
+
### Example 2: Logic / Reasoning Over Pasted Constraints
|
|
80
|
+
|
|
81
|
+
**Before Self-Consistency:**
|
|
82
|
+
"Given these seating constraints, who sits in seat 3?"
|
|
83
|
+
|
|
84
|
+
**After Self-Consistency** (source material supplied):
|
|
85
|
+
```
|
|
86
|
+
PROBLEM:
|
|
87
|
+
Determine who sits in seat 3, working only from the constraints in the
|
|
88
|
+
problem above.
|
|
89
|
+
|
|
90
|
+
SOURCE MATERIAL:
|
|
91
|
+
[Paste the full list of seating constraints — people, seats, and adjacency rules — here]
|
|
92
|
+
The figures and constraints in the material above are authoritative — extract from them rather than from assumed or recalled values.
|
|
93
|
+
|
|
94
|
+
Solve the problem above. Work through it step by step, showing your reasoning
|
|
95
|
+
at each stage. When you reach an answer, state it on its own line, in exactly
|
|
96
|
+
this form and with nothing else on that line:
|
|
97
|
+
|
|
98
|
+
FINAL ANSWER: [answer]
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
### Example 3: Business Calculation
|
|
102
|
+
|
|
103
|
+
**Before Self-Consistency:**
|
|
104
|
+
"If CAC is $1,200, MRR per customer is $150, and gross margin is 70%, what's the payback period in months?"
|
|
105
|
+
|
|
106
|
+
**After Self-Consistency** (no source material — every value is stated, so the `SOURCE MATERIAL` block is deleted):
|
|
107
|
+
```
|
|
108
|
+
PROBLEM:
|
|
109
|
+
Customer Acquisition Cost (CAC): $1,200. Average MRR per customer: $150.
|
|
110
|
+
Gross margin: 70%. What is the payback period in months?
|
|
111
|
+
|
|
112
|
+
Solve the problem above. Work through it step by step, showing your reasoning
|
|
113
|
+
at each stage. When you reach an answer, state it on its own line, in exactly
|
|
114
|
+
this form and with nothing else on that line:
|
|
115
|
+
|
|
116
|
+
FINAL ANSWER: [answer]
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Because a payback-period calculation admits several valid routes — dividing CAC by gross-margin-adjusted MRR, or building up contribution month by month — Self-Consistency is well suited here: independent runs that reach the same number through different arithmetic reinforce the result, and a run that fumbles the margin adjustment is outvoted.
|
|
120
|
+
|
|
121
|
+
## Best Use Cases
|
|
122
|
+
|
|
123
|
+
1. **High-Stakes Single Answers**
|
|
124
|
+
- Numerical problems where one wrong step flips the result
|
|
125
|
+
- Financial calculations where accuracy is worth the extra compute
|
|
126
|
+
- Answers you will act on and cannot easily re-check by hand
|
|
127
|
+
|
|
128
|
+
2. **Problems With Multiple Valid Solution Routes**
|
|
129
|
+
- Word problems solvable by more than one method
|
|
130
|
+
- Logic puzzles with several deduction orders
|
|
131
|
+
- Any task where convergent independent reasoning is meaningful evidence
|
|
132
|
+
|
|
133
|
+
3. **When Greedy Chain-of-Thought Is Almost Right**
|
|
134
|
+
- Prior single-path CoT gave inconsistent answers across attempts
|
|
135
|
+
- The model clearly can solve it but sometimes slips
|
|
136
|
+
- You can afford N samples and want to buy down the error rate
|
|
137
|
+
|
|
138
|
+
4. **Discrete, Comparable Answers**
|
|
139
|
+
- The final answer is a number, a label, or a short choice that can be compared for equality across runs and tallied
|
|
140
|
+
|
|
141
|
+
## Selection Criteria
|
|
142
|
+
|
|
143
|
+
**Choose Self-Consistency when:**
|
|
144
|
+
- ✅ The answer is discrete and comparable across runs (a number, a category, a choice)
|
|
145
|
+
- ✅ You can sample the prompt multiple times at non-zero temperature
|
|
146
|
+
- ✅ You can aggregate the results outside the model (script, notebook, or by hand)
|
|
147
|
+
- ✅ Accuracy justifies spending N times the compute of a single run
|
|
148
|
+
- ✅ Single-path CoT is close but not reliable enough
|
|
149
|
+
|
|
150
|
+
**Avoid Self-Consistency when:**
|
|
151
|
+
- ❌ You can only run the prompt once → use Chain-of-Thought instead
|
|
152
|
+
- ❌ The output is open-ended prose with no comparable "final answer" to vote on
|
|
153
|
+
- ❌ The task needs to explore and keep branching structure → use Tree of Thought
|
|
154
|
+
- ❌ The problem must be decomposed into dependent sub-problems → use Least-to-Most
|
|
155
|
+
- ❌ You need explicit variable extraction and planning within one run → use Plan-and-Solve
|
|
156
|
+
|
|
157
|
+
## Self-Consistency vs. Chain-of-Thought vs. Tree of Thought
|
|
158
|
+
|
|
159
|
+
Chain-of-Thought is the single-path neighbor of Self-Consistency: they use the *same* kind of step-by-step reasoning prompt. The distinction is entirely in how the answer is chosen. CoT commits to one path (typically greedy); Self-Consistency samples many CoT paths and lets them vote. Tree of Thought is different again — it explores and prunes a branching search *within* a single run rather than voting across independent runs.
|
|
160
|
+
|
|
161
|
+
| | Chain-of-Thought | Self-Consistency | Tree of Thought |
|
|
162
|
+
|---|---|---|---|
|
|
163
|
+
| Reasoning paths | One | Many, independently sampled | One branching search |
|
|
164
|
+
| Temperature | Any (often greedy) | Non-zero (~0.7) required | Any |
|
|
165
|
+
| How the answer is chosen | The single run's output | Majority vote across runs | Best surviving branch |
|
|
166
|
+
| Aggregation location | None | Outside the model | Inside the run |
|
|
167
|
+
| Runs needed | 1 | N (5-10 typical; 40 in paper) | 1 |
|
|
168
|
+
| Best for | General step-by-step reasoning | Discrete answers worth N-fold compute | Problems needing lookahead/backtracking |
|
|
169
|
+
|
|
170
|
+
## Common Mistakes
|
|
171
|
+
|
|
172
|
+
1. **Asking the model to vote on itself**
|
|
173
|
+
- Self-Consistency is *not* asking the model to generate several approaches in one response and then judge which it likes best. That is self-assessment — a different and weaker technique, because the model is not a reliable judge of its own reasoning. The vote must happen across independent runs, outside the model.
|
|
174
|
+
|
|
175
|
+
2. **Sampling at temperature 0**
|
|
176
|
+
- Greedy decoding produces the same path every time, so N identical runs vote unanimously for one answer and the method reduces to plain CoT with wasted compute. Sampling must be at non-zero temperature.
|
|
177
|
+
|
|
178
|
+
3. **Weighting the votes**
|
|
179
|
+
- The paper found an unweighted majority vote matched or beat every weighted scheme tested. Confidence- or probability-weighting adds complexity for no gain.
|
|
180
|
+
|
|
181
|
+
4. **Omitting or reformatting the FINAL ANSWER line**
|
|
182
|
+
- If runs end their answers differently, they cannot be extracted and compared programmatically. The `FINAL ANSWER: [answer]` line must appear verbatim, alone on its line, in every run.
|
|
183
|
+
|
|
184
|
+
5. **Forcing a tie-break**
|
|
185
|
+
- A near-tie means genuine uncertainty. Asking the model to break it discards the very signal the method surfaced; inspect the split reasoning instead.
|
|
186
|
+
|
|
187
|
+
## Quick Reference
|
|
188
|
+
|
|
189
|
+
| Component | Location | Purpose |
|
|
190
|
+
|-----------|----------|---------|
|
|
191
|
+
| Solve prompt | Emitted | One reasoning path plus a parseable `FINAL ANSWER:` line |
|
|
192
|
+
| Step-by-step instruction | Emitted | Forces reasoning so independent paths diverge meaningfully |
|
|
193
|
+
| Sampling (N, temperature) | Analysis-side | Generate independent paths; N≈5-10 (40 in paper), temp ~0.7 |
|
|
194
|
+
| Aggregation | Analysis-side | Plain unweighted majority vote over extracted answers |
|
|
195
|
+
| Tie handling | Analysis-side | Treat near-ties as uncertainty, not as a prompt to break the tie |
|
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
# Few-Shot / In-Context Learning (Composable Technique)
|
|
2
|
+
|
|
3
|
+
## What this is, and why it is not a framework
|
|
4
|
+
|
|
5
|
+
Few-shot prompting means showing the model a small number of worked examples —
|
|
6
|
+
input paired with the desired output — inside the prompt itself, so it induces the
|
|
7
|
+
pattern rather than working from an instruction alone. It is the single most
|
|
8
|
+
load-bearing prompting technique in general use, and it is deliberately **not** one
|
|
9
|
+
of this skill's routed frameworks.
|
|
10
|
+
|
|
11
|
+
The reason is structural. A framework answers "how should this request be shaped?"
|
|
12
|
+
and you pick exactly one. Few-shot answers "should this prompt carry examples, and if
|
|
13
|
+
so, which ones and in what order?" — and the answer can be yes for almost any
|
|
14
|
+
framework. It is a **layer**, not a choice. RACE-with-examples, CO-STAR-with-examples,
|
|
15
|
+
and Plan-and-Solve-with-examples are all still RACE, CO-STAR, and Plan-and-Solve. So
|
|
16
|
+
few-shot is applied *on top of* the chosen framework's emitted prompt, never selected
|
|
17
|
+
*instead of* it.
|
|
18
|
+
|
|
19
|
+
Two frameworks already carry a dedicated examples slot — **CARE** (its `E`) and
|
|
20
|
+
**RISE-IX** (its samples). Few-shot is the discipline of choosing, ordering, and
|
|
21
|
+
formatting those examples well. That discipline applies equally to the many frameworks
|
|
22
|
+
that have no examples slot at all, which is exactly why it lives here as a technique
|
|
23
|
+
instead of inside any one framework.
|
|
24
|
+
|
|
25
|
+
**Origin:** In-context / few-shot learning was named and popularized by Brown et al.,
|
|
26
|
+
"Language Models are Few-Shot Learners" (arXiv 2005.14165, NeurIPS 2020) — cite that
|
|
27
|
+
paper only for the *term*, not for any benchmark number; its abstract reports model
|
|
28
|
+
scale ("175 billion parameters"), not prompt-construction results. The practical
|
|
29
|
+
guidance below rests on later work that actually measured how example choice changes
|
|
30
|
+
outputs.
|
|
31
|
+
|
|
32
|
+
## When to add examples — and when not to
|
|
33
|
+
|
|
34
|
+
Add examples when:
|
|
35
|
+
|
|
36
|
+
- **The output format is easier to show than to describe.** A specific JSON shape, a
|
|
37
|
+
table layout, a citation style, a commit-message convention — one example settles
|
|
38
|
+
what a paragraph of instructions leaves ambiguous.
|
|
39
|
+
- **A consistent structure must repeat** across many outputs (every entry gets the same
|
|
40
|
+
four fields in the same order).
|
|
41
|
+
- **A voice or style must be matched** and the user can supply samples of it.
|
|
42
|
+
- **The task is classification or extraction with a fixed label set**, where examples
|
|
43
|
+
pin down the label space and the decision boundary.
|
|
44
|
+
- **Edge cases matter** that an instruction cannot cleanly enumerate — show the tricky
|
|
45
|
+
input handled correctly.
|
|
46
|
+
|
|
47
|
+
Do **not** add examples when:
|
|
48
|
+
|
|
49
|
+
- **The task is open-ended reasoning.** Few-shot examples can lock the model into
|
|
50
|
+
imitating the *surface pattern* of the examples instead of reasoning freshly;
|
|
51
|
+
zero-shot chain-of-thought often beats few-shot on novel multi-step problems.
|
|
52
|
+
- **One clear instruction already fixes the output.** Examples are tokens; every one
|
|
53
|
+
costs context and adds a pattern the model will over-fit to.
|
|
54
|
+
- **You would have to invent the examples.** A fabricated example teaches a fabricated
|
|
55
|
+
standard. If neither the user nor the source material supplies a real one, prefer a
|
|
56
|
+
precise instruction — do not manufacture a sample and present it as the target.
|
|
57
|
+
|
|
58
|
+
## How many, and in what order
|
|
59
|
+
|
|
60
|
+
- **Count:** 2–5 is the usual working range. Returns diminish fast; more examples is
|
|
61
|
+
not reliably better, and past a handful the marginal example mostly adds cost and
|
|
62
|
+
over-fitting risk. Start with the fewest that disambiguate the task.
|
|
63
|
+
- **Order is not neutral.** Models are measurably sensitive to example ordering, and to
|
|
64
|
+
the point of a strong **recency bias** — the last example carries extra weight (Lu et
|
|
65
|
+
al., "Fantastically Ordered Prompts…", ACL 2022, arXiv 2104.08786, which reports a 13%
|
|
66
|
+
relative improvement across eleven text-classification tasks purely from ordering).
|
|
67
|
+
Put the most representative example last, and do not accidentally sort all of one
|
|
68
|
+
class to the end.
|
|
69
|
+
|
|
70
|
+
## What the examples actually teach
|
|
71
|
+
|
|
72
|
+
- **Format and label *space* teach more than per-example correctness.** Min et al.
|
|
73
|
+
("Rethinking the Role of Demonstrations…", EMNLP 2022, arXiv 2202.12837) found that
|
|
74
|
+
randomly replacing the labels in demonstrations barely hurt performance — what the
|
|
75
|
+
model reads off the examples is the *shape* of the task: the input distribution, the
|
|
76
|
+
set of possible answers, and the output format. Practical consequence: keep every
|
|
77
|
+
example in the **exact** format you want back, and make sure the examples span the
|
|
78
|
+
real label set. A perfectly-labeled set of examples in the wrong format teaches the
|
|
79
|
+
wrong thing.
|
|
80
|
+
- **Balance the examples to counter built-in bias.** Zhao et al. ("Calibrate Before
|
|
81
|
+
Use…", ICML 2021, arXiv 2102.09690) documented majority-label bias, recency bias, and
|
|
82
|
+
common-token bias — models drift toward whichever label appears most, appears last, or
|
|
83
|
+
is lexically common (their contextual calibration recovered up to 30.0% absolute on
|
|
84
|
+
the tasks studied). You will not calibrate logits inside a pasted prompt, but you get
|
|
85
|
+
most of the benefit for free by **balancing label frequencies** and not stacking
|
|
86
|
+
same-class examples.
|
|
87
|
+
|
|
88
|
+
## How it composes with a framework's emitted prompt
|
|
89
|
+
|
|
90
|
+
Because section headers are stripped at emission (see SKILL.md step 6), examples cannot
|
|
91
|
+
live under an `EXAMPLES:` label that survives — they have to read as part of the flat
|
|
92
|
+
prompt. After the chosen framework's prompt is drafted, and only if examples earn their
|
|
93
|
+
place:
|
|
94
|
+
|
|
95
|
+
1. Draft the framework prompt as normal.
|
|
96
|
+
2. Insert the examples **before the final instruction**, introduced by a plain prose
|
|
97
|
+
line the stripped output can keep, e.g. *"Follow the format of these examples:"* —
|
|
98
|
+
each example showing input and the exact desired output.
|
|
99
|
+
3. End with the actual task so the model acts after seeing the pattern, not before.
|
|
100
|
+
|
|
101
|
+
Emitted shape (the `Follow the format…` line and the examples are ordinary text in the
|
|
102
|
+
flat block; there are no headers):
|
|
103
|
+
|
|
104
|
+
```
|
|
105
|
+
[The framework-shaped instruction and context.]
|
|
106
|
+
|
|
107
|
+
Follow the format of these examples:
|
|
108
|
+
|
|
109
|
+
Input: <real example input 1>
|
|
110
|
+
Output: <desired output 1, in the exact target format>
|
|
111
|
+
|
|
112
|
+
Input: <real example input 2>
|
|
113
|
+
Output: <desired output 2>
|
|
114
|
+
|
|
115
|
+
Now do the same for:
|
|
116
|
+
[Paste the actual input here]
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
If the user has no real examples to supply and you cannot draw them from pasted source
|
|
120
|
+
material, say so and fall back to a precise instruction rather than inventing samples —
|
|
121
|
+
the same rule the frameworks follow for facts about the user's world.
|